Retry Logic

FLUID automatically retries failed webhook deliveries to ensure reliable event notification. If your endpoint is temporarily unavailable or returns an error, FLUID will retry the webhook multiple times with exponential backoff.

How Retry Logic Works

Retry Schedule

FLUID uses exponential backoff with jitter for retry attempts:

Attempt	Delay	Total Time Elapsed	Status
1 (initial)	0s	0s	Immediate delivery
2	~2s	~2s	First retry
3	~4s	~6s	Second retry
4	~8s	~14s	Third retry
5	~16s	~30s	Fourth retry
6 (final)	~32s	~62s	Final retry

Default Configuration

Max Attempts: 6 (1 initial + 5 retries)
Max Delay: 300 seconds (5 minutes)
Timeout: 30 seconds per attempt
Connection Timeout: 10 seconds

These values can be customized per payment partner. Contact your integration manager for custom configurations.

Exponential Backoff Formula

base_delay = 2^(attempt_count)
jitter = random(0, base_delay * 0.1)
actual_delay = min(base_delay + jitter, 300)

Jitter (random delay) prevents thundering herd problem when multiple webhooks fail simultaneously.

When Webhooks Are Retried

FLUID automatically retries webhooks in these scenarios:

1. HTTP Status Codes (Retriable)

Status Code	Description	Retry?	Reason
408	Request Timeout	✅ Yes	Temporary timeout, likely recoverable
429	Too Many Requests	✅ Yes	Rate limited, retry after backoff
500	Internal Server Error	✅ Yes	Temporary server issue
502	Bad Gateway	✅ Yes	Gateway/proxy issue
503	Service Unavailable	✅ Yes	Temporary unavailability
504	Gateway Timeout	✅ Yes	Gateway timeout

2. Network Errors (Retriable)

Error Type	Retry?	Reason
Connection timeout	✅ Yes	Server didn't respond within 10s
Read timeout	✅ Yes	Response not received within 30s
Connection refused	✅ Yes	Server not accepting connections
DNS resolution failure	✅ Yes	Temporary DNS issue
SSL/TLS errors	✅ Yes	Certificate or handshake issues

3. HTTP Status Codes (Non-Retriable)

Status Code	Description	Retry?	Reason
200-299	Success	❌ No	Webhook delivered successfully
400	Bad Request	❌ No	Invalid request, won't succeed on retry
401	Unauthorized	❌ No	Authentication issue (check signature verification)
403	Forbidden	❌ No	Access denied
404	Not Found	❌ No	Endpoint doesn't exist
405	Method Not Allowed	❌ No	Wrong HTTP method
410	Gone	❌ No	Endpoint permanently removed

Client Errors Are Not Retried

4xx errors (except 408 and 429) indicate client-side issues that won't be resolved by retrying. Fix your endpoint and contact support to resend failed webhooks.

Webhook Delivery States

Each webhook delivery has one of these states:

`pending`

Initial state when webhook is created and waiting for delivery or retry.

json

{
  "id": 12345,
  "status": "pending",
  "attempt_count": 0,
  "max_attempts": 5,
  "next_attempt_at": "2025-01-28T10:15:00Z"
}

`delivered`

Successfully delivered (received 2xx response).

json

{
  "id": 12345,
  "status": "delivered",
  "attempt_count": 1,
  "response_code": 200,
  "response_body": "{\"received\":true}",
  "delivered_at": "2025-01-28T10:15:05Z"
}

`failed`

Failed temporarily and eligible for retry.

json

{
  "id": 12345,
  "status": "failed",
  "attempt_count": 2,
  "max_attempts": 5,
  "response_code": 500,
  "response_body": "Internal Server Error",
  "next_attempt_at": "2025-01-28T10:15:10Z"
}

`abandoned`

Failed permanently after exhausting all retries.

json

{
  "id": 12345,
  "status": "abandoned",
  "attempt_count": 6,
  "max_attempts": 5,
  "response_code": 503,
  "response_body": "Service Unavailable",
  "failed_at": "2025-01-28T10:16:30Z"
}

Handling Retries in Your Endpoint

1. Respond Quickly

Always respond with a 2xx status code within 30 seconds:

javascript

app.post('/webhooks/fluid', async (req, res) => {
  // ✅ CORRECT: Acknowledge receipt immediately
  res.status(200).json({ received: true });
  
  // Process asynchronously (don't block response)
  processWebhookAsync(req.body);
});

Don't do this:

javascript

app.post('/webhooks/fluid', async (req, res) => {
  // ❌ WRONG: Blocking response with slow processing
  await slowDatabaseOperation();
  await sendEmail();
  await updateInventory();
  
  res.status(200).json({ received: true });
});

2. Implement Idempotency

Use event_id to prevent duplicate processing when webhooks are retried:

javascript

const processedEvents = new Map(); // Or use database

app.post('/webhooks/fluid', async (req, res) => {
  const eventId = req.body.event_id;
  
  // Check if already processed
  if (processedEvents.has(eventId)) {
    console.log(`Event ${eventId} already processed`);
    return res.status(200).json({ received: true, duplicate: true });
  }
  
  // Verify signature
  if (!verifySignature(req.rawBody, req.headers['x-fluid-signature'])) {
    return res.status(401).json({ error: 'Invalid signature' });
  }
  
  // Mark as processing
  processedEvents.set(eventId, { status: 'processing', timestamp: Date.now() });
  
  // Acknowledge receipt
  res.status(200).json({ received: true });
  
  // Process asynchronously
  try {
    await processWebhook(req.body);
    processedEvents.set(eventId, { status: 'completed', timestamp: Date.now() });
  } catch (error) {
    console.error(`Error processing webhook ${eventId}:`, error);
    processedEvents.set(eventId, { status: 'error', error: error.message });
  }
});

3. Handle Rate Limiting

If your system has rate limits, return 429 status to signal FLUID to retry:

javascript

const rateLimiter = new RateLimiter({ maxRequests: 100, windowMs: 60000 });

app.post('/webhooks/fluid', async (req, res) => {
  // Check rate limit
  if (!rateLimiter.allow()) {
    console.log('Rate limit exceeded, requesting retry');
    return res.status(429).json({ 
      error: 'Rate limit exceeded',
      retry_after: 60 
    });
  }
  
  // Process webhook
  // ...
});

4. Return Appropriate Status Codes

Guide FLUID's retry behavior with correct HTTP status codes:

javascript

app.post('/webhooks/fluid', async (req, res) => {
  try {
    // Verify signature
    if (!verifySignature(req.rawBody, req.headers['x-fluid-signature'])) {
      // Don't retry - signature will never be valid
      return res.status(401).json({ error: 'Invalid signature' });
    }
    
    // Check if endpoint is being deprecated
    if (isEndpointDeprecated()) {
      // Don't retry - endpoint is gone
      return res.status(410).json({ error: 'Endpoint deprecated' });
    }
    
    // Temporary database issue
    if (isDatabaseUnavailable()) {
      // Retry - database might recover
      return res.status(503).json({ error: 'Database temporarily unavailable' });
    }
    
    // Process successfully
    await processWebhook(req.body);
    return res.status(200).json({ received: true });
    
  } catch (error) {
    console.error('Webhook processing error:', error);
    // Retry - unknown error might be temporary
    return res.status(500).json({ error: 'Internal server error' });
  }
});

Monitoring Webhook Deliveries

Check Delivery Status

Contact your FLUID integration manager to access webhook delivery monitoring dashboard:

Delivery Success Rate: Percentage of webhooks delivered successfully
Average Attempts: Average number of attempts before success
Abandoned Webhooks: Webhooks that failed all retries
Response Time: Time taken for your endpoint to respond
Failure Reasons: Common error codes and messages

Example Metrics

Last 24 Hours:
- Total Webhooks:    1,234
- Delivered:         1,198 (97.1%)
- Abandoned:            12 (1.0%)
- Pending:              24 (1.9%)
- Average Attempts:   1.3
- P95 Response Time: 850ms

Common Failure Patterns

Pattern	Cause	Solution
High timeout rate	Slow endpoint processing	Implement async processing
Abandoned during deployments	Server downtime	Use rolling deployments
401 errors	Wrong signature verification	Verify webhook secret
503 errors	Resource exhaustion	Scale infrastructure
Connection refused	Firewall/network issue	Check firewall rules

Troubleshooting Failed Deliveries

Abandoned Webhooks

If webhooks are being abandoned (failed all retries):

1. Check Endpoint Health

bash

# Test endpoint accessibility
curl -X POST https://your-domain.com/webhooks/fluid \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

# Expected: 200 OK (or 401 if signature verification fails)

2. Check Response Time

bash

# Measure response time
time curl -X POST https://your-domain.com/webhooks/fluid \
  -H "Content-Type: application/json" \
  -d '{"test": true}'

# Should complete within 30 seconds

3. Check Server Logs

Look for webhook processing errors:

bash

# Check application logs
tail -f /var/log/app.log | grep "webhooks/fluid"

# Check for errors
grep -i "error" /var/log/app.log | grep "webhook"

4. Verify Signature Implementation

javascript

// Add debug logging
function verifySignature(rawBody, signature, secret) {
  console.log('Raw body length:', rawBody.length);
  console.log('Signature:', signature);
  
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');
  
  console.log('Expected signature:', expectedSignature);
  console.log('Signatures match:', signature === expectedSignature);
  
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  );
}

Request Manual Resend

If webhooks are abandoned and you've fixed the issue, contact your FLUID integration manager to manually resend failed webhooks:

Provide:

Event ID(s) or date range
Transaction reference(s)
Reason for resend
Confirmation that endpoint is now healthy

Best Practices

✅ Do's

Return 2xx Immediately: Acknowledge receipt before processing
Process Asynchronously: Queue webhook for background processing
Implement Idempotency: Use event_id to prevent duplicate processing
Log Everything: Store webhook delivery IDs, timestamps, and outcomes
Monitor Metrics: Track success rates, response times, and failures
Handle Retries Gracefully: Recognize and handle duplicate webhooks
Return Appropriate Status Codes: Guide retry behavior correctly

❌ Don'ts

Don't Block Response: Never perform long operations before responding
Don't Return Errors for Business Logic Failures: Return 2xx even if your logic fails
Don't Ignore Idempotency: Always check for duplicate events
Don't Hardcode Timeouts: Be ready for 30-second timeout
Don't Skip Signature Verification: Never trust webhooks without verification

Advanced Configuration

Custom Retry Settings

Default retry settings work for most integrations, but custom configurations are available:

Setting	Default	Range	Description
`max_attempts`	5	1-10	Maximum retry attempts
`timeout`	30s	10-60s	Request timeout per attempt
`connection_timeout`	10s	5-30s	Connection establishment timeout

Contact your integration manager to customize these settings.

Retry for Specific Events Only

Configure selective retries based on event importance:

High Priority (max 10 retries):
- transaction.completed
- transaction.failed

Medium Priority (max 5 retries, default):
- transaction.processing
- transaction.reversed

Low Priority (max 2 retries):
- transaction.created
- transaction.pending

Testing Retry Behavior

Simulate Failures

Test retry logic by returning different status codes:

javascript

let attemptCount = 0;

app.post('/webhooks/fluid', (req, res) => {
  attemptCount++;
  
  // Simulate failures for first 3 attempts
  if (attemptCount <= 3) {
    console.log(`Attempt ${attemptCount}: Returning 503`);
    return res.status(503).json({ error: 'Service unavailable' });
  }
  
  // Succeed on 4th attempt
  console.log(`Attempt ${attemptCount}: Success`);
  res.status(200).json({ received: true });
});

Test Idempotency

javascript

const processedEvents = new Set();

app.post('/webhooks/fluid', (req, res) => {
  const eventId = req.body.event_id;
  
  if (processedEvents.has(eventId)) {
    console.log(`Duplicate event: ${eventId}`);
    return res.status(200).json({ received: true, duplicate: true });
  }
  
  processedEvents.add(eventId);
  console.log(`New event: ${eventId}`);
  
  res.status(200).json({ received: true, processed: true });
});

Retry Logic ​

How Retry Logic Works ​

Retry Schedule ​

Exponential Backoff Formula ​

When Webhooks Are Retried ​

1. HTTP Status Codes (Retriable) ​

2. Network Errors (Retriable) ​

3. HTTP Status Codes (Non-Retriable) ​

Webhook Delivery States ​

pending ​

delivered ​

failed ​

abandoned ​

Handling Retries in Your Endpoint ​

1. Respond Quickly ​

2. Implement Idempotency ​

3. Handle Rate Limiting ​

4. Return Appropriate Status Codes ​

Monitoring Webhook Deliveries ​

Check Delivery Status ​

Example Metrics ​

Common Failure Patterns ​

Troubleshooting Failed Deliveries ​

Abandoned Webhooks ​

1. Check Endpoint Health ​

2. Check Response Time ​

3. Check Server Logs ​

4. Verify Signature Implementation ​

Request Manual Resend ​

Best Practices ​

✅ Do's ​

❌ Don'ts ​

Advanced Configuration ​

Custom Retry Settings ​

Retry for Specific Events Only ​

Testing Retry Behavior ​

Simulate Failures ​

Test Idempotency ​

Next Steps ​

Webhooks Overview → ​

Event Types → ​

Error Handling Guide → ​

Retry Logic

How Retry Logic Works

Retry Schedule

Exponential Backoff Formula

When Webhooks Are Retried

1. HTTP Status Codes (Retriable)

2. Network Errors (Retriable)

3. HTTP Status Codes (Non-Retriable)

Webhook Delivery States

`pending`

`delivered`

`failed`

`abandoned`

Handling Retries in Your Endpoint

1. Respond Quickly

2. Implement Idempotency

3. Handle Rate Limiting

4. Return Appropriate Status Codes

Monitoring Webhook Deliveries

Check Delivery Status

Example Metrics

Common Failure Patterns

Troubleshooting Failed Deliveries

Abandoned Webhooks

1. Check Endpoint Health

2. Check Response Time

3. Check Server Logs

4. Verify Signature Implementation

Request Manual Resend

Best Practices

✅ Do's

❌ Don'ts

Advanced Configuration

Custom Retry Settings

Retry for Specific Events Only

Testing Retry Behavior

Simulate Failures

Test Idempotency

Next Steps

Webhooks Overview →

Event Types →

Error Handling Guide →