VERIFIED PUBLIC REPAIR
A CRM timeout turned one lead submission into two records.
A transparent demonstration of how an upstream timeout, an unsafe retry and missing idempotency protection can duplicate CRM records.
This is a public demonstration project, not a customer engagement or production incident.
FAILURE MODE
The first request succeeded. Its response did not.
The deterministic mock CRM committed the lead record and then returned a simulated upstream timeout. The application retried the request without a stable idempotency key, so one user submission created two CRM records.
FAILURE SEQUENCE
- 01User submits one valid lead
- 02CRM creates the first record
- 03Upstream response is lost behind a simulated timeout
- 04Unsafe retry creates a second CRM record
BUSINESS IMPACT
General operational risks illustrated by the demonstration, not measured customer impact.
- Duplicate records can fragment follow-up
- Ownership may become inconsistent
- Delivery status becomes unreliable
- Manual cleanup may be required
REPAIR
The retry became bounded, classified and idempotent.
- 01Server-side validation runs before CRM transport creation
- 02One submission ID is generated per accepted request
- 03The same idempotency key is reused across retry attempts
- 04Retry is limited to timeout, 502, 503 and 504
- 05Total CRM attempts are capped at two
- 06Permanent and authorisation failures fail closed
- 07Timeout cleanup uses AbortController
- 08Structured error metadata excludes lead PII
- 09UI prevents double-submit while a request is in flight
- 10Success and failure states reflect the actual sync result
EVIDENCE
The request count stayed the same. The record count did not.
The repaired flow retried the same submission safely and returned the existing CRM record instead of creating another one.
This result applies only to the demonstrated request-scope retry window.
BEFORE
- CRM attempts
- 2
- Records created
- 2
- Duplicate detected
- Yes
- Idempotency protected
- No
REPAIRED
- CRM attempts
- 2
- Records created
- 1
- Duplicate detected
- No
- Idempotency protected
- Yes
TECHNICAL RECORD
Repair Record
- Submission identity
- One server-generated ID per accepted request
- Retry policy
- Maximum two attempts
- Retryable failures
- Timeout · 502 · 503 · 504
- Non-retryable examples
- 400 · 401 · 403 · unclassified 500
- Idempotency
- Same key reused across attempts
- Validation
- Rejected before CRM transport creation
- Logging
- Structured and PII-safe
- Client protection
- In-flight double-submit guard
- Verification
- 41 tests · lint · production build · CI
VERIFICATION
Verified beyond the happy path.
Validation
- Valid payload accepted
- Missing fields rejected
- Invalid email rejected before CRM transport creation
Retry policy
- One retry after transient failure
- Stable key reused across attempts
- Non-retryable failures not retried
- Unlisted 500 status not retried
Idempotency
- Second attempt replays the existing record
- Record count remains one
- Duplicate prevention is reflected in diagnostics
Timeout
- Abort behaviour
- Timer cleanup before timeout completion
Logging
- Structured metadata
- No lead PII
UI
- Loading state
- In-flight double-submit protection
- Success state
- Safe failure state
- 41/41 tests passed
- Lint passed
- Production build passed
- npm audit --omit=dev clean
- GitHub Actions passed
PUBLIC EVIDENCE
Public Record
- Issue #1Single lead submissions can create duplicate CRM recordsStatus: Open
- Pull request #2Prevent duplicate CRM leads during retryable timeoutsStatus: Review-ready pull request · reviewed head frozen
- Quality checks41 tests · lint · buildStatus: Passed
- crm-sync-repair-demo-v1Reviewed and frozen demonstration headStatus: Frozen record
PRODUCTION BOUNDARY
The demo proves the repair pattern, not distributed persistence.
This demonstration uses a deterministic, request-scoped mock CRM. A production implementation would require durable idempotency storage across requests, processes, regions and restarts.
A production implementation would typically use:
- CRM-native idempotency support, or
- A durable application-side idempotency store
- An end-to-end submission key
- Expiration and replay rules
- Cross-process consistency
- Monitoring and operational alerting