A card charge can fail, time out, arrive twice, or succeed after the application has stopped waiting. Subscription billing has to handle all four without charging twice or granting access to the wrong customer.
I worked on this problem in a multi-tenant commerce platform where one merchant account can own multiple stores. The billing system needed monthly and annual plans, trials, automatic renewals, saved cards, failed-payment recovery, and store suspension. Two decisions shaped the rest of the implementation: where entitlement lives, and how to cross the gap between a PostgreSQL transaction and an external payment gateway.
Choosing the right subscription boundary
We first had to decide what the subscription covered.
A tenant account can own more than one store. Those stores may use different plans, start on different dates, or fail renewal independently. A subscription attached only to the tenant would force all stores to share one lifecycle or require billing rules to infer which part of an account a payment covered.
We made the store the unit of subscription entitlement. Each store has one mutable subscription record containing its current plan, billing cycle, period, status, pending plan change, and cancellation state. The database enforces that a store cannot have competing active subscription records.
Payment cards have a different boundary. They belong to the tenant account and can be reused across that tenant's stores. Entitlement follows the store; payment methods follow the tenant.
Invoices and payment attempts carry both tenant and store identifiers. That looks redundant, but settlement code can verify that the invoice, payment, subscription, and store all agree before granting access. In billing, a little redundant scope is useful when it turns a mistaken association into a rejected payment binding rather than entitlement for the wrong store.
The subscription row represents current state. Invoices, payment attempts, and audit records preserve the history, while the worker gets one stable subscription ID to process.
Separating what is owed from attempts to collect it
We modeled invoices and payment attempts as separate records.
An invoice records what a store owes for a service period: the amount, currency, dates, and a snapshot of the plan terms. A payment records one attempt to collect that invoice, including whether the attempt is pending, paid, failed, or requires customer action, plus its gateway identifiers and failure details.
When a charge fails, the invoice remains valid even though the first payment attempt is over. A retry creates another attempt against the same invoice instead of overwriting the failure or issuing a second invoice for the same period.
A subscription can have only one renewal invoice for a given period, and an invoice can have only one open payment attempt at a time. Completed failures remain as history, while the next attempt becomes a new row.
Initial checkout follows the same model. The system creates a pending invoice using the plan and billing cycle selected by the merchant. If checkout restarts with different terms, stale pending invoices are voided and their open attempts are closed. If the terms have not changed, the pending invoice can be reused.
The service period on an initial invoice is provisional until settlement. On success, both the invoice and subscription periods are rewritten from the same paid_at timestamp. This prevents time spent completing checkout from reducing the service period the customer paid for.
Monthly and annual subscriptions share one UTC date calculation. It uses calendar arithmetic and clamps month-end dates, so a subscription starting on January 31 renews on the last valid day of February instead of rolling into March.
Making renewals safe to retry
Automatic renewal is a sequence that can be repeated by a scheduler, a restarted process, a delayed webhook, or a merchant returning through a recovery page.
A scheduled worker finds subscriptions whose current period has ended. Before charging, it serializes renewal preparation, calculates the next period, and creates or reuses the renewal invoice and payment attempt. A local payment ID becomes the gateway idempotency key.
Database constraints guard the renewal flow against duplicate invoices, competing open attempts, repeated provider references, duplicate webhook events, and conflicting default payment methods. These constraints are the final line of defense when multiple workers reach the same decision at once; an application-level "find then create" check is not enough on its own.
Webhooks add another retry path. A worker records progress for each event so a duplicate can be acknowledged without running settlement twice, while abandoned work can be picked up safely by a later worker.
We bind each webhook to its local payment attempt and verify that the local and provider details agree. If the match is ambiguous, we reject the webhook rather than attach it to the newest open payment. Manual reconciliation is safer than applying money to the wrong store.
Local settlement treats success as terminal. If a paid webhook commits while the original request is still timing out, the later non-paid result cannot move the invoice or subscription backward.
Dealing with the database/payment-gateway boundary
PostgreSQL cannot atomically commit with an external payment gateway.
Holding database locks during the network request would lengthen the transaction and increase contention. The gateway could still charge successfully at the same moment the database transaction failed.
We split preparation from settlement:
Before the request leaves the application, a short transaction marks the charge as started. If the process crashes, a later worker sees the incomplete attempt instead of creating a fresh charge blindly.
When the gateway returns success, we perform a status lookup before marking the invoice paid. An unknown network outcome pauses automatic retries and leaves the attempt uncertain or requiring action. If a provider payment ID exists, the worker reconciles that payment directly. Retries respect the provider's idempotency guarantees; once those guarantees no longer apply, the system fails closed and requires authoritative reconciliation.
This flow adds bookkeeping, but every interruption leaves a local record of what may have happened and what the next worker can safely do.
Modeling failed payments in the subscription lifecycle
A failed renewal moves the subscription from active to past_due and leaves the renewal invoice pending.
The recovery policy uses spaced automatic attempts and a bounded grace window from the renewal period start. If the gateway requires an additional security step, the attempt stays in requires_action; automatic replacement attempts stop while that action remains valid.
The merchant recovery screen can continue the security step, use another saved card, or add a new card. It also refuses to start a competing manual payment while an automatic gateway charge is still in flight.
If payment succeeds during the valid recovery window, the same invoice becomes paid and the subscription returns to active for that invoice's service period. If retries are exhausted and grace has elapsed, the subscription becomes expired and an active store is suspended.
Billing recovery reverses only suspensions caused by billing and leaves administrative suspensions alone. A store resumes only when the related account is active and the failed invoice has been settled.
Email notifications and store pause or resume calls run after the billing transaction. The transaction writes small outbox markers first; later processing delivers them and clears the markers. Billing settlement commits independently, and later sweeps retry failed delivery.
Lessons from the implementation
- Put entitlement on the resource being sold. In a multi-store SaaS, the tenant pays and each store receives its own access.
- Keep invoices separate from payment attempts. What is owed should survive several efforts to collect it.
- Design unknown payment outcomes explicitly. A timeout after charge submission leaves the outcome uncertain; retrying it as a decline can create a duplicate charge.
- Layer retry safety. Idempotency keys help, but database constraints, serialized preparation, webhook deduplication, binding checks, and terminal success rules cover different failure modes.