Outbound webhooks

Champagne sends outbound HTTP POSTs to each club's configured backend endpoints. This is separate from the incoming Stripe endpoints at /api/webhooks/stripe. The receiving backend can decide whether to send email, SMS or push notifications; Champagne webhooks do not imply that any of those channels were used. Champagne also has an independent member inbox and email notification worker; see Member notifications.

Configure

An administrator opens Settings → Integrations → Webhook endpoints to add up to 10 endpoints per club. Each endpoint has its own URL, event selection and signing secret, so separate systems can each receive only what they need. The multi-select options and descriptions come from a global catalog table. Adding an endpoint generates its secret to store in the receiving backend; the endpoint menu reveals or resets it, sends a test, pauses or deletes the endpoint. Pause an endpoint or select no events to queue none. Only public HTTPS DNS names are accepted. Do not point a real club at webhook.site: embedded workers may deliver actual club events there. Redirects are not followed; Stripe's incoming webhook secret is unrelated.

Outbound delivery runs inside the API process locally and on Scaleway. Bootstrap initializes pg-boss's pgboss schema and outbound-webhooks queue before accepting requests, then starts four application-scoped delivery fibers polling every 500 ms. There is no separate worker image, container, CRON trigger, worker token or internal drain/test endpoint. The queue uses the same PGHOST, PGPORT, PGDATABASE, PGUSER, password secret (PGPASSWORD_FILE locally, PGPASSWORD in production) and DATABASE_CA_PEM as the application. Its separate pool is capped at three connections per API process. The runtime database role must be able to create the pgboss schema and its objects. Shutdown interrupts the scoped pollers and closes the queue pool.

Keep at least one API replica running (min_scale >= 1) to process deliveries and retries without incoming traffic. Each additional API process adds four pollers and up to three queue connections alongside five Database and five separate MailDatabase connections. Size database capacity for 13 connections per API process, with headroom for deployment overlap and operational clients. Scaling the API also scales webhook consumers; pg-boss coordinates claims across replicas. If every API process is stopped, jobs remain in PostgreSQL and processing resumes on startup, subject to queue retention.

Releases deploy the API image by digest and use the database-backed GraphQL readiness probe. Because the listener starts after queue initialization and fiber startup, this gates bootstrap; it does not prove receiver availability or ongoing queue progress. The disposable production-image check also submits a safe job referencing no business event, waits for the embedded consumer to complete it without HTTP delivery, and checks that queue sessions are absent after process exit. Post-exit absence does not prove the scoped finalizer ran: process termination also closes sockets. Backend integration coverage separately verifies live scoped disposal while the process remains alive. For a receiver-only smoke test, an authorized club administrator uses Test webhook or sendClubTestWebhook. This sends a synthetic signed event to the saved URL without member data; it bypasses the queue and does not test background retries. Use an isolated test club and a controlled receiver when validating the complete lifecycle.

Existing-environment cutover

This development change has no compatibility migration or automatic replay of legacy pending events. Choose the pending-event policy explicitly before deploying a version whose startup pushes the schema:

1. Pause business writes that could create outbound events. If pending legacy events must be retained, let the old worker drain them using the old schema, and verify no pending events or active claims remain. If this is disposable development data, explicitly approve discarding it instead; record that those notifications will not be delivered. Do not silently drop a backlog or assume pg-boss will adopt it.
2. Quiesce the old CRON trigger and any manual drain callers. Wait for all in-flight worker requests and claims to finish; stop the old worker before any schema push. Keep writes paused so the old producer cannot recreate a backlog between draining and cutover. Never run the legacy worker against the new schema.
3. Prepare the API's PG credentials/CA, schema privileges and capacity for 13 connections per process. Set min_scale >= 1 and an integer max_scale >= min_scale through a reviewed configuration change. Deploy the combined API implementation only after the legacy consumer is quiescent.
4. Verify GraphQL readiness, the embedded startup record and fresh queue snapshots. With a controlled receiver and isolated test club, verify a committed event is delivered and retry behavior works; the direct Test webhook mutation alone does not prove queue consumption. Resume business writes once the embedded lifecycle is verified.
5. Remove the old worker container, trigger, IAM key/policy/application and worker/synthetic credentials through a separately reviewed Terraform plan/apply against the reconciled existing state. Remove obsolete tfvars and protected GitHub environment settings after cutover. Ordinary image CD does not delete infrastructure or revoke credentials. Preserve unrelated API secrets, database and DNS.

All steps require the environment owner's approval; this source change performs no production operation.

Transactional queue

Events enter webhook_outbox and pg-boss in the same business transaction, once per enabled endpoint of the club that selected that event type. Each endpoint's copy is a separate delivery with its own job, retries and outcome, so a failing receiver never delays the others. The repository uses pg-boss's fromDrizzle transaction adapter, capturing the current Effect context so its SQL executes on the reserved transaction connection. A rollback removes both the business change and queued job. Job data contains the outbox delivery ID, which is also the job ID. Every delivery of one event carries the same envelope id. API requests never wait for the receiving backend.

Embedded delivery lifecycle

Business changes and queue jobs share one commit; each running API process consumes the same queue.

flowchart LR
  Request[Business request] --> Transaction[Drizzle Effect transaction]
  Transaction --> Business[Business rows]
  Transaction --> Outbox[One outbox delivery per subscribed endpoint]
  Transaction --> Adapter[pg-boss transaction adapter]
  Adapter --> Queue[PostgreSQL pg-boss queue]
  Queue --> Consumers[Four scoped fibers per API process]
  Consumers --> Receiver[Signed HTTPS delivery]
  Receiver --> Settlement[Transactional job and outbox settlement]
  Settlement --> Retry[Retry or complete]
  Retry --> Queue

A 2xx response acknowledges delivery. Timeouts, redirects and non-2xx responses use pg-boss's retry policy: 20 retries, exponential backoff starting at 2 seconds, capped at 1 hour, and a 60-second lease. Retry timing also depends on polling and expiration recovery; there is no immediate-delivery or cross-event ordering guarantee. After retry exhaustion, the job is failed and no longer retries automatically. Queue retention and completed/failed job deletion are configured for 30 days; delivered outbox records are pruned after 30 days. Events during an unconfigured or paused period are not queued or backfilled.

Deliveries are at least once: a crash after the receiver accepts a request but before Champagne records its acknowledgement can cause another POST. Persist id as a deduplication key before returning 2xx.

Concurrent deliveries and configuration changes

Each fiber fetches one job through pg-boss, which atomically claims due work across processes. No database connection or business-row lock is held while waiting for the receiver. The HTTP timeout is five seconds, with a ten-second overall delivery bound; the lease is not renewed.

Each delivery uses its endpoint's URL and signing secret read when it is claimed. A secret rotation can finish while an earlier request is in flight; that request still uses the previous secret. Receivers should allow a short overlap during rotation for in-flight requests. Subsequent claims use the updated configuration. Pausing or deleting the endpoint, or deselecting an event, makes subsequent attempts skip delivery. A request that already read the configuration may still reach the receiver.

Settlement requires an active job with the same retry count and an unexpired lease. Queue settlement and outbox outcome updates share a transaction. A late response from an expired or replaced attempt is ignored. A crash or interruption leaves the job active until pg-boss recovers it after expiration. This includes crashes after receiver acceptance, so receivers must deduplicate by event ID.

Request contract (v1)

The raw request body is UTF-8 JSON:

{
  "id": "a4e3d651-a06d-4e29-bd88-9870d6374a36",
  "type": "reservation.status_changed",
  "occurredAt": "2026-09-24T12:00:00.000Z",
  "clubId": "51000000-0000-4000-8000-000000000051",
  "data": {
    "reservationId": "05cb1d67-3935-4d87-a636-7ae1447f7060",
    "personId": "d60e28ed-8e79-4b9c-bc96-d1d93484af16",
    "categoryId": "1a3c3841-4d29-4394-8026-94341ef6f894",
    "previousStatus": "booked",
    "status": "confirmed",
    "startsAt": "2026-10-01T10:00:00.000Z",
    "endsAt": "2026-10-01T11:00:00.000Z",
    "guests": 1,
    "amountUnit": "credit_hundredths",
    "creditAmount": 2500,
    "monetaryAmount": null,
    "currency": null,
    "paid": false,
    "refunded": false
  }
}

Content-Type is application/json; X-Champagne-Event-Id repeats id. X-Champagne-Signature is t=<unix-seconds>,v1=<lowercase-hex-hmac-sha256>, calculated over the exact bytes of t + "." + rawBody using the club's one-time signing secret from Settings. Verify the HMAC in constant time, reject timestamps more than five minutes from your clock, then parse the JSON; do not reserialize before checking the signature. Keep a database unique constraint on the event ID. Return 2xx only after durable acceptance, and process notifications asynchronously on your side.

Emitted events

All events have clubId and stable object IDs; no names, email addresses, tokens, application answers or invoice details are included. Fetch further context through the authorized GraphQL API if needed.

Reservation status values are pending, booked, confirmed, completed, canceled; attendance is independent (unmarked, present, late, absent). startsAt and endsAt are null for undated conversation requests. paid means a Stripe payment intent has been recorded, not that a payment request has been created. Invoice amounts are in the smallest currency unit. Reservation amounts follow the explicit units below. A first sync of existing Stripe invoices can emit invoice.issued/invoice.paid; credit notes present on the initial projection are not emitted. A ticket scan does not mark attendance. Application payment/signature answers do not change application status and do not produce status events. Same-state retries and unchanged membership writes do not emit events.

Reservation amount units

Reservation lifecycle events (reservation.created, reservation.updated, and reservation.status_changed) publish a discriminated amount contract:

A booking priced at 25 credits emits creditAmount: 2500, even if the club sells 2 credits per euro. No conversion using the current exchange rate is performed. Zero-price credit bookings emit creditAmount: 0. This is the reservation price; it does not include the separate refundable credit deposit.

The retained direct Stripe reservation checkout path uses currency minor units. Reservations with a persisted Stripe account, checkout request/session, payment intent or refund identifier emit currency_minor, including before payment is confirmed. A Stripe purchase of credits does not mark individual credit bookings as monetary. paid and refunded indicate Stripe payment/refund records only; false does not mean a credit booking has an unpaid balance.

The former reservation amount field is removed. Previously queued or archived events without amountUnit retain their original payload and have ambiguous units: consumers must not interpret them as money or credits automatically. Invoice payloads retain their existing monetary amount and currency fields.

Member notification events

The following events supplement the original catalog. New catalog entries are available to club administrators; clubs with an existing saved selection must explicitly select the new types. Member inbox/email delivery does not depend on that selection or on a webhook URL.

reservation.proposed also includes an eventKey identifying the reservation and proposal version, plus validUntil set to the proposed start time. Member email delivery uses these fields to skip obsolete proposals; outbound webhooks retain the original lifecycle event.

Cancellation emits a recipient event per affected reservation/waitlist entry; an empty occurrence emits an event without a person. Recipient-level notification preferences do not suppress external webhooks. eventKey deduplicates scheduled/join milestones at production; delivery still requires receiver deduplication by envelope id. Existing webhook pause/no-backfill semantics remain. Sponsorship joining is deduplicated per sponsored person and sponsor, not per add-on or renewal. When a guest accepts a reservation invitation, the member inviter gets a "Your guest accepted" notification instead of a second guest.joined alert. Staff inviters have no member inbox and receive a direct email when their guest accepts a reservation or club guest invitation.

There are still no events for credit balances, sanctions, Mews/Book4Time stays or treatments, restaurant/SevenRooms no-shows, Lightspeed house accounts, Salto access or push/SMS preferences.

Operations

webhook_outbox retains one row per event and endpoint: the event, endpointId, attempt count, settlement timestamp and generic lastError (delivery-failed, not-selected, disabled, endpoint-deleted or not-configured, never a response body or secret). pg-boss owns scheduling, leases and terminal failure state; an unsettled outbox row alone does not prove a retry is pending. Inspect delivery metadata without selecting payloads or signing secrets:

select o.id, o.type, o."clubId", o."endpointId", o.attempts, o."deliveredAt", o."lastError",
       j.state, j.retry_count, j.retry_limit, j.start_after, j.started_on
from webhook_outbox o
left join pgboss.job j on j.id = o.id and j.name = 'outbound-webhooks'
where o."deliveredAt" is null
order by j.start_after nulls last;

Check webhook.queue logs for queued, active, retry, failed and completed counts and oldest queued age; webhook.delivery records sanitized outcomes. Investigate terminal failures and missing job rows before assuming automatic retry will recover them. There is no internal drain route or documented automatic replay operation. Protect database access, Terraform state and receiver logs; the receiver must not trust unsigned webhook bodies or assume delivery implies a user notification.

Delivery dashboard

The production bootstrap stack provisions the private Champagne webhook delivery dashboard from infra/scaleway/bootstrap/dashboards/webhooks.json using the existing managed Cockpit Loki source. Its 27 panels cover delivery outcomes and rates, retry attempts, duration percentiles, queue states and oldest queued age, snapshot freshness, worker failures/startup and event-type/log drilldowns. Review/apply the existing bootstrap stack to publish the dashboard, and deploy the core JSON logger contract before expecting data; a dashboard definition does not prove ingestion is live.

Effect's Logger.consoleJson emits message values webhook.delivery, webhook.queue and webhook.worker, with fields nested under annotations. Queries read annotations.module, annotations.outcome, annotations.eventType, annotations.attempt, annotations.durationMs, the queue counts/age, and worker operation/failure kind. Queue snapshots describe the same shared database from each API instance: panels take the MAX of the latest sample per instance, never sum replicas or accumulate repeated snapshots. The latest-sample window is three minutes; the MAX is a conservative sampled view, not an atomic live database count. Check freshness alongside counts; missing or stale samples are not healthy zeros. Native log retention limits historical views.

Local development continues using its existing worktree Waggle; no local Grafana is provisioned. The dashboard adds no alert rules, ingestion/retention billing changes or new compute.

Incoming Stripe webhook diagnostics

The /api/webhooks/stripe and /api/webhooks/stripe/accounts endpoints return a server-generated X-Request-Id. Unexpected processing failures remain HTTP 500 with Webhook processing failed and generate one correlated, sanitized Effect error log. Missing or invalid signatures remain HTTP 400 without an unexpected failure log. Raw event bodies, Stripe signatures and provider error messages are not logged. This concerns incoming Stripe callbacks, not the outgoing club webhook signature or event-ID contract described above.