Reliable Event Delivery

How ESS guarantees delivery of audit and other events to Kafka using a transactional outbox, including at-least-once delivery semantics, configuration, and metrics.

New in ESS 3.3

ESS services (including the Auditing Service) publish events — such as audit events — to Kafka using a transactional outbox. This guarantees that every committed database change produces a corresponding event, and that no event is published for a change that was rolled back.

What the Outbox Guarantees Because the event is written in the same database transaction as the change it describes, the two cannot diverge: a committed change always has a corresponding event, and a rolled-back change never produces one.

How It Works

A service writes its business data and the corresponding event(s) as part of a single database transaction. Once that transaction commits, the service attempts immediate delivery to Kafka; if delivery doesn't succeed right away, a background process retries it on a schedule (see Configuration).

Because delivery is retried until it succeeds, a message may occasionally be delivered more than once. Downstream consumers should treat delivery as at-least-once and deduplicate using the message's id field.

Multi-Replica Safety Multiple replicas of a service can retry delivery concurrently without any external coordination (e.g., leader election) and without duplicating work.

Failure Handling

Delivery failures are handled differently depending on whether they're likely to resolve on their own:

• Transient failures (e.g., a temporary Kafka connectivity issue) are retried indefinitely. Rather than logging on every attempt, a warning is logged periodically (see transient-warn-interval below) so persistent connectivity problems remain visible without flooding logs.

• Non-transient failures are retried up to a configurable maximum (max-persistent-retries). Once that maximum is reached, the message is moved to an internal dead-letter store.

Dead-Lettered Messages Dead-lettering does not occur under normal operation; it is a last-resort backstop for repeated non-transient delivery failures. Dead-lettered messages are retained in the service's database, where an operator can retrieve them. Use the inrupt_outbox_messages_dlq_total metric (see Observability) to alert when messages are dead-lettered.

Configuration

The following configuration options apply to every ESS service that publishes events to Kafka via the transactional outbox.

These options are set as environment variables on the corresponding service deployments. See Customize Configurations for how to apply configuration changes.

INRUPT_EVENTS_OUTBOX_POLL_INTERVAL

Default: 1s

How often the background delivery process checks for undelivered events.

INRUPT_EVENTS_OUTBOX_MAX_POLL_INTERVAL

Default: 30s

The upper bound the check interval may back off to when there is no outstanding work.

INRUPT_EVENTS_OUTBOX_BATCH_SIZE

Default: 100

The maximum number of events processed per delivery cycle.

INRUPT_EVENTS_OUTBOX_POLLER_GRACE_PERIOD

Default: 10s

The minimum age an event must reach before the background process will pick it up, so it doesn't race the immediate post-commit delivery attempt and cause a duplicate.

INRUPT_EVENTS_OUTBOX_MAX_PERSISTENT_RETRIES

Default: 3

The number of retry attempts for a non-transient delivery failure before the event is dead-lettered. Does not apply to transient failures, which retry indefinitely.

INRUPT_EVENTS_OUTBOX_TRANSIENT_WARN_INTERVAL

Default: 10

For transient failures, how many retry attempts elapse between logged warnings.

Observability

Services expose the following Prometheus metrics for event delivery. These are the exposition names, as scraped — use them directly in queries and alert rules:

inrupt_outbox_messages_delivered_total	Count of messages successfully delivered to Kafka.
inrupt_outbox_messages_retries_total	Count of delivery attempts that failed and were retried (transient or non-transient).
inrupt_outbox_messages_dlq_total	Count of messages dead-lettered after exhausting retries.

See also Service Metrics: Prometheus Configuration.