> For the complete documentation index, see [llms.txt](https://docs.inrupt.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.inrupt.com/ess/administration/reliable-event-delivery.md). # Reliable Event Delivery How ESS guarantees delivery of audit and other events to Kafka using a transactional outbox, including at-least-once delivery semantics, configuration, and metrics. {% hint style="info" %} **New in ESS 3.3** {% endhint %} ESS services (including the [Auditing Service](/ess/services/service-auditing.md)) publish events — such as audit events — to Kafka using a **transactional outbox**. This guarantees that every committed database change produces a corresponding event, and that no event is published for a change that was rolled back. {% hint style="info" %} **What the Outbox Guarantees**\ Because the event is written in the same database transaction as the change it describes, the two cannot diverge: a committed change always has a corresponding event, and a rolled-back change never produces one. {% endhint %} ## How It Works A service writes its business data and the corresponding event(s) as part of a single database transaction. Once that transaction commits, the service attempts immediate delivery to Kafka; if delivery doesn't succeed right away, a background process retries it on a schedule (see [Configuration](#configuration)). Because delivery is retried until it succeeds, a message may occasionally be delivered more than once. Downstream consumers should treat delivery as **at-least-once** and deduplicate using the message's **`id`** field. {% hint style="success" %} **Multi-Replica Safety**\ Multiple replicas of a service can retry delivery concurrently without any external coordination (e.g., leader election) and without duplicating work. {% endhint %} ## Failure Handling Delivery failures are handled differently depending on whether they're likely to resolve on their own: * **Transient failures** (e.g., a temporary Kafka connectivity issue) are retried indefinitely. Rather than logging on every attempt, a warning is logged periodically (see **`transient-warn-interval`** below) so persistent connectivity problems remain visible without flooding logs. * **Non-transient failures** are retried up to a configurable maximum (**`max-persistent-retries`**). Once that maximum is reached, the message is moved to an internal dead-letter store. {% hint style="info" %} **Dead-Lettered Messages**\ Dead-lettering does not occur under normal operation; it is a last-resort backstop for repeated non-transient delivery failures. Dead-lettered messages are retained in the service's database, where an operator can retrieve them. Use the **`inrupt_outbox_messages_dlq_total`** metric (see [Observability](#observability)) to alert when messages are dead-lettered. {% endhint %} ## Configuration The following configuration options apply to every ESS service that publishes events to Kafka via the transactional outbox. These options are set as environment variables on the corresponding service deployments. See [Customize Configurations](/ess/installation/customize-configurations.md) for how to apply configuration changes. #### INRUPT\_EVENTS\_OUTBOX\_POLL\_INTERVAL *Default*: **`1s`** How often the background delivery process checks for undelivered events. #### INRUPT\_EVENTS\_OUTBOX\_MAX\_POLL\_INTERVAL *Default*: **`30s`** The upper bound the check interval may back off to when there is no outstanding work. #### INRUPT\_EVENTS\_OUTBOX\_BATCH\_SIZE *Default*: **`100`** The maximum number of events processed per delivery cycle. #### INRUPT\_EVENTS\_OUTBOX\_POLLER\_GRACE\_PERIOD *Default*: **`10s`** The minimum age an event must reach before the background process will pick it up, so it doesn't race the immediate post-commit delivery attempt and cause a duplicate. #### INRUPT\_EVENTS\_OUTBOX\_MAX\_PERSISTENT\_RETRIES *Default*: **`3`** The number of retry attempts for a non-transient delivery failure before the event is dead-lettered. Does not apply to transient failures, which retry indefinitely. #### INRUPT\_EVENTS\_OUTBOX\_TRANSIENT\_WARN\_INTERVAL *Default*: **`10`** For transient failures, how many retry attempts elapse between logged warnings. ## Observability Services expose the following Prometheus metrics for event delivery. These are the exposition names, as scraped — use them directly in queries and alert rules:
inrupt_outbox_messages_delivered_totalCount of messages successfully delivered to Kafka.
inrupt_outbox_messages_retries_totalCount of delivery attempts that failed and were retried (transient or non-transient).
inrupt_outbox_messages_dlq_totalCount of messages dead-lettered after exhausting retries.
See also [Service Metrics: Prometheus Configuration](/ess/administration/service-metrics-prometheus-configuration.md).