> For the complete documentation index, see [llms.txt](https://docs.inrupt.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.inrupt.com/ess/services/appendix/appendix-kafka-configuration.md).

# ESS' Kafka Configuration

ESS’ services communicate with each other by sending messages through Kafka. For example:

* [Pod Storage Service](https://docs.inrupt.com/ess/services/service-pod-management/service-pod-storage.md) sends resource notification events through Kafka and the [Notification Delivery Service](https://docs.inrupt.com/ess/services/service-notification/notification-delivery-service.md) consumes these events.
* ESS services (including the Auditing service) send audit events through Kafka and the [Auditing Service](https://docs.inrupt.com/ess/services/service-auditing.md) consumes these events.

The following discusses some key Kafka configurations.

For more information on Kafka configuration, see <https://quarkus.io/guides/kafka#kafka-configuration>.

{% hint style="info" %}
**Tip**\
Kafka **MUST** be configured with topic auto-creation enabled (i.e. **`auto.create.topics.enable = true`** ). See [Custom MSK Configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.html).
{% endhint %}

## Configuring Bootstrap Kafka Brokers

You can configure ESS services to connect with Kafka either globally or per channel.

### Global Configuration

To configure globally such that all the ESS’ [message channels](https://quarkus.io/guides/kafka#kafka-configuration) use the same Kafka instance, you can set **`KAFKA_BOOTSTRAP_SERVERS`** configuration.

{% hint style="info" %}
**Note**\
Inrupt-provided [Kustomize overlays](https://docs.inrupt.com/ess/installation.md#step-1-initialize-the-installation-directory) read **`KAFKA_BOOTSTRAP_SERVERS`** from the **`kafka-config`** ConfigMap, which the Inrupt-provided Terraform creates.
{% endhint %}

### Per Channel Configuration

To configure per channel such that the message channel uses a separate Kafka instance, for the services that use the channel, configure the corresponding input and output bootstrap servers for that channel [**`MP_MESSAGING_[INCOMING|OUTGOING]_[CHANNEL]_BOOTSTRAP_SERVERS`**](https://quarkus.io/guides/kafka#kafka-configuration).

## Message Encryption

ESS does not encrypt the content of Kafka messages. With the Inrupt-provided Terraform, Kafka connections are encrypted in transit with TLS and authenticated with SASL/SCRAM, and the brokers encrypt their storage. If you run your own Kafka, enable TLS and encryption at rest on it.

### Audit Topics

ESS services publish audit events to **`auditv1eventsproducer`**. The [Audit Enricher](https://docs.inrupt.com/ess/services/service-auditing/service-audit-enricher.md) adds storage details and publishes the enriched events to **`auditv1events`**, which the [Auditing Service](https://docs.inrupt.com/ess/services/service-auditing.md) and the other services that read audit events consume. Before ESS 3.5, these topics were named **`auditv1eventsproducerencrypted`** and **`auditv1eventsencrypted`**.

### Deprecated Encryption Settings

The message encryption settings **`INRUPT_KAFKA_MASTER_KEYS_<key-id>`**, **`INRUPT_KAFKA_ACTIVE_KEY_ID`** and **`INRUPT_KAFKA_*_CIPHER_PASSWORD`**, and the serializer values **`com.inrupt.components.kafka.encryption.*`**, are deprecated. ESS services still accept them, but the Inrupt-provided overlays do not set them, and a future release removes them. Do not set them.
