ESS' Kafka Configuration
ESS’ services communicate with each other by sending messages through Kafka. For example:
Pod Storage Service sends resource notification events through Kafka and WebSocket Notification Service consumes these events.
ESS services (including the Auditing service) send audit events through Kafka and the Auditing Service consumes these events.
The following discusses some key Kafka configurations.
For more information on Kafka configuration, see https://quarkus.io/guides/kafka#kafka-configuration.
Configuring Bootstrap Kafka Brokers
You can configure ESS services to connect with Kafka either globally or per channel.
Global Configuration
To configure globally such that all the ESS’ message channels use the same Kafka instance, you can set KAFKA_BOOTSTRAP_SERVERS configuration.
Per Channel Configuration
To configure per channel such that the message channel uses a separate Kafka instance, for the services that use the channel, configure the corresponding input and output bootstrap servers for that channel MP_MESSAGING_[INCOMING|OUTGOING]_[CHANNEL]_BOOTSTRAP_SERVERS.
Configuring Password for Encryption and Decryption of Messages
By default, Inrupt enables data encryption for all data that pass through the Kafka messaging system.
Managing encryption keys through configuration
At any given point in time, ESS services are using an active master key to encrypt messages. Encrypted messages are tagged with the ID of the master key used to encrypt them. When a consumer receives a message, it searches for the key having the appropriate ID in its configuration, and uses it for decryption.
Here are the configuration variables to set to configure the multiple decryption key and the active key id:
INRUPT_KAFKA_MASTER_KEYS_<key-id>, with<key-id>set to an 8 alphanumeric characters value. Each key ID MUST be unique. This variable's value MUST be set to a strong password.INRUPT_KAFKA_ACTIVE_KEY_ID, set to one of the<key-id>of the defined master keys. This key being set to a non-existing value will prevent the services from starting. In the absence of this value, the legacy configuration is applied.
Unlike the legacy configuration, this is configured globally, not per-channel. There are measures in place to derive key material from this master key to maintain per-channel isolation, but that is an internal behavior which no longer requires operator management.
Rotating encryption key
It is good practice to regularily rotate keys. Here is the process we encourage operators to follow for minimal disruption:
Ensure the future key is present in the services configuration.
# Current active encryption key
INRUPT_KAFKA_MASTER_KEYS_140a39c3=e7e55b7a-7ce0-43ae-b26c-33f499c55a50
# Future encryption key
INRUPT_KAFKA_MASTER_KEYS_9e04c297=6a9832a0-d113-439f-8dcc-36f44043e44a
# Active key ID before the rotation
INRUPT_KAFKA_ACTIVE_KEY_ID=140a39c3Update the active key ID
# Rotated out legacy key, kept present for pending decryption
INRUPT_KAFKA_MASTER_KEYS_140a39c3=e7e55b7a-7ce0-43ae-b26c-33f499c55a50
# Current active encryption key
INRUPT_KAFKA_MASTER_KEYS_9e04c297=6a9832a0-d113-439f-8dcc-36f44043e44a
# Active key ID has been updated in the rotation
INRUPT_KAFKA_ACTIVE_KEY_ID=9e04c297After it has stopped being used (see the metrics section), remove the legacy key, and introduce the next encryption key to be able to restart the cycle when needed.
# Legacy key with ID 140a39c3 has been removed
# Current active encryption key
INRUPT_KAFKA_MASTER_KEYS_9e04c297=6a9832a0-d113-439f-8dcc-36f44043e44a
# Active key id is unchanged
INRUPT_KAFKA_ACTIVE_KEY_ID=9e04c297
# Newly introduced key, for future rotation (not used yet)
INRUPT_KAFKA_MASTER_KEYS_313e4426=be5bcbca-8f34-43ef-993d-647e496a1472Monitoring key usage
In order to safely remove keys that have been rotated out, key usage metrics are available to monitor when all the messages encrypted with a given key have been processed. The following counters are exposed: application_com_inrupt_components_kafka_encryption_KafkaChannelMonitor_decryption_total and application_com_inrupt_components_kafka_encryption_KafkaChannelMonitor_encryption_total, both tagged with the SHA256 hash of the key ID (keyId_sha256) and the topic name (topic). Messages encrypted with the legacy configuration use the special value legacy-password for the keyId_sha256 tag.
Last updated