Auditing Service#
ESS Auditing service audits various activities from the ESS services, including itself.
By default, auditing is enabled and logs the audit events to sysout
.
Note
Auditing service continues to run when auditing is disabled; instead, disabling auditing stops the ESS services from publishing audit events.
ESS uses asynchronous messaging for auditing.
Audit Events#
Note
To ensure that audit messages are delivered, the Audit service may send the message more than once.
For instance, multiple instances of the Audit service may exist when
applying configuration changes or when performing a rolling upgrade.
During this time, duplicate messages (i.e., messages with the same
unique id
value) can occur if the old Audit instance processes a
message but shuts down before acknowledging the message on Kafka,
and then, the new Audit instance processes the unacknowledged
message.
The following events are audited:
Services |
Event Name |
Notes |
---|---|---|
Most Services [4] |
|
Service Startup/Shutdown. |
Purgeable services (i.e. Access Grant, Authorization, Broker, Pod Provision, and WebID) |
|
Purge start/complete. |
Access Grant Service |
|
Access Request/Grant/Denial lifecycle events. Added in version 2.2.0: [3] |
Authorization Service |
|
ACR Lifecycle events. |
Pod Storage Service |
|
Resource Lifecycle events. |
|
Successful resource read events ( To enable, see
Added in version 2.1. |
|
Pod Provision Service |
|
Pod Provisioned event. |
Purger application |
|
Purge start/end (successful or not). |
Query Service |
|
Query events. |
Query Indexer |
|
Query Indexer events. |
Solid OIDC Broker Service |
|
Authentication/Authorization flow. |
UMA Service |
|
UMA Grant Flow. |
WebID Service |
|
WebID Profile events. |
The following services do not produce audit events:
Notification Gateway service
WebSocket Notification service
Start service
For examples, see Appendix: Audit Events Examples.
Audit Event Message Internal Format#
Internally, ESS’ audit event messages are in RDF and serialized as ActivityStreams 2.0 JSON-LD documents:
Note
Although the following document shows all possible fields for an event message, the specific events determine which fields appear.
Starting in version 2.2, Pod-related event messages include:
the actor information (WebID)
the application information (Client ID).
the Pod information (Pod root url and Pod data subject)
{
"@context":[
"https://www.w3.org/ns/activitystreams",
"https://schema.inrupt.com/audit/v1.jsonld"
],
"id":"<UUID of the event>",
"type": [ "Activity", <type2>,... ],
"name":"<event name>",
"summary": "<event description>",
"generator": <JSON document identifying the software producing the event>,
"actor": [ <JSON document identifying the actor associated with the event>, ... ],
"object": [ <JSON document identifying the object associated with the event>, ... ],
"instrument": [ <JSON document identifying the client/application associated with the event>,
<JSON document with associated OpenTelemetry data>,
<JSON document with associated application-defined metadata>,
<JSON document identifying the Pod root and data subject>, ... ],
"result": [ <JSON document containing associated results for the event, if any> ],
"published": "<datetime>",
"identifier":"<identifier to use for correlated events>"
}
|
Specifies the JSON-LD contexts. |
|
Universally Unique IDentifier (UUID) for the event. |
|
An array of event types; e.g., |
|
Name that denotes the event; e.g., See Audit Events for a list of audited events names. |
|
Short description associated with the message |
|
JSON document identifying the software (e.g., service)
producing the event. For example, the "generator": {
"id": "<service URL>"
"type": ["SoftwareApplication"],
"name": "<application name>",
"qualifiedAssociation": "<processId>",
"wasAssociatedWith": "<Kubernetes pod name>"
}
|
|
An array of JSON documents that identify the agents, if any, associated with the event. The actor’s identity can be denoted by various combination of fields, such as (list below is not exhaustive):
The field can also be empty Note Starting in version 2.2, For example:
|
|
An array of JSON documents that identify the objects associated with the event; that is, the object(s) against which the action is performed. The object can be denoted by various combination of fields, such as (list below is not exhaustive):
For example:
|
|
An array of JSON documents that identify:
|
|
An array of JSON documents that contains associated results. For example:
Added in version 2.1.5. |
|
The timestamp of the event. |
|
Identifier to use for correlated events from a single
service that have occurred within the same request. To correlate
events across services for a request, use the OpenTelemetry
|
For examples, see Appendix: Audit Events Examples.
Integration with External Logging Systems#
The ESS Auditing service can log to:
sysout
(default) [7]Syslog
When logging audit messages to sysout
(the default), Auditing
service uses the shared ESS logging components (available
starting in 2.2.0), which use JSON as the default
formatting for log messages. As such, starting in 2.2.0, audit
messages logged to sysout
are contained in the auditEvent
field of the log message. The audit event message itself (i.e., the
auditEvent
value in v2.2+) is serialized as a JSON-LD documents
(as in earlier releases).
Integration with Syslog#
The ESS Auditing service can integrate with Syslog. When integrating with Syslog, ESS audit events are converted to Syslog message format:
<priority>version timestamp hostname service processId messageId message
Where:
hostname
,service
,processId
andmessageId
values are taken extracted from the audit eventagent
field.message
is the full ESS audit event message in JSON.
For example (the event message has been abbreviated):
<110>1 2022-01-12T20:17:08.387Z ess-pod-storage-84648cfc95-qs865 inrupt-storage-postgres-s3 85 urn:uuid:579668c1-4e14-4fad-aea3-0000000005 { "@context": [ "https://www.w3.org/ns/activitystreams", "https://schema.inrupt.com/audit/v1.jsonld" ], "id": "urn:uuid:579668c1-4e14-4fad-aea3-0000000005", "type": ["Activity", "Create"], "name": "resource-created", "summary": "Resource has been created", ... }
By default, the Auditing service logs to sysout
. To have the
service output to Syslog instead, see
Manage Auditing.
See also Syslog configuration options.
For more information on Syslog, see RFC 5424.
Integration with Sentinel#
The ESS Audit service can integrate with Microsoft Sentinel
When integrating with Microsoft Sentinel, the ESS audit events are
converted into a Sentinel-specific format and POST
’ed to the
Sentinel service.
By default, the Auditing service logs to sysout
. To have the service
output to Sentinel instead, see
Manage Auditing.
See also Sentinel configuration options.
Configuration#
As part of the installation process, Inrupt provides base Kustomize overlays and associated files that require deployment-specific configuration inputs.
The following configuration options are available for the service and may be set as part of updating the inputs for your deployment. The Inrupt-provided base Kustomize overlays may be using updated configuration values that differ from the default values.
Auditing Service: General Configuration#
- QUARKUS_HTTP_PORT#
Default:
8080
The HTTP port the audit service runs on.
- QUARKUS_LOG_LEVEL#
Default:
INFO
Logging level.
Audit Service: Kafka#
Tip
See also ESS’ Kafka Configuration.
- INRUPT_KAFKA_AUDITV1EVENTSENCRYPTED_CIPHER_PASSWORD#
The strong cipher key to use when running auditing with encrypted messages over the
auditv1eventsencrypted
topic.
- INRUPT_KAFKA_AUDITV1EVENTSPRODUCERENCRYPTED_CIPHER_PASSWORD#
The strong cipher key to use when running auditing with encrypted messages over the
auditv1eventsproducerencrypted
topic.Added in version 2.2.0.
- KAFKA_BOOTSTRAP_SERVERS#
Default:
localhost:9092
Comma-delimited list of Kafka broker servers for use by ESS services, including this service.
Setting
KAFKA_BOOTSTRAP_SERVERS
configures ESS to use the same Kafka instance(s) for all its Kafka message channels (e.g.,solidresource
andauditv1out
message channels). This service uses theauditv1in
andauditv1out
channels.Note
Inrupt-provided overlays default to using
KAFKA_BOOTSTRAP_SERVERS
.To use a different Kafka instance for the
auditv1in
andauditv1out
channels, use specific message channel configuration.See also ESS’ Kafka Configuration.
Auditing Service: Syslog Configuration#
By default, the Auditing service logs to sysout
. To have the service
output to Syslog instead:
Customize your deployment to output to Syslog. See Manage Auditing for details.
Update configuration for Syslog integration. The following configuration options are available for integration with Syslog.
- INRUPT_AUDIT_SYSLOG_HOST#
Default:
localhost
The syslog server hostname that the audit service will connect to.
- INRUPT_AUDIT_SYSLOG_PORT#
Default:
514
The syslog server port that the audit service will connect to.
- INRUPT_AUDIT_SYSLOG_PROTOCOL#
Default:
SSL_TCP
The protocol used to connect to the syslog server. Valid values are:
TCP
SSL_TCP
(Default value starting in 2.2.1)
Auditing Service: Sentinel Configuration#
By default, the Auditing service logs to sysout
. To have the service
output to Microsoft Sentinel instead:
Customize your deployment to output to Sentinel. See Manage Auditing for details.
Update configuration for Sentinel integration. The following configuration options are available for integration with Microsoft Sentinel.
Log Redaction#
- INRUPT_LOGGING_REDACTION_NAME_ACTION#
Default: REPLACE
Type of the redaction to perform. Supported values are:
Action
Description
REPLACE
Default. Replaces the matching text with a specified replacement.
PLAIN
Leaves the matching field unprocessed. Only available if the redaction target is a field (i.e.,
INRUPT_LOGGING_REDACTION_{NAME}_FIELD
).DROP
Suppresses the matching field. Only available if the redaction target is a field (i.e.,
INRUPT_LOGGING_REDACTION_{NAME}_FIELD
).PRIORITIZE
Changes the log level of the matching message.
SHA256
Replaces the matching text with its hash.
If the action is
REPLACE
(default), see alsoINRUPT_LOGGING_REDACTION_{NAME}_REPLACEMENT
.If the action is to
PRIORITIZE
, see alsoINRUPT_LOGGING_REDACTION_{NAME}_LEVEL
.
For more information on log redaction, see Logging Redaction.
Added in version 2.2.0.
- INRUPT_LOGGING_REDACTION_NAME_ENABLED#
Default:
true
A boolean that determines whether the redaction configurations with the specified
INRUPT_LOGGING_REDACTION_{NAME}_
prefix is enabled.For more information on log redaction, see Logging Redaction.
Added in version 2.2.0.
- INRUPT_LOGGING_REDACTION_NAME_EXCEPTION#
Fully qualified name of the exception class to match in the log messages (includes inner exception). Configure to target an exception message class.
For more information on log redaction, see Logging Redaction.
Added in version 2.2.0.
- INRUPT_LOGGING_REDACTION_NAME_FIELD#
Exact name of the field to match in the log messages. Configure to target a specific log message field for redaction.
For more information on log redaction, see Logging Redaction.
Added in version 2.2.0.
- INRUPT_LOGGING_REDACTION_NAME_LEVEL#
A new log level to use for the log message if the
INRUPT_LOGGING_REDACTION_{NAME}_ACTION
isPRIORITIZE
.Added in version 2.2.0.
- INRUPT_LOGGING_REDACTION_NAME_PATTERN#
A regex (see Java regex pattern) to match in the log messages. Configure to target log message text that matches a specified pattern.
For more information on log redaction, see Logging Redaction.
Added in version 2.2.0.
- INRUPT_LOGGING_REDACTION_NAME_REPLACEMENT#
Replacement text to use if the
INRUPT_LOGGING_REDACTION_{NAME}_ACTION
isREPLACE
.If unspecified, defaults to
[REDACTED]
.For more information on log redaction, see Logging Redaction.
Added in version 2.2.0.
Application-Defined Metadata Propagation#
- INRUPT_AUDIT_PRODUCER_REQUEST_METADATA_ALLOW#
A comma-separated list [8] of application-defined properties that can be included in the associated audit events (unless specified in the corresponding
INRUPT_AUDIT_PRODUCER_REQUEST_METADATA_DENY
).This configuration is case-sensitive to the propagated properties in the baggage.
Tip
To include a propagated property that was added via the
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
configuration, ensure that the cases of these properties match.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_AUDIT_PRODUCER_REQUEST_METADATA_DENY#
A comma-separated list [8] of application-defined properties to exclude from the associated audit messages. This setting takes precedence over
INRUPT_AUDIT_PRODUCER_REQUEST_METADATA_ALLOW
.This configuration is case-sensitive to the propagated properties in the baggage.
Tip
To exclude a propagated property that was added via the
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
configuration, ensure that the cases of these properties match.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_LOGGING_REQUEST_METADATA_ALLOW#
A comma-separated list [8] of application-defined properties that can be included in the associated log messages (unless specified in the corresponding
INRUPT_LOGGING_REQUEST_METADATA_DENY
).This configuration is case-sensitive to the propagated properties in the baggage.
Tip
To include a propagated property that was added via the
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
configuration, ensure that the cases of these properties match.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_LOGGING_REQUEST_METADATA_DENY#
A comma-separated list [8] of application-defined properties to exclude from the associated log messages. This setting takes precedence over
INRUPT_LOGGING_REQUEST_METADATA_ALLOW
.This configuration is case-sensitive to the propagated properties in the baggage.
Tip
To exclude a propagated property that was added via the
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
configuration, ensure that the cases of these properties match.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW#
A comma-separated list [8] of non-baggage request headers to add to the baggage (unless specified in the corresponding
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_DENY
); i.e., include these non-baggage request headers as application-defined properties.The configuration is case-insensitive; i.e., the listed headers do not need to match the case of the client request headers. For example, a list that includes
x-correlation-id
can matchx-correlation-id
header,X-CoRrElAtIoN-Id
header, etc.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_DENY#
A comma-separated list [8] of non-baggage request headers to exclude from being added to the baggage; i.e., excludes these headers as application-defined properties. This setting takes precedence over
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
.The configuration is case-insensitive; i.e., the listed headers do not need to match the case of the client request headers. For example, a list that includes
x-correlation-id
can match (and exclude)x-correlation-id
header,X-CoRrElAtIoN-Id
header, etc.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_OVERRIDES#
A flag that determines ESS behavior when metadata property is defined both as a header and as a baggage entry:
If
true
, ESS updates/overrides the baggage entry with the header value.If
false
(the default), ESS keeps the baggage entry.
For details, Duplicate Property Definition.
Added in version 2.2.0.
- INRUPT_REQUEST_METADATA_REFLECTOR_HEADER_ALLOW#
A comma-separated list [8] of application-defined properties that can return as response headers (unless specified in the corresponding
INRUPT_REQUEST_METADATA_REFLECTOR_HEADER_DENY
).This configuration is case-sensitive to the propagated properties in the baggage.
Tip
To return a propagated property that was added via the
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
configuration, ensure that the cases of these properties match.You may need to update
QUARKUS_HTTP_CORS_EXPOSED_HEADERS
to extend the list of CORS-safelisted response headers.
See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
- INRUPT_REQUEST_METADATA_REFLECTOR_HEADER_DENY#
A comma-separated list [8] of application-defined properties to exclude from returning as response headers. This setting takes precedence over
INRUPT_REQUEST_METADATA_REFLECTOR_HEADER_ALLOW
.This configuration is case-sensitive to the propagated properties in the baggage.
Tip
To exclude a propagated property that was added via the
INRUPT_REQUEST_METADATA_PROPAGATOR_HEADER_ALLOW
configuration, ensure that the cases of these properties match.See:
Manage Application-Defined Metadata Propagation to configure.
Application-Defined Metadata for more information.
Added in version 2.2.0.
Additional Information#
See also Quarkus Configuration Options.
Whitespaces are preserved when parsing comma-delimited
lists (i.e., the parsed string values are not trimmed). For example, when
parsed, "value1, value2,value3 "
results in "value1"
, "
value2"
, "value3 "
.