Logging

ESS services uses an extension of Quarkus logging.

Note

ESS services refer specifically to ESS code and not its dependencies. Dependencies (such as Kafka) use their own logging libraries and as such may use another format and have different fields.

Important

Starting in version 2.2, ESS services log the startup configuration of the services.

Message Format

Starting in version 2.2, ESS services: [1]

• Use JSON as the default formatting for log messages.

• Use shared ESS logging components. The shared ESS logging components have loggerClassName and loggerName that start with com.inrupt.. The use of the shared logging components allow for the log messages to have a more consistent pattern across the ESS services.

The JSON log messages contain the following fields: [3]

{
  "timestamp":<value>,
  "sequence":<value>,
  "loggerClassName":<value>,
  "loggerName":<value>,
  "level":<value>,
  "message":<value>,
  "threadName":<value>,
  "threadId":<value>,
  "hostName":<value>,
  "processName":<value>,
  "processId":<value>,
  "messageId":<value>  // For messages using the ESS common logging module
  // additional relevant fields, if any, like ``mdc`` for OpenTelemetry information
}

[1] (1,2)

ESS services refer specifically to ESS code and not its dependencies. Dependencies (such as Kafka) use their own logging libraries and as such may use another format and have different fields.

MessageId

The JSON log messages emitted by ESS code (i.e., use the common ESS logging components) include a messageId. [2] The messageId consists of a prefix and an number <PREFIX><number>; for example, "CONFIG000001" or "AUTHORIZATION000016".

The messageId value appears in both:

• the messageId field

  "messageId" : "<messageId>"
  

• as part of the message field

  "message" : "<messageId>: <description>"
  

MessageId Prefix

The messageId consists of a prefix and an number <PREFIX><number>.

Most prefix align with the ESS service; for example:

• AUTHORIZATON for the Authorization service log messages;

• VC for the Access Grant service messages;

• WEBID for the WebID service log messages.

Other prefixes are shared across services; for example:

• CONFIG for startup configuration log messages for all ESS services.

• COMPONENT-<XXXXX> for log messages associated with an ESS component that can be used across ESS services; e.g., COMPONENT-KAFKAENCRYPTION that can be used across ESS services.

[2]

Log messages emitted by dependencies (such as those with loggerClassName of "org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger", "org.jboss.logmanager.Logger", "org.slf4j.impl.Slf4jLogger", etc.) do not use the ESS common logging module. As such, they may not contain the messageId field.

Startup Configuration Logging

Starting in version 2.2, ESS services [1] log the startup configuration of the services.

ESS services log each configuration property in its own log message:

{
   "timestamp" : <value>,
   "sequence" : <value>,
   "loggerClassName" : "com.inrupt.components.logging.filter.ConfigurationLog_$logger",
   "loggerName" : "com.inrupt.components.logging.filter.ConfigurationLogger",
   "level" : "INFO",
   "message" : "CONFIG000001: Configuration",
   "threadName" : <value>,
   "threadId" : <value>,
   "hostName" : <value>,
   "processName" : "<value>,
   "processId" : <value>,
   "configPropertyName" : "some.configuration.property.name",
   "configValue" : "<value>",
   "messageId" : "CONFIG000001",
}

You can configure which startup configuration properties to log by configuring the service’s INRUPT_LOGGING_CONFIGURATION_PREFIX_ALLOW and INRUPT_LOGGING_CONFIGURATION_PREFIX_DENY. See the configuration section of each service for details.

OpenTelemetry mdc Field

Starting in version 2.2, log messages may include an mdc field that contains the client-managed OpenTelemetry information:

"mdc": {"spanId": <value>,"traceId":<value>,"sampled":"true"}

Using the traceId value, operators can correlate log messages across services. However, as the traceId is subject to how the client specifies the traceId for its requests, correlation by traceId may not be suitable for audit trail purposes.

For an overview on OpenTelemetry, see: https://opentelemetry.io/docs/specs/otel/overview/.

See also Appendix: OpenTelemetry.

Application-Defined Metadata

Starting in 2.2, ESS can propagate application-defined request properties sent in client requests to include in associated log messages, associated audit events, and associated response to the request.

Depending upon the configuration, ESS log messages can include the application-defined request metadata in the requestMetadata field:

"requestMetadata":{"property1":"<value1>","property2":"<value2>", ...}

See Application-Defined Metadata for the configuration needed to include in log messages.

Log Redaction

Starting in 2.2, ESS supports configurable redaction of log messages. For more details, see Logging Redaction.

Modify the Log Level on an ESS Microservice

ESS services support a subset of log levels supported by Quarkus:

• FATAL

• ERROR

• WARN

• INFO (Default level)

• DEBUG

By default, ESS’ log level is configured to INFO level:

• This outputs logs with severity level INFO and higher (i.e., outputs INFO, WARN, ERROR and FATAL levels).

• This excludes logs with security level below INFO (i.e., excludes DEBUG).

To change a service’s log level, you can use Kustomize overlays to update the QUARKUS_LOG_LEVEL value to the lowest log level you want to log. [3]

You can use the following procedure to enable DEBUG level logging for pod-provisioning:

1. Go to your ESS installation directory:

   cd ${HOME}/ess
   

2. Modify the kustomization.yaml (i.e., step 3 of the Applying Your Customizations procedure).

   Specifically, add the highlighted content to the kustomization.yaml file under the patches key:

   Tip

   If patches list does not exist in kustomization.yaml, add the key patches as well.

   # kustomization.yaml in your ESS installation directory
   
   # ...  Preceding content omitted for brevity 
   # ...
   
   patches:
     - target:
         kind: Deployment
         name: ess-pod-provision
       patch: |-
         apiVersion: apps/v1
         kind: Deployment
         metadata:
           name: ess-pod-provision
         spec:
           template:
             spec:
               containers:
                 - env:
                   - name: QUARKUS_LOG_LEVEL
                     value: DEBUG
                   name: ess-pod-provision
   

3. Continue with the rest of the Applying Your Customizations procedure.

Tip

Remember to reset the log level when you’re finished debugging.

For more information on customizing ESS, see Customize ESS.

For more information on Quarkus-based logging, see https://quarkus.io/guides/logging.

[3] (1,2)

ESS also supports changing the level of the log message(s) using redaction. See Logging Redaction for details.

Retrieve Logs

ESS consists of multiple K8s pods (instances) running on multiple nodes (servers) in a K8s cluster. Kubernetes manages the orchestration of all these containers. You can retrieve the logs directly from Kubernetes or integrate into a centralized logging platform.

Retrieve Logs Directly from Kubernetes

To retrieve the logs displayed to stdout, run the kubectl logs command:

kubectl -n ess logs --follow <service app deployment>

Centralized Logging to a Backend Logging System

As your ESS deployment grows, logging to a centralized backend logging system can help manage your logs. You can set up a centralized logging system for your ESS deployment in Kubernetes. For more information, see Kubernetes: Logging.

See also:

• Kubernetes: Interacting with Running Pods

• Kubernetes: Monitoring, Logging, and Debugging

• Digital Ocean: How To Set Up an Elasticsearch, Fluentd and Kibana (EFK) Logging Stack on Kubernetes

• Splunk Connect for Kubernetes on EKS!