# Logging ESS services uses an [extension of Quarkus logging](https://quarkus.io/guides/logging#json-logging). {% hint style="info" %} **Note** ESS services refer specifically to ESS code and not its dependencies. Dependencies (such as Kafka) use their own logging libraries and as such may use another format and have different fields. {% endhint %} ## Message Format ESS services: * Use JSON as the default formatting for log messages. * Use shared ESS logging components. The shared ESS logging components have **`loggerClassName`** and **`loggerName`** that start with **`com.inrupt.`**. The use of the shared logging components allow for the log messages to have a more consistent pattern across the ESS services. The JSON log messages contain the following fields: ```json { "timestamp":, "sequence":, "loggerClassName":, "loggerName":, "level":, "message":, "threadName":, "threadId":, "hostName":, "processName":, "processId":, "messageId": // For messages using the ESS common logging module // additional relevant fields, if any, like ``mdc`` for OpenTelemetry information } ``` ESS also supports changing the level of the log message(s) using redaction. See [Logging Redaction](https://docs.inrupt.com/ess/2.5/administration/logging/logging-redaction) for details. ### MessageId The JSON log messages emitted by ESS code (i.e., use the common ESS logging components) include a **`messageId`**. The **`messageId`** consists of a prefix and an number **``**; for example, **`"CONFIG000001"`** or **`"AUTHORIZATION000016"`**. The **`messageId`** value appears in both: * the **`messageId`** field ``` "messageId" : "" ``` * as part of the **`message`** field ``` "message" : ": " ``` #### MessageId Prefix The **`messageId`** consists of a prefix and an number **``.** Most prefix align with the ESS service; for example: * **`AUTHORIZATON`** for the Authorization service log messages; * **`VC`** for the Access Grant service messages; * **`WEBID`** for the WebID service log messages. Other prefixes are shared across services; for example: * **`CONFIG`** for startup configuration log messages for all ESS services. * **`COMPONENT-`** for log messages associated with an ESS component that can be used across ESS services; e.g., **`COMPONENT-KAFKAENCRYPTION`** that can be used across ESS services. {% hint style="info" %} Log messages emitted by dependencies (such as those with **`loggerClassName`** of **`"org.apache.kafka.common.utils.LogContext$LocationAwareKafkaLogger"`**, **`"org.jboss.logmanager.Logger"`**, **`"org.slf4j.impl.Slf4jLogger"`**, etc.) do **not** use the ESS common logging module. As such, they may not contain the **`messageId`** field. {% endhint %} ### Startup Configuration Logging ESS services log the startup configuration of the services. ESS services log each configuration property in its own log message: 
{
   "timestamp" : <value>,
   "sequence" : <value>,
   "loggerClassName" : "com.inrupt.components.logging.filter.ConfigurationLog_$logger",
   "loggerName" : "com.inrupt.components.logging.filter.ConfigurationLogger",
   "level" : "INFO",
   "message" : "CONFIG000001: Configuration",
   "threadName" : <value>,
   "threadId" : <value>,
   "hostName" : <value>,
   "processName" : "<value>,
   "processId" : <value>,
   "configPropertyName" : "some.configuration.property.name",
   "configValue" : "<value>",
   "messageId" : "CONFIG000001",
}
You can configure which startup configuration properties to log by configuring the service’s **`INRUPT_LOGGING_CONFIGURATION_PREFIX_ALLOW`** and **`INRUPT_LOGGING_CONFIGURATION_PREFIX_DENY`**. See the configuration section of each service for details. ### OpenTelemetry `mdc` Field Starting in version 2.2, log messages may include an **`mdc`** field that contains the client-managed [OpenTelemetry information](https://opentelemetry.io/docs): ``` "mdc": {"spanId": ,"traceId":,"sampled":"true"} ``` Using the **`traceId`** value, operators can correlate log messages across services. However, as the **`traceId`** is subject to how the client specifies the **`traceId`** for its requests, correlation by **`traceId`** may not be suitable for audit trail purposes. For an overview on OpenTelemetry, see: . See also [Appendix: OpenTelemetry](https://docs.inrupt.com/ess/2.5/reference/appendix-opentelemetry). ### Application-Defined Metadata ESS can propagate [application-defined request properties](https://docs.inrupt.com/ess/2.5/administration/application-defined-metadata) sent in client requests to include in associated log messages, associated audit events, and associated response to the request. Depending upon the [configuration](https://docs.inrupt.com/ess/services/service-oidc/service-application-registration#configuration), ESS log messages can include the application-defined request metadata in the **`requestMetadata`** field: ```json "requestMetadata":{"property1":"","property2":"", ...} ``` See [Application-Defined Metadata](https://docs.inrupt.com/ess/2.5/administration/application-defined-metadata) for the configuration needed to include in log messages. ### Log Redaction ESS supports configurable redaction of log messages. For more details, see [Logging Redaction](https://docs.inrupt.com/ess/2.5/administration/logging/logging-redaction). ## Modify the Log Level on an ESS Microservice ESS services support a subset of log levels supported by Quarkus: * **`FATAL`** * **`ERROR`** * **`WARN`** * **`INFO`** (Default level) * **`DEBUG`** By default, ESS’ log level is configured to **`INFO`** level: * This outputs logs with severity level **`INFO`** and higher (i.e., outputs **`INFO`**, **`WARN`**, **`ERROR`** and **`FATAL`** levels). * This excludes logs with security level below **`INFO`** (i.e., excludes **`DEBUG`**). To change a service’s log level, you can use [Kustomize](https://github.com/kubernetes-sigs/kustomize) [overlays](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/#bases-and-overlays) to update the **`QUARKUS_LOG_LEVEL`** value to the lowest log level you want to log. You can use the following procedure to enable **`DEBUG`** level logging for **`pod-provisioning`**: 1. Go to your ESS installation directory: ```bash cd ${HOME}/ess ``` 2. Modify the **`kustomization.yaml`** (i.e., step 3 of the [Applying Your Customizations](https://docs.inrupt.com/ess/installation/customize-configurations#applying-your-customizations) procedure). Specifically, add the highlighted content to the **`kustomization.yaml`** file under the **`patches`** key:\\ {% hint style="info" %} **Tip** If **`patches`** list does not exist in **`kustomization.yaml`**, add the key **`patches`** as well. {% endhint %} ```yaml # kustomization.yaml in your ESS installation directory # ... Preceding content omitted for brevity # ... patches: - target: kind: Deployment name: ess-pod-provision patch: |- apiVersion: apps/v1 kind: Deployment metadata: name: ess-pod-provision spec: template: spec: containers: - env: - name: QUARKUS_LOG_LEVEL value: DEBUG name: ess-pod-provision ``` 3. Continue with the rest of the [Applying Your Customizations](https://docs.inrupt.com/ess/installation/customize-configurations#applying-your-customizations) procedure. {% hint style="info" %} **Tip** Remember to reset the log level when you’re finished debugging. {% endhint %} For more information on customizing ESS, see [Customize ESS](https://docs.inrupt.com/ess/2.5/installation/customize-configurations). For more information on Quarkus-based logging, see . ## Retrieve Logs ESS consists of multiple K8s pods (instances) running on multiple nodes (servers) in a K8s cluster. Kubernetes manages the orchestration of all these containers. You can retrieve the logs directly from Kubernetes or integrate into a centralized logging platform. ### Retrieve Logs Directly from Kubernetes To retrieve the logs displayed to **`stdout`**, run the **`kubectl logs`** command: ``` kubectl -n ess logs --follow ``` ### Centralized Logging to a Backend Logging System As your ESS deployment grows, logging to a centralized backend logging system can help manage your logs. You can set up a centralized logging system for your ESS deployment in Kubernetes. For more information, see [Kubernetes: Logging](https://kubernetes.io/docs/concepts/cluster-administration/logging/). See also: * [Kubernetes: Interacting with Running Pods](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#interacting-with-running-pods) * [Kubernetes: Monitoring, Logging, and Debugging](https://kubernetes.io/docs/tasks/debug/) * [Digital Ocean: How To Set Up an Elasticsearch, Fluentd and Kibana (EFK) Logging Stack on Kubernetes](https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes) * [Splunk Connect for Kubernetes on EKS!](https://www.splunk.com/en_us/blog/devops/monitor-amazon-eks-anywhere-with-splunk.html)