# Appendix: OpenTelemetry

ESS supports the use of [OpenTelemetry](https://opentelemetry.io/docs) for observability. Specifically, ESS has been [instrumented](https://opentelemetry.io/docs/concepts/instrumentation/) to support OpenTelemetry. For an overview on OpenTelemetry, see: <https://opentelemetry.io/docs/specs/otel/overview/>.

ESS also uses OpenTelemetry to propagate [application-defined properties](https://docs.inrupt.com/ess/2.3/administration/application-defined-metadata) that are defined in the client requests.

## Correlation

### By TraceId

When associated with a [span](https://opentelemetry.io/docs/concepts/signals/traces/#spans):

* ESS log messages can include OpenTelemetry information in the **`mdc`** field:

  ```json
  "mdc": {"spanId":<value>,"traceId":<value>,"sampled":"true"}
  ```

  The **`traceId`** field can be used to correlate log messages across services that are associated with a client request. However, as the **`traceId`** is subject to how the client specifies the **`traceId`** for its requests, correlation by **`traceId`** may not be suitable for audit trail purposes.
* ESS audit events can include OpenTelemetry information in the **`instrument`** field:

  ```json
  {
     ...
     "instrument" : [
        // ...
        {
           "spanId" : <value>,
           "traceId" : <value>,
           "parentId" : <value>,
           "name" : "OpenTelemetry Span Context",
           "isSampled" : true,
           "type" : [
              "SpanContext"
           ]
        }
     ],
  }
  ```

  The **`traceId`** field can be used to correlate audit events across services that are associated with a client request. However, as the **`traceId`** is subject to how the client specifies the **`traceId`** for its requests, correlation by **`traceId`** may not be suitable for audit trail purposes.

### By Application-Defined Property

ESS can propagate [application-defined properties](https://docs.inrupt.com/ess/2.3/administration/application-defined-metadata) sent in client’s request header to include in associated log messages, associated audit events, and associated response to the request.

Depending upon configuration:

* ESS log messages can include the **`requestMetadata`**:

  ```json
  "requestMetadata":{"property1":"<value1>","property2":"<value2>", ...}
  ```

  Correlation by client defined properties may not be suitable for audit trail purposes as these properties are subject to how the client manages these properties as well as the ESS [configuration](https://docs.inrupt.com/ess/services/service-oidc/service-application-registration#configuration).
* ESS audit events can include the application-defined request metadata in the **`instrument`** field:

  ```json
  "instrument" : [
     // ...
     {
       "name" : "Application-Defined Request Metadata",
       "items": [
          {
             "mediaType" : "text/plain",
             "name" : "x-correlation-id",
             "content" : "2049875809728750827498245084"
          },
          {
             "mediaType":"text/plain",
             "name":"my-client-version",
             "content":"1.0.3"
          }
       ],
       "type":[
          "urn:uuid:1a05e301-4013-40c9-bae7-5d719b7151c8"
       ]
     }
     // ...
  ],
  ```

  Correlation by client defined properties may not be suitable for audit trail purposes as these properties are subject to how the client manages these properties as well as the ESS configuration.
* ESS’ response can include the properties as headers.