Pod Storage Service#

Changed in version 2.0.

Starting in ESS 2.0, Pod provisioning and storage is handled by a separate Pod Provisioning Service and Pod Storage Service.

ESS’ Pod storage service hosts the Pods created by the Pod provisioning service and is responsible for reading and writing the resources stored in the Pod.

The Pod URL has the following pattern:

https://storage.{ESS Domain}/{Unique Root Container}/

Prior to version 2.0, ESS Pods had the URL of the form https://{ESS Domain}/{username}/

Pod Resources#

ESS Pods supports storing different types of Solid Resources, including:

a Container
Analogous to folders in a file system. A Container can contain other Containers as well as RDF and non-RDF resources. Container URLs always end with a slash /.

a Resource Description Framework (RDF) resource.
A file whose contents consists of statements (also known as triples) that describe some “subject” by its relationships:
<subject> <predicate> <object>

a non-RDF Resource
Any non-RDF binary or text file, such as .pdf , .jpeg, etc.

Access#

For access to the Pod resources, ESS’ Pod Storage service supports the use of the following tokens:

Discovery#

ESS provides Pod storage service (and related) metadata at the following .well-known/solid URI:

https://storage.{ESS Domain}/.well-known/solid

Its Response.body returns Resource Description Framework (RDF) statements. Depending on your configuration, the response can include information about:

List of application(s) that can perform read and write operations the Access Control Resources with the following caveat:

Disambiguation

The list displays the value of Pod Storage Service‘s INRUPT_AUTHORIZATION_CLIENT_ID_ALLOW_LIST configuration value, which is for discoverability purposes only.

The actual configuration that determines which applications can access the ACR is the Authorization Service‘s INRUPT_AUTHORIZATION_CLIENT_ID_ALLOW_LIST.

As such, if the two lists are not in sync, the returned list from Pod Storage Service may not accurately reflect the trusted apps.
Maximum number of Pods allowed per Agent.
Notification Gateway endpoint.
Provision service endpoint.
QPF service endpoint.

@prefix solid: <http://www.w3.org/ns/solid/terms#> .

[ a                          solid:DiscoveryDocument ;
  <http://www.w3.org/ns/auth/acl#trustedApp>
          <https://podbrowser.inrupt.com/api/app> ;
  solid:maxPodsPerOwner      10 ;
  solid:notificationGateway  <https://notification.{ESS DOMAIN}.com/> ;
  solid:provision            <https://provision.{ESS DOMAIN}.com/>;
  solid:qpf                  <https://fragments.{ESS DOMAIN}.com/
] .

Prune/Hard Delete Feature#

New in version 2.1.

Starting in 2.1, ESS includes a Prune feature to hard delete (i.e., permanently delete):

soft-deleted resources (i.e., files marked as deleted) and
orphan data (i.e., data that are no longer referenced by metadata).

For more information on soft-deleted resources and orphan data, see CRUD Operations.

Prune CronJobs#

Prune consists of two Kubernetes CronJobs :

One to delete soft-deleted resources. See Pruning Soft-Deleted Resources for details.
One to delete orphan data. See Pruning Orphan Data for details.

Pruning Soft-Deleted Resources#

Prune uses the following process to delete soft-deleted resources:

To find resources to delete, Prune queries for metadata entries that have been soft-deleted (i.e., marked as deleted) for longer than the configured INRUPT_STORAGE_PRUNE_RETENTION_WINDOW. These are “prunable” resources.

Prune uses the INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE to limit the number of results.
Using the identifiers (part of the metadata) from the query results,
1. Prune deletes the resources.
2. Prune deletes the associated metadata.

To configure, see Modify Prune Configuration.

Pruning Orphan Data#

Prune uses the following process to delete orphan data:

Prune starts by randomly selecting resource data identifiers.

Prune uses the INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE to limit the number of selected identifiers.
For the selected resource data identifiers, Prune queries for corresponding metadata to find those identifiers without corresponding metadata.

Prune uses the INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE to limit the number of results.
Prune deletes those resource data whose identifiers did not have corresponding metadata.

To configure, see Modify Prune Configuration.

Storage Metrics#

Starting in 2.1, ESS includes a Storage Metrics feature to gather the following metrics:

The total number of Pods
The number of Pods that have been “Created” (where the provision has been confirmed)
The number of Pods that have been “Deleted”(marked for deletion; i.e., soft-deleted).

The Storage Metrics is run as a Kubernetes CronJobs.

To modify the schedule, see Modify Storage Metrics Schedule.

Configuration#

As part of the installation process, Inrupt provides base Kustomize overlays and associated files that require deployment-specific configuration inputs.

The following configuration options are available for the service and may be set as part of updating the inputs for your deployment. The Inrupt-provided base Kustomize overlays may be using updated configuration values that differ from the default values.

Pod Storage Pruning Options#

COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE#

Default: 5000

Optional. The maximum amount of milliseconds Prune operations’ connection to the metadata database can remain open.

Adjust as needed to accommodate changes to:

INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE.
INRUPT_STORAGE_PRUNE_RETENTION_WINDOW.
INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE.

INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE#

An integer that limits the number of resource data identifiers selected by Prune during orphan data cleanup.

Important

For Prune cron job that prunes soft-deleted resources, set to 0.

An increase in the batch size may require a corresponding increase in COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE.

INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE#

An integer that limits the number of results returned when querying the metadata.

Important

For Prune cron job that prunes orphan data, set to 0.

An increase in the batch size may require a corresponding increase in COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE.

INRUPT_STORAGE_PRUNE_RETENTION_WINDOW#

The minimum amount of time (specified in a format supported by Java Duration.parse() method; e.g., PT23H for 23 hours) that resources have been soft-deleted (marked as deleted) before they become eligible for pruning. That is, resources that have been soft-deleted for longer than the specified time can be selected for hard deletion by Prune; i.e., these are “prunable” resources.

This configuration does not affect the pruning of orphaned data.

An increase in the retention window value may require a corresponding increase in COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE.

To configure Prune, see Modify Prune Configuration.

Pod Storage Resource Read Auditing Option#

INRUPT_STORAGE_AUDIT_RESOURCE_READ_ENABLED#

Default: false

A boolean that determines whether to audit successful read resource events (i.e., successful (GET and HEAD operations on resources).

Set to true to enable, and false to disable.

Important

When auditing of read operations is enabled, the total number of Audit events may increase substantially. Before enabling read operations auditing, consider allocating more compute and network resources to ESS.

Pod Storage UMA Configuration#

INRUPT_AUTHZ_AS_URI#

The URI of the UMA Authorization Server.

The value must match:

INRUPT_JWT_AUTHORIZATION_SERVER_ISSUER configuration for the service, and
INRUPT_UMA_ISSUER configuration for UMA Service.

INRUPT_AUTHZ_UMA_ANONYMOUS_ENABLED#

Default: false

A boolean flag that determines whether to support anonymous (i.e., unauthenticated) access to resources that have been granted public access.

INRUPT_AUTHZ_UMA_OIDC_ENABLED#

Default: false

A boolean flag that determines whether the Pod server supports OIDC access tokens. When set to false, clients will need access tokens from the associated UMA server.

INRUPT_JWT_AUTHORIZATION_SERVER_ISSUER#

The URI of the UMA token issuer.

The value must match INRUPT_AUTHZ_AS_URI.

INRUPT_JWT_AUTHORIZATION_SERVER_JWKS#: The JWKS endpoint of the INRUPT_JWT_AUTHORIZATION_SERVER_ISSUER.

SMALLRYE_JWT_ENCRYPT_KEY_ID#: The key id of the JWK key used to encrypt the ticket for the UMA Authorization Server. Required if using UMA.

SMALLRYE_JWT_ENCRYPT_KEY_LOCATION#: The location of the JWK key used to encrypt the ticket for the UMA Service. This should be configured to the SMALLRYE_JWT_SIGN_KEY_LOCATION on the UMA Service.

Pod Storage Options#

INRUPT_AUTHORIZATION_CLIENT_ID_ALLOW_LIST#

Default: https://podbrowser.inrupt.com/api/app

Comma-delimited list of Client IDs that are displayed in /.well-known/solid as the Client IDs of trusted applications. Trusted applications can perform read and write operations on the Access Control Resources.

Disambiguation

Both Authorization Service and Pod Storage Service have an INRUPT_AUTHORIZATION_CLIENT_ID_ALLOW_LIST setting.

Only the Authorization Service setting affects which clients are allowed. The Pod Storage Service is for Discovery purposes only.

The list should reflect the values set in the Authorization Service‘s INRUPT_AUTHORIZATION_CLIENT_ID_ALLOW_LIST since the authorization server’s configuration actually determines what clients are the trusted applications. That is, the Pod service’s configuration is for discoverability purposes only.

See Set Authorization Client Allow List.

INRUPT_STORAGE_HTTP_BASE_URL#

The base URL of the storage service. This is mainly for use by supporting services like Pod Provisioning service.

Important

The value requires a trailing slash /; e.g., https://storage.{ESS_DOMAIN}/.
Ensure that Pod Storage Service’s INRUPT_STORAGE_HTTP_BASE_URL value is consistent with the Pod Provision Service’s INRUPT_STORAGE_HTTP_BASE_URL value.

INRUPT_STORAGE_HTTP_CACHE_CONTROL_MAX_AGE#

Default: 0

The max-age directive value on the Cache-Control header.

For more information of Cache-Control directives, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control.

Solid-OIDC Issuer Configuration Options#

INRUPT_JWT_ALLOWED_SIGNATURE_ALGORITHMS#

Default: ES256, RS256

A comma-separated list that specifies the allowed encryption algorithms used to sign ID tokens.

INRUPT_JWT_ISSUER_ALLOW_LIST#

A comma-separated list of trusted issuers of Solid-OIDC tokens.

If unset, the service accepts Solid-OIDC tokens from all issuers with the exception of those in the INRUPT_JWT_ISSUER_DENY_LIST.
If set, the service accepts only the Solid-OIDC tokens from the issuers in the list with the following exception:
- If an issuer is in both INRUPT_JWT_ISSUER_ALLOW_LIST and INRUPT_JWT_ISSUER_DENY_LIST, the INRUPT_JWT_ISSUER_DENY_LIST supersedes the INRUPT_JWT_ISSUER_ALLOW_LIST and the issuer is not accepted by ESS.

JSON-LD Context Configuration Options#

INRUPT_JSONLD_CONTEXT_ALLOW_LIST#: A comma-delimited list of trusted JSON-LD context URIs allowed to be dereferenced.

INRUPT_JSONLD_CONTEXT_DENY_LIST#

A deny-list of JSON-LD context URIs. URIs listed in this configuration are prevented from being dereferenced.

Logging Configuration#

QUARKUS_LOG_LEVEL#

Default: INFO

Logging level.

Kafka Configuration#

Tip

AWS Options#

INRUPT_STORAGE_S3_BUCKET_NAME#

Default: inrupt.ess.storage

The name of the S3 bucket used for storage.

QUARKUS_S3_AWS_CREDENTIALS_STATIC_PROVIDER_ACCESS_KEY_ID#: AWS Access key id.

QUARKUS_S3_AWS_CREDENTIALS_STATIC_PROVIDER_SECRET_ACCESS_KEY#: AWS Secret access key.

QUARKUS_S3_AWS_REGION#: An Amazon Web Services region that hosts the S3 Bucket.

QUARKUS_S3_ENDPOINT_OVERRIDE#: Override S3 endpoint URL.

OpenTelemetry Options#

QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENABLED#

Default: false

The OpenTelemetry exporter can be enabled or disabled with this configuration.

QUARKUS_OPENTELEMETRY_TRACER_EXPORTER_OTLP_ENDPOINT#: The URL of the OpenTelemetry exporter.

Pod Storage Service#

Pod Resources#

Access#

Discovery#

Prune/Hard Delete Feature#

Prune CronJobs#

Pruning Soft-Deleted Resources#

Pruning Orphan Data#

Storage Metrics#

Configuration#

Pod Storage Pruning Options#

Pod Storage Resource Read Auditing Option#

Pod Storage UMA Configuration#

Pod Storage Options#

Solid-OIDC Issuer Configuration Options#

JSON-LD Context Configuration Options#

Logging Configuration#

Kafka Configuration#

AWS Options#

OpenTelemetry Options#

Additional Information#