Modify Prune Configuration#

Starting in 2.1, ESS includes a Prune feature to perform hard delete (i.e., permanently delete):

soft-deleted resources (i.e., files marked as deleted) and
orphan data (i.e., data that are no longer referenced by metadata).

Specifically, Prune consists of two Kubernetes CronJobs:

One to delete “prunable” resources. Prunable resources are resources that have been marked for deletion (i.e., soft-deleted) and are past their INRUPT_STORAGE_PRUNE_RETENTION_WINDOW.
One to delete orphan data.

You can use Kustomize to modify the two CronJobs.

Example Customizations#

Configure CronJob to Prune Soft-Deleted Resources#

Important

Pruning operations may negatively affect performance. If possible, schedule the CronJob to run at times when you can minimize its impact.

The provided overlays are configured to:

Use a job schedule of every 30 minutes.
Use a INRUPT_STORAGE_PRUNE_RETENTION_WINDOW of P3D (3 days).
Use INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE of 10000.
Use INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE of 0. Do not modify for pruning soft-deleted resources.
Use the default COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE of 5000 milliseconds.

If instead you wish to schedule the job to run every day at midnight (0 0 * * *) and to decrease the INRUPT_STORAGE_PRUNE_RETENTION_WINDOW to 2 days P2D:

Go to your ESS installation directory:
```
cd ${HOME}/ess
```

Create a kustomize-prune-resources.yaml file with the following content:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ess-prune-prunable
spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - env:
            - name: INRUPT_STORAGE_PRUNE_RETENTION_WINDOW
              value: P2D
            name: ess-prune-prunable
  schedule: '0 0 * * *'

Modify the kustomization.yaml (i.e., step 3 of the Applying Your Customizations procedure) to use kustomize-prune-resources.yaml.

Specifically, add the highlighted content to the kustomization.yaml file to the patches section:

Tip

If patches key does not exist in kustomization.yaml, add the patches key as well.
```
# kustomization.yaml in your ESS installation directory

# ...  Preceding content omitted for brevity 
# ...

patches:
  - path: kustomize-prune-resources.yaml
```
Continue with the rest of the Applying Your Customizations procedure.

Configure CronJob to Prune Orphan Data#

Important

Pruning operations may negatively affect performance. If possible, schedule the job to run at times when you can minimize its impact.

The provided overlays are configured to:

Use a job schedule of every 30th minutes from 10 minutes past the hour through 50 minutes past the hour.
Use a INRUPT_STORAGE_PRUNE_RETENTION_WINDOW of P3D (3 days). Does not affect the pruning of orphan data.
Use INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE of 0. Do not modify for pruning orphan data.
Use INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE of 80000.
Use the default COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE of 5000 milliseconds.

If instead you wish to increase the INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE to 140000 and COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE to 10000 milliseconds.

Go to your ESS installation directory:
```
cd ${HOME}/ess
```

Create a kustomize-prune-orphans.yaml file with the following content:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ess-prune-orphans
spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - env:
            - name:  INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE
              value: "140000"
            - name:  COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE
              value: "10000"
            name: ess-prune-orphans

Modify the kustomization.yaml (i.e., step 3 of the Applying Your Customizations procedure) to use kustomize-prune-orphans.yaml.

Specifically, add the highlighted content to the kustomization.yaml file to the patches section:

Tip

If patches key does not exist in kustomization.yaml, add the patches key as well.
```
# kustomization.yaml in your ESS installation directory

# ...  Preceding content omitted for brevity 
# ...

patches:
  - path: kustomize-prune-orphans.yaml
```
Continue with the rest of the Applying Your Customizations procedure.