Modify Prune Configuration

ESS includes a Prune feature to perform hard delete (i.e., permanently delete):

• soft-deleted resources (i.e., files marked as deleted) and

• orphan data (i.e., data that are no longer referenced by metadata).

Specifically, Prune consists of two Kubernetes CronJobs :

• One to delete “prunable” resources. Prunable resources are resources that have been marked for deletion (i.e., soft-deleted) and are past their INRUPT_STORAGE_PRUNE_RETENTION_WINDOW .

• One to delete orphan data.

You can use Kustomize to modify the two CronJobs.

Example Customizations

Configure CronJob to Prune Soft-Deleted Resources

Important Pruning operations may negatively affect performance. If possible, schedule the CronJob to run at times when you can minimize its impact.

The provided overlays are configured to:

• Use a job schedule of every 30 minutes.

• Use a INRUPT_STORAGE_PRUNE_RETENTION_WINDOW of P3D (3 days).

• Use INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE of 10000 .

• Use INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE of 0 . Do not modify for pruning soft-deleted resources.

• Use the default COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE of 5000 milliseconds.

If instead you wish to schedule the job to run every day at midnight ( 0 0 * * * ) and to decrease the INRUPT_STORAGE_PRUNE_RETENTION_WINDOW to 2 days P2D :

1. Go to your ESS installation directory:

   cd ${HOME}/ess

2. Create a kustomize-prune-resources.yaml file with the following content:

   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: ess-prune-prunable
   spec:
     jobTemplate:
       spec:
         template:
           spec:
             containers:
             - env:
               - name: INRUPT_STORAGE_PRUNE_RETENTION_WINDOW
                 value: P2D
               name: ess-prune-prunable
     schedule: '0 0 * * *'

3. Modify the kustomization.yaml (i.e., step 3 of the Applying Your Customizations procedure) to use kustomize-prune-resources.yaml . Specifically, add the highlighted content to the kustomization.yaml file to the patches section:

Tip

If patches key does not exist in kustomization.yaml , add the patches key as well.

# kustomization.yaml in your ESS installation directory

# ...  Preceding content omitted for brevity 
# ...

patches:
  - path: kustomize-prune-resources.yaml

4. Continue with the rest of the Applying Your Customizations procedure.

Configure CronJob to Prune Orphan Data

Important Pruning operations may negatively affect performance. If possible, schedule the job to run at times when you can minimize its impact.

The provided overlays are configured to:

• Use a job schedule of every 30th minutes from 10 minutes past the hour through 50 minutes past the hour.

• Use a INRUPT_STORAGE_PRUNE_RETENTION_WINDOW of P3D (3 days). Does not affect the pruning of orphan data.

• Use INRUPT_STORAGE_PRUNE_PRUNABLE_BATCH_SIZE of 0 . Do not modify for pruning orphan data.

• Use INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE of 80000 .

• Use the default COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE of 5000 milliseconds.

If instead you wish to increase the INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE to 140000 and COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE to 10000 milliseconds.

1. Go to your ESS installation directory:

   cd ${HOME}/ess

2. Create a kustomize-prune-orphans.yaml file with the following content:

   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: ess-prune-orphans
   spec:
     jobTemplate:
       spec:
         template:
           spec:
             containers:
             - env:
               - name:  INRUPT_STORAGE_PRUNE_ORPHAN_BATCH_SIZE
                 value: "140000"
               - name:  COM_INRUPT_STORAGE_METADATA_JDBC_CONNECTIONLIMITER_OPENCONNECTION_TIMEOUT_VALUE
                 value: "10000"
               name: ess-prune-orphans

3. Modify the kustomization.yaml (i.e., step 3 of the Applying Your Customizations procedure) to use kustomize-prune-orphans.yaml . Specifically, add the highlighted content to the kustomization.yaml file to the patches section:

Tip

If patches key does not exist in kustomization.yaml , add the patches key as well.

# kustomization.yaml in your ESS installation directory

# ...  Preceding content omitted for brevity 
# ...

patches:
  - path: kustomize-prune-orphans.yaml

1. Continue with the rest of the Applying Your Customizations procedure.