> For the complete documentation index, see [llms.txt](https://docs.inrupt.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.inrupt.com/ess/services/service-search/ingest-service.md).

# Ingest Service

{% hint style="success" %}
Added in version 3.2.0
{% endhint %}

The Ingest Service is the indexing pipeline for the ESS [Search Service](/ess/services/service-search.md). It automatically indexes Pod content as resources are created, updated, or deleted, and provides a REST API for on-demand re-indexing.

## How Indexing Works

The Ingest Service consumes encrypted audit events from a Kafka topic. When a resource change event is received:

1. The service fetches the resource content directly from object storage (S3 or Azure Blob Storage).
2. Text is extracted from the resource content. For PDFs and images, the service calls the [OCR Service](/ess/services/service-search/ocr-service.md). For Microsoft Office documents, it uses Apache POI. Plain text, HTML, CSV, JSON, and RDF formats are extracted directly.
3. The extracted text is split into overlapping chunks (default: 1000 characters with 200-character overlap).
4. Each chunk is sent to the [Embedding Service](/ess/services/service-search/embedding-service.md) to generate a 384-dimensional vector embedding.
5. The chunks, embeddings, and metadata are dual-written to pgvector (PostgreSQL) and OpenSearch.

For delete events, the service removes all indexed chunks for the deleted resource from both stores.

{% hint style="info" %}
**Maximum Resource Size**

Resources larger than 50 MB are skipped during indexing.
{% endhint %}

{% hint style="info" %}
**Dead-Letter Queue**

If a message cannot be processed after the configured number of retries (default: 3), it is sent to the Kafka dead-letter queue. Failed messages can be replayed after the underlying issue is resolved.
{% endhint %}

### Template-Based Indexing

The Ingest Service supports template-based indexing for structured data (such as JSON). Instead of indexing raw file content, a template defines how fields from each record are combined into searchable text and metadata.

#### How It Works

1. You create a **mapping resource** in the Pod — a JSON file that defines the template.
2. You attach the mapping to a data resource via a Solid auxiliary resource. The auxiliary resource contains an `indexingLink` (URL of the mapping) and an `indexingDataId` (data identifier for the structured content).
3. When the data resource is created or updated, the Ingest Service detects the auxiliary metadata in the audit event, fetches the mapping, and renders each record through the template.

#### Mapping Resource Format

A mapping resource is a JSON document with the following structure:

```json
{
  "version": 1,
  "strategy": "json-records",
  "recordPath": "$[*]",
  "template": [
    "Transaction: {description||reference||'Unknown'}",
    "Amount: {amount.value} {amount.currency}",
    "Date: {bookingDate}",
    "Account: {_meta.accountName}"
  ],
  "metadata": {
    "category": "{category}",
    "amount": "{amount.value}",
    "date": "{bookingDate}"
  }
}
```

| Field            | Type            | Required | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| ---------------- | --------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`version`**    | Integer         | Yes      | Must be **`1`**.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| **`strategy`**   | String          | Yes      | Must be **`json-records`**.                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| **`recordPath`** | String          | Yes      | A JSONPath expression that selects the array of records from the data resource. For example, **`$[*]`** for a top-level array or **`$.transactions[*]`** for a nested array.                                                                                                                                                                                                                                                                                                |
| **`template`**   | Array of String | Yes      | Lines of text that are rendered for each record. Field references use **`{fieldName}`** syntax.                                                                                                                                                                                                                                                                                                                                                                             |
| **`metadata`**   | Object          | No       | Key-value pairs where values can reference record fields. These are stored as searchable metadata alongside each indexed chunk. Field references resolve to their original JSON type (number, boolean, string); the [Search API](/ess/services/service-search/search-api.md) numeric filter operators (**`gt`**, **`gte`**, **`lt`**, **`lte`**) cast values at query time, so numeric comparisons work regardless of whether the source value was a JSON number or string. |

#### Template Syntax

Field references in template lines use curly braces:

* **`{fieldName}`** — simple field lookup.
* **`{nested.field}`** — dot-separated path for nested objects.
* **`{field1||field2||'fallback'}`** — fallback chain. Uses the first non-null value. Literal strings are quoted with single quotes.
* **`{_meta.fieldName}`** — references resource-level metadata instead of the current record.

If all field references in a template line resolve to **`null`**, the entire line is omitted from the rendered text.

#### Example: Indexing Bank Transactions

Given a data resource containing:

```json
[
  {
    "description": "Coffee Shop Purchase",
    "amount": {"value": -4.50, "currency": "GBP"},
    "bookingDate": "2026-07-15",
    "category": "dining"
  },
  {
    "reference": "SALARY JUL 2026",
    "amount": {"value": 3200.00, "currency": "GBP"},
    "bookingDate": "2026-07-01",
    "category": "income"
  }
]
```

And resource-level metadata (referenced via **`_meta`**) of:

```json
{
  "accountName": "Alice's Current Account"
}
```

With the mapping above, the Ingest Service produces two indexed chunks. Note that the first record has a **`description`** field, so the fallback chain **`{description||reference||'Unknown'}`** resolves to that value. The second record has no **`description`** but has **`reference`**, so it falls back to that.

**Chunk 1:**

```
Transaction: Coffee Shop Purchase
Amount: -4.50 GBP
Date: 2026-07-15
Account: Alice's Current Account
```

With metadata: `{"category": "dining", "amount": -4.50, "date": "2026-07-15"}`

**Chunk 2:**

```
Transaction: SALARY JUL 2026
Amount: 3200.00 GBP
Date: 2026-07-01
Account: Alice's Current Account
```

With metadata: `{"category": "income", "amount": 3200.00, "date": "2026-07-01"}`

These chunks are then embedded and indexed like any other content, making the records searchable by text and filterable by metadata fields.

## Ingest Service Endpoints

By default, the Ingest Service runs from the following root URL:

```none
https://ingest.<ESS Domain>
```

The Ingest Service provides the following endpoints:

| Endpoint                               | Description                                                      |
| -------------------------------------- | ---------------------------------------------------------------- |
| **`POST /api/reindex`**                | Synchronous single-resource re-index.                            |
| **`POST /api/reindex/all`**            | Start an asynchronous bulk re-index for the authenticated agent. |
| **`GET /api/reindex/status/{jobId}`**  | Get the status of a bulk re-index job.                           |
| **`POST /api/reindex/cancel/{jobId}`** | Cancel a running bulk re-index job.                              |

### Endpoint Access Control

All Ingest Service endpoints require authentication. The endpoints support ESS access tokens (Bearer JWT) with **`sub`** and **`client_id`** claims.

The Ingest Service enforces a concurrency limit (bulkhead) of 2 concurrent requests per endpoint. When the limit is exceeded, the service returns **`429 Too Many Requests`** with a **`Retry-After: 5`** header.

## `POST /api/reindex`

Triggers a synchronous re-index of a single resource. The service validates the resource URL, resolves the resource identity and ownership via the [Platform Management](/ess/services/service-platform-management.md) and [Pod Storage](/ess/services/service-pod-management/service-pod-storage.md) services, and runs the full indexing pipeline (text extraction, chunking, embedding, dual-write).

{% hint style="warning" %}
**Ownership Verification**

The service verifies that the authenticated agent owns the resource before re-indexing. If the resource does not belong to the agent, the service returns **`403 Forbidden`**.
{% endhint %}

### Input

| Field         | Value                                         |
| ------------- | --------------------------------------------- |
| Endpoint      | **`https://ingest.<ESS Domain>/api/reindex`** |
| Method        | **`POST`**                                    |
| Authorization | Bearer access token                           |
| Content-Type  | **`application/json`**                        |
| Payload       | Re-index request object                       |

### Request Body

| Field              | Type         | Required | Description                                                                                                                                                                                                                             |
| ------------------ | ------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`resourceUrl`**  | String (URI) | Yes      | The HTTPS URL of the resource to re-index. Must have a valid host.                                                                                                                                                                      |
| **`resourceType`** | String       | No       | A label identifying the kind of content. Use short, descriptive values such as **`document`**, **`invoice`**, or **`email`**. Defaults to **`unknown`** if not provided. See [Controlling Resource Types](#controlling-resource-types). |

### Example Request

```http
POST /api/reindex HTTP/1.1
Host: ingest.example.com
Authorization: Bearer <access-token>
Content-Type: application/json

{
  "resourceUrl": "https://storage.example.com/pods/alice/reports/q3-2026.pdf",
  "resourceType": "document"
}
```

### Output

Returns a **`ReindexResponse`**:

| Field               | Type    | Description                                                       |
| ------------------- | ------- | ----------------------------------------------------------------- |
| **`resourceUrl`**   | String  | The URL of the re-indexed resource.                               |
| **`agentId`**       | String  | The agent identifier.                                             |
| **`status`**        | String  | The result status: **`success`**, **`skipped`**, or **`failed`**. |
| **`chunksIndexed`** | Integer | The number of chunks indexed.                                     |
| **`message`**       | String  | A human-readable status message.                                  |

### Example Response

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "resourceUrl": "https://storage.example.com/pods/alice/reports/q3-2026.pdf",
  "agentId": "https://id.example.com/alice",
  "status": "success",
  "chunksIndexed": 12,
  "message": "Resource re-indexed successfully"
}
```

#### Error Responses

| Status    | Description                                                                                             |
| --------- | ------------------------------------------------------------------------------------------------------- |
| **`400`** | Invalid resource URL (not HTTPS, missing host, or malformed).                                           |
| **`401`** | Unauthorized. No valid access token.                                                                    |
| **`403`** | Forbidden. The agent does not own the resource.                                                         |
| **`429`** | Concurrency limit reached. Retry after the number of seconds indicated in the **`Retry-After`** header. |

## `POST /api/reindex/all`

Starts an asynchronous bulk re-index of all resources belonging to the authenticated agent. The service discovers all resources across the agent's storages and processes them in the background.

{% hint style="warning" %}
**One Job at a Time**

Only one bulk re-index job can run per agent at a time. If a job is already running, the service returns **`409 Conflict`**.
{% endhint %}

### Input

| Field         | Value                                             |
| ------------- | ------------------------------------------------- |
| Endpoint      | **`https://ingest.<ESS Domain>/api/reindex/all`** |
| Method        | **`POST`**                                        |
| Authorization | Bearer access token                               |

No request body is required.

### Output

Returns **`202 Accepted`** with a **`BulkReindexResponse`**:

| Field         | Type          | Description                                                                    |
| ------------- | ------------- | ------------------------------------------------------------------------------ |
| **`jobId`**   | String (UUID) | The unique identifier for the re-index job. Use this to poll status or cancel. |
| **`status`**  | String        | Always **`accepted`** on success.                                              |
| **`message`** | String        | A human-readable status message.                                               |

### Example Response

```http
HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "jobId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "status": "accepted",
  "message": "Bulk re-index job started"
}
```

#### Error Responses

| Status    | Description                                                      |
| --------- | ---------------------------------------------------------------- |
| **`401`** | Unauthorized.                                                    |
| **`409`** | Conflict. A bulk re-index job is already running for this agent. |
| **`429`** | Concurrency limit reached.                                       |

## `GET /api/reindex/status/{jobId}`

Returns the current status and progress of a bulk re-index job.

### Input

| Field         | Value                                                        |
| ------------- | ------------------------------------------------------------ |
| Endpoint      | **`https://ingest.<ESS Domain>/api/reindex/status/{jobId}`** |
| Method        | **`GET`**                                                    |
| Authorization | Bearer access token                                          |

| Parameter   | Type          | Required | Description                                         |
| ----------- | ------------- | -------- | --------------------------------------------------- |
| **`jobId`** | String (UUID) | Yes      | The job ID returned by **`POST /api/reindex/all`**. |

### Output

Returns a **`ReindexStatusResponse`**:

| Field                | Type              | Description                                                                                       |
| -------------------- | ----------------- | ------------------------------------------------------------------------------------------------- |
| **`jobId`**          | String (UUID)     | The job identifier.                                                                               |
| **`agentId`**        | String            | The agent identifier.                                                                             |
| **`status`**         | String            | Job status: **`accepted`**, **`in_progress`**, **`completed`**, **`failed`**, or **`cancelled`**. |
| **`totalResources`** | Integer           | Total number of resources discovered for re-indexing.                                             |
| **`processedCount`** | Integer           | Number of resources processed so far.                                                             |
| **`failedCount`**    | Integer           | Number of resources that failed to re-index.                                                      |
| **`skippedCount`**   | Integer           | Number of resources skipped (e.g., unsupported content type).                                     |
| **`createdAt`**      | String (ISO 8601) | When the job was created.                                                                         |
| **`updatedAt`**      | String (ISO 8601) | When the job was last updated.                                                                    |
| **`completedAt`**    | String (ISO 8601) | When the job completed. **`null`** if still running.                                              |
| **`errorMessage`**   | String            | Error details. Only set when status is **`failed`**.                                              |

### Example Response

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "jobId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "agentId": "https://id.example.com/alice",
  "status": "in_progress",
  "totalResources": 312,
  "processedCount": 145,
  "failedCount": 3,
  "skippedCount": 12,
  "createdAt": "2026-07-28T10:00:00Z",
  "updatedAt": "2026-07-28T10:05:30Z",
  "completedAt": null,
  "errorMessage": null
}
```

#### Error Responses

| Status    | Description                                      |
| --------- | ------------------------------------------------ |
| **`401`** | Unauthorized.                                    |
| **`403`** | Forbidden. The job belongs to a different agent. |
| **`404`** | Job not found.                                   |

## `POST /api/reindex/cancel/{jobId}`

Requests cancellation of a running bulk re-index job. The job transitions to **`cancelling`** and stops at the next resource boundary.

### Input

| Field         | Value                                                        |
| ------------- | ------------------------------------------------------------ |
| Endpoint      | **`https://ingest.<ESS Domain>/api/reindex/cancel/{jobId}`** |
| Method        | **`POST`**                                                   |
| Authorization | Bearer access token                                          |

| Parameter   | Type          | Required | Description           |
| ----------- | ------------- | -------- | --------------------- |
| **`jobId`** | String (UUID) | Yes      | The job ID to cancel. |

### Output

Returns a **`BulkReindexResponse`** with status **`cancelling`**.

### Example Response

```http
HTTP/1.1 200 OK
Content-Type: application/json

{
  "jobId": "a1b2c3d4-5678-90ab-cdef-1234567890ab",
  "status": "cancelling",
  "message": "Cancellation requested"
}
```

#### Error Responses

| Status    | Description                                                                         |
| --------- | ----------------------------------------------------------------------------------- |
| **`401`** | Unauthorized.                                                                       |
| **`403`** | Forbidden. The job belongs to a different agent.                                    |
| **`404`** | Job not found.                                                                      |
| **`409`** | Conflict. The job is already in a terminal state (completed, failed, or cancelled). |

## Controlling Resource Types

The **`resourceType`** field determines how indexed content is categorized in the [Search Service](/ess/services/service-search.md). The value you set during indexing controls how content can be filtered, faceted, and deleted through the Search API.

### Why Resource Types Matter

When a developer builds an application that writes data to Pods and wants that data to be searchable, the resource type label is the primary mechanism for distinguishing between different kinds of content. For example, a financial application might index transaction records with the type **`bank:transaction`** and account statements with the type **`document`**. Users of the Search API can then filter results to show only transactions or only documents.

### How to Set Resource Types

Use the single-resource re-index endpoint (**`POST /api/reindex`**) to control the resource type label. The **`resourceType`** field in the request body accepts any string, but for the value to be usable in Search API filters, it must match the pattern **`^[a-zA-Z][a-zA-Z0-9_:.-]*$`**. See [Search API: Resource Types](/ess/services/service-search/search-api.md#resource-types) for naming guidelines.

#### Example: Indexing Application Data with Resource Types

An application that manages invoices and receipts can index each document with an appropriate type:

```http
POST /api/reindex HTTP/1.1
Host: ingest.example.com
Authorization: Bearer <access-token>
Content-Type: application/json

{
  "resourceUrl": "https://storage.example.com/pods/alice/invoices/inv-2026-001.pdf",
  "resourceType": "invoice"
}
```

```http
POST /api/reindex HTTP/1.1
Host: ingest.example.com
Authorization: Bearer <access-token>
Content-Type: application/json

{
  "resourceUrl": "https://storage.example.com/pods/alice/receipts/receipt-coffeeshop.jpg",
  "resourceType": "receipt"
}
```

The application can then search for only invoices:

```http
POST /api/search HTTP/1.1
Host: search.example.com
Authorization: Bearer <access-token>
Content-Type: application/json

{
  "query": "overdue payment",
  "resourceTypes": ["invoice"]
}
```

Or get a breakdown of indexed content by type:

```http
POST /api/search/facets HTTP/1.1
Host: search.example.com
Authorization: Bearer <access-token>
Content-Type: application/json

{
  "facets": ["resourceType"]
}
```

{% hint style="info" %}
**Automatic Indexing**

Content that is indexed automatically via Kafka audit events (without using the re-index API) receives a resource type derived from the resource's RDF type URIs. These values typically contain characters (such as forward slashes) that do not pass Search API filter validation. To use resource type filtering with automatically indexed content, re-index the resources using **`POST /api/reindex`** with an explicit resource type label.
{% endhint %}

## Health Endpoints

The Ingest Service provides Kubernetes health probes on the management port (9000):

| Endpoint                  | Description                                                                              |
| ------------------------- | ---------------------------------------------------------------------------------------- |
| **`GET /q/health/live`**  | Liveness probe.                                                                          |
| **`GET /q/health/ready`** | Readiness probe. Includes custom health checks for OpenSearch and pgvector connectivity. |
| **`GET /q/metrics`**      | Prometheus metrics.                                                                      |

## Ingest Service Configuration

The following configuration options are available for the Ingest Service.

### Text Chunking

#### INRUPT\_INGEST\_CHUNKING\_MAX\_CHUNK\_SIZE

Default: **`1000`**

The maximum number of characters per text chunk. Larger values produce fewer, longer chunks; smaller values produce more, shorter chunks with finer-grained search results.

#### INRUPT\_INGEST\_CHUNKING\_OVERLAP

Default: **`200`**

The number of overlapping characters between consecutive chunks. Overlap helps ensure that search queries matching text near chunk boundaries still return relevant results.

### Service Timeouts

#### INRUPT\_INGEST\_OCR\_TIMEOUT\_MS

Default: **`60000`**

The timeout in milliseconds for calls to the [OCR Service](/ess/services/service-search/ocr-service.md).

#### INRUPT\_INGEST\_EMBEDDING\_TIMEOUT\_MS

Default: **`5000`**

The timeout in milliseconds for calls to the [Embedding Service](/ess/services/service-search/embedding-service.md).

### OpenSearch

#### INRUPT\_INGEST\_OPENSEARCH\_INDEX

Default: **`pod-chunks-text`**

The name of the OpenSearch index.

{% hint style="warning" %}
**Important**

This value must match [**`INRUPT_SEARCH_OPENSEARCH_INDEX`**](/ess/services/service-search.md#inrupt_search_opensearch_index) configured for the Search Service.
{% endhint %}

#### INRUPT\_INGEST\_OPENSEARCH\_SHARDS

Default: **`1`**

The number of primary shards for the OpenSearch index.

#### INRUPT\_INGEST\_OPENSEARCH\_REPLICAS

Default: **`1`**

The number of replica shards for the OpenSearch index.

### Re-index Jobs

#### INRUPT\_INGEST\_REINDEX\_MAX\_RESTARTS

Default: **`3`**

The maximum number of automatic restarts for a crashed re-index job.

#### INRUPT\_INGEST\_REINDEX\_STALE\_JOB\_THRESHOLD\_MINUTES

Default: **`120`**

The number of minutes after which a re-index job with no progress updates is considered stale and eligible for cleanup.

#### INRUPT\_INGEST\_REINDEX\_MAX\_RESOURCES\_PER\_JOB

Default: **`100000`**

The maximum number of resources that a single bulk re-index job can process. This is a safety cap to prevent runaway jobs.

#### INRUPT\_INGEST\_REINDEX\_DISCOVERY\_PAGE\_SIZE

Default: **`100`**

The page size used when discovering resources during a bulk re-index.

### Repair Queue

The Ingest Service maintains a repair queue to handle inconsistencies between the pgvector and OpenSearch stores (e.g., when a write succeeds in one store but fails in the other).

#### INRUPT\_INGEST\_REPAIR\_INTERVAL\_SECONDS

Default: **`30`**

The interval in seconds between repair queue processing cycles.

#### INRUPT\_INGEST\_REPAIR\_MAX\_ATTEMPTS

Default: **`5`**

The maximum number of repair attempts per chunk before it is abandoned.

#### INRUPT\_INGEST\_REPAIR\_PURGE\_AFTER\_DAYS

Default: **`7`**

The number of days to retain completed repair records before purging.

### Template Indexing

#### INRUPT\_INGEST\_TEMPLATE\_MAX\_RENDERED\_SIZE

Default: **`4096`**

The maximum size in characters of a rendered template. Templates that produce output exceeding this limit are rejected.

#### INRUPT\_INGEST\_TEMPLATE\_MAX\_RECORDS

Default: **`1000`**

The maximum number of records per template-based indexing operation.

### Service URLs

#### EMBEDDING\_SERVICE\_URL

The URL of the [Embedding Service](/ess/services/service-search/embedding-service.md) used to generate vector embeddings during indexing. Must use HTTPS.

#### OCR\_SERVICE\_URL

The URL of the [OCR Service](/ess/services/service-search/ocr-service.md) used for text extraction from PDFs and images. Must use HTTPS.

### Infrastructure

#### OPENSEARCH\_URL

Default: **`http://localhost:9200`**

The URL of the OpenSearch cluster.

#### OPENSEARCH\_AUTH\_MODE

Default: **`none`**

The authentication mode for OpenSearch. Values: **`none`** or **`aws-sigv4`**.

#### KAFKA\_BOOTSTRAP\_SERVERS

Default: **`localhost:9092`**

Comma-delimited list of Kafka broker servers. The Ingest Service consumes audit events from and produces audit events to Kafka.

See also [ESS' Kafka Configuration](/ess/services/appendix/appendix-kafka-configuration.md).

## Additional Information

* [Search Service](/ess/services/service-search.md) — Overview and search configuration.
* [Search API](/ess/services/service-search/search-api.md) — Search endpoint documentation.
* [Embedding Service](/ess/services/service-search/embedding-service.md) — Vector embedding generation.
* [OCR Service](/ess/services/service-search/ocr-service.md) — Text extraction from PDFs and images.