Ingest Service
Added in version 3.2.0
The Ingest Service is the indexing pipeline for the ESS Search Service. It automatically indexes Pod content as resources are created, updated, or deleted, and provides a REST API for on-demand re-indexing.
How Indexing Works
The Ingest Service consumes encrypted audit events from a Kafka topic. When a resource change event is received:
The service fetches the resource content directly from object storage (S3 or Azure Blob Storage).
Text is extracted from the resource content. For PDFs and images, the service calls the OCR Service. For Microsoft Office documents, it uses Apache POI. Plain text, HTML, CSV, JSON, and RDF formats are extracted directly.
The extracted text is split into overlapping chunks (default: 1000 characters with 200-character overlap).
Each chunk is sent to the Embedding Service to generate a 384-dimensional vector embedding.
The chunks, embeddings, and metadata are dual-written to pgvector (PostgreSQL) and OpenSearch.
For delete events, the service removes all indexed chunks for the deleted resource from both stores.
Maximum Resource Size
Resources larger than 50 MB are skipped during indexing.
Dead-Letter Queue
If a message cannot be processed after the configured number of retries (default: 3), it is sent to the Kafka dead-letter queue. Failed messages can be replayed after the underlying issue is resolved.
Template-Based Indexing
The Ingest Service supports template-based indexing for structured data (such as JSON). Instead of indexing raw file content, a template defines how fields from each record are combined into searchable text and metadata.
How It Works
You create a mapping resource in the Pod — a JSON file that defines the template.
You attach the mapping to a data resource via a Solid auxiliary resource. The auxiliary resource contains an
indexingLink(URL of the mapping) and anindexingDataId(data identifier for the structured content).When the data resource is created or updated, the Ingest Service detects the auxiliary metadata in the audit event, fetches the mapping, and renders each record through the template.
Mapping Resource Format
A mapping resource is a JSON document with the following structure:
version
Integer
Yes
Must be 1.
strategy
String
Yes
Must be json-records.
recordPath
String
Yes
A JSONPath expression that selects the array of records from the data resource. For example, $[*] for a top-level array or $.transactions[*] for a nested array.
template
Array of String
Yes
Lines of text that are rendered for each record. Field references use {fieldName} syntax.
metadata
Object
No
Key-value pairs where values can reference record fields. These are stored as searchable metadata alongside each indexed chunk. Field references resolve to their original JSON type (number, boolean, string); the Search API numeric filter operators (gt, gte, lt, lte) cast values at query time, so numeric comparisons work regardless of whether the source value was a JSON number or string.
Template Syntax
Field references in template lines use curly braces:
{fieldName}— simple field lookup.{nested.field}— dot-separated path for nested objects.{field1||field2||'fallback'}— fallback chain. Uses the first non-null value. Literal strings are quoted with single quotes.{_meta.fieldName}— references resource-level metadata instead of the current record.
If all field references in a template line resolve to null, the entire line is omitted from the rendered text.
Example: Indexing Bank Transactions
Given a data resource containing:
And resource-level metadata (referenced via _meta) of:
With the mapping above, the Ingest Service produces two indexed chunks. Note that the first record has a description field, so the fallback chain {description||reference||'Unknown'} resolves to that value. The second record has no description but has reference, so it falls back to that.
Chunk 1:
With metadata: {"category": "dining", "amount": -4.50, "date": "2026-07-15"}
Chunk 2:
With metadata: {"category": "income", "amount": 3200.00, "date": "2026-07-01"}
These chunks are then embedded and indexed like any other content, making the records searchable by text and filterable by metadata fields.
Ingest Service Endpoints
By default, the Ingest Service runs from the following root URL:
The Ingest Service provides the following endpoints:
POST /api/reindex
Synchronous single-resource re-index.
POST /api/reindex/all
Start an asynchronous bulk re-index for the authenticated agent.
GET /api/reindex/status/{jobId}
Get the status of a bulk re-index job.
POST /api/reindex/cancel/{jobId}
Cancel a running bulk re-index job.
Endpoint Access Control
All Ingest Service endpoints require authentication. The endpoints support ESS access tokens (Bearer JWT) with sub and client_id claims.
The Ingest Service enforces a concurrency limit (bulkhead) of 2 concurrent requests per endpoint. When the limit is exceeded, the service returns 429 Too Many Requests with a Retry-After: 5 header.
POST /api/reindex
Triggers a synchronous re-index of a single resource. The service validates the resource URL, resolves the resource identity and ownership via the Platform Management and Pod Storage services, and runs the full indexing pipeline (text extraction, chunking, embedding, dual-write).
Ownership Verification
The service verifies that the authenticated agent owns the resource before re-indexing. If the resource does not belong to the agent, the service returns 403 Forbidden.
Input
Endpoint
https://ingest.<ESS Domain>/api/reindex
Method
POST
Authorization
Bearer access token
Content-Type
application/json
Payload
Re-index request object
Request Body
resourceUrl
String (URI)
Yes
The HTTPS URL of the resource to re-index. Must have a valid host.
resourceType
String
No
A label identifying the kind of content. Use short, descriptive values such as document, invoice, or email. Defaults to unknown if not provided. See Controlling Resource Types.
Example Request
Output
Returns a ReindexResponse:
resourceUrl
String
The URL of the re-indexed resource.
agentId
String
The agent identifier.
status
String
The result status: success, skipped, or failed.
chunksIndexed
Integer
The number of chunks indexed.
message
String
A human-readable status message.
Example Response
Error Responses
400
Invalid resource URL (not HTTPS, missing host, or malformed).
401
Unauthorized. No valid access token.
403
Forbidden. The agent does not own the resource.
429
Concurrency limit reached. Retry after the number of seconds indicated in the Retry-After header.
POST /api/reindex/all
Starts an asynchronous bulk re-index of all resources belonging to the authenticated agent. The service discovers all resources across the agent's storages and processes them in the background.
One Job at a Time
Only one bulk re-index job can run per agent at a time. If a job is already running, the service returns 409 Conflict.
Input
Endpoint
https://ingest.<ESS Domain>/api/reindex/all
Method
POST
Authorization
Bearer access token
No request body is required.
Output
Returns 202 Accepted with a BulkReindexResponse:
jobId
String (UUID)
The unique identifier for the re-index job. Use this to poll status or cancel.
status
String
Always accepted on success.
message
String
A human-readable status message.
Example Response
Error Responses
401
Unauthorized.
409
Conflict. A bulk re-index job is already running for this agent.
429
Concurrency limit reached.
GET /api/reindex/status/{jobId}
Returns the current status and progress of a bulk re-index job.
Input
Endpoint
https://ingest.<ESS Domain>/api/reindex/status/{jobId}
Method
GET
Authorization
Bearer access token
jobId
String (UUID)
Yes
The job ID returned by POST /api/reindex/all.
Output
Returns a ReindexStatusResponse:
jobId
String (UUID)
The job identifier.
agentId
String
The agent identifier.
status
String
Job status: accepted, in_progress, completed, failed, or cancelled.
totalResources
Integer
Total number of resources discovered for re-indexing.
processedCount
Integer
Number of resources processed so far.
failedCount
Integer
Number of resources that failed to re-index.
skippedCount
Integer
Number of resources skipped (e.g., unsupported content type).
createdAt
String (ISO 8601)
When the job was created.
updatedAt
String (ISO 8601)
When the job was last updated.
completedAt
String (ISO 8601)
When the job completed. null if still running.
errorMessage
String
Error details. Only set when status is failed.
Example Response
Error Responses
401
Unauthorized.
403
Forbidden. The job belongs to a different agent.
404
Job not found.
POST /api/reindex/cancel/{jobId}
Requests cancellation of a running bulk re-index job. The job transitions to cancelling and stops at the next resource boundary.
Input
Endpoint
https://ingest.<ESS Domain>/api/reindex/cancel/{jobId}
Method
POST
Authorization
Bearer access token
jobId
String (UUID)
Yes
The job ID to cancel.
Output
Returns a BulkReindexResponse with status cancelling.
Example Response
Error Responses
401
Unauthorized.
403
Forbidden. The job belongs to a different agent.
404
Job not found.
409
Conflict. The job is already in a terminal state (completed, failed, or cancelled).
Controlling Resource Types
The resourceType field determines how indexed content is categorized in the Search Service. The value you set during indexing controls how content can be filtered, faceted, and deleted through the Search API.
Why Resource Types Matter
When a developer builds an application that writes data to Pods and wants that data to be searchable, the resource type label is the primary mechanism for distinguishing between different kinds of content. For example, a financial application might index transaction records with the type bank:transaction and account statements with the type document. Users of the Search API can then filter results to show only transactions or only documents.
How to Set Resource Types
Use the single-resource re-index endpoint (POST /api/reindex) to control the resource type label. The resourceType field in the request body accepts any string, but for the value to be usable in Search API filters, it must match the pattern ^[a-zA-Z][a-zA-Z0-9_:.-]*$. See Search API: Resource Types for naming guidelines.
Example: Indexing Application Data with Resource Types
An application that manages invoices and receipts can index each document with an appropriate type:
The application can then search for only invoices:
Or get a breakdown of indexed content by type:
Automatic Indexing
Content that is indexed automatically via Kafka audit events (without using the re-index API) receives a resource type derived from the resource's RDF type URIs. These values typically contain characters (such as forward slashes) that do not pass Search API filter validation. To use resource type filtering with automatically indexed content, re-index the resources using POST /api/reindex with an explicit resource type label.
Health Endpoints
The Ingest Service provides Kubernetes health probes on the management port (9000):
GET /q/health/live
Liveness probe.
GET /q/health/ready
Readiness probe. Includes custom health checks for OpenSearch and pgvector connectivity.
GET /q/metrics
Prometheus metrics.
Ingest Service Configuration
The following configuration options are available for the Ingest Service.
Text Chunking
INRUPT_INGEST_CHUNKING_MAX_CHUNK_SIZE
Default: 1000
The maximum number of characters per text chunk. Larger values produce fewer, longer chunks; smaller values produce more, shorter chunks with finer-grained search results.
INRUPT_INGEST_CHUNKING_OVERLAP
Default: 200
The number of overlapping characters between consecutive chunks. Overlap helps ensure that search queries matching text near chunk boundaries still return relevant results.
Service Timeouts
INRUPT_INGEST_OCR_TIMEOUT_MS
Default: 60000
The timeout in milliseconds for calls to the OCR Service.
INRUPT_INGEST_EMBEDDING_TIMEOUT_MS
Default: 5000
The timeout in milliseconds for calls to the Embedding Service.
OpenSearch
INRUPT_INGEST_OPENSEARCH_INDEX
Default: pod-chunks-text
The name of the OpenSearch index.
Important
This value must match INRUPT_SEARCH_OPENSEARCH_INDEX configured for the Search Service.
INRUPT_INGEST_OPENSEARCH_SHARDS
Default: 1
The number of primary shards for the OpenSearch index.
INRUPT_INGEST_OPENSEARCH_REPLICAS
Default: 1
The number of replica shards for the OpenSearch index.
Re-index Jobs
INRUPT_INGEST_REINDEX_MAX_RESTARTS
Default: 3
The maximum number of automatic restarts for a crashed re-index job.
INRUPT_INGEST_REINDEX_STALE_JOB_THRESHOLD_MINUTES
Default: 120
The number of minutes after which a re-index job with no progress updates is considered stale and eligible for cleanup.
INRUPT_INGEST_REINDEX_MAX_RESOURCES_PER_JOB
Default: 100000
The maximum number of resources that a single bulk re-index job can process. This is a safety cap to prevent runaway jobs.
INRUPT_INGEST_REINDEX_DISCOVERY_PAGE_SIZE
Default: 100
The page size used when discovering resources during a bulk re-index.
Repair Queue
The Ingest Service maintains a repair queue to handle inconsistencies between the pgvector and OpenSearch stores (e.g., when a write succeeds in one store but fails in the other).
INRUPT_INGEST_REPAIR_INTERVAL_SECONDS
Default: 30
The interval in seconds between repair queue processing cycles.
INRUPT_INGEST_REPAIR_MAX_ATTEMPTS
Default: 5
The maximum number of repair attempts per chunk before it is abandoned.
INRUPT_INGEST_REPAIR_PURGE_AFTER_DAYS
Default: 7
The number of days to retain completed repair records before purging.
Template Indexing
INRUPT_INGEST_TEMPLATE_MAX_RENDERED_SIZE
Default: 4096
The maximum size in characters of a rendered template. Templates that produce output exceeding this limit are rejected.
INRUPT_INGEST_TEMPLATE_MAX_RECORDS
Default: 1000
The maximum number of records per template-based indexing operation.
Service URLs
EMBEDDING_SERVICE_URL
The URL of the Embedding Service used to generate vector embeddings during indexing. Must use HTTPS.
OCR_SERVICE_URL
The URL of the OCR Service used for text extraction from PDFs and images. Must use HTTPS.
Infrastructure
OPENSEARCH_URL
Default: http://localhost:9200
The URL of the OpenSearch cluster.
OPENSEARCH_AUTH_MODE
Default: none
The authentication mode for OpenSearch. Values: none or aws-sigv4.
KAFKA_BOOTSTRAP_SERVERS
Default: localhost:9092
Comma-delimited list of Kafka broker servers. The Ingest Service consumes audit events from and produces audit events to Kafka.
See also ESS' Kafka Configuration.
Additional Information
Search Service — Overview and search configuration.
Search API — Search endpoint documentation.
Embedding Service — Vector embedding generation.
OCR Service — Text extraction from PDFs and images.
Last updated