Search Service

Added in version 3.2.0

ESS provides an optional Search Service that enables full-text and semantic search over content stored in Solid Pods. It lets an application find relevant resources across a user's Pod by keyword, meaning, or both — for example, surfacing a receipt by describing what was purchased rather than knowing its exact filename or location. The Search Service replaces the Query/QPF (Fragments) Service that was removed in ESS 3.0.

A Pod is a personal data store where each piece of data is a resource identified by a URL. Resources are organized into containers (similar to folders). Each Pod belongs to an agent (a person or application) identified by a unique agent IRI.

The Search Service consists of four cooperating microservices:

Service	Description
Search	Query API that provides hybrid (keyword + semantic), keyword-only, and semantic-only search, plus faceted discovery, metadata queries, and index management.
Ingest	Ingestion pipeline that listens for resource change events on Kafka, extracts text, generates embeddings, and writes to both OpenSearch and pgvector. Also provides a re-index API.
Embedding	ML inference sidecar that generates 384-dimensional vector embeddings using the BAAI/bge-small-en-v1.5 model.
OCR	Text extraction service that extracts text from PDFs (text layer with OCR fallback) and images.

How It Works

Content is indexed automatically as Pods change — no manual sync is required. See Implementation Notes for how the indexing pipeline works under the hood.

Supported Content Types

The Ingest Service extracts text from the following content types:

Content Type	Method
PDF (application/pdf)	Text layer extraction via pypdf, with OCR fallback for pages with insufficient text
Images (PNG, JPEG, GIF, WEBP)	OCR via RapidOCR
Microsoft Word (application/vnd.openxmlformats-officedocument.wordprocessingml.document)	Apache POI
Microsoft Excel (.xlsx, .xls)	Apache POI
Plain text (text/plain)	Direct extraction
HTML (text/html)	Text extraction
CSV (text/csv)	Text extraction
JSON (application/json)	Text extraction
RDF formats (Turtle, JSON-LD)	Text extraction

Search Modes

The Search Service supports three search modes:

Mode	Description
hybrid (default)	Combines BM25 keyword search (via OpenSearch) with semantic vector search (via pgvector). Results are ranked using Reciprocal Rank Fusion (RRF).
keyword	BM25 keyword search only, using OpenSearch.
semantic	Semantic vector search only, using pgvector. Requires the Embedding Service to be available.

Graceful Degradation

When the Embedding Service is unavailable, the Search Service automatically degrades to keyword-only mode for hybrid searches. The service logs a warning and opens a circuit breaker. Semantic-only searches return a 503 error. Recovery is automatic when the Embedding Service comes back online.

During an Embedding Service outage, new content indexing by the Ingest Service fails. Affected events are sent to the Kafka dead-letter queue and can be replayed after recovery.

Search Service Endpoints

By default, the Search Service runs from the following root URL:

https://search.<ESS Domain>

The Search Service provides the following endpoints:

Endpoint	Description
POST /api/search	Hybrid, keyword, or semantic search over Pod content.
POST /api/search/query	Metadata-based query with filters, sorting, deduplication, and optional semantic search.
POST /api/search/query/aggregate	Group-by aggregation on metadata fields.
POST /api/search/facets	Faceted counts grouped by resource type or container.
POST /api/search/resources	Resource lookup to check index status and metadata for specific resource URLs.
GET /api/search/stats	Index statistics including chunk counts, resource counts, and breakdown by type.
DELETE /api/search/all	Delete all indexed content for the authenticated agent.
DELETE /api/search/by-type	Delete by resource type.
DELETE /api/search/by-resource	Delete by resource URL.

See Search API for detailed endpoint documentation.

Endpoint Access Control

Access Tokens

All Search Service endpoints require the user to be authenticated. The endpoints support ESS access tokens obtained via OAuth 2.0 Token Exchange (RFC 8693). To obtain an access token, your application exchanges its OpenID Connect ID token with the ESS Token Exchange endpoint.

The access token must contain:

• A sub claim set to the caller's ESS agent IRI (the unique identifier for the user or application).

• A client_id claim identifying the calling application (registered with your identity provider).

Multi-Tenancy

All search operations are scoped to the authenticated agent's identity. The agent identity is extracted from the sub claim in the JWT access token. Cross-tenant data leakage is prevented by mandatory agent filtering on every query.

Client Allow List

To restrict which applications can access the Search API, configure the INRUPT_SEARCH_CLIENT_ID_ALLOW_LIST setting. By default, all authenticated clients are allowed.

Rate Limiting

The Search Service applies per-agent rate limiting. When the rate limit is exceeded, the service returns 429 Too Many Requests with the following headers:

Header	Description
Retry-After	Seconds to wait before retrying.
X-RateLimit-Limit	Maximum requests per second.
X-RateLimit-Remaining	Remaining requests in the current window.
X-RateLimit-Reset	Unix timestamp when the rate limit resets.

Caching

All Search Service responses include the Cache-Control: no-store header. Search results reflect the current state of the index and should not be cached by clients.

Health and Metrics Endpoints

Kubernetes health probes and metrics are available on the management port (9000) and do not require authentication:

Endpoint	Description
GET /q/health/live	Liveness probe.
GET /q/health/ready	Readiness probe. Checks connectivity to OpenSearch and pgvector.
GET /q/health/started	Startup probe.
GET /q/metrics	Prometheus metrics.

Search Service Configuration

As part of the installation process, Inrupt provides base Kustomize overlays and associated files that require deployment-specific configuration inputs.

The following configuration options are available for the Search Service.

Required

OPENSEARCH_URL

Default: http://localhost:9200

The URL of the OpenSearch cluster used for keyword search indexing.

OPENSEARCH_AUTH_MODE

Default: none

The authentication mode for connecting to OpenSearch. Supported values:

• none — No authentication.

• aws-sigv4 — AWS IAM authentication using SigV4 request signing.

When set to aws-sigv4, AWS_REGION must also be configured.

AWS_REGION

The AWS region for OpenSearch SigV4 signing. Required when OPENSEARCH_AUTH_MODE is set to aws-sigv4.

QUARKUS_DATASOURCE_JDBC_URL

The JDBC connection string for the PostgreSQL database with the pgvector extension.

See also: QUARKUS_DATASOURCE_USERNAME and QUARKUS_DATASOURCE_PASSWORD.

QUARKUS_DATASOURCE_USERNAME

The username for the PostgreSQL database.

See also: QUARKUS_DATASOURCE_JDBC_URL and QUARKUS_DATASOURCE_PASSWORD.

QUARKUS_DATASOURCE_PASSWORD

The password for the PostgreSQL database.

See also: QUARKUS_DATASOURCE_JDBC_URL and QUARKUS_DATASOURCE_USERNAME.

EMBEDDING_SERVICE_URL

The URL of the Embedding Service used to generate vector embeddings for semantic search queries. Must use HTTPS.

KAFKA_BOOTSTRAP_SERVERS

Default: localhost:9092

Comma-delimited list of Kafka broker servers. The Search Service uses Kafka to produce audit events.

See also ESS' Kafka Configuration.

Optional

INRUPT_SEARCH_CLIENT_ID_ALLOW_LIST

A comma-separated list of client identifiers allowed to access the Search API.

• If unset, all authenticated clients are allowed.

• Set to a list of specific Client IDs to restrict access.

• The special value ANY explicitly allows all clients.

INRUPT_SEARCH_DEADLINE_MS

Default: 10000

The server-side deadline for search operations in milliseconds. If the search does not complete within this deadline, the service returns 504 Gateway Timeout.

INRUPT_SEARCH_METADATA_DEADLINE_MS

Default: 15000

The server-side deadline for metadata query and aggregation operations in milliseconds.

INRUPT_SEARCH_EMBEDDING_TIMEOUT_MS

Default: 5000

The timeout for calls to the Embedding Service in milliseconds.

INRUPT_SEARCH_RATE_LIMIT_REQUESTS_PER_SECOND

Default: 10

The maximum number of requests per second per agent.

INRUPT_SEARCH_OPENSEARCH_INDEX

Default: pod-chunks-text

The name of the OpenSearch index used for keyword search.

Important

This value must match the index name configured for the Ingest Service.

The OpenSearch index is created by the Ingest Service. Shard and replica counts are configured there — see INRUPT_INGEST_OPENSEARCH_SHARDS and INRUPT_INGEST_OPENSEARCH_REPLICAS.

Advanced: Search Tuning

These settings control ranking behavior. The defaults work well for most deployments — adjust only after evaluating search quality against your own content.

INRUPT_SEARCH_RRF_K

Default: 60

The K parameter for Reciprocal Rank Fusion (RRF) used in hybrid search mode. Higher values give more weight to lower-ranked results from individual search backends.

INRUPT_SEARCH_KEYWORD_CUTOFF

Default: 0.3

The minimum BM25 keyword score threshold. BM25 scores are max-normalized to a 0–1 scale for each query — every score is divided by the highest score in that query's result set, so the best match always scores 1.0. The threshold is applied against this normalized score, and keyword candidates below it are discarded.

The cutoff applies in keyword mode and in hybrid mode. In hybrid mode it prunes the keyword candidates before deduplication and before RRF fusion, so discarded results cannot re-enter the merged result set.

INRUPT_SEARCH_SEMANTIC_CUTOFF

Default: 0.35

The minimum semantic similarity score threshold. The semantic score is 1 - cosine distance, on a 0–1 scale. Vector candidates scoring below the threshold are discarded.

The cutoff applies in semantic mode and in hybrid mode. As with the keyword cutoff, in hybrid mode it is applied before deduplication and before RRF fusion.

Keyword cutoffs are relative, not absolute

Because BM25 scores are normalized against the best match for each query, INRUPT_SEARCH_KEYWORD_CUTOFF sets a relative relevance floor rather than an absolute quality bar. The top-scoring result always scores 1.0 and therefore always survives the cutoff: raising the threshold narrows how far below the best match other results may fall, but it never empties the result set for a query that matched anything.

For the same reason, a given document may pass the cutoff for one query and fail it for another, depending on how strong the competing matches are.

INRUPT_SEARCH_RETRIEVAL_MULTIPLIER

Default: 3

The retrieval multiplier for internal over-fetching. The service retrieves limit * retrieval-multiplier results from each backend before applying RRF fusion and returning the final result set.

INRUPT_SEARCH_TOP_K

Default: 30

The maximum number of results retrieved from each search backend before fusion.

Implementation Notes

Content is indexed through the following pipeline:

1. When a resource is created, updated, or deleted, the Pod Storage Service publishes an audit event to an encrypted Kafka topic.

2. The Ingest Service consumes the event, fetches the resource content from object storage (S3 or Azure Blob Storage), and extracts text. For PDFs and images, it calls the OCR Service.

3. The extracted text is split into overlapping chunks.

4. Each chunk is sent to the Embedding Service to generate a 384-dimensional vector embedding.

5. The chunks, embeddings, and metadata are dual-written to pgvector (PostgreSQL with the vector extension) and OpenSearch.

6. The Search Service queries both stores and merges results using Reciprocal Rank Fusion (RRF) for hybrid search.

Error Responses

The Search Service returns errors in RFC 7807 Problem Details format:

{
  "type": "about:blank",
  "title": "Bad Request",
  "status": 400,
  "detail": "query: must not be blank"
}

Validation errors include an additional errors array:

{
  "type": "about:blank",
  "title": "Bad Request",
  "status": 400,
  "detail": "Validation failed",
  "errors": [
    {"path": "query", "message": "must not be blank"},
    {"path": "limit", "message": "must be greater than or equal to 1"}
  ]
}

Status	Description
400	Validation error. The request body failed validation.
401	Unauthorized. The request does not include a valid access token.
403	Forbidden. The agent or client is not authorized.
429	Too Many Requests. Rate limit exceeded. See rate limiting headers.
503	Service Unavailable. A required backend (OpenSearch or pgvector) is unreachable. The service fails closed rather than returning partial results.
504	Gateway Timeout. The search did not complete within the configured deadline.

Additional Information

• Search API — Detailed endpoint documentation with request/response shapes.

• Ingest Service — How content is indexed and how to trigger re-indexing.

• Embedding Service — Vector embedding generation.

• OCR Service — Text extraction from PDFs and images.