> For the complete documentation index, see [llms.txt](https://docs.inrupt.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.inrupt.com/ess/services/service-search.md).

# Search Service

{% hint style="success" %}
Added in version 3.2.0
{% endhint %}

ESS provides an optional Search Service that enables full-text and semantic search over content stored in Solid [Pods](https://github.com/inrupt/docs-gitbook/tree/main/reference/glossary.md#pod). It lets an application find relevant resources across a user's Pod by keyword, meaning, or both — for example, surfacing a receipt by describing what was purchased rather than knowing its exact filename or location. The Search Service replaces the Query/QPF (Fragments) Service that was removed in ESS 3.0.

A Pod is a personal data store where each piece of data is a [resource](https://github.com/inrupt/docs-gitbook/tree/main/reference/glossary.md#resource) identified by a URL. Resources are organized into [containers](https://github.com/inrupt/docs-gitbook/tree/main/reference/glossary.md#container) (similar to folders). Each Pod belongs to an [agent](https://github.com/inrupt/docs-gitbook/tree/main/reference/glossary.md#agent) (a person or application) identified by a unique agent IRI.

The Search Service consists of four cooperating microservices:

| Service                                                            | Description                                                                                                                                                                         |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Search**                                                         | Query API that provides hybrid (keyword + semantic), keyword-only, and semantic-only search, plus faceted discovery, metadata queries, and index management.                        |
| [**Ingest**](/ess/services/service-search/ingest-service.md)       | Ingestion pipeline that listens for resource change events on Kafka, extracts text, generates embeddings, and writes to both OpenSearch and pgvector. Also provides a re-index API. |
| [**Embedding**](/ess/services/service-search/embedding-service.md) | ML inference sidecar that generates 384-dimensional vector embeddings using the BAAI/bge-small-en-v1.5 model.                                                                       |
| [**OCR**](/ess/services/service-search/ocr-service.md)             | Text extraction service that extracts text from PDFs (text layer with OCR fallback) and images.                                                                                     |

## How It Works

Content is indexed automatically as Pods change — no manual sync is required. See [Implementation Notes](#implementation-notes) for how the indexing pipeline works under the hood.

### Supported Content Types

The Ingest Service extracts text from the following content types:

| Content Type                                                                               | Method                                                                              |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
| PDF (`application/pdf`)                                                                    | Text layer extraction via pypdf, with OCR fallback for pages with insufficient text |
| Images (PNG, JPEG, GIF, WEBP)                                                              | OCR via RapidOCR                                                                    |
| Microsoft Word (`application/vnd.openxmlformats-officedocument.wordprocessingml.document`) | Apache POI                                                                          |
| Microsoft Excel (`.xlsx`, `.xls`)                                                          | Apache POI                                                                          |
| Plain text (`text/plain`)                                                                  | Direct extraction                                                                   |
| HTML (`text/html`)                                                                         | Text extraction                                                                     |
| CSV (`text/csv`)                                                                           | Text extraction                                                                     |
| JSON (`application/json`)                                                                  | Text extraction                                                                     |
| RDF formats (Turtle, JSON-LD)                                                              | Text extraction                                                                     |

### Search Modes

The Search Service supports three search modes:

| Mode                   | Description                                                                                                                                      |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **`hybrid`** (default) | Combines BM25 keyword search (via OpenSearch) with semantic vector search (via pgvector). Results are ranked using Reciprocal Rank Fusion (RRF). |
| **`keyword`**          | BM25 keyword search only, using OpenSearch.                                                                                                      |
| **`semantic`**         | Semantic vector search only, using pgvector. Requires the Embedding Service to be available.                                                     |

{% hint style="info" %}
**Graceful Degradation**

When the Embedding Service is unavailable, the Search Service automatically degrades to keyword-only mode for hybrid searches. The service logs a warning and opens a circuit breaker. Semantic-only searches return a `503` error. Recovery is automatic when the Embedding Service comes back online.

During an Embedding Service outage, new content indexing by the Ingest Service fails. Affected events are sent to the Kafka dead-letter queue and can be replayed after recovery.
{% endhint %}

## Search Service Endpoints

By default, the Search Service runs from the following root URL:

```none
https://search.<ESS Domain>
```

The Search Service provides the following endpoints:

| Endpoint                               | Description                                                                                                                                                |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`POST /api/search`**                 | [Hybrid, keyword, or semantic search](/ess/services/service-search/search-api.md#post-apisearch) over Pod content.                                         |
| **`POST /api/search/query`**           | [Metadata-based query](/ess/services/service-search/search-api.md#post-apisearchquery) with filters, sorting, deduplication, and optional semantic search. |
| **`POST /api/search/query/aggregate`** | [Group-by aggregation](/ess/services/service-search/search-api.md#post-apisearchqueryaggregate) on metadata fields.                                        |
| **`POST /api/search/facets`**          | [Faceted counts](/ess/services/service-search/search-api.md#post-apisearchfacets) grouped by resource type or container.                                   |
| **`POST /api/search/resources`**       | [Resource lookup](/ess/services/service-search/search-api.md#post-apisearchresources) to check index status and metadata for specific resource URLs.       |
| **`GET /api/search/stats`**            | [Index statistics](/ess/services/service-search/search-api.md#get-apisearchstats) including chunk counts, resource counts, and breakdown by type.          |
| **`DELETE /api/search/all`**           | [Delete all indexed content](/ess/services/service-search/search-api.md#delete-apisearchall) for the authenticated agent.                                  |
| **`DELETE /api/search/by-type`**       | [Delete by resource type](/ess/services/service-search/search-api.md#delete-apisearchby-type).                                                             |
| **`DELETE /api/search/by-resource`**   | [Delete by resource URL](/ess/services/service-search/search-api.md#delete-apisearchby-resource).                                                          |

See [Search API](/ess/services/service-search/search-api.md) for detailed endpoint documentation.

### Endpoint Access Control

#### Access Tokens

All Search Service endpoints require the user to be authenticated. The endpoints support [ESS access tokens](https://github.com/inrupt/docs-gitbook/tree/main/reference/glossary.md#ess-access-token) obtained via OAuth 2.0 Token Exchange ([RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693)). To obtain an access token, your application exchanges its OpenID Connect ID token with the ESS [Token Exchange](/ess/services/service-platform-management/token-exchange.md) endpoint.

The access token must contain:

* A `sub` claim set to the caller's ESS agent IRI (the unique identifier for the user or application).
* A `client_id` claim identifying the calling application (registered with your identity provider).

{% hint style="info" %}
**Multi-Tenancy**

All search operations are scoped to the authenticated agent's identity. The agent identity is extracted from the `sub` claim in the JWT access token. Cross-tenant data leakage is prevented by mandatory agent filtering on every query.
{% endhint %}

#### Client Allow List

To restrict which applications can access the Search API, configure the [**`INRUPT_SEARCH_CLIENT_ID_ALLOW_LIST`**](#inrupt_search_client_id_allow_list) setting. By default, all authenticated clients are allowed.

#### Rate Limiting

The Search Service applies per-agent rate limiting. When the rate limit is exceeded, the service returns `429 Too Many Requests` with the following headers:

| Header                      | Description                                |
| --------------------------- | ------------------------------------------ |
| **`Retry-After`**           | Seconds to wait before retrying.           |
| **`X-RateLimit-Limit`**     | Maximum requests per second.               |
| **`X-RateLimit-Remaining`** | Remaining requests in the current window.  |
| **`X-RateLimit-Reset`**     | Unix timestamp when the rate limit resets. |

### Caching

All Search Service responses include the `Cache-Control: no-store` header. Search results reflect the current state of the index and should not be cached by clients.

### Health and Metrics Endpoints

Kubernetes health probes and metrics are available on the management port (9000) and do not require authentication:

| Endpoint                    | Description                                                      |
| --------------------------- | ---------------------------------------------------------------- |
| **`GET /q/health/live`**    | Liveness probe.                                                  |
| **`GET /q/health/ready`**   | Readiness probe. Checks connectivity to OpenSearch and pgvector. |
| **`GET /q/health/started`** | Startup probe.                                                   |
| **`GET /q/metrics`**        | Prometheus metrics.                                              |

## Search Service Configuration

As part of the [installation process](/ess/installation.md), Inrupt provides base Kustomize overlays and associated files that require deployment-specific configuration inputs.

The following configuration options are available for the Search Service.

### Required

#### OPENSEARCH\_URL

Default: **`http://localhost:9200`**

The URL of the OpenSearch cluster used for keyword search indexing.

#### OPENSEARCH\_AUTH\_MODE

Default: **`none`**

The authentication mode for connecting to OpenSearch. Supported values:

* **`none`** — No authentication.
* **`aws-sigv4`** — AWS IAM authentication using SigV4 request signing.

When set to **`aws-sigv4`**, [**`AWS_REGION`**](#aws_region) must also be configured.

#### AWS\_REGION

The AWS region for OpenSearch SigV4 signing. Required when [**`OPENSEARCH_AUTH_MODE`**](#opensearch_auth_mode) is set to **`aws-sigv4`**.

#### QUARKUS\_DATASOURCE\_JDBC\_URL

The JDBC connection string for the PostgreSQL database with the pgvector extension.

See also: [**`QUARKUS_DATASOURCE_USERNAME`**](#quarkus_datasource_username) and [**`QUARKUS_DATASOURCE_PASSWORD`**](#quarkus_datasource_password).

#### QUARKUS\_DATASOURCE\_USERNAME

The username for the PostgreSQL database.

See also: [**`QUARKUS_DATASOURCE_JDBC_URL`**](#quarkus_datasource_jdbc_url) and [**`QUARKUS_DATASOURCE_PASSWORD`**](#quarkus_datasource_password).

#### QUARKUS\_DATASOURCE\_PASSWORD

The password for the PostgreSQL database.

See also: [**`QUARKUS_DATASOURCE_JDBC_URL`**](#quarkus_datasource_jdbc_url) and [**`QUARKUS_DATASOURCE_USERNAME`**](#quarkus_datasource_username).

#### EMBEDDING\_SERVICE\_URL

The URL of the [Embedding Service](/ess/services/service-search/embedding-service.md) used to generate vector embeddings for semantic search queries. Must use HTTPS.

#### KAFKA\_BOOTSTRAP\_SERVERS

Default: **`localhost:9092`**

Comma-delimited list of Kafka broker servers. The Search Service uses Kafka to produce audit events.

See also [ESS' Kafka Configuration](/ess/services/appendix/appendix-kafka-configuration.md).

### Optional

#### INRUPT\_SEARCH\_CLIENT\_ID\_ALLOW\_LIST

A comma-separated list of client identifiers allowed to access the Search API.

* If unset, all authenticated clients are allowed.
* Set to a list of specific Client IDs to restrict access.
* The special value **`ANY`** explicitly allows all clients.

#### INRUPT\_SEARCH\_DEADLINE\_MS

Default: **`10000`**

The server-side deadline for search operations in milliseconds. If the search does not complete within this deadline, the service returns `504 Gateway Timeout`.

#### INRUPT\_SEARCH\_METADATA\_DEADLINE\_MS

Default: **`15000`**

The server-side deadline for metadata query and aggregation operations in milliseconds.

#### INRUPT\_SEARCH\_EMBEDDING\_TIMEOUT\_MS

Default: **`5000`**

The timeout for calls to the Embedding Service in milliseconds.

#### INRUPT\_SEARCH\_RATE\_LIMIT\_REQUESTS\_PER\_SECOND

Default: **`10`**

The maximum number of requests per second per agent.

#### INRUPT\_SEARCH\_OPENSEARCH\_INDEX

Default: **`pod-chunks-text`**

The name of the OpenSearch index used for keyword search.

{% hint style="warning" %}
**Important**

This value must match the index name configured for the [Ingest Service](/ess/services/service-search/ingest-service.md#inrupt_ingest_opensearch_index).
{% endhint %}

{% hint style="info" %}
The OpenSearch index is created by the [Ingest Service](/ess/services/service-search/ingest-service.md). Shard and replica counts are configured there — see [**`INRUPT_INGEST_OPENSEARCH_SHARDS`**](/ess/services/service-search/ingest-service.md#inrupt_ingest_opensearch_shards) and [**`INRUPT_INGEST_OPENSEARCH_REPLICAS`**](/ess/services/service-search/ingest-service.md#inrupt_ingest_opensearch_replicas).
{% endhint %}

### Advanced: Search Tuning

These settings control ranking behavior. The defaults work well for most deployments — adjust only after evaluating search quality against your own content.

#### INRUPT\_SEARCH\_RRF\_K

Default: **`60`**

The K parameter for Reciprocal Rank Fusion (RRF) used in hybrid search mode. Higher values give more weight to lower-ranked results from individual search backends.

#### INRUPT\_SEARCH\_KEYWORD\_CUTOFF

Default: **`0.3`**

The minimum BM25 keyword score threshold. BM25 scores are max-normalized to a `0`–`1` scale for each query — every score is divided by the highest score in that query's result set, so the best match always scores `1.0`. The threshold is applied against this normalized score, and keyword candidates below it are discarded.

The cutoff applies in **`keyword`** mode and in **`hybrid`** mode. In **`hybrid`** mode it prunes the keyword candidates before deduplication and before RRF fusion, so discarded results cannot re-enter the merged result set.

#### INRUPT\_SEARCH\_SEMANTIC\_CUTOFF

Default: **`0.35`**

The minimum semantic similarity score threshold. The semantic score is `1 - cosine distance`, on a `0`–`1` scale. Vector candidates scoring below the threshold are discarded.

The cutoff applies in **`semantic`** mode and in **`hybrid`** mode. As with the keyword cutoff, in **`hybrid`** mode it is applied before deduplication and before RRF fusion.

{% hint style="info" %}
**Keyword cutoffs are relative, not absolute**

Because BM25 scores are normalized against the best match for each query, [**`INRUPT_SEARCH_KEYWORD_CUTOFF`**](#inrupt_search_keyword_cutoff) sets a *relative* relevance floor rather than an absolute quality bar. The top-scoring result always scores `1.0` and therefore always survives the cutoff: raising the threshold narrows how far below the best match other results may fall, but it never empties the result set for a query that matched anything.

For the same reason, a given document may pass the cutoff for one query and fail it for another, depending on how strong the competing matches are.
{% endhint %}

#### INRUPT\_SEARCH\_RETRIEVAL\_MULTIPLIER

Default: **`3`**

The retrieval multiplier for internal over-fetching. The service retrieves `limit * retrieval-multiplier` results from each backend before applying RRF fusion and returning the final result set.

#### INRUPT\_SEARCH\_TOP\_K

Default: **`30`**

The maximum number of results retrieved from each search backend before fusion.

## Implementation Notes

Content is indexed through the following pipeline:

1. When a resource is created, updated, or deleted, the [Pod Storage Service](/ess/services/service-pod-management/service-pod-storage.md) publishes an audit event to an encrypted Kafka topic.
2. The **Ingest Service** consumes the event, fetches the resource content from object storage (S3 or Azure Blob Storage), and extracts text. For PDFs and images, it calls the **OCR Service**.
3. The extracted text is split into overlapping chunks.
4. Each chunk is sent to the **Embedding Service** to generate a 384-dimensional vector embedding.
5. The chunks, embeddings, and metadata are dual-written to **pgvector** (PostgreSQL with the vector extension) and **OpenSearch**.
6. The **Search Service** queries both stores and merges results using Reciprocal Rank Fusion (RRF) for hybrid search.

## Error Responses

The Search Service returns errors in [RFC 7807 Problem Details](https://datatracker.ietf.org/doc/html/rfc7807) format:

```json
{
  "type": "about:blank",
  "title": "Bad Request",
  "status": 400,
  "detail": "query: must not be blank"
}
```

Validation errors include an additional `errors` array:

```json
{
  "type": "about:blank",
  "title": "Bad Request",
  "status": 400,
  "detail": "Validation failed",
  "errors": [
    {"path": "query", "message": "must not be blank"},
    {"path": "limit", "message": "must be greater than or equal to 1"}
  ]
}
```

| Status    | Description                                                                                                                                      |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **`400`** | Validation error. The request body failed validation.                                                                                            |
| **`401`** | Unauthorized. The request does not include a valid access token.                                                                                 |
| **`403`** | Forbidden. The agent or client is not authorized.                                                                                                |
| **`429`** | Too Many Requests. Rate limit exceeded. See rate limiting headers.                                                                               |
| **`503`** | Service Unavailable. A required backend (OpenSearch or pgvector) is unreachable. The service fails closed rather than returning partial results. |
| **`504`** | Gateway Timeout. The search did not complete within the configured deadline.                                                                     |

## Additional Information

* [Search API](/ess/services/service-search/search-api.md) — Detailed endpoint documentation with request/response shapes.
* [Ingest Service](/ess/services/service-search/ingest-service.md) — How content is indexed and how to trigger re-indexing.
* [Embedding Service](/ess/services/service-search/embedding-service.md) — Vector embedding generation.
* [OCR Service](/ess/services/service-search/ocr-service.md) — Text extraction from PDFs and images.