Skip to main content

Semantic and Hybrid Search

This tutorial shows how to find the content chunks most relevant to a question with vector similarity search over the embeddings stored in the Cloud Content Repository, how to narrow the search with a document filter and a full-text match on the chunk text, and how to look up a single embedding by ID. These are the building blocks of a Retrieval-Augmented Generation (RAG) workflow.

Prerequisites​

  • A valid user access token for the signed-in user you are searching for, not a service user token: document ACLs are applied to the caller (see Authentication)
  • The base URL of your environment
  • A query vector produced by the same embedding model as the stored embeddings. The API does not compute embeddings from text

Set two variables before running the curl examples. The HTTP examples use the equivalent {{baseUrl}} and {{accessToken}} placeholders.

export BASE_URL="https://{environment_id}.content-lake.ai.experience.hyland.com"
export TOKEN="<access token of the signed-in user>"
About Embeddings

Embeddings are vector representations of text and images generated during ingestion. The vector dimensions depend on the model used. These embeddings power semantic search and are commonly used in Retrieval-Augmented Generation (RAG) workflows.

How Embeddings Work

During ingestion, documents are chunked and each chunk is converted into a vector embedding. Each embedding record contains the chunk text (sysembed_text), the vector (sysembed_vector), the location within the source document (sysembed_location — page/paragraph for text, bounding box for images, row/column for spreadsheets, timestamps for video), and a type label (sysembed_type, e.g. cohere.embed-multilingual-v3). There is no separate "chunks" API — embeddings are the chunks.

Tutorial Overview​

In this tutorial, you'll learn to:

  1. Run a vector similarity search
  2. Filter the search by document and embedding type
  3. Combine vector and full-text matching (hybrid search)
  4. Look up an embedding by ID
  5. Use the results in a RAG workflow

Perform a vector similarity (k-NN) search over embeddings. You supply the query vector; the API does not compute embeddings from text. Optionally combine it with a full-text match on the chunk text (hybrid search).

Send the query vector to POST /api/query/embeddings:

POST {{baseUrl}}/api/query/embeddings
Authorization: Bearer {{accessToken}}
Accept: application/vnd.hxcs.v1+json
Content-Type: application/json

{
  "vector": [0.0316, 0.0449, -0.0221, ...],
  "query": "SELECT * FROM SysContent WHERE sys_primaryType = 'SysFile'",
  "limit": 2
}
The vector must match the index dimension

The vector's length must equal the index's vector dimension. In the tested environment, whose embeddings are of type cohere.embed-multilingual-v3, the dimension is 1024; the dimension in your environment depends on its embedding model. Any other length returns 400 with a message containing Query vector has invalid dimension: 3. Dimension should be: 1024 (prefixed with Error performing OpenSearch query: failed to create query:), and a request without a vector returns 400 Invalid input.

Response​

Returns repositoryId, the supplied vector, query, limit, offset, count, and an embeddings array. With trackTotalCount: true the response includes totalCount (plus totalCountIsTruncated: true when the count was capped); otherwise neither field is returned. In vector search, totalCount is not the number of all matching chunks: it is capped at the number of nearest neighbours fetched, which is the larger of limit and trackTotalCountUpTo. count can be lower than limit when some matching chunks' text cannot be retrieved. Each entry is an embedding object (sysembed_* fields) as described in Embedding Object, with sysembed_score set to the similarity score.

-- Request (vector shortened; it must have 1024 values here)
{
  "vector": [0.0316, 0.0449, -0.0221, ...],
  "query": "SELECT * FROM SysContent WHERE sys_primaryType = 'SysFile'",
  "limit": 2
}

-- Response (trimmed)
{
  "query": "SELECT * FROM SysContent WHERE sys_primaryType = 'SysFile'",
  "limit": 2,
  "offset": 0,
  "count": 2,
  "embeddings": [
    {
      "sysembed_id": "ea20b9df-50fd-5687-acf3-ef90de020002",
      "sysembed_docId": "019546d0-32c9-487c-a75d-2ae160b6a93a",
      "sysembed_type": "cohere.embed-multilingual-v3",
      "sysembed_text": "## Master Services Agreement",
      "sysembed_location": { "text": { "page": 1, "paragraph": 0 }, "position": { "left": 448, "top": 260, "right": 712, "bottom": 319 }, "timestamp": null, "spreadsheet": null },
      "sysembed_score": 1.0,
      ...
    },
    { ..., "sysembed_score": 0.73336875 }
  ],
  "vector": [ ... ]
}

Search Fields​

FieldTypeRequiredDescription
vectornumber[]YesThe query vector. Missing returns 400 Invalid input. Its length must equal the index's vector dimension, otherwise 400 with a message containing Query vector has invalid dimension: 3. Dimension should be: 1024 (prefixed with Error performing OpenSearch query: failed to create query:) (1024 is the dimension in the tested environment, whose embeddings are of type cohere.embed-multilingual-v3)
repositoryIdstringNoRepository to search (default "default"; the response then echoes "repositoryId": null). An unknown repository returns 404 No such repository: <name>.
embeddingTypestringNoWildcard pattern on the embedding type (default *)
chunkFTSstringNoSpace-separated terms to match in the chunk text. When set, the search is hybrid: full-text matches are merged into the vector ranking (rank fusion). It does not exclude chunks that lack the terms, and sysembed_score becomes a fused rank score instead of a similarity.
querystringNoFilter on the parent document, in the syntax given by syntax (default SELECT * FROM SysContent). This is a filter, not a natural-language query
syntaxstringNo"hxql" (default) or "expr"
limitintegerNoMax embeddings to return (default 10)
offsetintegerNoNumber of embeddings to skip (default 0)
trackTotalCountbooleanNoWhether to compute the total count of embeddings (default false)
trackTotalCountUpTointegerNoMax number of embeddings to count, up to 10000. Larger values (and limit: 0) return 400 ([knn] requires k to be in the range (0, 10000]).

Both endpoints return JSON for Accept: application/vnd.hxcs.v1+json, application/json or */*, or when Accept is omitted. Non-JSON types such as text/plain or application/xml return 406 with an empty body.

Narrow the search with two optional fields:

  • query is an HXQL filter on the parent document (default SELECT * FROM SysContent). It is a filter, not a natural-language question. See Querying with HXQL.
  • chunkFTS holds space-separated terms to match in the chunk text. When set, the search is hybrid: chunks that match the terms are merged into the vector results by rank fusion. Chunks without the terms can still be returned, and scores become fused rank values (around 0.016) rather than similarities.

embeddingType is a wildcard pattern on the embedding type (default *).

POST {{baseUrl}}/api/query/embeddings
Authorization: Bearer {{accessToken}}
Accept: application/vnd.hxcs.v1+json
Content-Type: application/json

{
  "repositoryId": "default",
  "vector": [0.0123, -0.0456, 0.0789, ...],
  "embeddingType": "*",
  "chunkFTS": "agreement",
  "query": "SELECT * FROM SysContent WHERE sys_primaryType = 'SysFile'",
  "limit": 10
}

An embeddingType pattern that matches nothing (e.g. "nomatch*") returns 200 with "count": 0.

Look Up an Embedding by ID​

There are two ways to retrieve embeddings:

  • Vector search (POST /api/query/embeddings) — supply a query vector and get back the most similar chunks ranked by score. This is the primary API for semantic search and RAG workflows. The vector must have the index's dimension (1024 for cohere.embed-multilingual-v3 in the tested environment); any other length returns 400 "Query vector has invalid dimension: N. Dimension should be: 1024".
  • Direct lookup (below) — retrieve a specific embedding by its document ID and embedding ID, when you already know which chunk you want (for example, the sysembed_docId and sysembed_id of a vector-search hit).
GET {{baseUrl}}/api/documents/{docId}/embedding/{embeddingId}
Authorization: Bearer {{accessToken}}
Accept: application/vnd.hxcs.v1+json

Embedding Object​

A real response shape, with the text replaced and the 1024-element vector shortened:

{
  "sysembed_id": "7c1e2a90-1111-5aaa-8bbb-000000000001",
  "sysembed_docId": "3f6d0b4e-2222-4ccc-9ddd-000000000002",
  "sysembed_type": "cohere.embed-multilingual-v3",
  "sysembed_vector": [0.03161621, 0.044891357, -0.022125244, ...],
  "sysembed_text": "## Section heading from the source document",
  "sysembed_blobref": null,
  "sysembed_location": {
    "text": { "page": 1, "paragraph": 0 },
    "position": { "left": 448, "top": 260, "right": 712, "bottom": 319 },
    "timestamp": null,
    "spreadsheet": null
  },
  "sysembed_score": 0.0
}

Response Fields​

FieldTypeDescription
sysembed_idstringUnique identifier for this embedding
sysembed_docIdstringThe document this embedding belongs to
sysembed_textstringText label for the vector, or the original chunk text for text embeddings. Omitted when not set.
sysembed_typestringLabel for the embedding type, usually the model name (e.g. cohere.embed-multilingual-v3). It ensures semantic queries compare vectors of the same type.
sysembed_vectornumber[]The vector as a list of floating point numbers (1024 values for cohere.embed-multilingual-v3)
sysembed_locationobjectWhere in the source document this chunk came from (see Location Types below). Omitted when not set.
sysembed_blobrefstringBlob label for the embedding; returned as null when not set
sysembed_scorenumberSimilarity score. In vector search it ranks the hit (e.g. 1.0 for an identical vector, 0.73 for a close one). With chunkFTS, it is a fused rank score instead. A direct lookup is a filter-only match, so it returns 0.0.

Location Types​

The sysembed_location object always has four keys: text, position, timestamp and spreadsheet. Keys that don't apply are returned as null. More than one key can be populated: a chunk extracted from a page image, for example, has both text (page and paragraph) and position (bounding box), as in the example above.

Content TypeLocation FieldsExample
Text documentstext.page (integer), text.paragraph (integer){"text": {"page": 2, "paragraph": 5}, "position": null, "timestamp": null, "spreadsheet": null}
Images / page regionsposition.left, position.top, position.right, position.bottom (integers){"text": {"page": 1, "paragraph": 0}, "position": {"left": 448, "top": 260, "right": 712, "bottom": 319}, "timestamp": null, "spreadsheet": null}
Spreadsheetsspreadsheet.sheet (string), spreadsheet.row (integer), spreadsheet.column (integer){"spreadsheet": {"sheet": "Sheet1", "row": 3, "column": 1}, ...}
Video/Audiotimestamp.start, timestamp.end (decimal numbers)Seconds within the media file

Lookup Errors​

1. Unknown embedding ID

GET /api/documents/3f6d0b4e-2222-4ccc-9ddd-000000000002/embedding/nope
# 404
{"entity-type":"exception","status":404,"message":"Could not find embedding with id nope for document 3f6d0b4e-2222-4ccc-9ddd-000000000002"}

2. Path-traversal-like ID

GET /api/documents/{docId}/embedding/..%2F..
# 403 text/html — rejected by the gateway before it reaches the API
<html><head><title>403 Forbidden</title></head>...

Error Responses​

StatusDescription
403In the tested environment, the gateway blocks IDs containing encoded traversal sequences such as ..%2F.., with an HTML 403 Forbidden page.
401Not authenticated (empty body)
404Document or embedding not found, e.g. "Could not find embedding with id {embeddingId} for document {docId}". Also returned when the caller has no access to the document.

Use the Results in a RAG Workflow​

RAG Workflow

For Retrieval-Augmented Generation: (1) use POST /api/query/embeddings with a query vector to find the most relevant chunks, (2) use the sysembed_text from each result as context for your LLM, and (3) use sysembed_docId + sysembed_location to cite the source document and page in the response. If you need the full document, use the Download APIs with the sysembed_docId.

Next Steps​