Curating Documents

This topic describes the workflow for the Data Curation API: requesting presigned URLs, uploading a document, polling for job completion, and downloading structured results.

You can call the Data Curation API using one of the following methods:

Following the steps in this topic.
Use one of the sample scripts: Python, C#, JavaScript.

Overview

The Data Curation API uses a presigned URL pattern. You request URLs from the API first, then upload the file directly to the object store — the API server never receives the file content.

The following diagram shows the sequence of the curation workflow. The client requests presigned URLs from the Data Curation API, uploads the file directly to storage, polls the API until the job reaches a terminal state, then downloads the results from storage.

Prerequisites

A valid access token with the environment_authorization scope. For more information, see Authentication.
The base URL for your environment. For more information, see Data Curation API Reference.
A file to upload. The API supports more than 600 file formats, up to a maximum size of 5 GB. For more information about limits, see Data Curation API Reference.

Step 1: Request presigned URLs

Send a POST /presign request with the processing options in the request body. If you do not specify any options, the API uses default values. For the full list, see Configuration API Overview.

curl
HTTP

curl -X POST "https://knowledge-enrichment.ai.app.hyland.com/latest/api/data-curation/presign" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "normalization": { "quotations": true, "dashes": true },
    "chunking": { "chunk_size": 1000 },
    "embedding": false,
    "json_schema": false,
    "pii": false
  }'

POST /latest/api/data-curation/presign HTTP/1.1
Host: knowledge-enrichment.ai.app.hyland.com
Authorization: Bearer <access_token>
Content-Type: application/json

{
  "normalization": {
    "quotations": true,
    "dashes": true
  }
  "chunking": {
    "chunk_size": 1000
  },
  "embedding": false,
  "json_schema": false,
  "pii": false
}

The response contains three values you need for the next steps:

{
  "job_id": "API_6e1bb8a0-2bc3-43a2-b3a6-e87975799c8d",
  "put_url": "https://...",
  "get_url": "https://...",
  "options": {
    "normalization": {
      "quotations": true,
      "dashes": true
    },
    "chunking": {
      "chunk_size": 1000
    },
    "embedding": false,
    "json_schema": false,
    "pii": false
  }
}

Field	Type	Description
`job_id`	string	The unique identifier for this job. Use it to poll for status.
`put_url`	string	The presigned URL for uploading the file.
`get_url`	string	The presigned URL for downloading the results after processing completes.
`options`	object	The processing options applied to this job, including any default values.

For details on all processing options, see Configuration API Overview.

caution

The presigned URLs expire after 1 hour, so you must upload the file and download the results within that period. The job record itself expires after 24 hours. After that time, the API returns HTTP 404 with {"message": "Job not found"}.

For background on the presigned URL pattern, search for the article on using presigned URLs in the Amazon S3 User Guide.

Step 2: Upload the file

Send a PUT request to the put_url from the previous step. The file must be sent as binary data with Content-Type: application/octet-stream.

When creating the request to upload a file, also note the following:

The content type header must be application/octet-stream.

curl
HTTP

curl -X PUT "<put_url>" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @document.pdf

PUT <put_url> HTTP/1.1
Content-Type: application/octet-stream

<binary file content>

caution

Do not set any additional headers, such as Authorization, on the upload request. No authorization header is required, because the presigned URL already includes security credentials. Additional headers cause the upload to fail.

Step 3: Poll for job completion

To check the status of the job, use the job_id returned in Step 1 to poll GET /status/{job_id} until the status reaches a terminal state, either Done or Error.

Poll every 5–10 seconds.

curl
HTTP

curl "https://knowledge-enrichment.ai.app.hyland.com/latest/api/data-curation/status/API_6e1bb8a0-2bc3-43a2-b3a6-e87975799c8d" \
  -H "Authorization: Bearer <access_token>"

GET /latest/api/data-curation/status/API_6e1bb8a0-2bc3-43a2-b3a6-e87975799c8d HTTP/1.1
Host: knowledge-enrichment.ai.app.hyland.com
Authorization: Bearer <access_token>

Response:

{
  "jobId": "API_6e1bb8a0-2bc3-43a2-b3a6-e87975799c8d",
  "status": "Done"
}

note

The initial status for a file is Wait For Upload, while the finished status is Done.

Job status values

The Data Curation API reports status through multiple processing stages. Most intermediate states transition automatically — you only need to check for terminal states when polling.

Terminal states (processing complete):

Status	Description
`Done`	Processing is complete. The results are available at `get_url`.
`Error`	Processing failed with an error.

Intermediate states (processing still ongoing):

Status	Description
`Wait For Upload`	The job is created and the file is not yet uploaded.
`Processing`	The file is uploaded and processing is in progress.
`Chunking`	The document is being split into chunks.
`Embedding`	Embeddings are being generated. This stage takes longer for large documents.

Step 4: Download results

Send a GET request to the get_url returned in Step 1 to download the results. No authorization header is required.

curl
HTTP

curl "<get_url>" -o result.json

GET <get_url> HTTP/1.1

If the results are available, a JSON response similar to the following is displayed:

{
  "markdown": {
    "output": "Document Text",
    "chunks_with_embeddings": [
      {
        "chunk": "Chunk Text",
        "embeddings": [
          -0.042955305427312851, 0.077558189630508423, 0.0026660626754164696
        ]
      }
    ],
    "pii_matches": [
      {
        "start": 8339,
        "end": 8348,
        "match_type": "US_BANK_NUMBER",
        "redacted_start": 8292,
        "redacted_end": 8297
      }
    ],
    "pii_match_type_counts": {
      "US_BANK_NUMBER": 1
    }
  },
  "json": {
    "output": {
      "type": "root",
      "children": [
        {
          "type": "paragraph",
          "children": [
            {
              "type": "text",
              "value": "Document Text"
            }
          ]
        }
      ]
    }
  }
}

note

The example response includes extra line breaks for readability.

If processing fails, the get_url response contains an error message instead of structured output. For more information, see Data Curation API errors for details.

Next steps

Adjust processing options. Tune the chunking strategy or the chunk size. For more information, see Configuration API Overview.
Enable PII detection or redaction — For more information, see PII Redaction.
Use a code sample — For samples in Python, C#, and JavaScript, see Code Samples for Python, C#, and JavaScript.
Test interactively — For more information, see Testing Your Integration.

Overview​

Prerequisites​

Step 1: Request presigned URLs​

Step 2: Upload the file​

Step 3: Poll for job completion​

Job status values​

Step 4: Download results​

Next steps​