Skip to main content

PII Redaction

The Data Curation API can detect or redact personally identifiable information (PII) in your documents before it returns the extracted text. This topic describes when to use each mode, how to enable PII processing, and how to interpret the results.

Prerequisites​

  • A valid access token with the environment_authorization scope. For more information, see Authentication.
  • The base URL for your environment. For more information, see Data Curation API Reference.
  • A file to upload. The API supports more than 600 file formats, up to a maximum size of 5 GB. For more information about limits, see Data Curation API Reference.

When to use PII processing​

Enable PII processing when any of the following applies:

  • You must not store or expose sensitive identifiers, in order to meet obligations under data privacy regulations, such as the General Data Protection Regulation (GDPR), the California Consumer Privacy Act (CCPA), or the Health Insurance Portability and Accountability Act (HIPAA).
  • You want to sanitize documents before you index them in a search engine or an AI knowledge base.
  • You need an audit record of the PII entities found in a document, without removing them.

Enabling PII processing​

Enable PII processing in the pii field of the POST /presign request body. This is the same request that starts the Data Curation API workflow. For more information, see Step 1 in Curating Documents.

Set pii to one of the following values:

ValueBehavior
false (default)PII processing is disabled. The results contain no PII data.
"detection" or { "mode": "detection" }PII entities are detected and reported. The original text is not modified.
"redaction" or { "mode": "redaction" }PII entities are detected and the extracted markdown is redacted. Detected spans are replaced with [PII].
{ "mode": "redaction", "entity_redaction": true }Redaction mode with named entity recognition (NER) enabled for the PERSON, LOCATION, ORG, and AFFILIATION types.

Detection mode​

Detection locates PII and reports its position in the original text without modifying the output. You can use either the string shorthand or the object syntax in the POST /presign request body:

{
  "pii": "detection"
}

Or equivalently:

{
  "pii": {
    "mode": "detection"
  }
}

Redaction mode​

Redaction replaces PII entities in markdown.output with the placeholder [PII]. The positions in pii_matches refer both to the original text, in the start and end fields, and to the redacted text, in the redacted_start and redacted_end fields.

Include the following in the POST /presign request body:

{
  "pii": "redaction"
}

To enable entity redaction, which adds NER-based detection for person names, locations, organizations, and affiliations, use the object syntax:

{
  "pii": {
    "mode": "redaction",
    "entity_redaction": true
  }
}

PII types detected​

The API detects PII in the following categories.

Pattern-based detection​

Pattern-based detection is deterministic and uses rules and regular expressions. It covers the following types:

  • ADDRESS
  • EMAIL
  • PHONE
  • SSN
  • CREDIT_CARD
  • CVV
  • CREDITCARDISSUER
  • US_BANK_NUMBER
  • IBAN_CODE
  • IP_ADDRESS
  • URL
  • ZIPCODE
  • DATEOFBIRTH
  • GENDER
  • PASSPORT
  • MEDICAL_LICENSE
  • PREFIX
  • CRYPTO
  • BITCOINADDRESS

NER-based detection​

NER-based detection uses a machine learning model and runs only when entity_redaction is set to true. It covers the following types:

  • PERSON
  • LOCATION
  • ORG
  • AFFILIATION

Password detection​

Password detection runs automatically whenever PII processing is enabled, and you cannot disable it independently. Detected passwords are reported with a match_type value that identifies the pattern, such as password: base64_token, password: json_field, password: key_value, password: url_password, password: jwt_token, or password: high_entropy.

For descriptions of each type, see PII Handling Reference.

Reading PII results​

When PII processing is enabled, the response includes the pii_matches and pii_match_type_counts fields in the markdown object:

{
  "markdown": {
    "output": "The patient was admitted on [PII] by Dr. [PII].",
    "pii_matches": [
      {
        "start": 390,
        "end": 401,
        "match_type": "DATEOFBIRTH",
        "redacted_start": 24,
        "redacted_end": 29
      },
      {
        "start": 416,
        "end": 419,
        "match_type": "PREFIX",
        "redacted_start": 34,
        "redacted_end": 39
      }
    ],
    "pii_match_type_counts": {
      "DATEOFBIRTH": 1,
      "PREFIX": 1
    }
  }
}

pii_matches fields​

The pii_matches array contains one entry for each detected PII span:

FieldTypeRequiredDescription
startintegerYesThe start character position in the original text.
endintegerYesThe end character position in the original text.
match_typestringYesThe PII category, for example PERSON, EMAIL, SSN, or password: jwt_token.
redacted_startinteger | nullNoThe start character position in the redacted markdown.output. Present in redaction mode only, and null in detection mode.
redacted_endinteger | nullNoThe end character position in the redacted markdown.output. Present in redaction mode only, and null in detection mode.

In detection mode, redacted_start and redacted_end are null because the text is not modified.

pii_match_type_counts​

This field maps each match_type to the number of times it was detected. Use it for a summary of the results without iterating over pii_matches.

Limitations​

Note the following limitations of PII processing:

  • Detection accuracy depends on the quality and the language of the document. Scanned images with poor optical character recognition (OCR) quality reduce detection accuracy.
  • NER-based detection produces false positives for common names that also match locations or organizations.
  • PII processing increases job processing time.
  • Detection and redaction operate on the extracted text only. Metadata fields such as the file name and the author are not scanned.
  • Redaction uses the fixed placeholder token [PII]. You cannot customize the redaction string.
  • Password detection is always enabled when PII processing is enabled, and you cannot disable it independently.
  • When a table column has a sensitive header, such as birthdate, ssn, credit card, or address, the API redacts the entire value of every cell in that column.

For the complete reference — including all PII types and output structure — see PII Handling Reference.

Next steps​