PII Handling
The Data Curation API includes built-in Personally Identifiable Information (PII) processing capabilities to help organizations maintain compliance and data privacy when processing unstructured content.
To enable PII processing, include the pii field in the POST /presign request body. See Configuration API Overview for more information.
Processing modes
PII processing supports the following modes:
| Mode | Description |
|---|---|
"detection" | Identifies PII entities and reports their positions in the original text. The extracted text is not modified. |
"redaction" | Masks detected PII and removes it from the output content. Replaces all detected PII entities in markdown.output with the [PII] placeholder. Each match in pii_matches includes both original and redacted text positions. |
When enabled, masking is performed prior to any enabled chunking or embedding steps.
The optional entity_redaction: true flag enables Named Entity Recognition (NER)-based entity detection (names, organizations, locations) on top of the pattern-based detection. NER results are included in both detection and redaction modes.
Supported PII types
The Data Curation API supports the following categories of PII detection:
| Detection Type | Description |
|---|---|
| Pattern-Based | Uses deterministic rules and regular expressions. |
| Named Entity Recognition (NER) | Uses a machine-learning model to identify entities in text. |
Pattern-Based PII Types
The following types are detected using deterministic pattern matching and are included whenever PII processing is enabled.
| PII Type | Description |
|---|---|
Addresses (ADDRESS) | Street addresses including a street number; street name; and a street designator, such as Street, Avenue, Boulevard, Lane, Drive, Court, Plaza, Terrace, Place, Trail. Optional unit identifiers, such as Apt, Suite, Unit, Floor, and Building, are also recognized. Example: 123 Main Street Apt 4B. |
Email Addresses (EMAIL) | Standard email addresses in user@domain.tld format. Supports alphanumeric characters, dots, hyphens, underscores, and plus signs in the local part. |
Phone Numbers (PHONE) | US phone numbers in 10-digit formats with optional country code prefix (+1). Recognized separators include hyphens, dots, spaces, and parentheses around the area code. Examples: 555-123-4567, (555) 123.4567, +1 5551234567. |
Social Security Numbers (SSN) | US Social Security Numbers in 9-digit format. Supports hyphen-separated (for example, 123-45-6789), space-separated (for example, 123 45 6789), and consecutive digit (for example, 123456789) formats. Separators must be consistent within a single number. |
Credit Card Numbers (CREDIT_CARD) | Payment card numbers in 13-, 15-, 16-, or 19-digit formats. Supports Visa, Mastercard, American Express, Discover, JCB, Diners Club, and UnionPay card shapes. Recognized formats include consecutive digits (for example, 4111111111111111), hyphen-separated groups (for example, 4111-1111-1111-1111), space-separated groups (for example, 4111 1111 1111 1111), and the 4-6-5 grouping used by American Express (for example, 3782-822463-10005). |
CVV Numbers (CVV) | Card verification codes (3–4 digits) detected when preceded by a keyword, such as CVV, CVC, CSC, CID, security code, or card code. Also recognized in JSON fields (for example, "cvv": "123") and query string parameters (for example, ?cvv=123). |
Credit Card Issuers (CREDITCARDISSUER) | Explicit mentions of card network names: Visa, MasterCard, American Express, Amex, Discover, JCB, Diners Club, UnionPay, and Maestro. |
US Bank Account Numbers (US_BANK_NUMBER) | Numeric sequences of 8-, 10-, 11-, 12-, 14-, or 17-consecutive digits surrounded by whitespace or string boundaries. Digit lengths that overlap with SSNs (9) or credit cards (13, 15, 16, 19) are excluded to prevent conflicts. |
IBAN Codes (IBAN_CODE) | International Bank Account Numbers (IBANs) beginning with a two-letter country code, followed by two check digits and up to 30 alphanumeric characters (for example, GB29NWBK60161331926819). |
Passwords / Secrets (password: <pattern>) | Passwords, API keys, tokens, and secrets detected through multiple methods: key-value pairs with keywords (for example, password=, api_key:, or token=), JSON fields containing credential keywords, URLs with embedded credentials (for example, ://user:pass@host), JWT tokens (for example, eyJ...), Base64-encoded tokens, and high-entropy strings that exhibit characteristics of generated secrets. The match_type value includes the detection pattern: password: key_value, password: json_field, password: base64_token, password: url_password, password: jwt_token, or password: high_entropy. |
IP Addresses (IP_ADDRESS) | IPv4 addresses in dotted-decimal notation (for example, 192.168.1.1). Each octet is matched as 1–3 digits separated by dots. |
URLs (URL) | Web addresses beginning with http://, https://, or www. followed by a domain name with a valid TLD. |
Zip Codes (ZIPCODE) | US ZIP codes in 5-digit (for example, 12345) or ZIP+4 (for example, 12345-6789) format. Codes with prefixes that are unassigned in the US postal system are excluded to reduce false positives. |
Date of Birth (DATEOFBIRTH) | Dates in MM/DD/YYYY or MM-DD-YYYY format, detected only when preceded by a contextual keyword, such as birth, born, dob, date of birth, or birthdate. |
Gender Information (GENDER) | Gender identity terms, including but not limited to: transgender, cisgender, non-binary, genderqueer, genderfluid, two-spirit, pangender, and related compound forms. Simple terms like "male" or "female" in isolation are not matched. |
Passport IDs (PASSPORT) | US passport numbers matching the format of one letter followed by 7 digits (for example, A1234567). The letter range excludes certain characters (Q, X, Z) per US passport conventions. |
Medical License IDs (MEDICAL_LICENSE) | Medical license identifiers in the format of one letter followed by 5–7 digits (for example, D123456) or two digits, a hyphen, and five digits (for example, 12-34567). |
Prefixes and Titles (PREFIX) | Personal titles and honorifics, including common forms (Mr., Mrs., Ms., Dr., Prof., Rev.), military ranks (Capt., Col., Gen., Lt., Maj., Sgt.), nobility titles (Baron, Duke, Earl, Marquis), religious titles (Rabbi, Imam, Cardinal, Bishop, Archbishop, Pope), and civic titles (President, Chancellor, Dean, Judge, Justice, Ambassador). |
Crypto Wallet Addresses (CRYPTO) | Ethereum addresses in 0x prefix with 40 hex characters, and general cryptocurrency addresses starting with bc1, 1, or 3 followed by 25–39 Base58 characters. |
Bitcoin Addresses (BITCOINADDRESS) | Bitcoin-specific addresses in Legacy format (starting with 1 or 3, 25–34 Base58 characters) and Bech32/SegWit format (starting with bc1, 39–59 lowercase alphanumeric characters). |
Named Entity Recognition (NER) Types
The following types are detected using a Named Entity Recognition (NER) model. NER-based detection is controlled separately by the entity_redaction option and is disabled by default, enabled when entity_redaction: true is set.
| PII Type | Description |
|---|---|
Person Names (PERSON) | Full or partial names of individuals. |
Locations (LOCATION) | Geographic locations (cities, states, countries) — spaCy label GPE mapped to LOCATION. |
Organization Names (ORG) | Names of companies, agencies, or institutions. |
Affiliations (AFFILIATION) | National, religious, or political group identifiers — spaCy label NORP mapped to AFFILIATION. |
NER-based detection is not as deterministic as pattern-based detection and may produce false positives. Review NER output carefully before relying on it in production workflows.
Output structure
When PII processing is enabled, the pipeline result includes two additional fields in the markdown object: pii_matches and pii_match_type_counts. If no PII is detected, these fields are omitted.
pii_matches
Array of match objects describing each detected PII occurrence.
| Field | Type | Description |
|---|---|---|
start | integer | Start character offset of the match in the original text. |
end | integer | End character offset of the match in the original text. |
match_type | string | The PII category detected (for example, "SSN", "EMAIL", "PERSON", "ORG"). |
redacted_start | integer | null | Start offset in the redacted text. Present only in redaction mode. |
redacted_end | integer | null | End offset in the redacted text. Present only in redaction mode. |
pii_match_type_counts
Object where each key is a match_type string and the value is the count of detections of that type. Provides a quick summary without iterating through the full pii_matches array.
Detection Mode Output
In detection mode, the markdown.output field contains the original, unmodified text. The pii_matches array reports where PII was found, but no masking is applied.
{
"markdown": {
"output": "Contact John Smith at john.smith@example.com or 555-123-4567."
},
"pii_matches": [
{
"start": 8,
"end": 18,
"match_type": "PERSON",
"redacted_start": null,
"redacted_end": null
},
{
"start": 22,
"end": 44,
"match_type": "EMAIL",
"redacted_start": null,
"redacted_end": null
},
{
"start": 48,
"end": 60,
"match_type": "PHONE",
"redacted_start": null,
"redacted_end": null
}
],
"pii_match_type_counts": {
"PERSON": 1,
"EMAIL": 1,
"PHONE": 1
}
}
Redaction Mode Output
In redaction mode, each detected PII occurrence in markdown.output is replaced with the mask [PII]. The pii_matches array includes redacted_start and redacted_end fields indicating where each mask appears in the masked text.
Masking is performed prior to any enabled chunking or embedding steps.
{
"markdown": {
"output": "Contact [PII] at [PII] or [PII]."
},
"pii_matches": [
{
"start": 8,
"end": 18,
"match_type": "PERSON",
"redacted_start": 8,
"redacted_end": 13
},
{
"start": 22,
"end": 44,
"match_type": "EMAIL",
"redacted_start": 17,
"redacted_end": 22
},
{
"start": 48,
"end": 60,
"match_type": "PHONE",
"redacted_start": 26,
"redacted_end": 31
}
],
"pii_match_type_counts": {
"PERSON": 1,
"EMAIL": 1,
"PHONE": 1
}
}
The pii_matches and pii_match_type_counts fields are only present when at least one PII match is found. If no PII is detected, the output contains only the standard fields (for example, markdown, json).
Limitations
When using Data Curation's PII handling capabilities, also consider the following limitations:
- PII detection and redaction accuracy depends on the quality of the extracted text from the source document.
- Redaction masks PII in the output; it does not modify the original uploaded file.