PII detection has two well-understood failure modes. High recall systems catch everything that might be personal data, including a large volume of false positives, which produces alert volumes that either paralyze the security team or get tuned away entirely. High precision systems are more targeted and generate manageable alert volumes, but they miss the edge cases, the contextually sensitive content that does not pattern-match to a structured data format.
In email DLP and file transfer scanning, this tradeoff is painful but manageable. Incident volumes are bounded and the data being inspected has enough structure to make pattern matching tractable. In AI prompt inspection, the problem is qualitatively different. Prompt content is freeform natural language. Employees do not write in their prompts "I am now going to include a customer's email address." They paste a ticket thread that contains the email address as a subordinate element of a customer support context. The detection system needs to find that element without treating every prompt that mentions a person's name as a violation.
The Three Detection Layers and Where Each Fails
A production PII detection system for AI prompts needs at least three detection layers operating together, because no single layer handles the full distribution of sensitive content well.
Regex-based pattern matching handles structured PII formats well and poorly handles everything else. Email addresses, US phone numbers in standard formats, Social Security Number patterns, payment card numbers (with Luhn validation), and common API key formats (AWS Access Key IDs, GitHub PATs, Stripe keys) all have structure that makes regex both fast and accurate. False positive rates for well-constructed regexes on these formats are low. The failure mode is structured data written in non-standard formats, international phone and ID formats, and all unstructured content that is contextually sensitive but structurally opaque.
Named entity recognition (NER) using trained models handles the unstructured content that regex misses. A model trained on personal names, organization names, locations, and contextual person references can identify "John Smith called at 2pm about his account renewal" as containing a person reference even without any structured data field. NER recall is high for the categories it is trained on. Its precision depends heavily on domain and context. General-purpose NER models will flag mentions of historical figures, fictional characters, and public figures in contexts where there is no data risk. Domain-specific calibration matters substantially.
Context-aware ML classification is the third layer, and it is the one that handles the genuinely hard cases: a customer support ticket fragment that contains personal data embedded in conversational prose, a sales call note where the sensitive content is the context of the conversation rather than any structured data field, or a financial document summary where the sensitivity is the combination of figures rather than any single element.
The Recall-Precision Curve Is Not a Stable Tradeoff
In classification system design, the recall-precision tradeoff is usually presented as a stable curve with a threshold parameter that lets you choose your operating point. You move the threshold to increase recall at the cost of precision, or vice versa. At any given threshold setting, you have a predictable false positive rate and a predictable false negative rate.
In production AI prompt inspection, this assumption breaks down in two ways.
The input distribution shifts continuously. As new AI tools are adopted, as employee workflows evolve, and as the population of employees using AI tools grows, the character of prompt content changes. A model calibrated on support ticket content from early 2024 may perform differently on the mixture of content types flowing through a 2025 deployment that includes more developer and finance workflows. Threshold settings that produce acceptable false positive rates on the calibration data may not generalize.
The cost asymmetry between false positives and false negatives is not fixed. In some deployment contexts, a high false positive rate is acceptable because the cost of a blocked prompt is low. The employee redacts the flagged content and resubmits. In other contexts, high false positive rates create enough friction to drive employees toward unmonitored channels. At a certain false positive rate, the detection system becomes counterproductive: it reduces safety for the prompts it inspects while pushing higher volumes of traffic to unmonitored endpoints.
How We Think About Calibration Across Data Types
Different data categories warrant different detection postures, and the right operating point varies by data type rather than by a single global threshold.
Structured credentials (API keys, tokens, connection strings) should be detected at high recall with low tolerance for false negatives. A missed API key in a prompt is a high-severity incident. The cost of a false positive (flagging a clearly invalidated key or a test credential) is low compared to the cost of missing a live secret. We operate this category at high sensitivity with human review as the tier-two classification step for edge cases.
Customer PII categories (email, phone, name, account ID) warrant moderate recall with precision calibrated to the organizational context. The right detection threshold for a company that handles healthcare data is different from the right threshold for a company whose customer data is entirely B2B contact information. The regulatory exposure differs, the harm model differs, and the acceptable false positive rate differs accordingly. Deploying the same threshold across both contexts is a configuration choice that misserves at least one of them.
Unstructured sensitive content, internal document fragments, financial data in prose, strategic plan language: this category is the hardest to calibrate and the most context-dependent. We use ML classification as the primary detector, with confidence thresholds set conservatively and a logging-first policy for medium-confidence detections. High-confidence detections trigger block or redact actions. Medium-confidence detections generate audit log entries for periodic human review rather than immediate blocking. This preserves the high-sensitivity detection needed for real incidents while avoiding the false positive rate that would come from blocking on every medium-confidence match.
Contextual PII: The Category That Pattern Matching Cannot Reach
One category deserves specific attention because it is where pure pattern-matching systems have a structural blind spot: contextual PII that is sensitive not because of what it is but because of what it reveals in combination.
A prompt that says "I have a customer who was diagnosed with type 2 diabetes and is asking whether our insurance plan covers continuous glucose monitors" contains no email address, no phone number, no SSN. It contains a health condition and an implied association between a customer and a specific diagnosis. Under HIPAA, if the company operates in a covered entity context, this prompt contains protected health information. A regex-based PII scanner returns no matches. A basic NER system may flag "type 2 diabetes" as a medical entity but not classify the overall prompt as PHI.
We are not claiming that we have solved contextual PHI detection in all its variations. What we can say is that the ML classification layer is specifically calibrated to recognize the patterns of healthcare-adjacent and contextually sensitive content that appear most commonly in the B2B SaaS customer support workflows where this category shows up most often. The recall on pure structured PII is better than the recall on contextual PHI. That is honest about where the current technology sits.
Alert Volume and the Path to Actionable Operations
A detection system that generates 500 alerts per day for a 100-person company is not producing security operations value. It is producing a triage backlog that either consumes the security team entirely or gets resolved by lowering thresholds until the alert volume is manageable, which is just a manual way of increasing the false negative rate.
The operational model we use separates detection events into three tiers: automatic action (high-confidence, high-severity: block or redact in real time), audit log (medium-confidence or lower-severity: logged for periodic review), and dismiss (low-confidence matches that are informational only). Security teams see meaningful alerts for the first tier, can query the second tier on a schedule that matches their operational capacity, and do not see the third tier at all unless they are tuning detection rules.
The tier boundaries need to be calibrated per organization and per data category, and they need to be revisited periodically as the input distribution shifts. What was appropriate for 50 employees generating 200 prompts per day is not appropriate for 200 employees generating 2,000 prompts per day. The detection system needs to scale both in capacity and in how thresholds and tier boundaries are set.
Why We Do Not Claim Global Precision and Recall Numbers
Vendors in the DLP space commonly publish precision and recall numbers on benchmark datasets. Those numbers are real but they are not transferable to your deployment. The distribution of sensitive content in your employees' AI prompts is specific to your industry, your workflows, your data classification posture, and the AI tools your team uses. A benchmark dataset built from generic web text does not predict performance on a mix of healthcare support tickets and developer debugging sessions.
We share per-category performance data with customers based on their actual traffic samples during the onboarding process. That gives a realistic picture of how the detection system performs on the specific content types flowing in their environment, rather than on a public benchmark that may not resemble their data at all. Calibration is an ongoing process, not a one-time deployment decision.