lemonade-sdk/pii_policy_mmbert32k-pii-detector-merged-onnx

PII Router Policy - mmBERT32k PII Detector (ONNX)

A Lemonade collection.router policy JSON: routes any prompt containing PII to a local model, and everything else to a cloud model, using llm-semantic-router/mmbert32k-pii-detector-merged's ONNX export as the PII classifier.

This repo holds only the policy config (+ the manifest the classifier needs - see step 1 below) - not model weights. It's meant to be pulled directly into a running Lemonade server.

What this policy does

• Runs every prompt through the PII classifier first (34 non-"O" BIO labels across 17 entity types: AGE, CREDIT_CARD, DATE_TIME, DOMAIN_NAME, EMAIL_ADDRESS, GPE, IBAN_CODE, IP_ADDRESS, NRP, ORGANIZATION, PERSON, PHONE_NUMBER, STREET_ADDRESS, TITLE, US_DRIVER_LICENSE, US_SSN, ZIP_CODE, each with B-/I- variants).
• If any label crosses min_score: 0.5, the prompt routes to the local candidate (Qwen3.5-0.8B-GGUF by default).
• Otherwise it routes to the cloud candidate (fireworks.kimi-k2p6 by default).

On the Nemotron-PII benchmark (20,000 PII-bearing prompts, 20,001 total), this policy scored a 0.245% leak rate (49 leaks). All 49 misses were categories outside this classifier's 17-entity label set (demographic/soft-PII attributes - gender, religion, political view, ethnicity, employment/education status - that this checkpoint was never trained to recognize), not scoring/threshold errors. If your traffic includes those categories, consider lemonade-sdk/pii_policy_openmed-privacy-filter-multilingual-v2-onnx instead, which covers 54 entity types (including those) and scored a 0% leak rate on the same benchmark - at the cost of a much heavier classifier (5.6GB vs. this one's much smaller ONNX export).

Usage

1. Register the classifier. Lemonade's onnxruntime backend (ort-server) needs a manifest.json next to model.onnx to know this is a token-classification model - one score per token, not one per prompt - rather than the default it assumes otherwise (single-label text-classification, one score for the whole input, like a sentiment model). Without the right manifest, ort-server either errors on the shape mismatch or silently misreads the output. Unlike the classifier above (lemonade-sdk/openmed-privacy-filter-multilingual-v2-onnx), this one's ONNX export lives in the original author's repo, llm-semantic-router/mmbert32k-pii-detector-merged (under an onnx/ subfolder), which doesn't ship a manifest - ort-server's manifest is a Lemonade-specific artifact, not something HF repos ship by default - so this repo provides one, which needs to be placed alongside the ONNX files after registering the model:

# Register - downloads the full repo (includes the onnx/ subfolder)
curl -X POST http://localhost:13305/v1/pull -H "Content-Type: application/json" -d '{
  "model_name": "user.mmbert32k-pii-onnx",
  "checkpoint": "llm-semantic-router/mmbert32k-pii-detector-merged",
  "recipe": "onnxruntime"
}'

Then copy this repo's manifest.json into the resolved local cache snapshot's onnx/ subfolder (the directory containing model.onnx, tokenizer.json, config.json after the pull above) - typically:

~/.cache/huggingface/hub/models--llm-semantic-router--mmbert32k-pii-detector-merged/snapshots/<hash>/onnx/manifest.json

2. Register this policy:

hf download lemonade-sdk/pii_policy_mmbert32k-pii-detector-merged-onnx --local-dir .
curl -X POST http://localhost:13305/v1/pull -H "Content-Type: application/json" \
  --data-binary @pii_policy_mmbert32k-pii-detector-merged-onnx.json

3. Use it - send chat completions to "model": "user.PII-ONNX-Classifier-Router" and the server will route each request per the policy above.

Why the manifest keeps all 35 raw labels distinct

The manifest deliberately keeps B-/I- prefixed labels separate (B-EMAIL_ADDRESS, I-EMAIL_ADDRESS, ...) rather than collapsing them into one bare entity name. Per-label scores land in a map keyed by label string in ort-server - collapsing two label indices onto the same key would let the later-processed one silently overwrite the other's score instead of combining them. That's why this policy's match rule checks "does the B- OR the I- variant cross threshold" per entity type (34 leaf conditions) rather than 17 collapsed ones.

Adapting this policy

routing.candidates (local/cloud model names) and min_score are starting points, not fixed requirements - point them at whatever models you have registered, or tighten/loosen the threshold for your own precision/recall tradeoff.

For building a routing policy from scratch, or a more sophisticated one (multiple classifiers, LLM-based routing, custom match logic), see the lemonade-router-builder skill - it turns a natural-language description of routing intent into a valid collection.router policy JSON.

License

Apache 2.0.