Why we ran this
Every AI integration is a new way out for your data. An invoice summary, a support reply or a CRM export can carry names, IBANs and phone numbers to a model the company does not run. Penweir sits in that path and holds those values back.
For a gateway, three properties matter, in this order. It must never let a protected value through. It must not make every request noticeably slower. And it must keep working when the providers behind it do not.
A functional test shows that a filter works on one request. It does not show what happens with hundreds of concurrent clients, conversations of twenty turns, tool results full of customer fields and providers that time out. That is where shortcuts hide: a detector skipped under load, a token restored into the wrong answer, a fallback that resends the unfiltered original. So we pushed Penweir until it broke, on purpose, and watched what broke first.
What we did
We sent about 123,600 requests through ten isolated Penweir instances. The workloads grew from a three-step development pipeline to sixteen pipelines of 30–40 steps with 64 KB messages, with 30% to 70% of the requests streamed. From the second level on, the simulated providers returned server errors, rate limits, timeouts, unreadable answers and card numbers inside answers.
At the provider side sat an oracle. It recorded every raw sensitive value it received, including on fallback attempts to a second, third or fourth provider. A load generator checked every answer: was each tokenized value restored exactly, and was every masked, redacted or hashed value absent?
No real language model and no real personal data were used. Test values were synthetic but valid wherever a checksum exists.
What we found
| Question | Result |
|---|---|
| Did a protected value reach a provider? | No. 54,323 provider calls inspected, 0 leaks. |
| Were tokens restored exactly? | Yes. 43,966 streamed requests, tokens split across uneven chunks, 0 wrong restorations. |
| What does Penweir add? | 0.4 ms at the median and 1.0 ms at p95 for a 4 KB request. The target was 25 ms. |
| What happens when providers fail? | At 2–3% provider faults, clients saw 0% failures. 1,541 requests moved to the next provider. |
| What happens when the gateway is overloaded? | It fails closed. Requests ended in an error. None passed through unfiltered. |
Memory stayed between 116 and 206 MB at every load, with no growth over a run.
What it does not prove
This was one shared laptop, so absolute throughput is pessimistic. The providers were simulated, with uniform latency and independent faults, where real providers fail in bursts. The oracle checks literal values, so it would not catch a value damaged by a partial transformation. We did not measure how well detectors find identifiers in realistic free text, and we did not test the Postgres backend. Each level ran once, so there are no confidence intervals.
Penweir detects structured identifiers. It does not guarantee detection of names in free text. The full list of limits is in section 9 of the report.
What we are fixing
The test also found where one process runs out of room. From about 250 to 500 concurrent clients, requests began to fail with internal errors, caused by a cache stampede on the key lookup. Past that point some requests left no audit event. None of this touches the correctness of protection, but for a product whose audit log is a compliance feature, it comes first.
- A stampede-proof key cache and an explicitly sized connection pool.
- An audit event for every request with a valid key, including internal errors.
- Fast refusals with
503andRetry-After, instead of queues of up to 30 seconds. - Several worker processes per instance.
The harness, per-level reports and raw metrics contain no request or answer text, only counts, timings and status codes.