AI & Agents

Meta Has Put JiTTests Into Production to Catch Privacy Bugs Before They Ship

Leave a comment

Meta has moved Catching Just-in-Time Tests, or JiTTests, from a research prototype into production, MarkTechPost reports, and Meta’s own engineering blog fills in the results. These are LLM-written tests meant to fail when a risky code change introduces a bug, so the failure is executable evidence before the diff lands. That is the opposite of a typical hardening test, which is supposed to pass when you write it and then guard against later regressions.

Meta says the system caught 74 serious privacy bugs at the diff stage in the first half of 2026, and that privacy SEV occurrence was down 20% in that window. Engineers who engaged with the reports put the true-positive rate at 60%, which is self-reported. The earlier FSE 2026 paper on the initial deployment was more modest: only 8 of 41 reported catches were confirmed, about 19.5%. The production pipeline runs overnight on spare capacity, targets risky diffs, and uses both rule-based and LLM judges before a human is looped in. The tool is internal. There is no public model or API to try.

My take

A test that fails on purpose, on the change in front of you, is a better artifact than another review comment that may or may not be right. The early true-positive rate was not great, and the later numbers are still Meta’s own, but catching privacy bugs before they ship is exactly the kind of place I’d want this to work.

God Bless, and Happy hacking!

References & Further Reading

Subscribe (RSS) Follow AI & Agents via RSS

Leave a note

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.