OpenAI's vulnerability pipeline exploited 948 unique Linux root bugs; 20 of 200 early wins used the wrong bug

OpenAI's Cyber team described a five-stage pipeline built on an upcoming cyber model in a Nov 4 post. Scout agents picked 5,034 audit targets, about 2.8 million lines or roughly 7% of the Linux kernel. Analyst agents produced 2,308 candidate findings within 12 hours. Triage agents wrote a proof of concept for each candidate, ran it in a virtual machine with Kernel Address Sanitizer and reproduced 1,724 (about 74.7%) within 30 hours. Exploit agents then worked in virtual machines with SMEP, SMAP, KASLR and HARDENED_USERCOPY turned on. They wrote 291 information-leak proofs of concept and 1,169 working local privilege escalation exploits. After variant analysis, OpenAI wrote 948 distinct patches for the bugs behind those exploits and uses that number as its count of unique privilege escalation bugs. The mid-October run cost $1,213,430 in API credits, and OpenAI says its reported costs are "more an upper bound". Security researchers review each exploit and patch before reporting. OpenAI says it has sent hundreds of reports and patches for critical bugs and is working with Patch the Planet partners to send the rest. The post withholds technical details of the 10 new exploitation techniques it says the agents used, and the model is not released. Why it matters: OpenAI now says that, for the near and medium term, a low-privileged account on a system that is behind on patches "should be considered functionally equivalent to root access for a well-resourced threat actor". The post also shows how a pipeline like this can miscount its own successes. The pipeline counted an exploit as a success whenever a root shell opened. In an early run it reported 200 successes. Human reviewers found that 20 of them got root through a different 0-day bug than the one the agent was assigned. OpenAI fixed it with tighter task rules and an independent check that each exploit used its assigned bug. Its takeaway is to "verify the path to a result in addition to verifying the result itself". The same gap applies to any agent whose success test only looks at the final state.

Try this: Pick one agent workflow whose success check only looks at the end state, such as tests passing or a file existing. Add a second check that the agent got there the assigned way: the diff touches the assigned file, or the trace calls the assigned tool. Run your recent successful runs through that check and count how many fail it.