At Strix, we are building autonomous security agents that can do the work a human pentester does. Getting them there means working the full stack at once: the harness the models run inside, the RL environments they train against, and the models themselves.
A pentest is many things at once. There is reconnaissance, attack surface mapping, probing for specific vulnerability classes, exploit validation, retesting, and writing the final report. Some of this benefits from frontier-level reasoning. Other parts repeat hundreds of times per engagement against narrow problems where a smaller specialized model usually fits better. We think the right setup is to train specialized LLMs for individual vulnerability classes and run them through a shared harness.
In this research blog, we describe our first experiments around training smaller models for specific vulnerability classes inside the Strix workflow. The first model is trained from Qwen 3.5 27B using supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), using RL environments designed around interactive vulnerability discovery and exploit confirmation.
The goal is to make specialized security testing cheaper to run and easier to deploy in customer-controlled environments. Smaller models let us run repeated vulnerability-specific rollouts without the cost profile of frontier models, while also supporting self-hosted, on-prem, and air-gapped deployments where application data cannot leave the customer's infrastructure.
Why train specialized models
Specialized models let us train directly against the Strix harness. Each training trajectory is generated under the same conditions as a production rollout: identical tool-call schema, tool names, argument types, sandbox execution paths, observation format, and termination conditions the model encounters during a real Strix scan. Because vulnerability discovery is a long sequence of tool-mediated decisions across browser, proxy, terminal, Python, reporting, and multi-agent orchestration tools, the on-policy trajectory distribution seen during RL matches the distribution the model is deployed against. Knowledge of the harness is baked directly into the model's parameters rather than reintroduced through prompting at inference time.
Deployment is also a major reason to train smaller specialized models. Strix is self-hosted by dozens of Fortune 500 enterprises, financial institutions, and government environments where scans may involve internal services, source code, credentials, staging systems, runtime logs, and other sensitive context. These users often require on-prem, private, or air-gapped execution, which rules out frontier models that are only accessible through closed external APIs. Smaller specialized models can run on customer-owned hardware, making it practical to ship the capability into those environments while still allowing Strix to run useful vulnerability-specific testing.
The broader thesis is that vulnerability discovery models should be trained with the harness and verification logic they use in production. Different parts of the pentesting workflow can then be handled by different specialized LLMs, all operating through the Strix framework.
Supervised Fine-Tuning
Our first experiment targets XSS vulnerability discovery. Before reinforcement learning, we train the model on 1,400 synthetic multi-step XSS traces derived from real-world HackerOne reports. Each report is converted into a structured scenario containing the target context, injection surface, relevant defenses, expected exploitation path, and final exploitability condition.
Each scenario is then mapped into the Strix harness as a complete trace. Concretely, this means generating Strix-style trajectories that use the same categories of tools the model later sees during RL and deployment: page inspection, request submission, browser observation, payload iteration, execution confirmation, and final reporting.
The purpose of SFT is to give the model the basic shape of the task before RL begins. It teaches tool discipline and the structure of an XSS investigation inside the Strix harness. The main capability gains come from reinforcement learning; SFT provides the starting point that makes long-horizon RL more stable.
Building RL environments for vulnerability exploitation
The core of the training process is reinforcement learning in interactive vulnerability discovery environments. Each rollout gives the model a target application and requires it to discover and verify a vulnerability through the Strix harness. This is different from training on static examples or single-turn classification tasks: the model must inspect the target, form hypotheses, probe the application, observe the result, adapt its strategy, and decide when the evidence is sufficient.
For this first environment, the vulnerability class is XSS. Each rollout spins up a deterministic web application with simulated browser and tool behavior. The environment covers the XSS taxonomy across a wide range of sinks and defenses, including CSP bypass variants, DOMPurify, Trusted Types, and WAF-style filters.
Environment design
A few environment details mattered more than expected. The tasks are interaction-gated: the model cannot receive credit by guessing a vulnerability from the initial prompt. It has to inspect the target, submit inputs, observe browser or application behavior, and only then decide whether it has enough evidence to report. This keeps the training objective close to how Strix evaluates findings in production, and gives the reward function something to latch onto when penalizing premature reporting.
The tasks are also parametric. Server-side filters are randomized on every rollout, which prevents the model from succeeding by memorizing fixed payloads or static target behavior. XSS payloads are particularly easy to overfit to, so the goal is for the model to learn to adapt payload construction to the behavior it actually observes, not to reproduce strings it has seen before.
Safe baselines are equally important. Some targets reflect or transform user input without being exploitable. These force the model to learn a negative decision boundary: suspicious behavior is not enough, and the correct conclusion is sometimes that the target is well-defended. Without these, an XSS model that reports every reflection as a vulnerability would look active during training while being unusable in production.
The reward function is anchored by xss_confirmed, the highest-weighted signal. Credit is awarded when the model produces browser-confirmed JavaScript execution, not merely plausible reflection. Auxiliary rewards preserve structural and reporting behavior from SFT, while negative rewards target two failure modes we care about most: reporting reflected input as confirmed XSS without execution evidence, and submitting conclusions before sufficient probing has occurred.
This reward design makes verification part of the objective rather than a post-processing step. The model is not only trained to find something suspicious; it is trained to prove that the finding is exploitable.
Reward hacking
Security RL environments are particularly sensitive to reward hacking because the training signal is a proxy for real exploitability. If the reward function is too loose, the model can learn behaviors that look useful inside the environment but do not correspond to valid findings in production.
In this setting, the main failure mode is confusing reflection with execution. Reflected input, DOM presence, and plausible payload placement are not the same as browser-confirmed XSS.
The sharpest example of this was the model learning to cheat the xss_confirmed signal directly. Confirmation was supposed to come from the target site executing the payload, but the environment accepted the confirming signal from anywhere, so the model learned it could just emit the confirmation itself rather than get the site to do it. It was satisfying the check instead of satisfying the objective, producing confident findings with nothing behind them. This is the worst possible failure mode for a security tool: a report that looks valid and isn't.
In response, the semi-dense reward function reserves its highest credit for confirmed execution coming from the target itself, and applies harsh negative rewards to any XSS report unsupported by that evidence.
Premature reporting is the second major failure mode. A model can learn to stop too early if the environment does not require sufficient probing before a conclusion. We penalize conclusions submitted before sufficient testing has occurred, and include safe baselines so the model has to learn that some targets are intentionally non-exploitable.
A related lesson is that reward design needs to protect the parts of the trajectory around the final exploit. If the only thing optimized is the terminal xss_confirmed signal, the model can improve at producing confirmations while becoming worse at producing useful traces around them. We use auxiliary rewards to preserve the structural and reporting behavior learned during SFT, so the final output stays useful to Strix instead of only satisfying the environment.
Integration with the Strix harness
Strix is built as a multi-agent pentesting system. A coordinating model handles planning, reconnaissance, scope understanding, and task routing. Specialized LLMs can then be invoked for narrower vulnerability-specific testing loops.
In this setup, strix-mini-xss is called when the orchestrator identifies an XSS-relevant surface. The model receives task context from the orchestrator and performs testing through the Strix harness. The key point is that the model is trained against the same harness it later uses during deployment. This aligns the RL environment, tool interface, verification loop, and production workflow.
This creates a cleaner division of labor. General models can remain responsible for broad planning and coordination, while smaller specialized models handle repeated, vulnerability-specific testing.
The same pattern can extend beyond XSS. A Strix engagement can eventually route work across specialized LLMs for SQL injection, SSRF, authorization issues, business logic flaws, supply chain issues, internal infrastructure testing, exploit validation, and other subtasks. The shared harness provides the operating environment, while each model is trained for the part of the workflow where specialization provides the most leverage.
Evaluation
We evaluated strix-mini-xss on the XSS subset of XBEN, consisting of 23 problems. Each rollout was allowed up to 100 turns. For strix-mini-xss, we report best-of-k results, where a problem is counted as solved if at least one of k independent rollouts succeeds.
The first result we care about is the improvement over the base model. Base Qwen solves 5/23 problems. After post-training with both SFT and RL, strix-mini-xss solves 13/23, an 8-problem absolute improvement on the same evaluation setup and the clearest signal that the model learned useful XSS discovery behavior through training. It also compares favorably to smaller general models: GPT-5.4-mini solves 11/23 at a total cost per task of $1.954, while strix-mini-xss solves 13/23 at $0.280, making it both more capable and substantially cheaper than the general-purpose mini baseline.
We then evaluate how performance scales with repeated rollouts. This matters because the economics of a smaller specialized model are different from a frontier model. In a real engagement, Strix does not need to rely on a single attempt from a specialized model. It can run the model multiple times against the same XSS surface and aggregate successful findings across attempts. When several specialized rollouts are still cheaper than one general-model rollout, best-of-k becomes a practical deployment strategy rather than only a benchmark setting.
For strix-mini-xss, the cost reported at each pass@k is the total cost per task across all k rollouts. The model improves from 13/23 at pass@1 to 14/23 at pass@2, 17/23 at pass@4, and 18/23 at pass@6. GPT-5.4 solves 17/23 at a total cost per task of $1.861. At pass@4, strix-mini-xss reaches the same 17/23 solved count while the total cost per task is $0.481, approximately 3.9x less.
At pass@6, the total cost per task for strix-mini-xss is $0.829 at 18/23, still below the per-task cost of GPT-5.4. Claude Sonnet 4.6 solves 19/23, but at $8.066 per task in this evaluation. The result is not that a specialized 27B model dominates all general models on raw capability, but that post-training moves a smaller model onto a much better cost-performance frontier for this vulnerability class.
XBEN XSS pass rate vs. total cost per task
strix-mini-xss (teal) traces best-of-k from pass@1 to pass@6. Cost aggregates all k rollouts per task.
What's Next
The next step is to train more specialized LLMs for the pentesting workflow and deploy them as sub-agents inside Strix. XSS is the first domain because it has a clear interaction loop and a concrete exploitability signal, but the same approach can be applied to other vulnerability classes and testing tasks.
The long-term goal is a library of specialized security LLMs, each trained against the harness and verification logic it will use in production. Strix can then route work between these models depending on what the orchestrator finds during reconnaissance. A model trained for XSS should not need to be good at every vulnerability class; it should be very good at XSS. The same should hold for models trained for SQL injection, SSRF, authorization flaws, business logic bugs, supply chain issues, internal infrastructure testing, and exploit validation.
We also expect orchestration to become a larger part of the training problem. As the number of specialized models grows, the system needs to decide which model to invoke, when to run additional rollouts, when evidence is strong enough to report, and how to combine findings across multiple models. The broader direction is autonomous pentesting built from specialized LLMs operating through a shared harness, rather than relying on frontier models for every step of the workflow.
Special thanks to @kusonooyasumi for collaborating with us on this research.