ML Tooling / 2026
Agent evaluation environments
Testing how agents investigate and reason.
I built OpenEnv investigation environments with procedural cases, MCP tools, evolving state, and adversarial tests.
Hackathon projects
Follow the investigation.
Walk through an investigation and see why evidence matters to evaluation. Click a step, connection, or trace to inspect it.
Start the walkthrough, or inspect any step before running it.
Simplified from the project architecture. Runs locally; no live models, credentials, or external services.
The engineering
Inside
the decisions.
Evaluate the investigation, including the route to an answer.
Evaluating an agent requires more than checking its final answer. The environment must test how it gathers evidence, sequences actions, and responds as the situation changes.
01The design decision
+
Represent scenarios as evolving state with explicit tools and rewards. Test for shortcuts through adversarial cases.
02The trade-off
+
A simulated reward is an approximation of good behaviour. Procedural variation and adversarial testing help expose weaknesses, but cannot establish real-world security or compliance performance.
03Building a stateful investigation
+
Built a compliance investigation environment with MCP tools, scenario state, procedural generation, and a multi-component reward.
04Testing shortcuts in the reward
+
Added adversarial tests targeting reward shortcuts and insufficient evidence gathering.
05Advancing an adaptive adversary
+
Built WarRoom with an adversary that advances during an investigation and adapts to containment actions.
↳Read the system flow
+
Agent evaluation environments · Public explanation of the implementation boundaries.
- ObserveInspect the system and the current case state.
- Gather evidenceUse investigation tools and cross-reference the findings.
- ActClassify and remediate, or contain the simulated incident.
- EvaluateScore the actions and evidence, not only the final answer.
↳Compliance Auditor
+
A procedural compliance investigation with MCP tools and a multi-component reward. Tests challenge skipped investigation, invented findings, incorrect classification, and missing remediation.
Environment and reward logic ↗↳Adversarial checks
+
Reward tests ask whether an agent can score well by guessing, flooding the evaluator with findings, or skipping evidence. Procedural cases vary scenarios so repeating one memorized action sequence is insufficient.
Reward shortcut tests ↗↳WarRoom
+
A simulated security operations environment. The adversary advances during the investigation and responds to containment, so the sequence of actions changes the case.
WarRoom implementation ↗↳Validation scope
+
Public projects built for the Meta PyTorch OpenEnv Hackathon 2026. No placement, benchmark score, or real-world security and compliance performance is claimed.
Hackathon projects
Inspect
the work.
Public simulation environments built for the Meta PyTorch OpenEnv Hackathon 2026. The repositories contain environment state, investigation tools, reward logic, and adversarial tests.
Next project: Repository intelligence ↗Source material
Compliance environmentWarRoom environment