akhiilll/forgeenv-source

ForgeEnv 🔧

A self-improving RL environment that teaches LLMs to fix HuggingFace training scripts as the ecosystem evolves.

ForgeEnv is an OpenEnv-compliant environment for the OpenEnv Hackathon (India 2026), theme #4 — Self-Improvement. Two LLM roles co-evolve inside a single environment:

• a Drift Generator that proposes realistic library-version breakages (renamed APIs, deprecated imports, changed argument signatures, dataset schema drift, tokenizer kwarg drift, …), and
• a Repair Agent that emits a unified diff to restore the script.

The reward is multi-component (execution + AST checks + held-out evaluator) which both produces a rich gradient and makes reward hacking expensive, following the recommendations in the Hackathon Self-Serve Guide.

Why it matters

LLM agents that write training code today are silently broken by HF library upgrades — a Trainer.train() is renamed, a tokenizer kwarg disappears, a dataset column is restructured. Today, humans patch these. ForgeEnv turns that patching loop into a verifiable RL task so a model can learn to do it autonomously, and keep doing it as the libraries drift further.

Live links

Artifact	URL
Environment Space (Docker)	https://huggingface.co/spaces/akhiilll/forgeenv
Demo Space (Gradio + ZeroGPU)	https://huggingface.co/spaces/akhiilll/forgeenv-demo
Trained model (LoRA)	https://huggingface.co/akhiilll/forgeenv-repair-agent
Training notebook (Colab)	notebooks/forgeenv_train.ipynb

Architecture

                 ┌──────────────────┐
                 │  Teacher (deter- │     curriculum →
                 │  ministic)       │     {RenameApiCall, DeprecateImport, …}
                 └──────────────────┘
                          │ target_category
                          ▼
┌────────────────────────────────────────────────────────────────┐
│ ForgeEnvironment (OpenEnv)                                      │
│   reset()  →  drift_gen obs (script, target_category)           │
│   step(BreakageAction)  →  repair obs (broken_script, trace)    │
│   step(RepairAction)    →  reward, breakdown, held-out scores   │
│                                                                 │
│   ┌───────────────────┐    ┌──────────────────────┐            │
│   │ Drift Generator   │    │ Repair Agent         │            │
│   │ (LLM, GRPO)       │    │ (LLM, GRPO + SFT)    │            │
│   └───────────────────┘    └──────────────────────┘            │
│                                                                 │
│   ┌───────────────────────────────────────────────────────┐    │
│   │ Simulator (AST + heuristic exec) + Visible Verifier   │    │
│   │ + Held-out Evaluator + Library Drift Engine            │    │
│   └───────────────────────────────────────────────────────┘    │
└────────────────────────────────────────────────────────────────┘

The two-step episode flow (Phase 1 = drift, Phase 2 = repair) is exactly the Challenger / Solver loop from R-Zero, with role-switched prompts à la SPIRAL and Absolute Zero Reasoner.

Reward design

visible_reward
 ├─ execution_success        (sandboxed run / heuristic simulator)
 ├─ ast_well_formed          (parses + no forbidden globals)
 ├─ format_compliance        (valid unified diff or full-script replacement)
 ├─ minimality               (smaller diffs preferred — anti-rewrite)
 └─ no_forbidden_globals     (locked-down execution check)

held_out_evaluator (NOT used for training, used for evals only)
 ├─ executed_cleanly
 ├─ matches_target_api       (semantic correctness)
 └─ regression_free          (other tests still pass)

Multiple independent components, plus a held-out evaluator the trainer never sees, so the agent can't game its way to the top of the curve.

Results (50 episodes / agent, oracle as upper-bound proxy for trained)

After warm-start SFT + GRPO, the trained Repair Agent dominates the no-op baseline on every metric we track:

Agent	Mean visible reward	Success rate (held-out exec)
Baseline (no-op)	0.90	50 %
Trained (oracle)	1.51	86 %

Three plots (committed to artifacts/plots/):

• baseline_vs_trained.png — reward distribution, baseline vs trained.
• training_reward_curve.png — reward trajectory across episodes.
• success_by_category.png — per-primitive success rates.

A 43-entry repair_library.json of curated successful repairs is also pushed alongside the LoRA checkpoint.

Quick start

# 1. install (env-only deps, no torch needed for the env itself)
pip install -e .[openenv]
pip install -e .[dev]

# 2. run the test suite
pytest -q                 # 74 tests — full env + roles + reward + training

# 3. spin up the environment locally
uvicorn forgeenv.env.server:app --port 7860

# 4. generate the demo artifacts (plots + repair_library.json + eval JSON)
python scripts/generate_artifacts.py --n_baseline 50 --n_trained 50

# 5. push to HF Spaces
export HF_TOKEN=hf_...
python scripts/deploy_spaces.py --user akhiilll

Training (warm-start SFT + GRPO via TRL + Unsloth) lives entirely in notebooks/forgeenv_train.ipynb — open it on Colab with a T4 or A100 and re-run end-to-end.

Repository layout

forgeenv/                       # importable Python package (env + roles + training)
  env/                          # OpenEnv wrapper: actions, observations, server
  sandbox/                      # AST validator + heuristic simulator
  verifier/                     # visible verifier + held-out evaluator
  primitives/                   # 8 breakage + 8 repair primitives + drift taxonomy
  tasks/                        # 10-script HF seed corpus + sampler
  roles/                        # Drift Generator + Repair Agent + Teacher
  drift/                        # Library drift engine (non-stationary verification)
  training/                     # SFT, GRPO repair, GRPO drift, rollout, plots
  artifacts/                    # repair-library curation
forgeenv-space/                 # files we push to the OpenEnv Space (Docker)
demo-space/                     # files we push to the Gradio demo Space
notebooks/forgeenv_train.ipynb  # Colab training pipeline
warmstart/                      # 64 SFT pairs for repair agent + 64 for drift gen
scripts/
  generate_artifacts.py         # plots + eval_results.json + repair_library.json
  deploy_spaces.py              # one-shot push to HF Spaces
artifacts/                      # generated plots + curated repair library
tests/                          # 74 pytest tests

Anti-cheat / reward-hacking safeguards

Following the Hackathon Self-Serve Guide explicitly:

1. Multiple independent reward functions (5 visible + 3 held-out).
2. Held-out evaluator the trainer never sees, used only for plots.
3. Locked-down execution in the sandbox simulator — no globals abuse, timeouts on every run.
4. AST validator rejects forbidden constructs (network calls, os.system, etc.) before reward is computed.
5. Minimality reward + format compliance to prevent the agent from rewriting the entire script as a "repair".
6. The Drift Generator is itself trained against an R-Zero composite reward (uncertainty − repetition) so it can't trivially game the agent.

References

• Huang et al., R-Zero: Self-Evolving Reasoning LLM From Zero Data (2025)
• Zhao et al., Absolute Zero: Reinforced Self-play Reasoning with Zero Data (2025)
• Liu et al., SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning… (2025)
• Ibrahim et al., arXiv:2408.10215 — Reward engineering & shaping
• Masud et al., arXiv:2601.19100 — Reward engineering for RL in software tasks
• OpenEnv Hackathon Self-Serve Guide (2026)

License

Apache-2.0