RSIBench

A modular framework for RSI research

Self-improvement needs an experimental control.

RSIBench asks whether a system can improve itself when only one part of it is allowed to change, and the rest is held still.

See the method Open RSIBench-Data

A result becomes useful when its comparison survives.

RSIBench is built so that a difference in outcome has exactly one possible cause.

A cloud of runs collapses into two measured columns that reveal one attributable difference under a fixed contract.

The platform is built on three constraints.

  1. 01

    One interface for every component.

    Each component declares the same contract: its input, its output, its configuration, the artifact it produces, the resources it consumes, and how it is scored. A method can replace one component without rebuilding the training and evaluation system around it.

    One component slot with six labelled contract edges, input and output at the sides, configuration and artifact above, cost and evaluation below, and two implementations beneath it, method a and method b, with different internals and the same notched mating profile.

  2. 02

    The evaluator cannot be read.

    A system submits a model or an agent and receives a controlled result. Hidden tasks, reference answers and scoring code stay inside an isolated service, as do training execution, the sandbox, the budget and the artifact registry.

    The RSI system on the near side of a ticked boundary marked isolated service. Submit, model or agent, crosses inward and return, controlled result, crosses back, while a read attempt stops at the near face of the boundary with a blocked mark. On the far side, sealed items: hidden tasks, reference answers, scoring code, training execution, sandbox, budget and artifact registry.

  3. 03

    A smaller space is a searchable space.

    Given an unbounded task, current models tend to edit surface details, repeat near identical experiments, and settle early on shallow solutions. Narrowing the scope of each study makes the search tractable and the resulting ability measurable.

    An unbounded task region, open at its edges, holding faint scattered marks: paired surface edits, near identical repeated experiments and paths that stop early. A funnel with a lens at its throat, marked bounded scope, leads to a small closed frame marked smaller study and searchable space, holding a few complete candidate paths and a measurement axis marked measurable.

Choose where the system is allowed to evolve.

The research question is defined by the component under study. Everything outside that boundary remains part of the control.

Synthesis, filtering, mixture, curriculum.
  1. Best verified candidate
  2. Curriculum revision
  3. Selection policy
  4. Difficulty schedule
  5. Mixture policy
  6. Verification pass
  7. Synthesis seed
  8. Baseline
  9. Search ends below its best: the strongest verified candidate is retained while the search cursor rests two ranks lower.

Current release

RSIBench-Data

Data is the currently unlocked research surface. Training, evaluation, execution and budget remain shared controls.

Agents find stronger strategies. Search management remains unstable.

Open RSIBench-Data

Widen what may change, one component at a time.

Each phase opens more of the system to the search. The platform control is closed in every one of them.

  1. 01

    Modular infrastructure

    Services, artifacts, budgets, sandboxes, audit, replay.

    open:
    none
    control:
    closed

  2. 02

    RSIBench-Data now

    The synthetic data pipeline is the only thing the system may rewrite.

    open:
    data
    control:
    closed

  3. 03

    Specialized tracks

    Each component is studied on its own, with the others fixed.

    open:
    algorithm, agent, arch, separately
    control:
    closed

  4. 06

    Joint optimization

    Coordination, credit assignment, resource allocation.

    open:
    two or more, jointly
    control:
    closed

  5. 07

    Full-system RSI

    Long-horizon, reproducible improvement under hidden evaluation.

    open:
    all four
    control:
    closed

Make improvement interpretable before making it recursive.

Every RSIBench study opens one component, holds the rest at recorded values, and returns evidence another team can replay.

Open RSIBench-Data Evolvent AI