Open Source

Runtime Intervention Engine

Open-source modified inference engine and SDK for token-level interventions during runtime. Modify generation behavior without retraining.

Resources

Playground

Inspect token timelines and intervention behavior on live shared requests.

Open →

SDK Docs

Implement Python mods, action builders, and event-scoped control patterns.

Open →

GitHub

Open-source engine + SDK for practical runtime intervention workflows.

Open →

Research

Findings from token injection experiments across reasoning and non-reasoning models.

Open →

Core Runtime Features

Event-driven intervention surface for modifying generation behavior without model retraining.

Mod Hook

Define an event-driven mod with ActionBuilder control.

runtime_control.py

1
2
3
4
5@mod
def runtime_control(event, actions: ActionBuilder, tokenizer):
    if isinstance(event, (Prefilled, ForwardPass, Added)):
        return actions.noop()
    return actions.noop()

Adjust Prefill

Rewrite prompt context before generation starts.

prefill_patch.py

1
2
3
4
5
6
7
8@mod
def patch_prefill(event, actions: ActionBuilder, tokenizer):
    if isinstance(event, Prefilled):
        replacement = "System: answer concisely\n" + event.prefilled_text
        return actions.adjust_prefill(
            tokenizer.encode(replacement)
        )
    return actions.noop()

Adjust Logits

Constrain output distribution during forward pass.

logits_guard.py

1
2
3
4
5
6
7
8@mod
def logits_guard(event, actions: ActionBuilder, tokenizer):
    if isinstance(event, ForwardPass):
        logits = event.logits.clone()
        blocked = tokenizer.encode("unsafe")[0]
        logits[blocked] = -1e9
        return actions.adjusted_logits(logits)
    return actions.noop()

Backtrack

Rewind and replace recent tokens after a sampled event.

trajectory_rewrite.py

1
2
3
4
5
6@mod
def revise_suffix(event, actions: ActionBuilder, tokenizer):
    if isinstance(event, Added) and event.step > 30:
        replacement = tokenizer.encode("Let's restate clearly:")
        return actions.backtrack(n=8, replacement=replacement)
    return actions.noop()

Tool Call

Trigger structured tool calls from runtime conditions.

tool_router.py

1
2
3
4
5
6
7
8@mod
def route_tools(event, actions: ActionBuilder, tokenizer):
    if isinstance(event, Prefilled):
        return actions.tool_calls([{
            "name": "search_docs",
            "arguments": {"query": "token injection"}
        }])
    return actions.noop()

Self-Prompt Flow

Use runtime prompts to route into tool or intervention logic.

flow_engine.py

1
2
3
4
5
6
7
8
9
10
11
12runtime_tool_question = FlowQuestion(
    name="runtime_tool_router",
    prompt="Should I call a runtime tool?",
    responses=["yes", "no"],
    erase_mode="all",
)
@mod
def runtime_flow(event, actions: ActionBuilder, tokenizer):
    if isinstance(event, (Prefilled, ForwardPass, Added)):
        return ENGINE.handle_event(event, actions, tokenizer)
    return actions.noop()

1 / 6

Get Started

Tell us about your workflow and we will help you get started with runtime interventions.