research · the runtime stack

The agent runtime, layer by layer.

An agent is only as strong as the runtime under it, so we research the whole execution path: from the agent's own compiled program down to the CUDA kernels. In a compound AI system every millisecond and every dollar compound through the chain. Six layers, one target. Each layer below states what we claim, what proves it, and what is still open.

L5Agent programcompiled from traces L4Run loopinterpreter that never stalls L3Tools and contextone tool in 1500+ L2Reasoning and modelscritics that pay for themselves L1Inference compilerour own kernels L0Serving fabricthe GPU floor

compile.logapr - sep 2026

$ flymy compile --from trace --tier 1
✓ trace read       11 steps · 5 model · 4 tool
✓ def-use pruned   2 calls (no path to output)
✓ program frozen   input → tools[2] → output
✓ served           393 runs off one compile

# fleet, production
programs_compiled   346
runs_from_compiled  1,729
model_calls/run     5 → 3       -40%
tool_call_errors    5.7% → 2.6% 2.2x lower
status              live

Read the study Go to L5

the stack six layers, one target

Own every layer and each one makes the next stronger.

Faster kernels make critics affordable. Affordable critics make agents smarter. Smarter agents produce traces worth compiling. A compiled agent stops paying for the layers below on every run. Cost flows down the stack; evidence flows back up.

stack.mapcost ↓ · evidence ↑

layerwhat it ishands on

L5agent programa run distilled into a typed program-40% model calls per run

L4run loopinterpreter, recovery ladder, step traces↑ traces worth compiling

L3tools and context1500+ connectors, searched not dumped0.91 recall@5, leaner prompts

L2reasoning and modelscritique-and-refine loops, media models↑ cheaper critics, better edits

L1inference compilercustom CUDA and Triton kernels3x faster than stock PyTorch

L0serving fabricGPUs, scheduling, cold starts1.4 s/image, #1 on FLUX.1 dev

L5 · Agent program live in production

Compiling the agent.

thesis: the agent loop is an interpreter; freezing it against one task is compilation

Most of what an agent does is deterministic, and today you pay a model to rediscover it on every run. Watch one successful run, distill the deterministic path into a typed program, and the model is called only where judgment matters. Futamura wrote this down in 1971: specialize an interpreter to a fixed program and a compiled program falls out. We built that for agents, and it has carried production traffic since April 2026.

2026agent compilertier 1 in production

Two tiers, like a JIT

Tier 1 freezes a successful run into a typed program: goal, ordered tool calls with templated arguments, declared input and output schema. The model executes it instead of planning it. Tier 2 turns the same trace into executable code that calls the tools directly, with no model in the loop. 346 programs compiled so far, 303 tier 1 and 18 tier 2. The shift Yohei Nakajima calls the "AGI compiler" is running here today.

programs compiled346

production runs from compiled programs1,729

most runs off a single compile393

breaks evenon run 2

Read the study Our take on X

Visualization of an agent compiler: thousands of execution paths crystallizing into compiled circuits around a single reasoning core

paired · n = 78

-40%

Model calls per run

Each compiled program measured against its own source session: median model calls fall from 5 to 3, tool calls from 4 to 2. The planning is compiled away; the judgment stays.

Method

reuse

393runs

Off one compile

Compilation is a single model pass over a trace the platform already recorded, so tier 1 is cheaper than interpreting from the second run onward and converges on a 40% saving.

Cost model

reliability

2.2x fewer

Tool-call errors

2.6% of tool calls fail on a compiled program against 5.7% in an interactive session, and completion holds at 95.4% with nobody watching. Only argument shapes that already worked get shipped.

Reliability data

foundations

None. This layer did not exist five years ago: there was no agent loop to compile. Everything here is new work, which is why it carries the only production numbers on this page.

open problem

Path determinism sits at 34% run-weighted across 1,534 production runs: a frozen program still lets the model wander, and one program took 170 distinct tool paths across 385 runs. Tier 2 removes that by construction but covers 18 of 346 programs, and it has no guards yet. When a tool renames a field, a compiled program should trip a precondition and deoptimize back into the interpreter mid-run. It does not yet.

L4 · Run loop

The interpreter that never stalls.

thesis: a loop that logs nothing has nothing to compile

Before you can compile an agent you need a loop worth compiling. Ours has to survive a hung tool, a dead worker, a provider that returns nothing three hundred times in a row, and a context window that fills halfway through a long job, without losing the work already done. A recovery ladder, context compression, completion control, and liveness judged by whether the log is still growing rather than whether a process is still alive.

2026agent runtime

9,279 agent runs in production

Every execution, interactive or unattended, carries a step-level trace. That trace is what the compiler above reads.

unattended runs reaching a saved result95.4%

tool calls traced, step by step35k

Run an agent

2025AI agents

World's first media agent to beat OpenAI Image

Add a hat edit comparison: original photo, FlyMyAI result preserving the face, main competitor changing the face

Media Agent M1, the first multi-domain GenAI media agent: identity-preserving edits, chained actions, every media domain in one chat.

Article Try it

foundations

Triton Inference Server scheduling 2019

open problem

Recovering from a stalled tool without discarding partial work is still not solved cleanly, and our context-compression trigger measures the wrong quantity, so it fires late on exactly the long runs that need it most. Under concurrency, one slow run can hold the head of the queue.

L3 · Tools and context

Finding one tool in 1500+.

thesis: tool selection is retrieval, and retrieval is only as fast as its embeddings

An agent with 1500+ connectors and 500+ models attached has no context budget left for the task. The tool surface has to be searched and scoped on every turn instead of dumped into the prompt.

2026agent retrieval

Tool search across 1500+ connectors

Postgres full-text search and trigram matching over the whole action catalog, plus a named-resolution pass for tools the user asks for by name, at a fraction of the context an attached catalog would cost.

recall@5 on our tool-retrieval benchmark0.91

models reachable through the same search500+

Browse the catalog

2025AI compiler

World's fastest embeddings

ModernBERT-base latency: HuggingFace 1x, Flash-Attention 1.3x, optimized on FlyMyAI 2.8x faster (NVIDIA A10, fp16)

ModernBERT embeddings accelerated a further 2.8x on top of the original stack: the fastest text-to-vector we know of, powering real-time retrieval and memory for agent chains.

Announcement

foundations

RuCLIP-tiny multimodal embeddings 2022

open problem

Discovery precision. The agent still escalates "no such tool exists" against a catalog that demonstrably has the tool, because a query phrased in the user's words misses an action named in the provider's. Ranking has to learn from what the agent went on to call, not just from what the query looked like.

L2 · Reasoning and models

Image models can think.

thesis: once critics are cheap, a critique-and-refine loop beats a bigger model

When the layers below make critics cheap enough to run, wrapping generation in a critique-and-refine loop beats a bigger single model without retraining anything. And the models an agent calls have to be worth calling: an agent is only as good as the best edit, the best video and the best voice it can reach for.

compound AI · CRAFT

4x win rate

Image models can think

CRAFT wraps generation in a critique-and-refine loop and lifts preference win rate from 0.19 to 0.76 on Parti-Prompts. No retraining.

Generatebase model→Critiquevision QA→Refine0.19 → 0.76

CRAFT pipeline block diagram: prompt rewriting, generation, visual question answering, image comparison and prompt editing loop

CRAFT result: baseline Earth render vs refined Earth-from-the-Moon render after the reasoning loop

arXiv Report Live demo

GenAI media

0.85identity

World's most precise face transfer

FLUX LoRA training that keeps identity intact under strong style shifts: 0.85 identity fidelity and 0.89 prompt adherence in our published benchmark, against 0.81 and 0.69 for the alternatives.

Identity fidelity benchmark chart: FlyMyAI 0.85 vs competitors 0.81 and 0.69

Read the study Trainer

open source

750+ stars

World's first Qwen-Image LoRA trainer

Open-source trainer adopted by the community at record pace.

Side by side: base Qwen-Image vs Qwen-Image with FlyMyAI LoRA Realism

GitHub Model

foundations

StarGAN-v2 fast-train at GTC 2021 Kandinsky-2.0 / 2.1 / 2.2 2022-2023 Kandinsky-4 diffusion 2024 Kandinsky Video 2024 VideoDALL-E 2022 Imagen-PyTorch 2022 Emotion-Embedding TTS 2023 PitchFlow expressive TTS 2024 ImproveYourVideos architecture 2024 AlexNet-3D video understanding 2018

open problem

A critique loop costs a full extra generation on every pass, and we have no learned stopping rule for when refining stops paying, so the loop runs a fixed number of times whether the image needed it or not. Routing has the same gap: the runtime picks a model by capability, not by expected cost per successful result.

L1 · Inference compiler

Custom kernels under every step.

thesis: a millisecond saved here is paid back fifteen times in a fifteen-step run

This is the floor the agent stands on. Whatever the kernel layer gives back is multiplied by every step above it, which is why we write our own CUDA and Triton kernels instead of renting someone else's.

AI compiler

Faster than PyTorch kernels

Custom CUDA and Triton non-linearity kernels: the floor of the stack, cutting GPU hours for every agent step above it.

Benchmark chart: custom Triton kernels vs stock Torch kernels on Nvidia A100

Read the article

GenAI media

24fps

World's first real-time video style transfer

Live video restyled frame by frame on 2 GPUs: the first real-time video-to-video diffusion pipeline.

Real-time video to video style transfer demo at 24 fps

Watch the demo

AI compiler

50ms

World's fastest Stable Diffusion

Stable Diffusion at about 50 ms per image, regenerating live as you type: 14x faster than baseline PyTorch, 3x faster than the nearest competitor, at $0.15 per 1K images.

Live demo: Stable Diffusion XL Turbo regenerating the image on every keystroke at about 50 ms per image on Nvidia H100

Demo Preprint Model

2024AI compiler

5x faster FLUX.Schnell on H100

200 ms for 4 steps at 640x640: fast enough to sit inside an interactive agent loop.

Model

2025GenAI media

Real-time avatars at 25 fps

8x acceleration of LatentSync lip-sync, 2.5x faster than the nearest competitor. Production-grade avatars for real-time agents.

Demo video Model

2025GenAI media

Kandinsky Video, accelerated

Production-quality text-to-video, 2x faster generation than the nearest competitor.

Demo video Model

foundations

NVIDIA TensorRT compiler 2018 NVIDIA

NVIDIA FasterTransformer 2020 NVIDIA

Megatron GPT-3 inference at GTC 2021 Outperforming PyTorch with Triton 2024

open problem

Kernel coverage is still per-model handwork. Every new architecture needs a human to find the fusions, and no pass we have written generalizes to the next one on its own, so the floor of the stack moves at the speed of the team rather than the speed of the model release cycle.

L0 · Serving fabric

The GPU floor.

thesis: this layer decides whether one agent step costs a cent or a dollar

Scheduling, cold starts, scale-to-zero economics. The public scoreboard where this shows up is diffusion serving, because it is the one workload the whole industry benchmarks in the open.

Artificial Analysis · Jan 2026

Diffusion inference leader across popular open models

Seconds per image, lower is better. FlyMy.AI: 1.4 s, the fastest of all tracked providers. #1 in Qwen-Image price per image and top-2 on FLUX.1 [schnell]. Our own kernels, our own GPU fabric, in production today.

#1 speed · FLUX.1 dev #1 price · Qwen-Image

Artificial Analysis benchmark: FLUX.1 dev API generation time - FlyMyAI is the fastest provider at 1.4 seconds per image

source: Artificial Analysis, official benchmark artificialanalysis.ai/image/providers/flux-1-dev

foundations

NVIDIA Megatron-LM 2019 NVIDIA

NVIDIA Triton Inference Server 2019 NVIDIA

Pipeline-parallel GPT-3 & GPT-J serving 2021

open problem

Cold start still dominates first-call latency on scale-to-zero models, which is the single worst experience an agent can hand a user: the step that should be cheapest is the one that feels broken. Keeping a pod warm fixes latency and breaks the economics. We want both.

provenance a decade before FlyMy.AI

A team that built what the industry still runs on.

Before FlyMy.AI, this team spent a decade building the compilers, serving stacks and generative models the industry still runs on, at NVIDIA, Sber AI and beyond. Each item is filed under its layer above; here it is in one list.

NVIDIA pipeline-parallel GPT-3 & GPT-J serving L0 · 2021 NVIDIA

NVIDIA Megatron GPT-3 inference at GTC L1 · 2021 NVIDIA

NVIDIA StarGAN-v2 fast-train at GTC L2 · 2021 NVIDIA

NVIDIA TensorRT compiler L1 · 2018 NVIDIA

NVIDIA Megatron-LM L0 · 2019 NVIDIA

NVIDIA Triton Inference Server L0 · 2019 NVIDIA

NVIDIA FasterTransformer L1 · 2020 Outperforming PyTorch with custom Triton kernels L1 · 2024 AlexNet-3D L2 · 2018 ImproveYourVideos architecture L2 · 2024 RuCLIP-tiny L3 · 2022 VideoDALL-E L2 · 2022 Imagen-PyTorch L2 · 2022 Emotion-Embedding TTS L2 · 2023 PitchFlow expressive TTS L2 · 2024 Kandinsky-2.0 L2 · 2022 Kandinsky-2.1 L2 · 2023 Kandinsky-2.2 L2 · 2023 Kandinsky-4 diffusion L2 · 2024 Kandinsky Video L2 · 2024