# FlyMy.AI Research: the agent runtime, layer by layer

> FlyMy.AI research across the whole agent runtime: a production agent compiler (346 programs, 1,729 runs, -40% model calls), tool retrieval over 1500+ connectors, image models that reason, custom CUDA and Triton kernels and the fastest diffusion serving on Artificial Analysis.

Page: https://flymy.ai/Research/

research · the runtime stack

# The agent runtime, layer by layer.

An agent is only as strong as the runtime under it, so we research the whole execution path: from the agent's own compiled program down to the CUDA kernels. In a compound AI system every millisecond and every dollar compound through the chain. Six layers, one target. Each layer below states what we claim, what proves it, and what is still open.

[L5 Agent program compiled from traces](https://flymy.ai/Research/#layer-l5), [L4 Run loop interpreter that never stalls](https://flymy.ai/Research/#layer-l4), [L3 Tools and context one tool in 1500+](https://flymy.ai/Research/#layer-l3), [L2 Reasoning and models critics that pay for themselves](https://flymy.ai/Research/#layer-l2), [L1 Inference compiler our own kernels](https://flymy.ai/Research/#layer-l1), [L0 Serving fabric the GPU floor](https://flymy.ai/Research/#layer-l0)

compile.log apr - sep 2026

```
$ flymy compile --from trace --tier 1
✓ trace read       11 steps · 5 model · 4 tool
✓ def-use pruned   2 calls (no path to output)
✓ program frozen   input → tools[2] → output
✓ served           393 runs off one compile

# fleet, production
programs_compiled   346
runs_from_compiled  1,729
model_calls/run     5 → 3       -40%
tool_call_errors    5.7% → 2.6% 2.2x lower
status              live
```

[Read the study](https://flymy.ai/blog/compiling-ai-agents/) [Go to L5](https://flymy.ai/Research/#layer-l5)

**the stack** six layers, one target

## Own every layer and each one makes the next stronger.

Faster kernels make critics affordable. Affordable critics make agents smarter. Smarter agents produce traces worth compiling. A compiled agent stops paying for the layers below on every run. Cost flows down the stack; evidence flows back up.

stack.map cost ↓ · evidence ↑

layer, what it is, hands on

L5 agent program a run distilled into a typed program **-40%** model calls per run

L4 run loop interpreter, recovery ladder, step traces ↑ traces worth compiling

L3 tools and context 1500+ connectors, searched not dumped **0.91** recall@5, leaner prompts

L2 reasoning and models critique-and-refine loops, media models ↑ cheaper critics, better edits

L1 inference compiler custom CUDA and Triton kernels **3x** faster than stock PyTorch

L0 serving fabric GPUs, scheduling, cold starts **1.4 s** /image, #1 on FLUX.1 dev

L5 · Agent program live in production

## Compiling the agent.

thesis: **the agent loop is an interpreter; freezing it against one task is compilation**

Most of what an agent does is deterministic, and today you pay a model to rediscover it on every run. Watch one successful run, distill the deterministic path into a typed program, and the model is called only where judgment matters. Futamura wrote this down in 1971: specialize an interpreter to a fixed program and a compiled program falls out. We built that for agents, and it has carried production traffic since April 2026.

2026 agent compiler tier 1 in production

### Two tiers, like a JIT

Tier 1 freezes a successful run into a typed program: goal, ordered tool calls with templated arguments, declared input and output schema. The model executes it instead of planning it. Tier 2 turns the same trace into executable code that calls the tools directly, with no model in the loop. 346 programs compiled so far, 303 tier 1 and 18 tier 2. The shift Yohei Nakajima calls the "AGI compiler" is running here today.

programs compiled **346**

production runs from compiled programs **1,729**

most runs off a single compile **393**

breaks even **on run 2**

[Read the study](https://flymy.ai/blog/compiling-ai-agents/) [Our take on X](https://x.com/cop_on_fire/status/2074705081405182030)

![Visualization of an agent compiler: thousands of execution paths crystallizing into compiled circuits around a single reasoning core](https://flymy.ai/Research/img/agent-compiler-epic.jpg)

paired · n = 78

-40%

### Model calls per run

Each compiled program measured against its own source session: median model calls fall from 5 to 3, tool calls from 4 to 2. The planning is compiled away; the judgment stays.

[Method](https://flymy.ai/blog/compiling-ai-agents/#results)

reuse

393 runs

### Off one compile

Compilation is a single model pass over a trace the platform already recorded, so tier 1 is cheaper than interpreting from the second run onward and converges on a 40% saving.

[Cost model](https://flymy.ai/blog/compiling-ai-agents/#economics)

reliability

2.2 x fewer

### Tool-call errors

2.6% of tool calls fail on a compiled program against 5.7% in an interactive session, and completion holds at 95.4% with nobody watching. Only argument shapes that already worked get shipped.

[Reliability data](https://flymy.ai/blog/compiling-ai-agents/#reliability)

foundations

None. This layer did not exist five years ago: there was no agent loop to compile. Everything here is new work, which is why it carries the only production numbers on this page.

open problem

Path determinism sits at 34% run-weighted across 1,534 production runs: a frozen program still lets the model wander, and one program took 170 distinct tool paths across 385 runs. Tier 2 removes that by construction but covers 18 of 346 programs, and it has no guards yet. When a tool renames a field, a compiled program should trip a precondition and deoptimize back into the interpreter mid-run. It does not yet.

L4 · Run loop

## The interpreter that never stalls.

thesis: **a loop that logs nothing has nothing to compile**

Before you can compile an agent you need a loop worth compiling. Ours has to survive a hung tool, a dead worker, a provider that returns nothing three hundred times in a row, and a context window that fills halfway through a long job, without losing the work already done. A recovery ladder, context compression, completion control, and liveness judged by whether the log is still growing rather than whether a process is still alive.

2026 agent runtime

### 9,279 agent runs in production

Every execution, interactive or unattended, carries a step-level trace. That trace is what the compiler above reads.

unattended runs reaching a saved result **95.4%**

tool calls traced, step by step **35k**

[Run an agent](https://app.flymy.ai/)

2025 AI agents

### World's first media agent to beat OpenAI Image

![Add a hat edit comparison: original photo, FlyMyAI result preserving the face, main competitor changing the face](https://flymy.ai/Research/img/m1-edit-compare.jpg)

Media Agent M1, the first multi-domain GenAI media agent: identity-preserving edits, chained actions, every media domain in one chat.

[Article](https://medium.com/flymy-ai/flymy-ai-media-agent-1-next-level-genai-tools-for-developers-6754c446e493) [Try it](https://app.flymy.ai/chat)

foundations

![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) Triton Inference Server scheduling 2019

open problem

Recovering from a stalled tool without discarding partial work is still not solved cleanly, and our context-compression trigger measures the wrong quantity, so it fires late on exactly the long runs that need it most. Under concurrency, one slow run can hold the head of the queue.

L3 · Tools and context

## Finding one tool in 1500+.

thesis: **tool selection is retrieval, and retrieval is only as fast as its embeddings**

An agent with 1500+ connectors and 500+ models attached has no context budget left for the task. The tool surface has to be searched and scoped on every turn instead of dumped into the prompt.

2026 agent retrieval

### Tool search across 1500+ connectors

Postgres full-text search and trigram matching over the whole action catalog, plus a named-resolution pass for tools the user asks for by name, at a fraction of the context an attached catalog would cost.

recall@5 on our tool-retrieval benchmark **0.91**

models reachable through the same search **500+**

[Browse the catalog](https://app.flymy.ai/mcp)

2025 AI compiler

### World's fastest embeddings

![ModernBERT-base latency: HuggingFace 1x, Flash-Attention 1.3x, optimized on FlyMyAI 2.8x faster (NVIDIA A10, fp16)](https://flymy.ai/Research/img/modernbert-latency.jpg)

ModernBERT embeddings accelerated a further 2.8x on top of the original stack: the fastest text-to-vector we know of, powering real-time retrieval and memory for agent chains.

[Announcement](https://www.linkedin.com/feed/update/urn:li:activity:7282892798364635136)

foundations

[RuCLIP-tiny multimodal embeddings 2022](https://github.com/cene555/ru-clip-tiny)

open problem

Discovery precision. The agent still escalates "no such tool exists" against a catalog that demonstrably has the tool, because a query phrased in the user's words misses an action named in the provider's. Ranking has to learn from what the agent went on to call, not just from what the query looked like.

L2 · Reasoning and models

## Image models can think.

thesis: **once critics are cheap, a critique-and-refine loop beats a bigger model**

When the layers below make critics cheap enough to run, wrapping generation in a critique-and-refine loop beats a bigger single model without retraining anything. And the models an agent calls have to be worth calling: an agent is only as good as the best edit, the best video and the best voice it can reach for.

compound AI · CRAFT

4 x win rate

### Image models can think

CRAFT wraps generation in a critique-and-refine loop and lifts preference win rate from 0.19 to 0.76 on Parti-Prompts. No retraining.

Generate base model *→* Critique vision QA *→* Refine 0.19 → 0.76

![CRAFT pipeline block diagram: prompt rewriting, generation, visual question answering, image comparison and prompt editing loop](https://flymy.ai/Research/img/craft-pipeline.jpg)

![CRAFT result: baseline Earth render vs refined Earth-from-the-Moon render after the reasoning loop](https://flymy.ai/Research/img/craft-result.jpg)

[arXiv](https://arxiv.org/abs/2512.20362), [Report](https://huggingface.co/blog/flymy-ai/craft-1), [Live demo](https://craft-demo.flymy.ai/)

GenAI media

0.85 identity

### World's most precise face transfer

FLUX LoRA training that keeps identity intact under strong style shifts: 0.85 identity fidelity and 0.89 prompt adherence in our published benchmark, against 0.81 and 0.69 for the alternatives.

![Identity fidelity benchmark chart: FlyMyAI 0.85 vs competitors 0.81 and 0.69](https://flymy.ai/Research/img/lora-identity-fidelity.png)

[Read the study](https://flymy.ai/blog/flux-lora-training-flymyai-vs-fal-ai-vs-nanobanana/) [Trainer](https://app.flymy.ai/models/flymyai/flux-lora-trainer-fast)

open source

750 + stars

### World's first Qwen-Image LoRA trainer

Open-source trainer adopted by the community at record pace.

![Side by side: base Qwen-Image vs Qwen-Image with FlyMyAI LoRA Realism](https://flymy.ai/Research/img/qwen-lora-compare.jpg)

[GitHub](https://github.com/FlyMyAI/flymyai-lora-trainer) [Model](https://app.flymy.ai/models/flymyai/qwen-image-lora-trainer)

foundations

![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) StarGAN-v2 fast-train at GTC 2021, [Kandinsky-2.0 / 2.1 / 2.2 2022-2023](https://github.com/ai-forever/Kandinsky-2), [Kandinsky-4 diffusion 2024](https://github.com/ai-forever/Kandinsky-4), [Kandinsky Video 2024](https://github.com/ai-forever/KandinskyVideo), [VideoDALL-E 2022](https://github.com/ai-forever/ru-dalle/blob/master/jupyters/videodalle.ipynb), [Imagen-PyTorch 2022](https://github.com/cene555/Imagen-pytorch), [Emotion-Embedding TTS 2023](https://www.isca-archive.org/interspeech_2023/shaheen23_interspeech.html), [PitchFlow expressive TTS 2024](https://www.isca-archive.org/interspeech_2024/sadekova24_interspeech.html), [ImproveYourVideos architecture 2024](https://ieeexplore.ieee.org/document/10815947), [AlexNet-3D video understanding 2018](https://github.com/denti/AlexNet3D)

open problem

A critique loop costs a full extra generation on every pass, and we have no learned stopping rule for when refining stops paying, so the loop runs a fixed number of times whether the image needed it or not. Routing has the same gap: the runtime picks a model by capability, not by expected cost per successful result.

L1 · Inference compiler

## Custom kernels under every step.

thesis: **a millisecond saved here is paid back fifteen times in a fifteen-step run**

This is the floor the agent stands on. Whatever the kernel layer gives back is multiplied by every step above it, which is why we write our own CUDA and Triton kernels instead of renting someone else's.

AI compiler

3 x

### Faster than PyTorch kernels

Custom CUDA and Triton non-linearity kernels: the floor of the stack, cutting GPU hours for every agent step above it.

![Benchmark chart: custom Triton kernels vs stock Torch kernels on Nvidia A100](https://flymy.ai/Research/img/triton-vs-torch.png)

[Read the article](https://medium.com/flymy-ai/how-to-outperform-pytorch-kernels-with-openais-triton-compiler-ba2babce8cc5)

GenAI media

24 fps

### World's first real-time video style transfer

Live video restyled frame by frame on 2 GPUs: the first real-time video-to-video diffusion pipeline.

![Real-time video to video style transfer demo at 24 fps](https://flymy.ai/Research/img/style-transfer-24fps.gif)

[Watch the demo](https://x.com/cop_on_fire/status/1895533228691505532)

AI compiler

50 ms

### World's fastest Stable Diffusion

Stable Diffusion at about 50 ms per image, regenerating live as you type: 14x faster than baseline PyTorch, 3x faster than the nearest competitor, at $0.15 per 1K images.

![Live demo: Stable Diffusion XL Turbo regenerating the image on every keystroke at about 50 ms per image on Nvidia H100](https://flymy.ai/Research/img/sdxl-50ms-demo.gif)

[Demo](https://www.linkedin.com/feed/update/urn:li:activity:7209575137996619778/), [Preprint](https://arxiv.org/abs/2503.04871), [Model](https://app.flymy.ai/models/flymyai/SDXLTurbo)

2024 AI compiler

### 5x faster FLUX.Schnell on H100

200 ms for 4 steps at 640x640: fast enough to sit inside an interactive agent loop.

[Model](https://app.flymy.ai/models/flymyai/flux-schnell)

2025 GenAI media

### Real-time avatars at 25 fps

8x acceleration of LatentSync lip-sync, 2.5x faster than the nearest competitor. Production-grade avatars for real-time agents.

[Demo video](https://www.loom.com/share/57fb086da93b476c9f1c8846a156d7e7) [Model](https://app.flymy.ai/models/flymyai/latentsync)

2025 GenAI media

### Kandinsky Video, accelerated

Production-quality text-to-video, 2x faster generation than the nearest competitor.

[Demo video](https://www.loom.com/share/0fe33dc1283e4c2988429e2564ef2379)

foundations

![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) NVIDIA TensorRT compiler 2018,![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) NVIDIA FasterTransformer 2020,![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) Megatron GPT-3 inference at GTC 2021, [Outperforming PyTorch with Triton 2024](https://medium.com/flymy-ai/how-to-outperform-pytorch-kernels-with-openais-triton-compiler-ba2babce8cc5)

open problem

Kernel coverage is still per-model handwork. Every new architecture needs a human to find the fusions, and no pass we have written generalizes to the next one on its own, so the floor of the stack moves at the speed of the team rather than the speed of the model release cycle.

L0 · Serving fabric

## The GPU floor.

thesis: **this layer decides whether one agent step costs a cent or a dollar**

Scheduling, cold starts, scale-to-zero economics. The public scoreboard where this shows up is diffusion serving, because it is the one workload the whole industry benchmarks in the open.

![Artificial Analysis](https://flymy.ai/Research/img/artificial-analysis.png) Artificial Analysis · Jan 2026

### Diffusion inference leader across popular open models

Seconds per image, lower is better. FlyMy.AI: 1.4 s, the fastest of all tracked providers. #1 in Qwen-Image price per image and top-2 on FLUX.1 [schnell]. Our own kernels, our own GPU fabric, in production today.

#1 speed · FLUX.1 dev #1 price · Qwen-Image

![Artificial Analysis benchmark: FLUX.1 dev API generation time - FlyMyAI is the fastest provider at 1.4 seconds per image](https://flymy.ai/Research/img/aa-flux-dev-api-time.jpg)

source: Artificial Analysis, official benchmark [artificialanalysis.ai/image/providers/flux-1-dev](https://artificialanalysis.ai/image/providers/flux-1-dev)

foundations

![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) NVIDIA Megatron-LM 2019,![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) NVIDIA Triton Inference Server 2019,![NVIDIA](https://flymy.ai/Research/img/nvidia.svg) Pipeline-parallel GPT-3 & GPT-J serving 2021

open problem

Cold start still dominates first-call latency on scale-to-zero models, which is the single worst experience an agent can hand a user: the step that should be cheapest is the one that feels broken. Keeping a pod warm fixes latency and breaks the economics. We want both.

**provenance** a decade before FlyMy.AI

## A team that built what the industry still runs on.

Before FlyMy.AI, this team spent a decade building the compilers, serving stacks and generative models the industry still runs on, at NVIDIA, Sber AI and beyond. Each item is filed under its layer above; here it is in one list.

[NVIDIA pipeline-parallel GPT-3 & GPT-J serving L0 · 2021](https://developer.nvidia.com/blog/deploying-gpt-j-and-t5-with-fastertransformer-and-triton-inference-server/), [NVIDIA Megatron GPT-3 inference at GTC L1 · 2021](https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s31578/), [NVIDIA StarGAN-v2 fast-train at GTC L2 · 2021](https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s31203/), [NVIDIA TensorRT compiler L1 · 2018](https://github.com/NVIDIA/TensorRT), [NVIDIA Megatron-LM L0 · 2019](https://github.com/NVIDIA/Megatron-LM), [NVIDIA Triton Inference Server L0 · 2019](https://github.com/triton-inference-server/server), [NVIDIA FasterTransformer L1 · 2020](https://github.com/NVIDIA/FasterTransformer), [Outperforming PyTorch with custom Triton kernels L1 · 2024](https://medium.com/flymy-ai/how-to-outperform-pytorch-kernels-with-openais-triton-compiler-ba2babce8cc5), [AlexNet-3D L2 · 2018](https://github.com/denti/AlexNet3D), [ImproveYourVideos architecture L2 · 2024](https://ieeexplore.ieee.org/document/10815947), [RuCLIP-tiny L3 · 2022](https://github.com/cene555/ru-clip-tiny), [VideoDALL-E L2 · 2022](https://github.com/ai-forever/ru-dalle/blob/master/jupyters/videodalle.ipynb), [Imagen-PyTorch L2 · 2022](https://github.com/cene555/Imagen-pytorch), [Emotion-Embedding TTS L2 · 2023](https://www.isca-archive.org/interspeech_2023/shaheen23_interspeech.html), [PitchFlow expressive TTS L2 · 2024](https://www.isca-archive.org/interspeech_2024/sadekova24_interspeech.html), [Kandinsky-2.0 L2 · 2022](https://github.com/ai-forever/Kandinsky-2), [Kandinsky-2.1 L2 · 2023](https://github.com/ai-forever/Kandinsky-2), [Kandinsky-2.2 L2 · 2023](https://github.com/ai-forever/Kandinsky-2), [Kandinsky-4 diffusion L2 · 2024](https://github.com/ai-forever/Kandinsky-4), [Kandinsky Video L2 · 2024](https://github.com/ai-forever/KandinskyVideo)

---

For agents: the FlyMyAI operating guide (sign up, connect, run models, tools and agents) is https://flymy.ai/skill.md, and https://flymy.ai/llms.txt lists the other pages and the documentation.
