AI runtime architecture

The Runtime Boundary Is the Product: Why AI Workflows Fail Outside the Model

AI teams often ask whether a model can do the job. The better question is whether the surrounding runtime can make the model usable, observable, recoverable, and repeatable.

A model demo can succeed while the product fails. That sounds obvious until you watch it happen. The model is capable. The sample output is impressive. The benchmark suggests there is a better path. Then the real system has to start the runtime, fit the model into memory, coordinate storage, stream progress, recover from interruption, and explain errors to a human operator.

That is where many AI projects become fragile. They treat the runtime boundary as plumbing, when it is actually part of the product.

In one local image-generation workflow, the goal was simple on paper: generate product-design images with FLUX.2 and a project LoRA, store the resulting images in object storage, and make the work visible in a queue. The stable path used ComfyUI as the image-generation runtime. A proposed replacement used vLLM-Omni because it promised a faster server style interface for diffusion models.

The base model was nominally the same. The business task was the same. The hardware was the same. The operational reality was not the same at all.

The Model Was Not the Whole System

It is tempting to describe an AI workflow by naming the model: "we use FLUX.2" or "we run Qwen locally." That shorthand hides the parts that decide whether the workflow can run reliably.

A generation runtime is not just a model file. It is a bundle of contracts:

• which checkpoint and auxiliary components are loaded
• which text encoder, VAE, scheduler, sampler, and LoRA path are used
• how weights are quantized, cast, moved, pinned, cached, or offloaded
• how startup and warmup are represented to the user
• how outputs are written, ingested, verified, and cleaned up
• how the system behaves when it is interrupted halfway through a job

In this case, ComfyUI had a mature path for the exact kind of oversized diffusion workload the workstation needed. It reported normal VRAM mode, async weight offloading, pinned memory, RAM pressure caching, and dynamic VRAM support. The app could submit a prompt graph, wait for Comfy history, upload the final images to MinIO, and clean up GPU memory at the end of the job.

The vLLM-Omni attempt was different. The runtime had to load enormous FLUX.2 components, handle text-encoder pressure, apply offloading, and serve a compatible image-generation API. The startup path alone became a long-running operation that pushed system memory and swap hard enough to threaten the machine. That is not a mere implementation detail. It is a product risk.

Same model, different runtime, different product.

Runtime Fit Is a Product Requirement

A runtime can be theoretically better and still be wrong for a particular system. The evaluation has to include the whole operating shape, not just promised throughput.

For local AI systems, runtime fit includes at least four questions.

These questions are not infrastructure trivia. They affect product confidence, batch planning, staffing, cost, and the amount of human supervision the system requires.

The Boundary Contract

A useful AI runtime boundary should be boring. That is a compliment. Boring means the app knows how to ask for work, how to observe work, how to stop work, and how to verify the result.

In the image-generation system, the boundary contract needs to cover the full lifecycle:

• preflight: confirm the runtime, model files, LoRA files, and storage are present
• submission: send the exact prompt graph or manifest, with seed and settings recorded
• progress: show loading, encoding, sampling, saving, ingesting, and cleanup phases
• storage: upload final app-owned images to MinIO and verify object keys
• recovery: resume, retry, delete, or repair jobs without hidden state
• cleanup: release memory deliberately, not as an unexplained side effect

This is why queues, logs, storage audits, and runtime profiles belong in the product conversation. They are not administrative screens. They are how the user learns whether the system can be trusted with expensive long-running work.

Why the Stable Path Won

The ComfyUI path was not chosen because it was conceptually elegant. It was chosen because it matched the workload and the hardware better. It already had the dynamic memory behavior needed to run an oversized FLUX.2 workflow on the available GPU. It already understood the local model layout, LoRA file, scheduler, and VAE choices. It already exposed a queue/history model the app could integrate with.

The vLLM-Omni path may still become interesting on a different hardware profile, especially a production machine with separate GPUs for image generation and review. It may also become more attractive as official support for the exact diffusion model, offload strategy, and quantization path matures.

But a production architecture cannot be based on "this should work because the model can generate images." It has to be based on what the actual runtime can do repeatedly without damaging the operator's day.

Portability Without Fantasy

There is a subtle trap in portability discussions. Teams often try to make everything interchangeable too early. They create multiple runtime paths, multiple fallbacks, and multiple storage routes so the system can "support" every environment.

Real portability comes from stable contracts, not from pretending every runtime is equivalent. The prompts, seeds, settings, object keys, review decisions, training metadata, and storage contents should be portable. The hardware profile can change. The runtime profile can change. But each profile still needs one honest execution path with explicit health, explicit errors, and explicit metadata.

A local 5090 workstation and a two-GPU production box should not be forced to behave identically. They should share business contracts: what a generation job means, what successful storage means, what a review means, and what recovery looks like.