Long-Running AI Jobs

Three shapes, and picking the wrong one is the common mistake. Inline streaming when a human is waiting and first-token latency is the product — a chat turn. A background job when the work is minutes long or multi-step: document ingestion, an agent run, a batch of classifications. The provider's batch API when it can wait hours, because it is dramatically cheaper and the right answer for backfills, evals, and re-embedding a corpus.

What makes these jobs different from a thumbnail resize:

Retries cost money. Re-running a job that made forty model calls and failed on the forty-first pays for all forty again. So checkpoint: persist each step's output keyed by the job and step, and on retry resume from the last completed step. This is exactly what durable-execution engines (Temporal, Inngest, Restate) provide out of the box, and it is the main reason to consider one for agent workloads rather than a plain queue.

Partial output has value. A summarisation job that fails at chunk 9 of 10 should surface nine chunks, not nothing. Persist as you go and model the job with real states — queued, running, partial, failed, cancelled — because "processing" and "failed" alone can't express what actually happened.

Cancellation must propagate. A user who closes the tab or clicks stop should stop the spend. That means a cancellation flag the worker checks between steps and an AbortSignal passed into the provider SDK, so the in-flight request is actually torn down rather than merely ignored.

Cost is per-attempt and belongs in the record. Store input and output tokens, model, and latency per step on the job row. Without it, "why did our bill triple" is unanswerable and per-tenant quota enforcement is impossible.

Two more: provider rate limits are the real concurrency ceiling, so the worker pool should be sized to the token-per-minute budget and back off on 429 rather than racing itself; and timeouts must be generous but finite — a hung streaming connection can otherwise hold a worker slot indefinitely.

This is the defining backend workload at the companies these loops hire for, and it's where generic queue knowledge stops being enough. Checkpointing, cancellation and per-job cost tracking are the three things interviewers listen for, because each one only occurs to someone who has actually run model calls in the background.

• Match the shape to the wait: stream inline for a human, queue for minutes of work, batch API for hours.
• Checkpoint per step — a retry that replays completed model calls charges you twice for the same tokens.
• Durable execution engines exist precisely to make multi-step, resumable AI jobs ordinary code.
• Model partial success as a first-class state; discarding nine good chunks because the tenth failed is a design bug.
• Cancellation needs both a checked flag and a propagated AbortSignal, or the spend continues after the user leaves.
• Record tokens, model and cost per attempt on the job, or billing and quotas are guesswork.
• Size worker concurrency to the provider's token-per-minute limit, not to your CPU count.