Turning documents into filled forms: an async, LLM-powered backend

I spent the last stretch building a backend that does something deceptively simple to describe and surprisingly fiddly to build well: you upload a person's documents — a resume, an ID, a certificate — and the system reads them, pulls out structured data, and uses that data to fill out whatever form template you throw at it (Excel, Word), rendering a finished file you can download. On top of that it keeps a knowledge graph of every person so you can query the whole dataset later.

The domain it shipped for doesn't matter much, so I'll keep it generic: think "resume in, filled form out," multi-tenant, with an LLM doing the reading and mapping. What I want to write about is the architecture, because most of the interesting decisions had nothing to do with the AI part.

The one decision that shaped everything: nothing heavy happens in the request

Every operation here is slow. Extraction is a multimodal LLM call over a PDF. Filling a form is several LLM calls plus rendering an Excel file. None of that belongs in an HTTP request that a browser is waiting on.

So the rule is: the API never does heavy work synchronously. Every "do something" endpoint:

1. Writes a row to Postgres in a ready / queued state,
2. Uploads any file to S3,
3. Enqueues a message on SQS,
4. Returns 202 Accepted with the row's id.

A separate worker process does the actual work and updates the row's status and result columns. Clients poll the GET endpoint until the status is terminal. The status enum is the contract between the API and the worker — neither shares memory with the other, they only talk through Postgres, S3, and SQS.

The stack underneath: Fastify + Zod for the API (every request and response is schema-validated), Postgres via Supabase with Drizzle, AWS S3 for files, AWS SQS for the async backbone, Neo4j for the graph, and Google Gemini through the Vercel AI SDK for everything LLM. Two long-running processes ship from one repo — an API and a worker — which matters later.

The queue is where the real engineering lives

It's easy to draw "API → queue → worker" on a whiteboard. Making it survive crashes, redeliveries, and 10-minute LLM jobs is the actual work. A few invariants carry the whole thing.

Idempotent claims. SQS guarantees at-least-once delivery, which means every message can arrive twice. So the first thing every handler does is an atomic conditional update — it only proceeds if it was the one that flipped the row to processing:

const claimed = await db
  .update(jobs)
  .set({ status: "processing", updatedAt: now() })
  .where(and(eq(jobs.id, id), inArray(jobs.status, ["ready", "review", "error"])))
  .returning();
 
if (claimed.length === 0) return; // someone else already has it

A duplicate message finds nothing to claim and quietly drops. No double-processing, no locks, no coordination service.

Self-healing stale claims. That same claim query also re-acquires rows that have been stuck in processing for too long. If a worker dies mid-job, the row would otherwise be orphaned forever. Instead the next delivery (or a retry) sees a stale processing row, reclaims it, and continues. The crash heals itself.

Keeping long jobs alive. An LLM extraction can run for minutes, but SQS will redeliver a message if you don't finish within its visibility timeout. The fix is a visibility extender: while a message is in flight, a timer pushes its visibility deadline out periodically so the message never reappears underneath the worker that's still processing it.

This only works if three timers are ordered correctly:

visibility-extend interval  <  stale-processing window  <  visibility timeout
        (≈ timeout / 2)              (~10 min)                 (~15 min)

Get that ordering wrong — say the stale window is shorter than the extend interval — and a healthy job gets reclaimed out from under itself. This single inequality is the thing I'd put on a sticky note for anyone maintaining the system.

Terminal vs. retryable errors. Handlers normalize every failure into one of two buckets. A retryable error (a timeout, a transient 5xx) is rethrown — SQS retries it, and after a few attempts it lands in a dead-letter queue. A terminal error (the document is missing a required field) is persisted as status: "error" and not rethrown, so the message is deleted instead of being retried forever against a problem that won't fix itself. Encoding "is this worth retrying?" into the handler is what keeps the DLQ meaningful instead of full of garbage.

One worker process runs five independent consumers — one per queue — concurrently, each with its own bounded concurrency. Worst case is a handful of jobs in flight at once, which is deliberate; the LLM and rendering work is memory-hungry.

The LLM layer is three jobs, not one

People assume "AI backend" means one big prompt. Here it's three narrow, structured-output jobs, each with temperature: 0 and a strict schema so the model can only return shapes I can actually use:

1. Extract — read the raw document and emit a canonicalized structured representation (sections, tables, fields) plus a quality score. Low confidence routes the row to a human review state instead of silently shipping bad data.
2. Enrich the form schema — when a new template is onboarded, the LLM turns its blank fields into a typed schema: each field gets a type, a description, and a list of aliases. I deliberately over-include synonyms and abbreviations here so the next step can bridge messy real-world labels ("D.O.B." vs "Date of Birth" vs "Born") without me hand-maintaining a mapping table.
3. Map — given a person's extracted data and a form's schema, produce the filled values. One call per source document, run in parallel.

Onboarding a form is itself a nice trick: I parse the template for placeholder tokens like {{FIELD_NAME}}, where dots mean nesting and a numeric suffix means an array index, build a skeleton schema from the field paths, and only then hand it to the LLM to enrich. When a batch of fields is too big for the model's structured-output limits, the enrichment recursively splits the batch and retries, preserving structure exactly.

Filling a form is a merge problem

Mapping each source document separately gives me several partial answers for the same form. Folding them together has a clear rule:

• arrays concatenate,
• objects deep-merge,
• scalars take the first non-empty value.

Because of that last rule, ordering the sources matters — the most authoritative document goes first, so its values win ties. After merging I coerce the result back to the schema shape, compute which fields are still missing and an overall confidence score, then render the values back into the original Excel/Word template and upload the finished file to S3. The client gets a presigned URL and downloads it directly.

The knowledge graph: rebuild, don't patch

Every successful extraction also kicks off a graph sync into Neo4j, projecting the person and their documents into typed nodes and relationships. The counterintuitive choice: each sync deletes the person's entire derived subgraph and rewrites it from scratch rather than trying to diff-and-patch.

Patching a graph incrementally is a swamp of edge cases — stale edges, orphaned nodes, partial updates. A clean rebuild is trivially correct: the graph always reflects the person's latest documents, full stop. The cost is that you must never run two rebuilds for the same person at once, or the delete-then-write steps race. So this one queue is pinned to concurrency 1. Correctness for cheap, at the price of throughput on the one path that doesn't need it.

Two processes, because memory is the bottleneck

The API and the worker ship from one repo but run as two PM2 processes that can live on one box or two. The split exists for one reason: the worker is the memory-heavy half — concurrent multimodal LLM calls, whole files buffered in RAM, Excel/Word parsing — and I didn't want a worker running out of memory to take the API down with it. In production they're separate boxes; the worker has no public inbound at all. Same code, same config, driven entirely by environment variables.

What generalizes

Strip away the documents and the forms and what's left is a pattern I'd reach for again:

• Async-first with status as the contract. If work is slow, return 202 and a row id, and let clients poll. Your API stays responsive no matter how slow the backend gets.
• Make handlers idempotent at the database, not with locks. An atomic conditional claim is simpler and more robust than any distributed lock, and it self-heals stale state for free.
• Mind the timer ordering when you extend message visibility. One inequality decides whether long jobs survive.
• Narrow LLM jobs with strict schemas beat one mega-prompt — they're testable, debuggable, and each can fail into a review state instead of corrupting data.
• Prefer clean rebuilds over incremental patches when correctness matters more than throughput, and serialize the path that demands it.

None of the hard parts were the AI. The AI was the easy, fun bit you bolt on at the end. The architecture is what makes it hold together.