How my AI workflow led me to a team of specialists
The next model I'm adding to my AI factory answers bounded questions and nothing else. The process asked for that tool, and it reminded me of something I knew from other processes long before: AI is bigger than LLMs.
The factory runs my development and my writing with agents, and both are LLM sweet spots. LLMs draft and review the code and the articles, and a model from a second vendor reviews the first one's work. I wrote about that in Different models, different roles.
My agents already search my knowledge base with 2 small models on a machine of my own, one that finds text (an embedding model) and one that ranks it (a reranker).
The next step is routing by complexity. A cheap judgment model will decide how hard each step is and which model gets it. It will also help the deterministic engine, the software that applies the factory's rules, with the bounded questions a rule can't settle, such as whether a passage supports a claim.
The same shape in other processes
Other processes I've been part of needed different AI tools, and had them before LLMs existed. I'd split each one this way today, kept to its main steps; the real ones have many more systems behind them.
Onboarding a customer. OCR reads the documents and image recognition checks the ID against the face. Behind that, software verifies the ID with the government registry, runs sanctions and AML screening, classifies the customer in the CRM, enrols them in partner programmes and creates the account, while scoring models rate risk, credit and fraud.
A judgment model decides whether everything matches and whether anything is missing. An LLM explains the gap to the customer and asks for what's needed, and a person handles the exceptions.
Customer support. A judgment model classifies each request and routes it. An LLM answers the ones that need language.
Software executes the actions it's allowed to, such as a refund within limits or an address change. Anything outside the limits goes to a person.
The call centre. Speech models listen and speak, and an LLM carries the conversation. A judgment model watches intent and risk (is this a complaint, a vulnerable customer, a fraud attempt?) and pulls a person in.
Fraud and credit risk. In banking, the scoring model was there to protect the customers, by identifying the operation's inherent risk based on previous behaviour, in real time and long before LLMs existed. A rules engine held the hard limits.
Today I'd keep the scoring model predicting and the investigator deciding, and add a judgment model to triage the alerts and an LLM to write the case narrative.
IT operations. Anomaly detection on time series spots the problem, and a judgment model sets the severity and routes it. An LLM drafts the incident summary and walks the runbook, software executes the safe actions, and a person takes the consequential ones.
The job decides the tool
Each step of a process has a job: perceive, find, predict from history, allocate under constraints, understand and generate content, decide a bounded question, apply an exact rule, own the consequence. That's 8 jobs. LLMs take 1 of them.
The other 7 belong to specialists. OCR, image recognition and speech models perceive; embeddings and rerankers find; tabular and forecasting models predict from history; optimisation allocates; a judgment model decides the bounded question; software applies the exact rule; a person owns the consequence.
A multimodal model can cover several jobs in this map. Cost and quality per task decide which ones it gets.
So the redesign of a process starts by naming the job in each step, and deciding what stays, what changes and what goes. Only then comes the tech.
Cost per task and quality per task
When 2 tools could both do a step, I choose with 2 numbers: what an accepted result costs and what quality it reaches. Cost per accepted task includes the retries, the review and the human repair.
Routing by complexity means sending each step to the cheapest tool that reaches the quality it needs.
What to be careful with
Perception models read clean documents and clear speech well. Poor scans, accents and background noise can lead to errors, and a speech model can invent a phrase during a pause. Confidence scores can help route uncertain results to a person.
Retrieval finds what is similar, and similar can be stale or wrong. A reranker sharpens the match and a freshness check catches what's out of date.
Prediction models work well when tomorrow resembles the history they learned from. When the world moves they drift, and someone has to own the monitoring and the retraining.
Optimisation can find the best allocation within the model you give it. A missing constraint can produce a plan that breaks a real-world limit.
LLMs handle unstructured text and generate explanations. They can guess when unsure and vary from run to run, so I keep them off the deciding step or check their output.
Judgment models answer bounded questions cheaply, with a probability attached, and Jev from TypeSafe is the one I've followed. They can pick a wrong option with confidence. On unfamiliar tasks that confidence can overstate their accuracy, so "none of these" has to be an option and the threshold gets tuned per task with your own labelled cases.
Software applies written rules exactly and cheaply. Unhandled cases can fail silently. A person is slow and inconsistent, and the only one who can be accountable, which is why every exception ends with one.
I think the field is moving the same way: a frontier LLM for the open-ended work, and a fleet of specialists around it, many of them small open-weight models tuned to do 1 thing well. The steps that are a yes or a no are the first ones I'd hand to a specialist.
Have you reviewed who does what in your process?