Daniel Concepcion
← All Thinking

AI evaluation

Different models, different roles

Separating execution and review improves focus; using a different model may add useful difference, but task-specific evaluations decide which model belongs in each role.

View on LinkedIn

One spec in my personal agentic AI project passed 4 rounds of AI review and was approved for implementation. Then a model from another provider found 24 more issues in it.

That left me with a question I couldn't answer: why had the earlier reviews missed them?

I mentioned this in my first post: recurrent issues had survived one model doing the work and another reviewing it.

At first I thought splitting the roles was the important change: one agent to execute, another to challenge the result against the evidence.

  • That separated the jobs. Model choice was still its own decision.

  • The same model can do two different jobs when the instructions and acceptance criteria change. A different model may bring different strengths and blind spots.

So I looked for research that might explain part of it.

  • A recent study found self-preference even with verifiable rubrics. Some judges were more than 50% more likely to mark a failed criterion as satisfied when reviewing their own output.

  • An Apple study counted both what review fixed and what it broke. Results changed with the reviewer model and prompt. Its GPT-4o and o3-mini comparison is already dated.

I didn't find a definitive answer. The studies gave me hypotheses. (I left their links in the first comment)

For this project, the multi-provider setup is now part of the foundation. Reviews have been better when the executor and reviewer use models from different providers. That is still my current feeling. I haven't run a controlled comparison yet. But I can change the model behind a role without rebuilding the workflow.

In the day-to-day work on this personal project, I use several coding agents: Claude Code, Codex and Pi, with OpenRouter as a gateway to open-weight models such as DeepSeek, GLM and Kimi. I follow the same pattern by passing work between models.

My next step is to test that observation with data. I need evaluations built from the work I actually do. Coding and research may need different model pairings and evidence.

I expect model rankings to move as versions change. I want the role to stay stable while I test and replace the model behind it.

How are you deciding which model gets which role?

#AgenticAI #LLMEvaluation #ModelSelection #LearningAIByUsingAI

Infographic titled ‘The review passed. The issues remained.’ comparing four passed AI review rounds with 24 additional issues found when a model from another provider reviewed the same specification.
Illustration titled ‘The same work. A different angle.’ showing two desk lamps illuminating different parts of one document, suggesting that different reviewers can expose different blind spots.
Diagram titled ‘One workflow. Two different jobs.’ separating an executor that produces the artifact from a reviewer that challenges the evidence, while keeping models replaceable within stable roles.
Research summary titled ‘A clue. Not a verdict.’ noting self-preference in self-review, the possibility that review can both correct errors and damage correct work, and the effect of reviewer model and prompt.
Evaluation framework titled ‘Stable roles. Replaceable models.’ pairing models by task while measuring the accepted result by quality, cost and elapsed time.

Sources and links

Companies, products and research

Claude CodeCodexOpenRouter