I don't trust one AI model's answer to anything that matters. I'll ask ChatGPT something, then paste the same question into Claude, then Gemini, just to see where they agree and where they don't. So I built the tool that does that for me automatically. Ask one question, watch three models answer it side by side in real time, then a fourth model reads all three and writes one final answer.
That fourth model was the whole design decision, really. My first instinct was to have one of the three council members also write the summary. But that's like asking a debate participant to also be the judge, they're going to unconsciously favor their own argument. So the synthesizer is a separate model that never appears as a council member itself. It costs more to run a fourth model, but I think it's worth it for an honest comparison.
I built it in phases instead of trying to do everything at once. First was just login working end to end, nothing smart yet. Then the actual council: a workflow that fans a question out to three models in parallel, streams each answer into its own pane as it's generated, and waits for all three before triggering the synthesis pass.
That streaming piece is where I hit my first real bug. Every so often a pane would just sit there showing nothing, even though the database showed the model had already finished answering. Took me a while to track down. The problem was timing, if a fast model finished before the browser had even opened its connection to receive updates, the answer got published into the void and nobody was listening yet. Fixed it by opening that connection the moment someone logs in instead of waiting until they ask a question, so nothing fast can outrun it.
After the council I built a separate admin tool for managing which models are on the roster, editing prompt templates, and checking costs, without redeploying code every time I want to swap a model. Then multi-turn conversations, so a follow-up question actually builds on what came before instead of starting from scratch.
Most recently I added content moderation, and that's where the second bug showed up. I tested it with a question containing a phone number, figuring it would pass through fine. Instead it got rejected outright, like I'd asked something dangerous. Turned out my code was treating two different things the filter can do, quietly masking a phone number, or actually blocking harmful content, as the same signal. Once I taught the code to tell those apart, the phone number sailed through masked and fine, and only genuinely bad content got stopped.
The tool works today. Log in, ask a question, watch three models think out loud at once, and a fourth quietly reconciles them. What's left is less exciting: usage limits, a way to browse past conversations, real monitoring. Nothing that changes what the tool does, just what keeps it running well once more than one person is using it.
← Back to Articles