Watching the tapes

Every conversation with Caissa is traced — the question, the engine analysis it saw, the tools it called, and the answer it gave. This month we sat down and read all of it. Not a sample. All of it.

The good news: when Caissa answered, it was almost always right, and the safety nets we wrote about last time were catching real mistakes before they reached you. The bad news: the worst failures weren't wrong answers at all. They were moments where Caissa refused to engage, and our test suite never caught them because they weren't chess mistakes — they were product mistakes.

Four failures, four fixes

1. "what should i do now" → "I'm here to help you with chess!"

A player in the middle of a live game asked the most natural coaching question there is, and got brushed off with a canned "ask me about chess!" reply. Why? A little pre-filter decides whether a question is about chess at all, and it misfired on a question with no chess keywords in it.

The fix is embarrassingly obvious in hindsight: if you're asking a question while sitting at a chess board mid-game, it's a chess question. The filter now only rejects clearly off-topic requests (no, Caissa still won't write your homework) and otherwise assumes you're talking about the position in front of you.

2. Asking about a finished game → nothing

Someone was checkmated, asked "what should I have done better in this game?", and got nothing useful. When a game is over there are no moves left to analyze, and Caissa's briefing for the position was essentially two lines: "Checkmate. You lose." Not exactly coaching material.

Now, when a game has ended, Caissa gets the full move history and game review data in its briefing, plus explicit instructions on how to walk back through the game and find where things went wrong. The post-mortem is often the most valuable conversation in chess — it deserved better than a shrug.

3. "I played Be2, was it a good square?" → "Which move do you mean?"

A player asked about the move they had just played. The board was already showing the position after the move, so when Caissa checked "Be2" against the current position, it came back illegal — because it had already been played. Caissa punted and asked a clarifying question. The player had to rephrase to get their answer.

Now, when you ask about a move that isn't legal right now but was played earlier in the game, Caissa automatically figures out which move you mean, rewinds to that position, and analyzes it there.

4. Plan questions got opening-book recitals

Questions like "I'm Black, what's a solid plan here?" were the least reliable category in our traces. The reason: Caissa's briefing was full of concrete move rankings but had almost nothing positional in it — so when you asked about plans, the model reached into its memory of opening theory and started reciting. Sometimes that theory fit your position. Sometimes it didn't.

Caissa's briefing now includes a deterministic positional repstaort computed straight from the board: pawn structure (doubled, isolated, passed pawns), open and half-open files, king safety and pawn shields, and which pieces are still undeveloped. And the rules changed: plans must be built from those facts and the engine's actual lines — reciting memorized theory as concrete advice is off-limits.

Did it work? We measured.

Before shipping any of this, we ran the full evaluation suite — 200 real Lichess puzzle positions plus 30 deliberately nasty adversarial cases (deep tactics, tricky endgames, positions designed to bait the model into making things up) — on both the old and new versions, with the same graders.

The results, honestly framed:

• Legality and material claims: 100% on every run, before and after. The validators from the last post are still doing their job — no illegal moves, no false material claims reached the "send" button in 230 test conversations.
• When Caissa answered, it named the verified best move in 197 out of 197 lichess positions. (Three positions hit a temporary API outage mid-test and got a polite "I'm having trouble reaching my analysis engine" message instead — which is itself one of this release's fixes. Previously that failure mode was a blank error.)
• On the adversarial set, the LLM-judge scores went up across the board — accuracy 0.97 → 0.98, helpfulness 0.91 → 0.93. The single biggest jump was the exact case type fix #4 targets: a quiet position with no tactics where the only good answer is a real plan. That probe went from 0.55 to 0.93.

And the honest caveats, because numbers without caveats are marketing: the "names the best move" metric measures whether Caissa faithfully relays the engine's verified analysis — that's the design (Stockfish thinks, Caissa explains), but it's a narrower claim than "Caissa is always right." The judge scores come from an LLM grading an LLM; we treat them as directional, not gospel. The real ground truth is you telling us when something's wrong.

What's still on the list

The trace review also showed us things we haven't fixed yet: a proper "review my whole game" experience (the current fix makes the conversation possible, not effortless), and answers still take longer than we'd like — verification before shipping costs seconds, and we're not willing to trade it away for speed.

"I have suggestions!"

You know the drill: sebastian@soleinnovations.com. Half of this release exists because real players asked real questions our test suite never thought of. Keep them coming.