Chatbot and Conversation Evaluation

Multi-turn changes everything

A single-turn eval asks "is this response good for this question?" A conversation eval asks "is this conversation going well?" The second is much harder, because quality depends on context, memory, consistency, and trajectory.

Conversation-specific metrics

Turn-level quality — each response evaluated independently (still useful).
Context retention — does the bot remember what was said earlier?
Topic coherence — do later turns follow logically from earlier ones?
Repetition — does the bot repeat itself?
Conversation completion rate — what fraction of multi-turn flows reach a successful end state?
User-effort signals — turns to resolution, escalation rate, abandonment rate.

Two evaluation patterns

Frozen-conversation eval — pre-recorded conversations replayed against the new system. Fast, cheap, deterministic. Misses behaviors that depend on the bot's actual responses.
Live-simulation eval — a "user simulator" (another LLM) holds a conversation with your bot. Captures emergent behavior. Slower, more expensive, more realistic.

Principle: Run both frozen and simulated conversation evals. Frozen catches regressions cheaply; simulated catches what frozen can't see.

Production conversation telemetry

Track turns-to-resolution, abandonment rate, and escalation rate as live metrics. They are user-effort proxies — when they degrade, the conversation experience has gotten worse, even if turn-level quality looks unchanged.

Code

Frozen-conversation replay eval·python

def replay_conversation(conversation, bot):
    """Replay a recorded conversation, scoring each new bot response."""
    history = []
    scores = []
    for turn in conversation:
        if turn["role"] == "user":
            history.append(turn)
        else:  # role = assistant
            new_response = bot.complete(history)
            score = judge(turn, new_response, history=history)  # compare to recorded
            scores.append(score)
            history.append({"role": "assistant", "content": new_response})
    return scores

User simulator for live multi-turn eval·python

USER_SIMULATOR_PROMPT = """
You are role-playing a user trying to accomplish this goal: {goal}
Be realistic — sometimes ask follow-up questions, sometimes get confused.
End the conversation when you are satisfied or give up.

Keep messages short (1-2 sentences). Reply ONLY with what the user would say.
"""

def simulate_conversation(goal, bot, simulator, max_turns=10):
    history = []
    for turn in range(max_turns):
        sim_msg = simulator.complete(USER_SIMULATOR_PROMPT.format(goal=goal), history=history)
        if sim_msg.strip().lower() in ("thanks", "goodbye", "that's enough"):
            break
        history.append({"role": "user", "content": sim_msg})
        bot_msg = bot.complete(history)
        history.append({"role": "assistant", "content": bot_msg})
    return history

Chatbot and Conversation Evaluation

Multi-turn changes everything

Conversation-specific metrics

Two evaluation patterns

Production conversation telemetry

Code

External links

Exercise

Progress

Comments 0