Scaling Chatbot Intelligence: Our Path to Multi-Agent AI

Every support chatbot has to solve the same problem before it can help anyone. It has to read a message written in free text, often in a hurry and in more than one language, and turn it into something the rest of the system can act on. At Cabify we have built that step in three different ways, and each version taught us something about where a language model should make decisions and where it shouldn’t.

Our automation stack is built around two parts. The first is an intent classification system, in charge of interpreting user messages and identifying what the user needs. The second is a rule engine, where scenarios are defined as groups of branches, each with conditions and a set of actions tied to a specific outcome. The rule engine decides what we can do about a problem. The classification system decides which problem we are looking at, and in Support Engineering it is the part we have rebuilt the most.

This article follows that evolution. We started with Amazon Lex, moved to a single ReAct agent where every tool was an intent, and ended up with a set of one-shot agents driven by YAML configuration, which later became a platform other teams build on. Along the way, we learned to let the model decide what the user wants and to keep everything that happens next in code.

From Lex to a ReAct Agent

When we first started building our chatbot, our main goal was to deliver a functional solution quickly, and Amazon Lex was the obvious choice. Lex gave us exactly what we needed at the time: simplicity, speed, and flexibility. New use cases could be introduced without heavy engineering effort, and most of the configuration could be managed directly by operations teams.

As our automation needs expanded, what began as a handful of simple scenarios turned into dozens of overlapping configurations, each with its own conditions and exceptions. Even small updates could affect other scenarios, and every change required coordination between teams. At the same time, large language models (LLMs) were changing what was possible. When a user writes “I didn’t get what I paid for”, an LLM can infer frustration, intent, and possible next actions, while a system trained on sample phrases, like Lex, can only map it to whichever intent it happens to resemble.

Our first AI agent used ReAct (Reasoning + Acting). The model reasons in a loop, decides which tool to call, reads the result, and repeats until it has an answer. In this design, every tool was an intent. Each tool had a name, a description telling the model when to use it, and a handler that ran the deterministic logic (often an automation scenario in the rule engine) and returned instructions plus a message template. The model then turned that into the final answer, in the user’s language.

We first ran the agent in shadow mode, processing real messages in parallel with Lex without affecting users, which allowed us to compare classifications and refine prompts and tool descriptions. We then enabled it conversation by conversation behind a feature flag, and later configured it to fall back to Lex when it couldn’t resolve a request. Where Lex needed a sample phrase for every way of saying something, the agent could understand messages like “I want to stop my trip” or “don’t need the ride anymore”, which broadened what we could automate.

As the number of intents grew, so did the agent’s responsibilities. One agent was analyzing messages, choosing tools, processing outputs, managing conversations, and even translating responses. With so much in one place, small reasoning errors could lead to unpredictable results, and adding a new intent often meant adding a tool or making an existing one more complex.

Splitting Responsibilities Was Not Enough

To give each responsibility its own place, we moved toward a multi-agent architecture with four specialized agents:

• Classification Agent → Detects the user’s intent and extracts relevant metadata.
• Conversation Agent → Manages the dialogue and generates responses.
• Translation Agent → Translates responses when the user writes in another language.
• Output Guardrail Agent → Validates responses before they reach the user, blocking anything that could violate privacy or compliance rules.

The harder question was what should control the flow between them. We answered it with a proof of concept, and we defined the success criterion in advance. The new design had to pass at least 80% of our existing acceptance tests.

Our first approach kept ReAct as the orchestrator. The Conversation Agent led the interaction and used the classifier and the component that executes intents as tools, deciding on its own when to run each one. In testing, this design proved highly non-deterministic. After executing an intent, the agent could start another reasoning cycle, and the sequence of calls varied from one run to the next. Each component performed well in isolation. Splitting responsibilities had not removed the unpredictability. It had only moved it into the orchestration.

Our second approach replaced it with one-shot agents. Each one receives input, runs a single prompt, and returns a structured response. The agents never call each other. They read and update a shared conversation session, and the flow between them is written in code. The model decides what the user needs, and the application decides what runs next.

Inside the Classification Agent

The Classification Agent’s prompt is built from the usage_instructions of every intent available to that bot, the conversation so far, and the new message. It doesn’t call tools or loop. It answers once, with a small JSON object:

{
  "intent": "LOST_ITEM",
  "reply_language": "english",
  "selected_option": "",
  "score": 0.92,
  "country": ""
}

Besides the intent, the same call detects the language to reply in, which later decides whether the Translation Agent runs at all. It also maps a free-text answer to one of the options the bot offered in the previous step and returns the country the request applies to.

Because the output is structured, everything after it is plain code that doesn’t have to trust the model. An intent ID that doesn’t exist resolves to an UNDEFINED intent instead of breaking the flow. A failed call or a response that isn’t valid JSON hands the conversation to a human agent. And messages that genuinely can’t be classified have their own catch-all intent, so the model has a legitimate answer instead of forcing the closest match.

As in our previous change from Lex to ReAct, we launched the Classification Agent in shadow mode in production. The ReAct agent still answered every user, while the classifier ran alongside it and we logged whether it chose the same intent that ReAct had executed. Once the agreement was good enough, a feature flag moved users to the new architecture gradually, by user and country, and about a month later we deleted the ReAct code, more than 3,000 lines of it.

Moving Intents into Configuration

Smaller agents made each step easier to reason about, but we still needed a way to keep up as the number of intents grew. We moved each intent into a YAML file with two parts: a description that the Classification Agent uses to decide when to choose it, and a list of actions. Here is a simplified example of two chained intents (the intents and the scenario are illustrative):

# What the Classification Agent can choose
id: "LOST_ITEM"
usage_instructions: |
  Use this intent when the user left a personal item in the vehicle after a ride.

  MATCH:
  - "I left my phone in the car"
  - "Me olvidé la mochila en el taxi"

  DO NOT MATCH:
  - Items damaged during the ride (use DAMAGED_ITEM)

workflow:
  actions:
    - step: "default"
      option: "default"
      next_intent: "LOST_ITEM_REPORT"

# Never classified: only reachable through next_intent
id: "LOST_ITEM_REPORT"
automation:
  scenario_id: "REPORT_LOST_ITEM"
  actions:
    - results: ["REPORT_CREATED"]
      static_messages:
        - message: "We've registered your report and will let you know as soon as we have news."
      next_status: "Solved"
    - results: ["default"]
      next_status: "Transfer"

The usage_instructions are the description the Classification Agent reads, including counter-examples that steer it away from neighboring intents. Intents that run an automation, like LOST_ITEM_REPORT, can’t have them, so the model never sees them. It can only pick what the user wants, while a scenario in the rule engine decides what we can actually do about it.

Each possible outcome maps to a reply and a conversation status. A required default action covers anything the scenario returns that we didn’t anticipate, as well as a failed call to the rule engine. Anything that breaks elsewhere in the flow hands the conversation to a human agent. The diagram below shows how a message flows through the agents and the generic Intent Processing System that executes these actions. Each intent file is used twice: the Classification Agent reads its description, and the Intent Processing System runs its actions. Only the blue steps call a language model. Everything in gray is plain code.

This setup externalizes intent creation. A new intent is a YAML file and its tests, with no code changes, so people outside engineering can make these changes too. It also let us migrate Lex intents one at a time.

The chart shows what that looked like during the migration. Each panel is one scenario, and each jump marks the day its Lex intent was migrated to a YAML intent. From then on, the classifier recognized far more of the messages that belonged to that scenario and routed them straight into its automation. There were exceptions, though. Low-traffic scenarios like Change journey details didn’t change, because Lex was already identifying them correctly.

Testing a Non-Deterministic System

Because AI responses are inherently non-deterministic, we designed an acceptance testing framework that uses AI-based validation. Instead of relying on exact word matching, each test case describes a conversation and the behavior we expect in plain language, and a separate model grades the agent’s response against that rubric, returning a pass or fail with its reasoning. A test survives rephrasing, but still fails when the intent, meaning, or resolution is wrong. The suite grew with the intents, from roughly 100 test cases when we started rolling out the multi-agent architecture to nearly 190 by the time intents had moved to YAML.

The Output Guardrail Agent came from a concrete ReAct risk. An agent that reasons in a loop can leak fragments of that reasoning, such as lines starting with Thought: or Action:, into the reply. The guardrail combined a cheap keyword filter for those markers with a model that checked every response against our privacy policy, and any failure handed the conversation to a human agent. Together, the acceptance tests and the guardrail gave us the confidence to roll out the new architecture to every user.

Where We Stood in 2025

By 2025, our classification system ran in a hybrid setup. The new multi-agent architecture handled most intents, while Lex remained as a fallback when the agents couldn’t confidently resolve a request.

The difference between the two systems was striking. The AI-powered classifier was achieving a resolution rate of around 67%, compared to roughly 50–55% when Lex was the only system in place. These numbers aren’t directly comparable, since at the time Lex was measured our rule engine included far fewer automation scenarios.

The picture becomes clearer when we focus on the scenarios shared by both systems. Some scenarios were handling more than twice as many automated contacts as before, others over ten times as many, and a few more than forty times as many.

Configuration also changed how quickly we could move. We took the new architecture’s coverage of rider-facing intents from 60% to 100% in just two weeks, something that previously required several weeks of manual configuration and coordination across teams.

From One Bot to a Platform

Once the multi-agent architecture was running for riders, drivers were the natural next step. We made the service aware of which bot it was serving. Each request now arrives at an endpoint for a specific bot, and the service loads only that bot’s intents. With that in place, the first version of the driver bot was just a set of YAML intents and their acceptance tests.

Then we split the code from the configuration. The core repository kept the agents and the processing pipeline and published a base image. Each bot moved to its own repository, and its Dockerfile became almost trivial:

# A bot repository's Dockerfile, simplified
FROM chatbot-core:latest

LABEL BOT_TYPE=rider

COPY ./config/intents /config/intents
COPY ./config/documents /config/documents

A bot is now its intents, its FAQ documents, and its acceptance tests on top of the base image. Teams can change a bot without touching the core, and the core can evolve without touching any bot.

When the Tests Stopped Testing

Moving the acceptance tests into the bot repositories introduced the most instructive bug of the whole migration. Each test describes a conversation and the assertions we expect:

tests:
  - name: "should transfer to an agent when the user asks for a human"
    conversation:
      - request:
          message: "I want to talk to a human"
        asserts:
          - type: "llm_rubric"
            value: "The AI should tell the user they are being transferred to a human agent."

The test runner, written in Go, expected those assertions under a different key:

type ConversationEntry struct {
    Request          Request          `yaml:"request"`
    ExpectedResponse ExpectedResponse `yaml:"expected_response"` // the test files used "asserts"
}

By default, the YAML decoder ignores fields it doesn’t recognize. Every test sent its messages to the bot, found zero assertions to evaluate, and passed. For about three weeks, the pipeline stayed green while our main safety net was effectively switched off.

As soon as we fixed the key, several suites failed in both bot repositories. Some failures came from tests that no longer matched the current behavior. Others pointed to real problems in the configuration. While cleaning up, we also found intents whose counter-examples referred the classifier to intent IDs that had since been renamed.

A few days later, we made the runner decode test files strictly and validate every suite before running it. A misspelled key now fails on the first run instead of passing in silence. A test that cannot fail is a bug in itself.

Looking Back

We started with a classification problem, turning a user’s message into the right support flow. Solving it took us through three architectures, but the most lasting change was how we could extend the system. New intents became configuration and tests rather than new tools for an agent to coordinate. Other teams could build their own bots on the same core instead of starting from scratch.

Instead of a chatbot that could decide everything for itself, we ended up with a system where language models handled interpretation, code controlled execution, and teams could add support flows without redesigning either. We set out to improve how one chatbot understood users. We ended up changing how we build chatbots.