Building a Production AI Customer Support Agent: FastAPI, LangGraph, RAG & Tool Calling

Most systems that call themselves an "AI customer support agent" are a thin wrapper around a single call:

User → LLM → Response

That's fine for FAQs — refund windows, shipping cutoffs, how to reset a password. It falls apart the moment a customer asks something that requires looking anything up:

My payment was deducted, but my order is still showing as pending. Can you check it?

The model has no way to answer that honestly. It doesn't know this customer's order, it doesn't know their payment status, and if you don't constrain it, it will confidently make something up rather than say "I don't know." A production support agent needs four things a bare chat completion doesn't give you: a way to search your own knowledge base instead of guessing at policy, a way to call your own backend for customer-specific facts, a memory of the conversation that survives more than one HTTP request, and — the part most demos skip — a way to stop and hand off to a human when it's genuinely out of its depth.

This is the stack I reach for when a client needs exactly that: Python, FastAPI, LangGraph, RAG, Redis, and tool calling, wired together as one stateful agent rather than a single prompt.

What actually makes this an "agent"

The distinction that matters isn't the framework name, it's who makes the decision. A workflow hardcodes it:

if "payment" in message:
    search_payment()

An agent hands that decision to the model, and constrains what it's allowed to do about it. For the payment-pending message above, the agent needs to understand the intent, pull the relevant policy from the knowledge base, call an order-status tool, call a payment-status tool, reconcile the two results against each other, and then either answer or escalate. None of that sequence is fixed in code — the model decides which tools it needs and in what order, and the application decides what those tools are allowed to touch. That combination of reasoning, tool access, grounded knowledge, persistent state, and a hard stop for escalation is what separates an agent from a chatbot.

Tech stack

A production-leaning stack for this looks like:

• FastAPI — the API surface the frontend or channel integration talks to.
• LangGraph — orchestrates the agent as a stateful graph, not a single prompt.
• LangChain — tool definitions and the LLM client.
• A vector database — pgvector on Postgres, Qdrant, Pinecone, or Weaviate for retrieval.
• Redis — short-lived conversation and session state.
• PostgreSQL — durable business data: orders, payments, tickets.
• An OpenAI-compatible LLM — the reasoning engine behind the agent node.

I'll keep the vector-search interface abstract below — swap in whichever of those you already run.

Project structure

customer-support-agent/
├── app/
│   ├── main.py
│   ├── graph.py
│   ├── state.py
│   ├── agents/
│   │   └── support_agent.py
│   ├── tools/
│   │   ├── orders.py
│   │   ├── payments.py
│   │   └── escalation.py
│   ├── rag/
│   │   └── retriever.py
│   └── memory/
│       └── redis.py
├── requirements.txt
├── Dockerfile
└── docker-compose.yml

Dependencies are deliberately minimal — this doesn't need a heavyweight agent framework on top of LangGraph:

pip install fastapi uvicorn langgraph langchain langchain-openai redis pydantic

Defining the agent's state

LangGraph is built around a shared state object that every node in the graph reads from and writes to. For a support agent, that state needs to carry the conversation so far, who the customer is, whatever the retriever found, and whether the graph has decided a human needs to take over:

from typing import TypedDict, List, Any


class SupportState(TypedDict):
    messages: List[Any]
    customer_id: str
    retrieved_context: str
    tool_results: List[Any]
    needs_human: bool

That state flows through the graph roughly like this:

START → understand request → retrieve knowledge → decide/execute tools → generate response → END

The RAG layer: grounding the agent in real policy

Support agents need access to internal knowledge — refund policy, shipping cutoffs, cancellation rules, subscription terms, account recovery steps. The wrong way to handle that is stuffing all of it into the system prompt. The right way is Retrieval-Augmented Generation: embed the query, search a vector store, pull back the handful of documents that are actually relevant, and only then let the LLM answer.

A minimal retriever, with the vector-search internals abstracted out:

# rag/retriever.py

def search_knowledge_base(query: str) -> str:
    """
    Replace this with pgvector, Qdrant, Pinecone, etc.
    In production: embed the query, run a similarity
    search, return the top-k documents (optionally reranked).
    """

    documents = [
        {
            "content": """
            If a payment is successful but an order remains pending,
            the order may take several minutes to update.
            If the status remains pending after 30 minutes,
            customer support should investigate the transaction.
            """
        },
        {
            "content": """
            Refunds for cancelled orders are normally processed
            within 5-7 business days.
            """
        },
    ]

    return "\n\n".join(doc["content"] for doc in documents)

The important part in production isn't this stub, it's the pipeline behind it: query in, embedding out, top-k documents back, optionally through a reranker before they ever reach the LLM.

Giving the agent tools, not database access

The agent should never touch your database directly. It gets a small, explicit set of functions — each one a controlled boundary the application owns, not the model.

# tools/orders.py

from langchain_core.tools import tool


@tool
def get_order_status(order_id: str) -> dict:
    """Get the current status of an order."""

    # In production this calls your backend API.
    return {
        "order_id": order_id,
        "status": "pending",
        "amount": 4999,
    }

# tools/payments.py

from langchain_core.tools import tool


@tool
def get_payment_status(order_id: str) -> dict:
    """Check payment status for an order."""

    return {
        "order_id": order_id,
        "payment_status": "successful",
        "transaction_id": "txn_12345",
    }

# tools/escalation.py

from langchain_core.tools import tool


@tool
def escalate_to_human(customer_id: str, reason: str) -> dict:
    """Open a support ticket for a human agent."""

    # Production: create a Zendesk/Intercom/Salesforce ticket.
    return {
        "success": True,
        "ticket_id": "SUP-12345",
        "message": "Ticket created successfully.",
    }

With these three defined, the LLM has everything it needs to decide when a question is answerable from policy alone and when it needs to check a specific customer's data.

Binding tools to the LLM

from langchain_openai import ChatOpenAI

from app.tools.orders import get_order_status
from app.tools.payments import get_payment_status
from app.tools.escalation import escalate_to_human


tools = [get_order_status, get_payment_status, escalate_to_human]

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
llm_with_tools = llm.bind_tools(tools)

Binding doesn't give the model the ability to execute anything. It gives the model the ability to request a tool call with a name and arguments; your application is what actually runs it.

The LangGraph agent loop

This is the core of the system: a graph with an agent node, a tool-execution node, and a conditional edge that routes between them until the model has enough information to answer.

# graph.py

from langgraph.graph import StateGraph, START, END
from langgraph.prebuilt import ToolNode, tools_condition

from app.state import SupportState


def agent_node(state: SupportState):
    response = llm_with_tools.invoke(state["messages"])
    return {"messages": state["messages"] + [response]}


tool_node = ToolNode(tools)

builder = StateGraph(SupportState)
builder.add_node("agent", agent_node)
builder.add_node("tools", tool_node)

builder.add_edge(START, "agent")
builder.add_conditional_edges(
    "agent",
    tools_condition,
    {"tools": "tools", "__end__": END},
)
builder.add_edge("tools", "agent")

graph = builder.compile()

tools_condition is what makes this a loop instead of a straight line: after every agent turn, it checks whether the model's response requested a tool call. If it did, execution routes to the tool node, the result gets appended back onto messages, and control returns to the agent — which now has the tool's output and can either call another tool or produce a final answer. If it didn't, the graph ends.

Wiring RAG into the graph

Add a retrieval node in front of the agent, and the graph becomes:

START → retrieve knowledge → agent ⇄ tools → response → END

def retrieve_node(state: SupportState):
    user_message = state["messages"][-1].content
    context = search_knowledge_base(user_message)
    return {"retrieved_context": context}

Then the retrieved context gets injected straight into the system prompt for that turn:

from langchain_core.messages import SystemMessage

SYSTEM_PROMPT = """
You are a customer support AI agent.

Rules:
1. Use the knowledge base when answering policy questions.
2. Use tools when customer-specific information is required.
3. Never invent order or payment information.
4. Escalate when the issue cannot be safely resolved.
5. Keep responses concise and professional.
"""


def agent_node(state: SupportState):
    context = state["retrieved_context"]

    messages = [
        SystemMessage(content=f"{SYSTEM_PROMPT}\n\nKnowledge Base:\n{context}"),
        *state["messages"],
    ]

    response = llm_with_tools.invoke(messages)
    return {"messages": state["messages"] + [response]}

Rule 3 in that prompt is doing real work. Retrieval gives the model policy to ground on; tools give it customer-specific facts to ground on. Between the two, there's no reason for it to be inventing an order status.

Redis for conversation memory

Without memory, every message the customer sends is processed cold — the agent has no idea what was said two turns ago. Redis fixes that cheaply: it's a good fit for short-lived, per-conversation state, while durable customer and order records stay in Postgres where they belong.

# memory/redis.py

import json
import redis

redis_client = redis.Redis(host="localhost", port=6379, decode_responses=True)


def save_conversation(conversation_id: str, messages: list):
    redis_client.set(f"conversation:{conversation_id}", json.dumps(messages))


def get_conversation(conversation_id: str):
    data = redis_client.get(f"conversation:{conversation_id}")
    return json.loads(data) if data else []

Exposing it through FastAPI

# main.py

from fastapi import FastAPI
from pydantic import BaseModel

from app.graph import graph

app = FastAPI(title="AI Customer Support Agent")


class SupportRequest(BaseModel):
    conversation_id: str
    customer_id: str
    message: str


@app.post("/support")
async def support(request: SupportRequest):
    result = graph.invoke(
        {
            "messages": [{"role": "user", "content": request.message}],
            "customer_id": request.customer_id,
            "retrieved_context": "",
            "tool_results": [],
            "needs_human": False,
        }
    )

    final_message = result["messages"][-1]

    return {
        "conversation_id": request.conversation_id,
        "response": final_message.content,
    }

uvicorn app.main:app --reload

Example request against that endpoint:

POST /support
{
  "conversation_id": "conv_123",
  "customer_id": "customer_456",
  "message": "My payment was deducted but my order is still pending."
}

Behind that single response, the graph did four to six model/tool round trips — the caller never sees the intermediate steps, only the final grounded answer. For the payment example, that answer might read something like: "Your payment was successful, but the order is still pending — that status can take a few minutes to update, and there's no indication the transaction failed." The key property isn't the wording, it's that every fact in that sentence came from a tool call, not from the model's imagination.

Human-in-the-loop

A production agent shouldn't try to resolve everything on its own. Fraud, refund disputes, account takeover, legal complaints, high-value transactions, and repeated failed resolutions belong with a person, not a model. That boundary is enforced with a routing function on needs_human:

def should_escalate(state: SupportState):
    return "human" if state["needs_human"] else "continue"

When it routes to "human", the graph calls escalate_to_human and hands the conversation off with a ticket ID instead of trying one more time. Agent autonomy has to be bounded by business rules the model doesn't get to override — this is where that boundary actually lives in the code.

Tool security: the model requests, your backend decides

The agent decides what it wants to do. Your backend decides whether it's allowed to do it, and what data it's allowed to touch.

The most common mistake in agent design is treating the LLM as a trusted caller. It isn't one — prompt injection from a malicious message, a hallucinated argument, or just an overly broad tool definition can turn "helpful agent" into "unauthenticated database client" fast. Every tool needs the same guarantees you'd put around any other internal API:

@tool
def cancel_order(order_id: str):
    # Validate ownership
    # Check cancellation policy
    # Call the order service
    # Record an audit event
    ...

Authentication, authorization tied to the requesting customer, input validation, audit logging, rate limiting, and idempotency where the action isn't safely repeatable. The model decides which tool to call and with what arguments; your application still owns the decision about whether that call is actually permitted.

RAG is more than a vector search call

A production retrieval pipeline usually earns its keep beyond a single embedding lookup. Query rewriting cleans up an ambiguous customer message before it's embedded. Running vector search and a keyword method like BM25 in parallel and reranking the combined results catches cases pure semantic search misses — policy documents are often short and keyword-heavy, which vector search alone can underweight. And metadata filtering (product line, region, language, policy version) keeps a query about Indian payment methods from surfacing a shipping policy written for a different market.

Observability

A single customer request here can trigger several LLM calls and several tool calls before anything comes back — that fan-out is exactly what makes agentic systems harder to debug than a single request/response API. Worth tracking per request:

• Latency, broken down by graph node
• Token usage and LLM cost
• Which tools were called, with what arguments, and whether they failed
• Retrieval quality — were the right documents actually returned
• Escalation rate and resolution rate over time

Tracing tools like LangSmith or Langfuse make this fan-out inspectable instead of a black box.

Evaluation

"Does the answer look reasonable" isn't a test suite. What's worth asserting on is whether the agent picked the right tool for a given question, since that's the decision most likely to go wrong silently:

TEST_CASES = [
    {"question": "Where is my order?", "expected_tool": "get_order_status"},
    {"question": "Can I get a refund?", "expected_tool": None},
    {"question": "My payment was charged twice.", "expected_tool": "get_payment_status"},
]

Beyond tool-selection accuracy, track argument accuracy, retrieval quality, final-answer correctness, whether escalation fired when it should have, latency, and cost — in that rough order of how often they actually break in practice.

Production architecture

A deployment of this shape scales the same way any stateless API does:

Customer
   │
   ▼
FastAPI  (horizontally scalable, behind a load balancer)
   │
   ├── Redis          conversation / session state
   ├── Vector DB       knowledge base for RAG
   └── LangGraph agent
         ├── LLM API
         └── Tool calls → internal REST APIs → PostgreSQL
                                              → Ticketing system (human escalation)

Locally, that's a two-service Docker Compose file — the API and Redis:

services:
  api:
    build: .
    ports:
      - "8000:8000"
    environment:
      REDIS_HOST: redis
      OPENAI_API_KEY: ${OPENAI_API_KEY}
    depends_on:
      - redis

  redis:
    image: redis:7
    ports:
      - "6379:6379"

Not every support flow needs an agent

A password reset doesn't need an LLM in the loop — it's API → reset password, full stop, and introducing a model there only adds latency and a new failure mode. But "my account was charged twice, my order hasn't arrived, and I want to know if I can get a refund" bundles three intents into one message and genuinely benefits from a model that can reason across them.

Agentic AI doesn't mean using an LLM everywhere. It means giving the model controlled autonomy exactly where dynamic decision-making earns its cost — and routing everything else straight to deterministic code.

The practical version of that is an intent router in front of the agent: known, single-step operations go straight to a deterministic backend call; anything ambiguous, multi-intent, or requiring judgment goes to the LangGraph agent. Most of the traffic a support system sees is the former. The agent earns its complexity on the smaller slice that actually needs it.

Key takeaways

A production-grade AI customer support agent isn't an LLM with a system prompt — it's a small distributed system with the LLM as one component inside it:

• FastAPI — the API layer
• LangGraph — stateful orchestration and the agent/tool loop
• RAG — company-specific knowledge, grounded and cited
• Tool calling — controlled, audited access to real backend systems
• Redis — conversation and session state
• PostgreSQL — durable business data
• Human-in-the-loop — a hard boundary the model can't reason its way past
• Observability and evaluation — because a multi-hop agent fails in ways a single API call never does

The principle underneath all of it: keep the LLM inside a controlled system. The model gets to decide what it wants to do. Your application always decides whether it's allowed to, and what data it can see while doing it. That's the line between a demo chatbot and something you can actually put in front of paying customers.