Phoenix: see what your agents do

Phoenix is a trace viewer. A trace is the record of one request to an agent: each step the agent took, each LLM call with its prompt and answer, each tool call, and each call to another agent, with how long it took and how many tokens it used.

Agents built with the BAT ADK send their traces to Phoenix when telemetry is on. You can use the Phoenix of your cluster, or run your own on your computer.

Agents in your cluster

Your cluster runs Phoenix at http://phoenix-svc.monitoring.svc.cluster.local:6006.

Send the traces

An AIFabric turns telemetry on for all its agents with its telemetry part:

spec:
  telemetry:
    endpoint: http://phoenix-svc.monitoring.svc.cluster.local:6006
    projectName: observability-lab

• endpoint: the address of Phoenix.
• projectName: the Phoenix project that groups the traces of this fabric. Without it, the project has the name of the AIFabric.

Open Phoenix

Forward its port, on the cluster master node:

kubectl -n monitoring port-forward svc/phoenix-svc 6006:6006

If you work on your own computer, carry the port over SSH:

ssh -L 6006:localhost:6006 <user>@<master-node>

Open http://localhost:6006 in your browser and choose the project of your fabric.

An agent on your computer

When you develop an agent on your computer, you can run your own Phoenix.

Start Phoenix

In a new terminal:

uvx --from arize-phoenix phoenix serve

Keep this terminal open. Phoenix is now at http://localhost:6006.

Send the traces

In the agent's config.yaml, replace the telemetry part with:

telemetry:
  project_name: tutorial
  output:
    - type: remote
      endpoint: http://localhost:6006

• project_name: the Phoenix project for your traces. Choose any name, for example the name of your blueprint.
• output: where the traces go. type: remote sends them to Phoenix, at endpoint.

Restart your agent and ask it a question. Then open http://localhost:6006 and choose your project.

To use the Phoenix of your cluster instead of your own, forward its port as in Open Phoenix, and keep the same config.yaml.

Read a trace

When you open a project, Phoenix shows two tabs:

• Traces: one row per question you asked, newest first, with how long it took and how many tokens it used. Click a row to open the trace. On the left is the tree of its steps: the agent's nodes, each LLM call and each tool call, in the order they ran. Click a step to see what went in and what came out: for an LLM call, the prompt and the answer; for a tool call, its arguments and its result.
• Spans: every step of every trace, in one table. Use it to filter, for example to show only the LLM calls.

Privacy levels

A trace can show everything an agent does, including your data and the agent's instructions. The privacy level decides what stays out of the traces. Hidden text reads __BLOCKED__.

Level	What it hides
none	Nothing. This is the default.
content	Prompts, answers, messages between agents, tool descriptions and tool arguments, and error messages. Step names and tool names stay visible.
names	Also the step names: each step shows only its kind, LLM, CHAIN or TOOL.
full	Also the tool names.

At every level, the trace keeps its shape: the tree of steps, their kind, how long each took, and how many tokens it used.

Choose a level

On your computer, set the level in the agent's config.yaml:

telemetry:
  privacy: content

The developer of an agent can also build a minimum level into it, in its app.py:

AgentApplication(
    AgentGraphType=ThroughputCheckerAgentGraph,
    AgentStateType=ThroughputCheckerAgentState,
    telemetry_privacy_floor="content",
)

The agent uses the stricter of the two levels: config.yaml can hide more than the built-in level, never less. In the cluster, only the built-in level applies.

Some agents released by BubbleRAN have a built-in level. Their traces show each step as LLM, CHAIN or TOOL, with its timing and tokens, while the prompts, answers, tool arguments and tool names read __BLOCKED__.