Observability Blueprint — Agents Catalog
Agents that answer questions about live 5G networks (BubbleRAN MX-PDK on Kubernetes). Eight specialists each answer about one kind of evidence; the RCA Agent investigates failures by asking them. Every agent is an A2A server built with BAT-ADK: ask it directly, or put it behind a supervisor in an AIFabric.
All nine agents and the Docs DB MCP server are modes of one CompositionModel, observability: in an AIFabric, reference each one as model: observability/<mode>. KPI reads VictoriaMetrics, Logs reads VictoriaLogs, and Alerts reads both, through the platform's MCP servers in the monitoring namespace: list them in the agent's mcpServers. The Docs Agent uses the Docs DB MCP server. RCA uses no MCP server: it asks the eight specialists. List them in its dependencies under their mode names (topology, k8s-health, alerts, logs, kpi, config, pcap, br-docs), because RCA finds them by these names. The blueprint is included in MX-PDK O-RAN, AI-RAN and CAMPUS. The catalog overview explains these terms and how to install a blueprint.
Every agent except Docs and RCA reads the Kubernetes API, with read-only rules. Config and PCAP also need pods/exec, to read files and capture packets inside the networks' Pods.
Agents
KPI Agent — Network performance metrics
model: observability/kpi · role: worker
Measures how the networks perform, from VictoriaMetrics. It covers each UE's throughput, PRB usage and congestion, latency, MCS, CQI and radio quality (from the E2SM-KPM monitoring xApp), and the CPU, memory and network use of every network Pod. The model chooses what to measure, and code builds the query. For what the agent's metric list does not cover, such as SINR, RSRP or RSRQ, the model writes the query itself. Code checks every query and limits it to the gNBs and Pods of the networks asked about. Answers give current values, averages, peaks and trends with their units, compare UEs and networks, and say so when a metric has no data.
Logs Agent — Log search and summaries
model: observability/logs · role: worker
Searches the container logs of the networks' components and UEs in VictoriaLogs, by network, component, severity, topic words and period. Earlier Pod instances are included. Answers give exact line counts per component and severity, the most frequent message patterns with their first and last time, and the newest lines, quoted as logged, with long lines cut. When a search cannot finish (a component flooding its logs), it says what it could not count.
Topology Agent — What the networks are made of
model: observability/topology · role: worker
Shows each network's declared composition, from the live Network and Terminal resources: core, gNBs and cells, RIC and xApps, slices, E2 node IDs, and which UEs prefer which gNB and target which core. It checks every reference across networks and flags dangling references and shared E2 node IDs. Readiness comes from the status the platform writes on each resource.
Alerts Agent — Alerts and alert rules
model: observability/alerts · role: worker
Tells which alerts are firing or pending now, what fired in a past period (the last 24 hours unless you name another, at most 7 days), and what each rule checks (expression, threshold, duration, severity), from vmalert and VictoriaMetrics. It places each alert on the gNB, network workload or cluster resource it concerns, and flags rules whose input metrics have no data. For "why" questions it adds the affected component's logs from just before the alert started.
Config Agent — Live configuration and consistency checks
model: observability/config · role: worker
Reads the rendered configuration of every element from the running Pods (core functions and database, gNBs and E2 agents, RIC, xApps, UEs), with credentials removed, together with the operator's declared intent. Answers name each setting's file and line and convert radio values to MHz, bandwidth and TDD patterns. Code checks the configuration across components:
- addresses against live Pod IPs;
- PLMN, TAC, slices, DNN and address pools;
- UE radio settings against the gNB's;
- declared against rendered values;
- whether each process runs its current file.
PCAP Agent — Live packet capture and analysis
model: observability/pcap · role: worker
Captures a few seconds of live packets at the elements a question is about (gNB, AMF, SMF, UPF, RIC, xApp or UE), at all points at once, and explains what tshark decoded and code measured:
- signaling message by message (NGAP/NAS, PFCP, GTP-U, SBI, E2AP and E42, SCTP), with failed procedures and their causes;
- connectivity (SCTP associations, sockets) and signs of loss;
- per-UE and per-flow user-plane throughput.
It shows a full decode when asked; credentials are removed.
K8s Health Agent — Kubernetes health of the networks
model: observability/k8s-health · role: worker
Checks the Kubernetes state of every network's Pods and of the rest of the namespace. Per Pod: status, readiness and which container is not ready, restarts with each container's last termination, failing probes with their message, and scheduling problems. It also reports rollouts, Services and their endpoints, and Events, with totals and rankings per network.
Docs Agent — BubbleRAN documentation
model: observability/br-docs · role: worker
Answers from BubbleRAN's documentation, product pages, release notes and news, through the Docs DB MCP server. It searches the latest release or the one named, compares releases on request, and cites the pages it answers from. A command, flag or release that is not documented is said to be so.
RCA Agent — Root cause analysis
model: observability/rca · role: worker
Finds the root cause of a failure or degradation: a UE that does not attach, a gNB without its core or RIC, a crashing Pod, a firing alert, degraded throughput. It collects its own evidence by asking the eight specialists, then tests competing explanations against their answers. It answers with the cause and the chain from it to the symptom, with evidence for each step, plus what it ruled out and a fix with how to check it. Read-only: it proposes fixes and never applies them.
MCP Servers
Docs DB — Documentation search
model: observability/br-docs-db
Indexes BubbleRAN's documentation release by release, the product, solution and news pages, the release notes and the agent catalogs. Tools:
- search: search by question, filtered by release and kind of page;
- read: read the section a search result comes from, or its whole page;
- catalog: list the documented releases, the kinds of pages and the products.
Two older tools, retrieve_context and list_documents, stay for existing clients. Used by the Docs Agent.
Evaluation Results
Each task is one question to an agent about a live MX-PDK testbed: four networks with OAI and Open5GS cores and emulated radios, including deliberate faults (a gNB that names an undeclared core, a gNB left on a stale AMF address). The expected answer of every task was read from the cluster independently of the agents and checked again during the runs. Tasks were run per model with bat eval run (BAT-CLI). The scores come from a judge model that reads the whole run and the expected answer; they go from 0 to 1, higher is better, and are averaged over the tasks of the section.
- Task completion: how close the run came to the expected answer, and how cleanly it got there;
- Relevance: the agent stayed on the subject the user raised, without detours;
- Groundedness: every value in the reply and in the tool calls comes from the user or from a tool result, nothing invented;
- Time (s) and Tokens: wall time and total tokens, averaged per conversation. For the RCA Agent and the supervisor, tokens include the agents they call, which ran on gpt-6-luna.
Overall
All 100 tasks: ten per agent and ten for the supervisor.
| Model | Task completion |
|---|---|
| gpt-6-luna | 0.83 |
| gpt-4.1-mini | 0.75 |
| gemma4 | 0.58 |
| qwen3.8 | 0.64 |
KPI Agent
Ten questions about network performance: throughput per UE now and in a past hour, 7-day peaks, PRB usage, CPU and memory of a network's Pods, an unusually busy gNB, a UPF's memory trend, the collected KPIs, a metric that is not collected (packet loss) and a network without KPM data.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.74 | 0.96 | 0.76 | 38 | 7,247 |
| gpt-4.1-mini | 0.68 | 0.87 | 0.74 | 19 | 13,872 |
| gemma4 | 0.53 | 0.85 | 0.70 | 42 | 8,793 |
| qwen3.8 | 0.54 | 0.87 | 0.50 | 56 | 17,133 |
Logs Agent
Ten log questions: error lines per component in a past hour, a gNB's newest lines, warnings over 24 hours, SCTP and connection lines, a network that does not exist, how often a message occurred in a window, what a component logged at a given time, an AMF's recent lines, the noisiest Pod and the count of one error pattern.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.82 | 0.90 | 0.84 | 30 | 4,902 |
| gpt-4.1-mini | 0.70 | 0.84 | 0.76 | 21 | 7,055 |
| gemma4 | 0.10 | 0.80 | 0.50 | 35 | 5,221 |
| qwen3.8 | 0.20 | 0.77 | 0.30 | 36 | 5,661 |
Topology Agent
Ten questions about what the networks are made of: the deployed networks, one network's composition, UEs and their gNBs, a gNB that names an undeclared core, a network that does not exist, radio settings, RIC controllers, slice address pools, gNB identities and monitoring models.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.94 | 0.96 | 0.96 | 3 | 4,357 |
| gpt-4.1-mini | 0.93 | 0.84 | 0.82 | 3 | 5,249 |
| gemma4 | 0.94 | 0.92 | 0.96 | 11 | 5,583 |
| qwen3.8 | 0.98 | 0.94 | 1.00 | 7 | 4,615 |
Alerts Agent
Ten questions: alerts firing now, what fired on a network in the last 24 hours, what a rule checks, rules that can never fire on the testbed, rule groups, a node outage in the last 7 days, why an alert fires and since when, alerts for one network, critical rules, and the node and interface of a past alert.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.88 | 0.98 | 0.88 | 7 | 6,640 |
| gpt-4.1-mini | 0.83 | 0.94 | 0.76 | 4 | 6,519 |
| gemma4 | 0.86 | 0.99 | 0.90 | 20 | 7,870 |
| qwen3.8 | 0.80 | 0.94 | 0.78 | 19 | 7,979 |
Config Agent
Ten configuration questions: a gNB's radio settings when the network has two gNBs, a misconfigured network, the differences between two networks' gNBs, a full configuration file, an AMF address checked against the live Pod, PLMN and TAC, a UE's secret key (which must not be shown), an SMF's UPF address, DNN and address pool, and a process command line.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.93 | 0.92 | 0.98 | 19 | 21,397 |
| gpt-4.1-mini | 0.92 | 0.94 | 1.00 | 19 | 31,879 |
| gemma4 | 0.70 | 0.71 | 0.90 | 21 | 2,959 |
| qwen3.8 | 0.52 | 0.62 | 0.82 | 29 | 14,497 |
PCAP Agent
Ten capture questions: a gNB's signaling with its AMF, whether every gNB is connected to its AMF, why a UE never registers, user-plane traffic at a UPF, a gNB's E2 link to its RIC, the protocols seen at an AMF, per-UE throughput, PFCP between an SMF and its UPF, an ambiguous capture target, and whether the UEs reach the internet.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.89 | 0.95 | 0.94 | 26 | 9,062 |
| gpt-4.1-mini | 0.82 | 0.85 | 0.76 | 32 | 14,171 |
| gemma4 | 0.68 | 0.85 | 0.76 | 44 | 8,756 |
| qwen3.8 | 0.60 | 0.55 | 0.78 | 37 | 6,020 |
K8s Health Agent
Ten questions: whether all network Pods are running and ready, why a UE is not ready and since when, the most-restarted Pods and why they stopped, node state, an AMF's health, a gNB Pod's containers, why an AMF restarted, a network that does not exist, a RIC's health, and warning events.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.97 | 0.87 | 1.00 | 35 | 20,280 |
| gpt-4.1-mini | 0.87 | 0.80 | 0.82 | 23 | 34,018 |
| gemma4 | 0.77 | 0.83 | 0.84 | 58 | 33,708 |
| qwen3.8 | 0.90 | 0.86 | 0.88 | 81 | 28,889 |
Docs Agent
Ten documentation questions: how to deploy a network, the difference between two products, what is new in the latest release, slice configuration compared across two releases, a live-data question that belongs to another agent, supported radios, adding a UE, the CLI's main commands, a product overview and enabling KPM monitoring.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.82 | 0.97 | 1.00 | 16 | 12,664 |
| gpt-4.1-mini | 0.75 | 0.82 | 0.92 | 10 | 14,458 |
| gemma4 | 0.30 | 0.36 | 0.72 | 27 | 4,676 |
| qwen3.8 | 0.80 | 0.81 | 0.88 | 42 | 15,206 |
RCA Agent
Ten investigations: a UE that does not attach, a whole-network diagnosis, the same fault with a follow-up question, UEs that are never ready, a request to fix a network (it must propose, not act), KPM data that stopped, a firing storage alert, a false premise (a crash that did not happen), a CPU difference between two gNBs, and GTP error indications. Only the RCA Agent's own model varies; the specialists it asks ran on gpt-6-luna.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.66 | 0.78 | 0.66 | 228 | 151,543 |
| gpt-4.1-mini | 0.44 | 0.79 | 0.54 | 139 | 100,682 |
| gemma4 | 0.43 | 0.67 | 0.50 | 153 | 63,660 |
| qwen3.8 | 0.47 | 0.68 | 0.54 | 275 | 92,574 |
Through a supervisor
Ten questions to a supervisor (supervisor/base) with the nine agents behind it, each needing several of them: a network's TDD pattern with its KPIs, logs and what FlexRIC is; a network's health; a UE that cannot connect with the documentation on linking a gNB to its core; firing alerts with the log storage and error lines; two networks compared; the busiest component with its health and logs; a gNB's AMF link checked in the configuration and in a capture; enabling KPM and which networks have it; an overall health report; and a two-turn root-cause question. Only the supervisor's model varies; the agents ran on gpt-6-luna.
| Model | Task completion | Relevance | Groundedness | Time (s) | Tokens |
|---|---|---|---|---|---|
| gpt-6-luna | 0.67 | 0.93 | 0.72 | 195 | 132,107 |
| gpt-4.1-mini | 0.56 | 0.90 | 0.66 | 95 | 70,328 |
| gemma4 | 0.54 | 0.83 | 0.54 | 158 | 94,981 |
| qwen3.8 | 0.62 | 0.83 | 0.56 | 242 | 122,782 |