Status Assessment and Troubleshooting
Blueprint pre-checks
Before deploying a network using 7.2, certain pre-checks are recommended, especially after reboot.
The blueprints do not name the node of the gNB/CU-DU: Athena assigns a Radio Unit (RU) reported by RU discovery to each access section and places the gNB/CU-DU on the node that reaches it.
That node needs an SFP NIC (for a pre-installed MX-PDK, normally 1 or 2 worker nodes have an Intel E810-XXV NIC), connected to the PTP-grandmaster fronthaul switch with both SFP interfaces, which are generally called sync0 and dpdk0.
Athena also fills the fronthaul parameters of the gNB/CU-DU (DPDK PCI IDs, DU and RU MACs) from what RU discovery reports, so the blueprints do not set them. Start with the RU discovery checks below, then use the node checks to compare the node with what discovery reports.
Manage RU discovery with brc config ru
RU discovery works in three steps:
- Each worker node declares the RUs it is attached to in
/etc/pontus/ru.json. - The
pontus-ru-reportservice on the node probes every declared RU every 15 seconds: PTP state of the node, SSH login to the RU, and its sync state. - Pontus publishes the RUs that pass the probe as device resources and in the
radio.gaia.t9s.io/inventoryannotation of the node. Athena only assigns those.
An RU that is not declared is invisible to the whole stack, and an RU that is declared but fails the probe can never be assigned.
The brc config ru commands manage both the declarations and the RU credentials.
See the brc config ru reference for every option.
brc config ru list, add, remove and set-password read or write /etc/pontus/ru.json on the worker nodes over SSH, with the Kubernetes node name as SSH host.
Run them from a machine that can SSH to the worker nodes with a user that has sudo, and use --host and --user when the SSH host or user differ.
brc config ru list --observed needs only the kubeconfig.
List the RUs
brc config ru list
╭──────────────────┬───────┬─────────┬──────────────┬──────────┬──────────┬───────┬─────────────────────────────────────────────────────────────────┬────────╮
│ NAME │ NODES │ VENDOR │ IP │ DECLARED │ OBSERVED │ READY │ RESOURCE │ REASON │
├──────────────────┼───────┼─────────┼──────────────┼──────────┼──────────┼───────┼─────────────────────────────────────────────────────────────────┼────────┤
│ Benetel-RAN550-1 │ node1 │ benetel │ 192.168.1.51 │ yes │ yes │ yes │ gaia.t9s.io/ru-8c-1f-64-d1-16-a0-02-11-22-33-44-68-0000-01-11-0 │ │
│ │ node2 │ │ │ yes │ yes │ yes │ gaia.t9s.io/ru-8c-1f-64-d1-16-a0-02-11-22-33-44-66-0000-01-11-0 │ │
├───────────────── ─┼───────┼─────────┼──────────────┼──────────┼──────────┼───────┼─────────────────────────────────────────────────────────────────┼────────┤
│ LiteON-ORU-1 │ node1 │ liteon │ 192.168.1.41 │ yes │ yes │ yes │ gaia.t9s.io/ru-e8-c7-4f-25-83-25-02-11-22-33-44-68-0000-01-11-0 │ │
│ │ node2 │ │ │ yes │ yes │ yes │ gaia.t9s.io/ru-e8-c7-4f-25-83-25-02-11-22-33-44-66-0000-01-11-0 │ │
╰──────────────────┴───────┴─────────┴──────────────┴──────────┴──────────┴───────┴─────────────────────────────────────────────────────────────────┴────────╯
Each row is one RU, with one line per node that declares or reaches it.
The RESOURCE is the device resource Athena assigns, named ru-<RU MAC>-<DU VF MAC>-<PCI ID>.
In this example both RUs sit on a fronthaul segment shared by node1 and node2, so each one is offered twice, once per node.
Within one network, Athena recognizes such an RU by its MAC and IP and assigns it only once.
Two networks deployed at the same time can still each get it, one on each node, so do not run two networks that need the same RU at the same time.
| DECLARED | OBSERVED | READY | Meaning and next step |
|---|---|---|---|
| yes | yes | yes | The RU can be assigned. |
| yes | yes | no | The probe failed. Read the REASON column, see the table below. |
| yes | no | The RU was never published: check the reporter service and Pontus as described in Check Pontus RU discovery. | |
| no | yes | The RU was removed from ru.json and the annotation is stale. It clears within 15 seconds. | |
| (no row) | The RU is not declared on any node: add it with brc config ru add. |
The REASON column carries the reason reported by ru.py:
| Reason | Cause |
|---|---|
service ptp4l not active, service phc2sys not active | PTP is not running on the node, see Check PTP status. |
ptp4l metrics out-of-bounds (...), phc2sys metrics out-of-bounds (...) | PTP is running but not locked to the grandmaster. |
no matching interface subnet for <ip> among: sync0, bubbleran0 | No interface of the node is in the subnet of the RU IP. Check the IP of the RU and the cabling. |
ssh: ... | The node cannot log into the RU. For a Benetel RU this is usually a wrong or missing password, see Set the Benetel RU password. |
sync-state not locked: ... | The RU answers but is not synchronized yet. Wait, then check the PTP of the fronthaul switch. |
Use brc config ru list --node <node> to look at one node, and brc config ru list --output json to get the RU MACs, the PCI IDs and the fronthaul interface that each node reports.
With --observed, brc config ru list does not read ru.json, so the DECLARED column shows no for every RU.
Declare an RU
The RUs of a node are normally written in ru.json when the node is provisioned.
To declare one more, give its name, its vendor (liteon, benetel or benetel-650) and its IP:
brc config ru add ru-01 benetel 192.168.1.52 --node node1
Leave out any argument to be asked for it. For a Benetel RU the command also asks for the password ru.py logs in with.
No restart is needed: the RU appears as declared at once and, if it passes the probe, as observed within about 30 seconds.
An RU added with brc config ru add is a local change: re-provisioning the node rewrites ru.json and drops it.
Ask BubbleRAN support to add the RU to your cluster configuration to keep it.
To stop offering an RU to new networks, remove it by name or by IP. Elements already using it keep running.
brc config ru remove ru-01 --node node1
Set the Benetel RU password
From Benetel firmware 2.x a Benetel RU has a password, the one set by its operator. Two components log into the RU and both need that password:
ru.pyon each worker node, with thepasswordof the RU entry in/etc/pontus/ru.json. Without it the RU never becomes ready.- Athena's manager, with the vendor's key in the
ru-secretsSecret (ocudu-ru-secretsfor OCUDU). Without it the RU is assigned but cannot be configured.
brc config ru set-password writes both:
# asks for the password without echo
brc config ru set-password benetel
# for a Benetel RAN650
brc config ru set-password benetel-650
# from a script
printf '%s' "$RU_PASSWORD" | brc config ru set-password benetel --stdin
Use --secret-only or --inventory-only to update only one of the two, and --empty for firmware without a password.
LiteON RUs have fixed credentials and are refused.
The password in the Secret is reset to the shipped default when the RAN model (oai-ran or ocudu) is reinstalled or upgraded. Run brc config ru set-password again afterwards.
Check the NIC's PCI IDs for DPDK on gNB/CU-DU node
Check the PCI IDs of the NIC.
dpdk-devbind.py --status
which should provide an output like the following:
Network devices using DPDK-compatible driver
============================================
0000:01:11.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
0000:01:11.1 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
...
Check that these 2 PCI IDs (in this case 0000:01:11.0 and 0000:01:11.1) are the pci_ids reported for this node by brc config ru list --output json, and the last part of the RU RESOURCE. In the case that the command fails, you may need to reconfigure DPDK on that node:
sudo dpdk-setup
This command will give the same output as dpdk-devbind.py --status.
Check VF MACs obtained for dpdk0 on gNB/CU-DU node
ip link show
which should provide an output like the following:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: bubbleran0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether d8:43:ae:ca:bf:50 brd ff:ff:ff:ff:ff:ff
3: sync0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 50:7c:6f:66:ec:6e brd ff:ff:ff:ff:ff:ff
4: dpdk0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 50:7c:6f:66:ec:6f brd ff:ff:ff:ff:ff:ff
alias dpdk0
vf 0 link/ether 02:11:22:33:44:66 brd ff:ff:ff:ff:ff:ff, vlan 564, spoof checking off, link-state auto, trust off
vf 1 link/ether 02:11:22:33:44:67 brd ff:ff:ff:ff:ff:ff, vlan 564, spoof checking off, link-state auto, trust off
altname enp1s0f1
...
Check that the MAC reported for vf 0 (in this case 02:11:22:33:44:66) is the DU VF MAC in the RESOURCE of the RU on this node, as shown by brc config ru list.
If no vfs are reported in the output it means that you perhaps skipped the previous step, if so, run sudo dpdk-setup.
Check connectivity with Benetel RU on gNB/CU-DU node
Try logging in to the RU via SSH. For pre-installed MX-PDK the IP of the RU is preconfigured as specified in the configuration spreadsheet (see here). In the case of a remote installation the Benetel RU needs to be configured (in this case contact BubbleRAN support).
ssh root@<benetel-ru-ip>
- Username: root
- Password: the one set on the RU (firmware 2.x). It must also be set with
brc config ru set-password.
Once logged in, you can check the status of the RU using the following command:
cat /tmp/logs/radio_status
Once logged in, you can check the MAC of the RU using the following command:
ifconfig
which should provide an output like the following:
root@benetelru:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 metric 1
inet 192.168.1.50 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::8e1f:64ff:fed1:1202 prefixlen 64 scopeid 0x20<link>
ether 8c:1f:64:d1:12:02 txqueuelen 1000 (Ethernet)
RX packets 21819032440 bytes 150092895720236 (136.5 TiB)
RX errors 0 dropped 21760 overruns 0 frame 0
TX packets 2467612690 bytes 14558104214036 (13.2 TiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device memory 0xff200000-ff2002ff
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 metric 1
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 2906529 bytes 301349892 (287.3 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2906529 bytes 301349892 (287.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Check that the MAC address (in this case 8c:1f:64:d1:12:02) and IP address (in this case 192.168.1.50) of eth0 are the RU MAC and the IP reported for this RU by brc config ru list.
Check connectivity with LITEON RU on gNB/CU-DU node
Try logging in to the RU via SSH. For pre-installed MX-PDK the IP of the RU is preconfigured as specified in the configuration spreadsheet (see here). In the case of a remote installation the LITEON RU needs to be configured (in this case contact BubbleRAN support).
ssh user@<liteon-ru-ip>
- Username: user
- Password: user
Once logged in, you can check the status of the RU using the following command:
enable
A password will be requested, which is liteon168 by default. Once logged in, you can check the MAC of the RU using the following command:
show eth-info
which should provide an output like the following:
# show eth-info
eth0 Link encap:Ethernet HWaddr e8:c7:4f:28:29:38
inet addr:10.101.131.59 Bcast:10.101.131.255 Mask:255.255.255.0
inet6 addr: fe80::eac7:4fff:fe28:2938/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:278 (278.0 B)
Interrupt:29
eth1 Link encap:Ethernet HWaddr e8:c7:4f:25:80:e1
inet addr:192.168.1.40 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::eac7:4fff:fe25:80e1/64 Scope:Link
UP BROADCAST RUNNING MTU:1500 Metric:1
RX packets:10419101 errors:0 dropped:26 overruns:0 frame:0
TX packets:4494120 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:830061534 (791.6 MiB) TX bytes:289941104 (276.5 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:12684 errors:0 dropped:0 overruns:0 frame:0
TX packets:12684 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1116192 (1.0 MiB) TX bytes:1116192 (1.0 MiB)
Check that the MAC address (in this case e8:c7:4f:25:80:e1) and IP address (in this case 192.168.1.40) of eth1 are the RU MAC and the IP reported for this RU by brc config ru list.
Check Pontus RU discovery
brc extract infra and brc config ru list report RUs from Gaia/Pontus. If the RU is declared and reachable manually but is not observed, check the two Pontus pieces on the gNB/CU-DU node:
pontus-ru-report.service: a hostsystemdservice that runsru.pyevery 15 seconds.pontus: a Kubernetes DaemonSet in thegaianamespace. Itsdevice-plugincontainer reads the RU report and exposes RU resources to kubelet.
The host service reads the RU inventory from:
/etc/pontus/ru.json
and writes the generated readiness report to:
/var/run/pontus/ru-report.json
Pontus consumes that report through its ConfigMap. You can verify the configured path with:
kubectl -n gaia get configmap pontus-config -o jsonpath='{.data.config\.yaml}'
The ru section should contain:
ru:
enabled: true
report-path: "/var/run/pontus/ru-report.json"
Check the RU inventory used by ru.py (brc config ru list shows the same entries):
sudo cat /etc/pontus/ru.json | jq .
It should contain a rus array with the expected RU name, management IP, and vendor:
{
"rus": [
{
"name": "benetel-ru",
"ip": "192.168.1.50",
"vendor": "benetel"
}
]
}
A Benetel RU with a password also has a password field, written by brc config ru set-password. Without that field, ru.py tries root's SSH key on the node, then an empty password.
Run the reporter manually to see the exact readiness output and error reasons:
sudo /usr/bin/ru.py -f /etc/pontus/ru.json | jq .
Check the last report produced by the service:
sudo cat /var/run/pontus/ru-report.json | jq .
Check and restart the host reporter service:
sudo systemctl status pontus-ru-report.service
sudo journalctl -u pontus-ru-report.service -n 100 --no-pager
sudo systemctl restart pontus-ru-report.service
After restarting it, wait a few seconds and re-check the report:
sleep 20
sudo cat /var/run/pontus/ru-report.json | jq .
Check the Pontus DaemonSet:
kubectl -n gaia get pods -l app.kubernetes.io/name=pontus -o wide
kubectl -n gaia logs ds/pontus -c device-plugin --tail=200
Restart the DaemonSet only after confirming that /var/run/pontus/ru-report.json is correct but the RU still does not appear in the cluster inventory:
kubectl -n gaia rollout restart ds/pontus
kubectl -n gaia rollout status ds/pontus --timeout=120s
Then verify the node resources again:
brc extract infra
kubectl describe node <gNB-CU-DU-node-name> | grep -i -A20 "Capacity"
Pontus also writes the RU inventory into the node annotation consumed by Athena:
kubectl get node <gNB-CU-DU-node-name> -o json \
| jq -r '.metadata.annotations["radio.gaia.t9s.io/inventory"] // empty' \
| jq .
SSH key instead of a password
The recommended way to give ru.py access to a Benetel RU is the password, set with brc config ru set-password.
An SSH key is only used for an RU entry without a password field. Athena's manager always logs in with the password from the Secret, so the RU still needs it there.
For Benetel RUs, ru.py checks the RU over SSH as root. The reporter service runs as root on the gNB/CU-DU node, so the SSH identity must be available under /root/.ssh, not only under the normal user account.
Test key-only SSH first:
sudo ssh -i /root/.ssh/id_ed25519 \
-o IdentitiesOnly=yes \
-o PasswordAuthentication=no \
root@<benetel-ru-ip> 'hostname; cat /var/syncmon/sync-state'
If no root SSH key exists on the gNB/CU-DU node, create one with no passphrase:
sudo mkdir -p /root/.ssh
sudo ssh-keygen -t ed25519 -f /root/.ssh/id_ed25519 -N "" -C "$(hostname)-pontus-ru"
Do not put a hostname or username in -N: that option sets the key passphrase. Use -C for the comment.
Install the public key on the RU:
sudo ssh-copy-id -i /root/.ssh/id_ed25519.pub root@<benetel-ru-ip>
If SSH asks for Enter passphrase for key '/root/.ssh/id_ed25519', the key was created with a passphrase. Remove it:
sudo ssh-keygen -p -f /root/.ssh/id_ed25519
When prompted for the new passphrase, press Enter twice to store an empty passphrase. Then test again with PasswordAuthentication=no.
If you know the current passphrase, the same operation can be done non-interactively:
sudo ssh-keygen -p -f /root/.ssh/id_ed25519 -P "<old-passphrase>" -N ""
If key-only SSH still fails, log into the RU with its password and check the authorized-key permissions:
chmod 700 /root/.ssh
chmod 600 /root/.ssh/authorized_keys
chown -R root:root /root/.ssh
After fixing SSH, restart the reporter service and verify the generated report:
sudo systemctl restart pontus-ru-report.service
sleep 20
sudo cat /var/run/pontus/ru-report.json | jq .
Check PTP status on gNB/CU-DU node
Check the status of PTP services in gNB/CU-DU with the following commands:
sudo systemctl status ptp4l.service
sudo systemctl status phc2sys.service
which should provide the following outputs:
$ sudo systemctl status ptp4l.service
● ptp4l.service - Precision Time Protocol (PTP) service
Loaded: loaded (/etc/systemd/system/ptp4l.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2025-07-23 14:22:15 CEST; 1 week 2 days ago
Docs: man:ptp4l
Main PID: 210606 (ptp4l)
Tasks: 1 (limit: 49542)
Memory: 604.0K
CPU: 3h 23min 33.257s
CGroup: /system.slice/ptp4l.service
└─210606 /usr/sbin/ptp4l -f /etc/ptp4l.conf
Aug 01 18:49:27 node1 ptp4l[210606]: [810004.243] rms 4 max 7 freq -7677 +/- 4 delay 94 +/- 1
Aug 01 18:49:28 node1 ptp4l[210606]: [810005.368] rms 3 max 7 freq -7679 +/- 4 delay 94 +/- 1
Aug 01 18:49:29 node1 ptp4l[210606]: [810006.493] rms 4 max 8 freq -7671 +/- 5 delay 95 +/- 1
Aug 01 18:49:30 node1 ptp4l[210606]: [810007.618] rms 3 max 8 freq -7668 +/- 5 delay 94 +/- 1
Aug 01 18:49:32 node1 ptp4l[210606]: [810008.743] rms 3 max 7 freq -7677 +/- 4 delay 94 +/- 1
Aug 01 18:49:33 node1 ptp4l[210606]: [810009.872] rms 3 max 6 freq -7678 +/- 5 delay 94 +/- 1
Aug 01 18:49:34 node1 ptp4l[210606]: [810010.993] rms 2 max 4 freq -7677 +/- 3 delay 94 +/- 0
Aug 01 18:49:35 node1 ptp4l[210606]: [810012.118] rms 3 max 6 freq -7673 +/- 4 delay 94 +/- 1
Aug 01 18:49:36 node1 ptp4l[210606]: [810013.243] rms 2 max 4 freq -7677 +/- 3 delay 95 +/- 1
Aug 01 18:49:37 node1 ptp4l[210606]: [810014.368] rms 3 max 6 freq -7671 +/- 5 delay 95 +/- 0
Check that the service is running (i.e. the status is running) and that the values reported after max are not higher than +/- 10.
$ sudo systemctl status phc2sys.service
● phc2sys.service - Synchronize system clock or PTP hardware clock (PHC)
Loaded: loaded (/etc/systemd/system/phc2sys.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2025-08-01 18:49:53 CEST; 13s ago
Docs: man:phc2sys
Main PID: 3287995 (phc2sys)
Tasks: 1 (limit: 49542)
Memory: 440.0K
CPU: 1ms
CGroup: /system.slice/phc2sys.service
└─3287995 /usr/sbin/phc2sys -a -r -r -n 24
Aug 01 18:49:57 node1 phc2sys[3287995]: [810033.903] CLOCK_REALTIME phc offset 11 s2 freq -9386 delay 541
Aug 01 18:49:58 node1 phc2sys[3287995]: [810034.923] CLOCK_REALTIME phc offset 15 s2 freq -9379 delay 531
Aug 01 18:49:59 node1 phc2sys[3287995]: [810035.923] CLOCK_REALTIME phc offset -12 s2 freq -9401 delay 521
Aug 01 18:50:00 node1 phc2sys[3287995]: [810036.923] CLOCK_REALTIME phc offset -12 s2 freq -9405 delay 531
Aug 01 18:50:01 node1 phc2sys[3287995]: [810037.923] CLOCK_REALTIME phc offset 17 s2 freq -9379 delay 541
Aug 01 18:50:02 node1 phc2sys[3287995]: [810038.923] CLOCK_REALTIME phc offset 13 s2 freq -9378 delay 541
Aug 01 18:50:03 node1 phc2sys[3287995]: [810039.923] CLOCK_REALTIME phc offset -9 s2 freq -9396 delay 531
Aug 01 18:50:04 node1 phc2sys[3287995]: [810040.923] CLOCK_REALTIME phc offset -5 s2 freq -9395 delay 531
Aug 01 18:50:05 node1 phc2sys[3287995]: [810041.923] CLOCK_REALTIME phc offset 2 s2 freq -9389 delay 531
Aug 01 18:50:06 node1 phc2sys[3287995]: [810042.923] CLOCK_REALTIME phc offset -9 s2 freq -9400 delay 531
Check that the service is running (i.e. the status is running) and that the values reported after offset are not higher than +/- 30.
Sample output of a failed service:
$ sudo systemctl status phc2sys.service
× phc2sys.service - Synchronize system clock or PTP hardware clock (PHC)
Loaded: loaded (/etc/systemd/system/phc2sys.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2025-07-26 05:38:23 CEST; 6 days ago
Docs: man:phc2sys
Main PID: 3622303 (code=exited, status=255/EXCEPTION)
CPU: 2.302s
Jul 26 05:38:16 node1 phc2sys[3622303]: [244133.013] CLOCK_REALTIME phc offset 9 s2 freq -9085 delay 551
Jul 26 05:38:17 node1 phc2sys[3622303]: [244134.013] CLOCK_REALTIME phc offset -1 s2 freq -9093 delay 511
Jul 26 05:38:18 node1 phc2sys[3622303]: [244135.013] CLOCK_REALTIME phc offset 1 s2 freq -9091 delay 551
Jul 26 05:38:19 node1 phc2sys[3622303]: [244136.013] CLOCK_REALTIME phc offset 12 s2 freq -9080 delay 521
Jul 26 05:38:20 node1 phc2sys[3622303]: [244137.013] CLOCK_REALTIME phc offset -1 s2 freq -9089 delay 551
Jul 26 05:38:21 node1 phc2sys[3622303]: [244138.013] CLOCK_REALTIME phc offset 7 s2 freq -9081 delay 490
Jul 26 05:38:22 node1 phc2sys[3622303]: [244139.014] CLOCK_REALTIME phc offset 6 s2 freq -9080 delay 541
Jul 26 05:38:23 node1 systemd[1]: phc2sys.service: Main process exited, code=exited, status=255/EXCEPTION
Jul 26 05:38:23 node1 systemd[1]: phc2sys.service: Failed with result 'exit-code'.
Jul 26 05:38:23 node1 systemd[1]: phc2sys.service: Consumed 2.302s CPU time.
If either of the services have failed or values are exceptionally high you may try restarting the services :
sudo systemctl restart ptp4l.service
sudo systemctl restart phc2sys.service
Healthy UE attach and data with OAI RAN
- On LITEON RU Once the gNodeB connects to RU, the
DuConnectedfield should change toReadyas shown below when we run:brc extract logs <liteon-element-name>
Sync State : SYNCHRONIZED
RF State : Ready
DPD : Ready
DuConnected : Ready
- On Benetel Once the gNodeB connects to RU, the antena transmission can be checked using:
TXMeanPowerand if all 4 in the 4RX4TX configuration are up, one should see:
TX 1 Mean Power: 10.7029 dBm
TX 2 Mean Power: 4.37732 dBm
TX 3 Mean Power: 9.40766 dBm
TX 4 Mean Power: 10.2364 dBm
In addition, KPI measurements can be checked using: kpi.sh
Good transmission means the LATE and EARLY packet counters are zeros as shown in the example:
SAMPLE_TIME | RX_TOTAL | RX_ON_TIME | RX_EARLY | RX_LATE | RX_ON_TIME_C | RX_EARLY_C | RX_LATE_C | TX_TOTAL
10:32:25.066903 | 92959 | 83396 | -0 | -0 | 9604 | -0 | -0 | 32400
10:32:26.065888 | 92672 | 83032 | -0 | -0 | 9592 | -0 | -0 | 32400
10:32:27.065919 | 92883 | 83272 | -0 | -0 | 9596 | -0 | -0 | 32400
10:32:28.067132 | 92930 | 83136 | -0 | -0 | 9608 | -0 | -0 | 32400
10:32:29.070562 | 92837 | 83384 | -0 | -0 | 9620 | -0 | -0 | 32505
If the RX_ON_TIME counters is 0, with RX_EARLY , RX_LATE, populated as shown below:
SAMPLE_TIME | RX_TOTAL | RX_ON_TIME | RX_EARLY | RX_LATE | RX_ON_TIME_C | RX_EARLY_C | RX_LATE_C | TX_TOTAL
11:07:48.053547 | 92978 | -0 | 41868 | 41600 | -0 | 4408 | 4424 | -0
11:07:49.048914 | 92817 | -0 | 41340 | 41600 | -0 | 4392 | 4384 | -0
11:07:50.046409 | 92844 | -0 | 41720 | 41600 | -0 | 4400 | 4392 | -0
11:07:51.048411 | 92781 | -0 | 41518 | 41600 | -0 | 4400 | 4416 | -0
Resolve by checking the ptp4l and phc2sys services in the DU server; If any of them is in a failed state, restart:
sudo systemctl restart ptp4l.service
sudo systemctl restart phc2sys.service
- OAI gNB/CU-DU
On successful DU RU connection, you should see the following tx rx counters in the gnb log:
pps 127488 kbps 4214777][Total Msgs_Rcvd 14284785]
[NR_PHY] [o_du0][pusch0 3225585 prach0 345632]
[NR_PHY] [o_du0][pusch1 3225592 prach1 345600]
[NR_PHY] [o_du0][pusch2 3225590 prach2 345600]
[NR_PHY] [o_du0][pusch3 3225586 prach3 345600]
[NR_MAC] Frame.Slot 256.0
UE RNTI 8f6d CU-UE-1 ID 1 in-sync PH 47 dB PCMAX 24 dBm, average RSRP -71 (32 meas)
CQI 15, RI 2, PMI (0,1)
UE 8f6d: dlsch_rounds 1278/692/678/665, dlsch_errors 597, pucch0_DTX 2632, BLER 0.54206 MCS (1) 3
UE 8f6d: dlsch_rounds 1118/117/74, dlsch_errors 3, ulsch_DTX 0, BLER 0.00000 MCS (1) 6
UE 8f6d: MAC: TX = 175737 RX = 165266 bytes
LCID 1: TX 907 RX 3878 bytes
LCID 2: TX 126 RX 28 bytes
LCID 4: TX 4371 RX 1578 bytes
- RSRP close to -70 dBm (typical indoor)
- Low uplink BLER (0.00000) – good uplink channel
- Downlink BLER ~0.53 – moderate downlink quality
- MAC layer TX/RX activity confirms user-plane traffic
Troubleshooting
- If below logs is observed, with zero tx and rx counters, the
devmemcommands that set the transmission mode and the PRACH compression may not be well set. Setextras.t9s.io/liteon-ru-rebootto true and re-deploy your network so that the RU is rebooted and devmem settings are applied. If the above devmem settings are not made, pusch and prach counters will have zeros as below;
[NR_PHY] [o-du 0][rx 0 pps 0 kbps 0][tx 1020100 pps 127488 kbps 4717971][Total Msgs_Rcvd 0]
[NR_PHY] [o_du0][pusch0 0 prach0 0]
[NR_PHY] [o_du0][pusch1 0 prach1 0]
[NR_PHY] [o_du0][pusch2 0 prach2 0]
[NR_PHY] [o_du0][pusch3 0 prach3 0]
- If above problem persists and/or errors regarding received time not matching the expected time:
[PHY] Received Time doesn't correspond to the time we think it is (slot mismatch, received 480.5, expected 475.8)
[PHY] Received Time doesn't correspond to the time we think it is (frame mismatch, 480.5 , expected 475.5)
[PHY] Jump in frame counter last_frame 480 => 519, slot 19
[PHY] Received Time doesn't correspond to the time we think it is (slot mismatch, received 519.19, expected 480.12)
Restart the ptp4l and phc2sys services on the gNB/CU-DU node with:
sudo systemctl restart ptp4l.service
sudo systemctl restart phc2sys.service
After restarting these services, remove and re-install the network with brc remove network <network.yaml> and brc install network <network.yaml>.
Checking Status of CU-DU nodes
Here are the commands for CU-DU (gNB) nodes using O-RAN 7.2 to be run after each reboot.
-
Check the IP and interfaces:
ip address show -
Initialize the DPDK (after every reboot):
sudo dpdk-setup # produces the PCI addresses createdyou may view the contents of the script via:
sudo vim /bin/dpdk-setup -
Check the virtual MAC addresses
ip link show # under dpdk0, vf0 and vf1 -
Synchronization checks:
sudo systemctl status ptp4l.serviceshould show low numerical values for the diff -- Ideally -10 < x < +10
sudo systemctl status phc2sys.serviceShould show low numerical values for the diff -- Ideally -10 < x < +10
-
If restarting the services is needed, you may run:
sudo systemctl restart ptp4l.servicesudo systemctl restart phc2sys.service -
Check the journal logs for the PTP or PHC2sys
sudo journalctl --unit ptp4l --unit phc2sys --follow