> For the complete documentation index, see [llms.txt](https://docs.gxc.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.gxc.io/docs/troubleshooting/onyx-troubleshooting-guide/service-issues.md).

# Service Issues

This chapter provides troubleshooting information for service-affecting issues in GXC Onyx networks.

For Onyx Portal navigation paths and status indicator definitions, see [*Onyx Portal Navigation & Indicators Reference*](/docs/troubleshooting/onyx-troubleshooting-guide/onyx-portal-navigation-and-indicators-reference.md).

## UE Connectivity & Data Plane Issues

This section covers issues affecting UE connectivity, network registration, and data plane operations. These issues typically represent the highest-priority service-impacting conditions.

### UE Not Seeing Signal / UE Not Latching to Network

<details>

<summary><strong>Onyx AP Offline or Unable to Communicate with Onyx Edge</strong></summary>

**Symptoms**

UE displays "*No Service*" or "*Searching*" within a known coverage area.

**Indicators**

* Alerts: "AP Offline" alert (Major) is active.
* **Dashboard** > **Access Point** summary card: AP's Online Status is red.
* AP summary page and AP details page: AP's Online Status and Interface Status are red.
* Onyx Edge details page: The associated Onyx Edge Interface Status is red.

**Possible Causes**

* The AP has lost network connectivity to the Onyx Edge (cabling, switch port, or PoE issue).
* The AP has lost power or was unexpectedly power cycled.
* The AP has crashed or is otherwise unresponsive.

**Actions**

1. Verify AP Online Status — In the AP summary page or AP details page, confirm the AP's Online Status is green.
2. Ping the AP from the Onyx Edge — In the Onyx Edge details page, click **Actions** > **Ping**, select the **NR** filter, enter the AP's management IP address, then click **Ping**. The test verifies connectivity between the Onyx Edge and the AP and helps distinguish a network-layer issue from an AP-side fault before assuming a cabling or power issue. The result includes Average Response (ms), Error, Host or IP, and packet sent/received/transmitted counts.
3. Restart the AP — If the AP is offline, restart the AP and verify that it reconnects to the Onyx Portal.
4. Check the physical connection to the AP — Verify the following:
   * Cabling
   * Switch port
   * PoE/PoE++ delivery to the AP
5. Verify the interface status — Verify that both Onyx Edge Interface Status and AP Interface Status are green.

**Resolution**

The issue is resolved when all of the following conditions are met:

* AP Online Status returns to green.
* The AP reconnects to the Onyx Portal.
* The UE is able to detect the network.

**Escalation**

Escalate to GXC Technical Support if AP Online Status remains red after restart. Provide the following information:

* AP's serial number
* Active Onyx Portal alerts
* Onyx Edge diagnostic bundle
* AP Online Status

</details>

<details>

<summary><strong>AP Online but Not Broadcasting / RF Transmission Issue</strong></summary>

**Symptoms**

A UE displays "*No Service*" even though the AP appears to be online.

**Indicators**

* Alerts: One or more of the following alerts may be active:
  * "Cell Service Down"
  * "Timing Out of Sync"
  * "CBSD Registration Failure"
  * "SAS Connection Failure"
* AP summary page and AP details page: AP's Online Status is green, but Service Status is red or amber; Sync Status is red.
* (CBRS deployments) AP details page > **Overview** tab > **RAN** section > **SAS Status**: The **Registered**, **Granted**, and/or **Authorized** indicators are not green.
* (CBRS deployments) AP details page > **Logs** tab: **DP Logs** shows the actual CBSD-to-SAS message exchange, filterable by CBSD ID. On 4G APs, a separate generic **Logs** table (Timestamp/Severity/Message) is also available.

**Possible Causes**

* RF transmission is disabled, or the cell is not broadcasting
* The AP has lost PTP or GPS synchronization
* (CBRS deployments) The AP failed to obtain or maintain a SAS grant

**Actions**

1. Verify AP Service Status — In the AP summary page, verify that the AP's Service Status is green.
2. Verify Cell Transmit configuration — In the AP details page > **Overview** tab > **RAN** section, verify the following:

   * The **Transmit** parameter is set to **Enabled**.
   * The configured operating frequency is correct.
   * The configured transmit power matches the deployment requirements.

   If RF transmission is disabled, the UE will not detect the network.
3. Verify the AP's Sync Status — In the AP summary page or AP details page, verify the Sync Status. Also check **Alerts** > **Active Alerts** for the "Timing Out of Sync" alert.
4. (CBRS deployments only) Verify the SAS status:
   * In the AP details page > **Overview** tab > **RAN** section > SAS Status, verify that the **Registered**, **Granted**, and **Authorized** indicators are green.
   * Check **Alerts** > **Active Alerts** for the "CBSD Registration Failure" and "SAS Connection Failure" alerts.
   * Review the DP logs for the SAS grant request and response exchange.

**Resolution**

The issue is resolved when all of the following conditions are met:

* AP Online Status, Service Status, and Sync Status return to green.
* RF transmission is enabled with the correct frequency and power configuration.
* For CBRS deployments, SAS Status shows Authorized in green.
* The UE is able to detect the network.

**Escalation**

Escalate to GXC Technical Support if the AP's Service Status remains red with no configuration issue identified, or if the AP's Sync Status remains red after verifying the synchronization source. Provide the following information:

* AP vendor-assigned serial number
* Active Onyx Portal alerts details
* Onyx Edge diagnostic bundle
* AP Service Status
* AP Sync Status

(CBRS deployments only) Escalate to SAS vendor support if SAS grant failures persist after applying the workaround. Provide the following information:

* CBSD ID
* SAS Status screenshot
* Grant request and response logs
* SAS event logs (if available)

</details>

<details>

<summary><strong>UE Registration/Attach Rejected by Onyx Edge</strong></summary>

**Symptoms**

The UE detects the network but fails to attach or register, remaining in "*No Service*" or "*Searching*" despite adequate signal strength.

**Indicators**

* **Dashboard** > **Subscribers**: The subscriber count does not increment, and no active subscriber session is displayed.
* Subscriber summary page: The **Status** column does not show an active or connected state for the UE's IMSI.
* (4G) AP details page > **Overview** tab: The **Subscribers** panel does not list the UE.
* (5G) Onyx Edge details page > **Overview** tab: The **Subscribers** panel does not list the UE.
* Onyx Edge details page > **Events & Alerts** tab > **Events**: `attach_reject`, `attach_reject_imei_check` (4G), `registration_reject`, `session_rejected` (5G), or `ip_allocation_failure` events are recorded for the UE's IMSI.
* Onyx Edge details page > **Logs** tab (Core): 4G EMM Reject Cause Codes or 5G AMF Reject Cause Codes recorded for the attach or registration attempt.

**Possible Causes**

* The UE SIM is not provisioned in the Onyx Edge subscriber database.
* The subscriber is inactive or is not assigned a valid APN profile.
* The UE PLMN or APN configuration does not match the Onyx Edge network configuration.
* The Onyx Edge is rejecting the UE attach or registration request.

**Actions**

1. Verify subscriber provisioning — In the Subscriber details page, verify:
   * The IMSI is provisioned
   * The subscriber is active
   * A valid APN profile is assigned
2. Verify UE PLMN and APN configuration — Cross-check the following:
   * **PLMN**: In the Network details page > **Core** section, note the configured PLMN ID (e.g., "315 010"). Compare this against the first five to six digits of the subscriber's IMSI, displayed in the Subscriber details page. These digits represent the UE's provisioned PLMN (MCC+MNC) and should match the network's PLMN ID.
   * **APN**: In the Subscriber details page, confirm the APN matches the provisioned subscriber profile. An incorrect PLMN or APN configuration can prevent UE registration.
3. Check UE registration logs — In the Onyx Edge details page > **Logs** tab, select the **Core** radio button and review the 4G EMM Reject Cause Codes or 5G AMF Reject Cause Codes. Use the reject cause to determine whether the failure is related to subscriber provisioning, authentication, or network configuration.
4. Check attach or registration reject events — In the Onyx Edge details page > **Events & Alerts** tab > **Events** section, select the **Core** filter and look for `registration_reject`, `session_rejected`, or `ip_allocation_failure` events associated with the UE's IMSI.
5. Confirm manual network selection — If the UE is configured for automatic network selection, it may attempt to attach to a neighboring or incompatible network instead of the intended one. Disable automatic network selection on the UE and manually select the correct PLMN before continuing to troubleshoot.
6. Check for RAN connection-setup failure events — In the Onyx Edge details page > **Events & Alerts** tab > **Events** section, select the **NR** filter and look for `f1_ue_context_setup_failure` or `ng_initial_context_setup_failure` events.\
   A high volume of these events, rather than an isolated rejection, typically indicates connection-attempt resource exhaustion or a fronthaul or connection setup issue on the cell rather than a subscriber provisioning issue, and should be investigated separately from individual attach-reject cases.
7. Test with a known-good SIM — Replace the UE SIM with a known-good, provisioned GXC test SIM.
   * If the test SIM successfully attaches, the issue is likely related to the original SIM provisioning or UE configuration.
   * If the test SIM also fails, continue troubleshooting the network infrastructure.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The UE detects the network
* The UE successfully attaches or registers
* An active subscriber session is visible in the Subscriber details page

**Escalation**

Escalate to GXC Technical Support if the known-good test SIM also fails to attach, indicating a network infrastructure issue. Provide the following information:

* Onyx Edge diagnostic bundle
* UE IMSI (if available)
* UE registration logs
* Active Onyx Portal alerts

</details>

<details>

<summary><strong>Insufficient RF Coverage or UE/Band Incompatibility</strong></summary>

**Symptoms**

The UE displays a weak signal, no signal, or is unable to detect the network on the configured operating band.

**Indicators**

* AP summary page and AP details page: AP's Online Status and Service Status are both green, ruling out AP-side faults.

**Possible Causes**

* Interference
* Insufficient RF coverage at the UE location because of distance, obstructions, or antenna placement.
* Incorrect antenna placement or antenna connections.
* The UE does not support the configured operating band.
* Incorrect cell frequency or transmit power configuration for the deployment.

**Actions**

1. Use Signal Analyzer to rule out interferencing frequencies, and change the frequencies accordingly.
2. For SAS deployments, use SAS portal or other available (third-party cellular network scanners, etc) means to verify neighboring frequencies and interferences.
3. If no interference is detected, measure the received signal strength (RSRP) at the UE location using the GXC Survey App or any UE diagnostic application tool.
4. If the measured signal level is below the recommended range, verify the following:
   * Antenna placement (is as per RF design recommendations, LOS obstructions, etc.)
   * Antenna connections (check the *AP Installation Guide* for cabling and cable connections)
   * Reposition the AP or UE if necessary
5. Verify UE band support — Confirm that the UE supports the operating frequency band configured on the AP.

{% hint style="info" %}
**NOTE**: 5G APs operate only in Single Carrier mode. Carrier Aggregation (CA) and Dual Connectivity (DC) modes are not supported.
{% endhint %}

**Resolution**

The issue is resolved when all of the following conditions are met:

* RSRP is within the acceptable range for the UE location.
* The UE successfully attaches on a supported band.

**Escalation**

Escalate to GXC Technical Support if RF coverage and operating band compatibility have been verified, but the UE still cannot attach, or if a coverage gap is identified that requires additional AP placement or RF design review. Provide the following information:

* AP vendor-assigned serial number
* Measured RSRP values and measurement location
* Antenna configuration details
* Active Onyx Portal alerts details

</details>

### Data Not Working

<details>

<summary><strong>PDN/PDU Session Not Established (No IP Address Assigned)</strong></summary>

**Symptoms**

The UE is attached to the network and displays signal strength, but no IP address is assigned, and the PDN/PDU session is either not established or is immediately terminated.

**Indicators**

* Subscriber details page > **Overview** tab:
  * **IP Address** — No IP address assigned.
  * PDN/PDU session is not established or is immediately terminated.
* On a commercial UE, the device typically shows signal bars but no data-connection icon (for example, no "4G"/"5G" indicator) when no IP address has been assigned.

**Possible Causes**

* Onyx Edge packet core misconfiguration preventing PDN/PDU session establishment.
* APN profile mismatch between the UE and the subscriber group's APN assignment.
* IP address pool exhausted in Onyx Edge DHCP/IP allocation.

**Actions**

1. Check PDN/PDU session — In the Subscriber details page > **Overview** tab, verify that a data session is active and an IP address has been assigned.
2. Verify APN profile — In the Subscriber Group details page > **APN**, verify the configured APN. Compare it with the APN configured on the UE. Even a single-character mismatch can prevent the PDN/PDU session from being established.
3. In VLAN deployments, verify the enterprise DHCP sever is configured correctly to assign IP addresses to the UE that are mapped to the corresponding APN.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The UE has an active data session.
* An IP address is assigned.

**Escalation**

Escalate to GXC Technical Support if the PDN/PDU session is not established and no configuration issue has been identified. Provide the following information:

* Onyx Portal diagnostic bundle
* APN configuration export
* Subscriber details page screenshot showing session state and IP assignment

</details>

<details>

<summary><strong>Session Established but Data Traffic Fails</strong></summary>

**Symptoms**

The UE has an active data session and an assigned IP address, but application traffic fails, or a Subscriber Speed Test fails or returns 0 Mbps.

**Indicators**

* The UE must be in Connected mode (not Idle) with an assigned IP address; a UE in Idle mode will not produce a meaningful data-traffic test result.
* The UE displays signal strength and has an assigned IP address, but all application traffic fails.
* Subscriber Speed Test at `http://uespeedtest.gxc.io` fails or returns 0 Mbps.

**Possible Causes**

* Networking mode misconfiguration (NAT/Bridge mode mismatch) causes data traffic to fail even though the session is established.
* (VLAN deployments) Incorrect VLAN-to-DHCP mapping or VLAN-to-gateway routing prevents traffic from reaching the correct gateway.
* Firewall rules on the Onyx Edge or upstream enterprise firewall block traffic from the UE subnet.
* DNS is unreachable from the UE's assigned IP address.
* The Onyx Edge uplink interface (connection to enterprise LAN/WAN) is down.

**Actions**

Base checks (all deployments):

1. Run a Subscriber Speed Test — On the UE, open a web browser and navigate to `http://uespeedtest.gxc.io`. This test verifies the UE-to-Onyx Edge data path independently of Internet access. Test results, including uplink, downlink, jitter, RTT, are recorded in the Subscriber details page > **Overview** tab > **Speed Test Details** section.
2. Ping the UE from the Onyx Portal — Confirms the UE is reachable from the Onyx Edge, independently of internet access. If steps "Run a Subscriber Speed Test" and "Ping the UE from the Onyx Portal" both succeed, the UE-to-radio path is confirmed healthy; any remaining data-traffic failure is on the network/internet side, and the deployment-specific checks below apply.

For NAT deployments:

1. Verify the Networking Mode is set to NAT and matches the deployment design. In NAT mode, the UE cannot be pinged directly from an external enterprise server; rely on steps "Run a Subscriber Speed Test" and "Ping the UE from the Onyx Portal" above to confirm the UE-to-radio path.

For VLAN deployments:

1. Verify the enterprise DHCP server is mapped to the correct VLAN and is assigning IP addresses correctly (see [*PDN/PDU Session Not Established*](#pdn-pdu-session-not-established-no-ip-address-assigned)).
2. Ping the UE from the enterprise VLAN network — This confirms the enterprise-side path and is only applicable to VLAN deployments; NAT deployments cannot ping the UE directly from an external server.
3. Verify that VLAN-to-gateway routing is configured correctly so that traffic from the UE's VLAN reaches the correct gateway.

If steps "Run a Subscriber Speed Test" and "Ping the UE from the Onyx Portal" succeed but Internet access still fails:

1. Run an Internet speed test using a third party tool (for example, Ookla) to confirm whether the failure is Internet-specific rather than a base connectivity issue.
2. Check firewall rules for the UE subnet — Verify firewall rules on the Onyx Edge and upstream enterprise firewall permit traffic from the UE subnet. Refer to the *Firewall Configuration Application Note*.
3. Test DNS resolution from the UE — On the UE, run `nslookup` or `dig`. Verify that the DNS server configured in the APN profile is reachable and resolves DNS queries correctly from the UE's allocated IP address.
4. Capture traffic at the Onyx Edge data plane — Use the Onyx Edge CLI packet capture tools to verify that GTP-U/N3 user-plane traffic is flowing. Compare uplink and downlink traffic to determine which direction is affected.

**Resolution**

The issue is resolved when all of the following conditions are met:

* Data traffic flows successfully.
* A Subscriber Speed Test completes successfully.

**Escalation**

Escalate to GXC Technical Support if the Onyx Edge uplink has been verified and data traffic continues to fail with no firewall, DNS, or networking mode issue identified. Provide the following information:

* Onyx Portal diagnostic bundle
* Onyx Edge uplink interface status
* Subscriber details page screenshot showing the session state and IP address assignment

**Reference**

* *Firewall Configuration Application Note*
* *Networking Modes Application Note*

</details>

### Intermittent Connection Drops

<details>

<summary><strong>Poor Radio Conditions (No Handover Involved)</strong></summary>

**Symptoms**

The UE repeatedly loses connectivity even when it is not moving between coverage areas and no handover is involved.

**Indicators**

* Subscriber details page > **Events** tab: Detach or drop events are not correlated with handover events.
* (5G) Subscriber details page > **Analytics** tab > **Plots** tab: CQI and RSRP are degraded leading up to the drop.
* (4G) A UE diagnostic application reports degraded RSRP or CQI at the UE location leading up to the drop.

**Possible Causes**

* Poor radio conditions alone (weak signal, interference, or marginal RSRP/CQI) can cause a UE to intermittently drop even with no mobility or handover involved.

**Actions**

1. Check the Events log for handover correlation — In the Subscriber details page > **Events** tab, confirm whether the drop coincides with a handover attempt. If it does not, treat the drop as a radio-quality issue rather than a handover issue.
2. Check radio quality graphs —&#x20;
   * (5G) In the Subscriber details page > **Analytics** tab > **Plots** tab, review CQI and RSRP.
   * (4G) Use a UE diagnostic application to check RSRP and CQI at the UE location. Degraded CQI or RSRP leading up to a drop indicates a radio-quality issue rather than an equipment fault.

**Resolution**

The issue is resolved when the UE maintains a stable session with no unexpected drops, and CQI/RSRP remain within the expected range in the Subscriber details page > **Analytics** tab > **Plots** tab.

**Escalation**

Escalate to GXC Technical Support if connection drops continue after confirming poor radio conditions with no handover correlation and no equipment or configuration issue identified. Provide the following information:

* Subscriber details page > **Analytics** tab > **Plots** tab screenshot showing the CQI/RSRP trend
* Subscriber details page > **Events** tab screenshot showing drop timestamps
* Active Onyx Portal alerts details

</details>

<details>

<summary><strong>Handover Failure Between APs</strong></summary>

**Symptoms**

The UE repeatedly loses connectivity while moving between coverage areas served by neighboring APs.

**Indicators**

* Subscriber details page > **Events** tab: UE detach and reattach events coincide with AP handover attempts.
* **Alerts** > **Active Alerts**: Handover-related alerts correlate with connection drop timestamps.

**Possible Causes**

* Coverage gaps or poor RF conditions prevent successful handovers between neighboring APs.

**Actions**

1. Check UE session history — In the Subscriber details page > **Events** tab, verify whether the connection drops coincide with AP handover attempts.
2. Check for AP-level alerts during connection drops — In **Alerts** > **Active Alerts**, filter the alerts by the connection drop timestamps and look for handover-related alerts.

**Resolution**

The issue is resolved when the UE maintains a stable session during handovers, with no unexpected detach events in the Subscriber details page > **Events** tab.

**Escalation**

Escalate to GXC Technical Support if connection drops continue with no AP alerts or configuration issues identified. Provide the following information:

* Active Onyx Portal alert details
* Subscriber details page > **Events** tab screenshot showing the connection drop timestamps

</details>

<details>

<summary><strong>(5G) RF Interference / Radio Link Failure (5G TDD Band)</strong></summary>

**Symptoms**

A UE connected to a 5G network experiences intermittent connection drops with no correlation to handovers or equipment status.

**Indicators**

* Subscriber details page > **Events** tab: Connection drops occur with no correlation to handover or detach events.
* No corresponding AP, Mesh Node, or FHM status indicator is red/amber.

**Possible Causes**

* (5G) High interference in the TDD band causes intermittent Radio Link Failures (RLFs).

**Actions**

1. Run Spectrum Analyzer — External interference in the TDD band can increase the noise floor, resulting in intermittent Radio Link Failures (RLFs) and session drops. Refer to the *5G Spectrum Analyzer Application Note* to perform a spectrum scan.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The spectrum scan shows no significant interference.
* Connection drops no longer occur.

**Escalation**

Escalate to GXC Technical Support if connection drops continue with no AP alerts or configuration issue identified. Provide the following information:

* Active Onyx Portal alerts details
* Spectrum scan results

**Reference**

* *5G Spectrum Analyzer Application Note*

</details>

<details>

<summary><strong>Mesh Node / FHM Backhaul Link Degradation</strong></summary>

**Symptoms**

All UE served through a specific Mesh Node or FHM lose connectivity simultaneously or experience intermittent connectivity, rather than a single UE being affected.

**Indicators**

* Status Indicators: The Mesh Node or FHM status indicators are red or amber during the connection drops.

**Possible Causes**

* (4G) Onyx Mesh backhaul link instability interrupts connectivity for all UE served through the affected Mesh Node.
* (5G) FHM fiber or synchronization loss causes connected RUs to go offline.

**Actions**

1. (4G Mesh deployments) Check Mesh Node link health — In the Mesh Node details page, review the Interface Status indicator.

   * A red Interface Status indicates the GRE tunnel is unavailable.
   * An amber Interface Status indicates the Mesh Node is online, but the Mesh AP is offline.

   A degraded mesh backhaul link can cause all downstream UE to disconnect. Reposition the Mesh Node if the signal quality is low.
2. (5G FHM deployments) Verify FHM status — In the FHM summary page or FHM details page, verify that the FHM's Online Status, Interface Status, and Sync Status indicators are green.\
   A red Sync Status indicates that the FHM is out of synchronization, affecting all connected RUs.
3. (5G) Inspect the fiber connections and SFP modules — Dirty or damaged fiber connector ends are a common cause of intermittent fronthaul packet loss and brief (roughly 1–30 second) service glitches. Inspect the fiber connector ends for contamination or damage, and clean or replace them as necessary.\
   Verify that the installed SFP modules are compatible with the FHM and AP hardware. Where possible, use vendor-approved single-mode SFP+ modules.\
   In previous deployments, replacing a copper SFP with a fiber SFP+ module has resolved recurring intermittent connectivity issues.
4. (5G) Verify PTP transport compatibility — Verify that the AP firmware version supports the configured PTP transport mode. \
   PTP-over-Ethernet in Single Carrier mode is the recommended and supported configuration. Some PTP transport modes are not supported on older AP firmware versions and require a firmware upgrade rather than a configuration change.
5. (5G) Correlate packet loss with logs — In the Onyx Edge details page > **Logs** tab, filter by the logs for the fronthaul management service and review packet loss events. Correlate the event timestamps with the reported service interruptions to confirm that the fronthaul link is the source of the issue before escalating.

**Resolution**

The issue is resolved when all of the following conditions are met:

* Mesh Node or FHM status indicators are green.
* All previously affected UE maintain stable sessions.

**Escalation**

Escalate to GXC Technical Support if Mesh Node link degradation is confirmed and repositioning does not resolve the issue, or if an FHM synchronization failure persists after rebooting the FHM. Provide the following information:

* FHM and AP serial numbers (FHM-related issues)
* Mesh Node topology screenshot (Mesh-related issues)

**Reference**

* *4G Mesh Node Application Note*

</details>

<details>

<summary><strong>Onyx Edge Process Restart or Watchdog Recovery</strong></summary>

**Symptoms**

All UE at the site briefly lose connectivity simultaneously and then recover without any AP, Mesh Node, or FHM reporting a persistent fault.

**Indicators**

* **Alerts** > **Active Alerts**: No AP/Mesh/FHM-specific alerts correspond to the connection drop. The outage is brief and affects the entire site.

**Possible Causes**

* An Onyx Edge process restart or Watchdog recovery causes a temporary core network outage.

**Actions**

1. Check the Onyx Edge process health — In the Onyx Edge summary page or Onyx Edge details page, verify the Service Status indicator. A red Service Status indicates that a critical service has gone down or restarted more than three times in the last five minutes, which can result in a brief service interruption for all attached UE.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The Onyx Edge Service Status is green.
* No further site-wide brief service interruptions occur.

**Escalation**

Escalate to GXC Technical Support if the Onyx Edge's Service Status remains red or if repeated service restarts recur. Provide the following information:

* Active Onyx Portal alerts details
* Onyx Edge diagnostic bundle

</details>

<details>

<summary><strong>UE Idle Timer Mismatch</strong></summary>

**Symptoms**

A single UE repeatedly detaches and reattaches, while no other UE connected to the same AP are affected, and no AP, Mesh Node, FHM, or Onyx Edge status indicator is red.

**Indicators**

* Subscriber details page > **Events** tab: Repeated attach and detach events are recorded for a single UE, with no corresponding alerts or equipment status changes.

**Possible Causes**

* A mismatch between the UE idle timer (T3412/T3412-ext) and the Onyx Edge configuration causes unexpected UE detach events.

**Actions**

1. Check UE session history — In the Subscriber details page > **Events** tab, verify that the detach pattern is isolated to the affected UE and occurs periodically, indicating an idle timer mismatch rather than an RF or equipment-related issue.

**Resolution**

The issue is resolved when the UE maintains a stable attached session with no unexpected detach events.

**Escalation**

Escalate to GXC Technical Support if the periodic detach pattern persists after confirming that no RF or equipment-related issue exists. Provide the following information:

* Subscriber details page > **Events** tab screenshot showing drop timestamps
* UE make/model
* UE timer configuration (if available)

</details>

<details>

<summary><strong>CBRS SAS Grant Expiry During Active Session</strong></summary>

**Symptoms**

In a CBRS deployment, the UE repeatedly and intermittently loses connectivity to a specific AP, recovering after several minutes each time, and the AP periodically reports SAS-related alerts.

**Indicators**

* **Alerts** > **Active Alerts**: One or more of the following alerts may be active, indicating SAS grant renewal activity:
  * "CBSD Registration Failure"
  * "SAS Connection Failure"
* AP details page > **Overview** tab > **RAN** section > SAS Status: Authorized indicator is not green.

**Possible Causes**

* The SAS is frequently changing the CBRS grant's frequency or power parameters. Each grant change causes the Onyx Edge to push a new configuration to the AP and reboot it, which briefly disconnects all UE served by that AP until it comes back online (typically a few minutes). This differs from a single grant expiry, which would leave the AP offline rather than intermittently cycling.

**Actions**

1. Check the CBRS SAS grant status — In the AP details page > **Overview** tab > **RAN** section > SAS Status, verify whether the Authorized indicator is green. Also, check **Alerts** > **Active Alerts** for "CBSD Registration Failure" and "SAS Connection Failure" alerts over time, which indicate recurring grant changes rather than a single expiry event.
2. Verify the CBSD certificates and identifiers — Verify the following:
   * The CBSD SAS CA certificate, CBSD certificate, and CBSD key are valid and have not expired.
   * The FCC ID and serial number configured on the AP match the values registered with the SAS.
   * For Category B CBSDs, Certified Professional Installer (CPI) validation has been completed. An incomplete CPI validation prevents CBSD registration regardless of certificate validity.
3. Verify connectivity to the SAS server — Verify that the Onyx Edge can resolve and connect to the configured SAS provider. Confirm that DNS resolution is working and that outbound connectivity on the required port is available. A SAS grant failure that appears to be an expired grant can also result from the Onyx Edge being unable to communicate with the SAS.
4. Consider requesting an alternate frequency channel — If the SAS grant has been denied or terminated because of incumbent activity or interference on the current channel, requesting a different frequency channel for the CBSD may restore transmission more quickly than waiting for the incumbent condition to clear.

**Resolution**

The issue is resolved when all of the following conditions are met:

* SAS Status shows Authorized in green.
* No further grant-related connection drops occur.

**Escalation**

Escalate to the SAS Vendor Support if SAS grant renewal failures persist. Provide the following information:

* CBSD ID
* SAS Status screenshot
* Grant request and response logs
* SAS event logs (if available)

</details>

### Low Data Throughput or Poor Performance

<details>

<summary><strong>Poor RF Conditions or Interference</strong></summary>

**Symptoms**

A single UE experiences low throughput while other UE on the same AP perform normally.

**Indicators**

* Alerts:
  * "PHC2SYS Offset is too High" or "TS2PHC Offset is too High" alerts (Warning, Onyx Edge (NR)), or "Early eCPRI Packets," "Late eCPRI Packets," or "eCPRI Rx Down" alerts (Warning, Onyx Edge (NR)), are active. Any of these can indicate a timing or fronthaul condition that degrades throughput and RF KPIs even when the radio link itself looks otherwise healthy.
  * "Cell High DL BLER" or "Cell High UL BLER" alerts (Warning, fires when BLER exceeds 50% for at least 15 minutes) are active.
  * "Insufficient SAS Power" alert (Major) is active — Indicates the SAS granted power is below the AP hardware minimum and transmission has been disabled as a result.
* (5G) Subscriber details page > **Analytics** tab > **Plots** tab > **5G Only KPIs**: Review the full KPI set — DL and UL RSRP, CQI, DL and UL MCS, DL and UL BLER, and DL and UL Rank — rather than downlink metrics alone. Reviewing all graphs helps identify whether the uplink or the downlink is affected before proceeding to *Actions*.
* (4G) A UE diagnostic application reports low RSRP at the UE location.

**Possible Causes**

* Poor RF conditions reduce RSRP or SINR, resulting in a lower Modulation and Coding Scheme (MCS) and reduced throughput.
* (5G) External interference in the TDD band increases the noise floor, reducing SINR and MCS.
* (5G, G105/G505 deployments) MIMO rank degradation — persistent RI=1 on a 4T4R radio typically indicates antenna placement, polarization, or obstruction issues.
* Poor timing synchronization or fronthaul packet loss can degrade throughput and RF-related KPIs independently of RF conditions.
* (Multi-Cell deployments) Two or more cells configured with the same operating frequency can cause self-interference.
* (Multi-Port deployments) Overlapping coverage between adjacent sectors can degrade performance for a UE located within the overlap zone, even when the antennas are correctly oriented.
* Insufficient line of sight, significant RF reflections, or distance beyond the expected range for the configured frequency and power can limit signal quality independently of any hardware, software, or configuration issue.

**Actions**

1. Check RF signal quality for the affected UE — Review the full KPI set described above (both link directions), not DL metrics alone, to determine which link is affected and narrow down the likely cause.
   * (5G) In the Subscriber details page > **Analytics** tab > **Plots** tab > **5G Only KPIs**, review DL/UL RSRP, DL/UL MCS, DL/UL BLER, and DL/UL Rank. Low RSRP or SINR below 10 dB significantly reduces MCS and throughput. Persistent RI=1 on a 4T4R radio typically indicates an antenna placement, polarization, or obstruction issue.
   * (4G) Use a UE diagnostic application to measure RSRP at the UE location.
2. Check for timing and fronthaul alerts — In **Alerts** > **Active Alerts**, check for "PHC2SYS Offset is too High," "TS2PHC Offset is too High," "Early eCPRI Packets," "Late eCPRI Packets," and "eCPRI Rx Down" alerts. Any of these firing can also account for degraded throughput or KPIs and should be ruled out alongside RF conditions.
3. Check for BLER and SAS power alerts — In **Alerts** > **Active Alerts**, check for "Cell High DL BLER," "Cell High UL BLER" (Warning), and "Insufficient SAS Power" (Major). Any of these confirm the condition seen in the KPI graphs rather than requiring a manual read of the raw BLER values.
4. (5G) Run Spectrum Analyzer to check for external interference in the TDD band. Refer to the *5G Spectrum Analyzer Application Note* for performing and interpreting a spectrum scan on the affected RU.
5. Identify high-uplink-consumption UE — When uplink throughput or UL BLER issues affect multiple UE on a cell rather than a single UE, review the per-UE uplink PRB utilization for the affected cell.
   * For each UE with high uplink resource utilization, review the following: RSRP, SINR, UL MCS, Throughput, BLER/HARQ retransmissions, and Power headroom.
   * A UE located near the cell edge with poor signal quality can consume a disproportionate amount of uplink resources, reducing throughput for other UE on the same cell. In this case, additional AP coverage in the weak-signal area is a more effective solution than further RF tuning of the existing cell.
6. Verify the AP transmit power and SAS grant — Verify that the AP transmit power has not been reduced below the configured or supported value for the deployed AP model. Also, verify whether the current SAS grant provides the full authorized bandwidth or has been reduced (for example, following a Carrier Aggregation-to-Single Carrier transition).\
   A reduced SAS grant can produce symptoms similar to RF coverage or interference issues. If the AP is not already transmitting at its maximum supported power, consider increasing the transmit power in incremental steps to determine whether higher power improves signal quality.\
   Skip this step if the AP is already at maximum power, or if the SAS grant is confirmed to be limiting available power. In the AP details page > **Overview** tab > **RAN** section, check **Preferred Max Pwr (dBm)**.
7. Verify multi-cell frequency configuration — In multi-cell deployments, verify that each cell is configured with a distinct operating frequency. The Onyx Portal does not currently prevent two or more cells from being configured with the same frequency; if this occurs, the cells can interfere with each other, which may appear as interference in a spectrum scan even though no external signal is involved.
8. Verify multi-port antenna orientation — In multi-port deployments, verify that antennas are oriented orthogonally to each other. Some coverage overlap between adjacent sectors is expected even with correct orientation; a UE located within an overlap zone may experience degraded performance as a result of this expected overlap rather than a fault.
9. Verify the UE's physical location and line of sight — If no interference is detected and the AP is transmitting at or near its maximum power but signal quality remains poor, verify the UE's location: confirm line of sight to the AP, check for an environment with significant RF reflections or obstructions, and confirm the UE is within the expected coverage distance for the configured frequency and power. Reduced signal quality at long range, given the configured frequency and power, can be an expected result of distance rather than an issue requiring resolution.
10. Physically inspect the antenna installation — As a final step, once all other causes have been ruled out, verify the antenna's mechanical and electrical installation: antenna direction, antenna gain, and antenna connections. This step typically requires an on-site technician.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The UE achieves throughput consistent with the configured subscriber QoS profile.
* Confirm the expected performance with a Subscriber Speed Test.

**Escalation**

Escalate to GXC Technical Support if throughput remains degraded after verifying RF signal quality and checking for interference. Provide the following information:

* Subscriber details page > **Analytics** tab screenshot (**Plots** > **5G Only KPIs**, if applicable)
* AP placement and antenna details

**Reference**

* *5G Spectrum Analyzer Application Note*

</details>

<details>

<summary><strong>(4G) Carrier Aggregation Issues</strong></summary>

**Symptoms**

A 4G UE on a G100/G101/G501 AP is not getting the expected Carrier Aggregation (CA) throughput. This may present as CA never activating, or as throughput that correlates with frequent CA activation/deactivation cycles rather than a hard connectivity drop.

**Indicators**

* **Analytics** page: CA KPIs for the affected AP indicate that CA is not activating, or show frequent CA activation and deactivation cycles.

**Possible Causes**

* GXC 4G APs (G100/G101/G501) support two-component Carrier Aggregation (2CC CA) to increase peak throughput. CA may fail to activate, or may activate and deactivate frequently, for the reasons below — in either case, throughput will be lower than the expected CA-enabled level.

**Actions**

1. Verify both carriers are enabled — In the AP details page > **Overview** tab > **RAN** section, confirm that both the primary and secondary carrier are configured and set to **Transmit**: Enabled. If the secondary carrier is disabled, CA cannot activate regardless of RF conditions.
2. Check the secondary carrier's SAS grant status (CBRS deployments) — In the AP details page > **Overview** tab > **RAN** section > SAS Status, verify that the **Registered**, **Granted**, and **Authorized** indicators are green for the secondary carrier. CA cannot activate without an active grant on the secondary carrier.
3. Confirm UE CA capability — Verify the UE supports 2CC Carrier Aggregation and the specific band combination configured on the AP.
4. Check RF signal quality on the secondary carrier — Poor signal on the secondary carrier alone can prevent activation even when it is enabled and granted. Poor or marginal RF conditions on the secondary carrier can also cause the system to deactivate it automatically once active, producing the frequent activation/deactivation cycles described above; this is expected system behavior in response to degraded conditions, not a fault.
5. Check CA stability — In the Analytics page, review the CA KPIs for the affected AP. Frequent CA activation and deactivation indicate RF instability that may require AP repositioning or transmit power adjustments.

**Resolution**

The issue is resolved when all of the following conditions are met:

* CA activates as expected and remains stable.
* Throughput reaches the expected CA-enabled level, and no longer correlates with CA state changes.

**Escalation**

Escalate to GXC Technical Support if CA does not activate despite confirmed correct AP and UE configuration, or if instability persists after repositioning or transmit power adjustments. Provide the following information:

* Analytics page PRB/CA KPI export for the affected AP
* AP placement and antenna details
* Active Onyx Portal alerts details

</details>

<details>

<summary><strong>(5G) Cell Congestion / Uplink Bandwidth Saturation (High PRB Utilization)</strong></summary>

**Symptoms**

Multiple UE on the same AP experience reduced throughput simultaneously, or all UE across the site share a throughput ceiling that matches the enterprise WAN capacity regardless of radio performance.

**Indicators**

* **Analytics** page (AP-level): **DL PRB Utilization** or **UL PRB Utilization** remains consistently above 80% for the serving AP. PRB Utilization is available only at the AP level and is not displayed in the Subscriber details page > **Analytics** tab.
* Site-wide throughput is consistently limited to the known enterprise WAN bandwidth, independent of RF conditions or Physical Resource Block (PRB) utilization — this points to the Onyx Edge uplink rather than cell congestion (see *Actions*).

**Possible Causes**

* Cell congestion caused by too many UE competing for the available PRBs on the serving AP.
* Onyx Edge uplink bandwidth saturation — The enterprise WAN connection, rather than the radio network, limits throughput.

**Actions**

1. Check the AP PRB utilization — In the **Analytics** page, review the **DL PRB Utilization** and **UL PRB Utilization** for the affected AP. PRB utilization consistently above 80% indicates cell congestion. Consider adding additional APs or redistributing subscribers across neighboring cells to reduce the load on the affected AP.
2. If PRB utilization is within normal range but throughput is limited site-wide to a value consistent with the enterprise WAN capacity, test the Onyx Edge uplink independently. Run iPerf3 between the Onyx Edge and an enterprise server. If the enterprise uplink is the bottleneck, UE throughput cannot exceed the available WAN bandwidth regardless of radio performance.

**Resolution**

The issue is resolved when all of the following conditions are met:

* PRB utilization returns to a sustainable level, or the Onyx Edge uplink is confirmed not to be the bottleneck (or the enterprise WAN capacity is increased to meet the required throughput).
* Throughput for the affected UE returns to the expected level.

**Escalation**

Escalate to GXC Technical Support if PRB utilization is low but throughput remains degraded, with no RF or QoS issue identified, or if the uplink bandwidth is confirmed saturated and the available WAN capacity cannot be increased. Provide the following information:

* Analytics page PRB utilization export for the affected AP

</details>

<details>

<summary><strong>QoS Policy Misconfiguration (MBR Limiting)</strong></summary>

**Symptoms**

Throughput is consistently capped at a fixed rate for a subscriber or subscriber group, regardless of RF conditions or cell load.

**Indicators**

* The observed throughput consistently matches the configured Maximum Bit Rate (MBR) rather than varying with RF conditions or PRB utilization.

**Possible Causes**

* QoS policy misconfiguration causes the configured MBR to limit subscriber throughput.

**Actions**

1. Check subscriber QoS profile — In the Subscriber Group details page, review the following parameters:

   * Aggregate BW (Upload)
   * Aggregate BW (Download)
   * QoS Category

   An overly restrictive MBR limits subscriber throughput regardless of RF conditions. Refer to the *QoS Management Application Note*.

**Resolution**

The issue is resolved when the UE achieves throughput consistent with the intended subscriber QoS profile.

**Escalation**

Escalate to GXC Technical Support if throughput remains below the configured MBR after profile correction. Provide the following information:

* Subscriber group QoS profile details

**Reference**

* *4G Onyx QoS Management Application Note*

</details>

<details>

<summary><strong>Backhaul Bottleneck (Mesh Node)</strong></summary>

**Symptoms**

All UE served through a specific Mesh Node experience reduced throughput simultaneously, rather than a single UE being affected.

**Indicators**

* Reduced throughput is observed across all UE served by the same Mesh Node, rather than being isolated to a single UE.

**Possible Causes**

* Onyx Mesh backhaul capacity is insufficient, reducing the aggregate throughput available to all downstream UE.

**Actions**

* Check Mesh Node backhaul capacity — Review the Mesh Node Interface Status and backhaul link health. The combined throughput of all UE served by the Mesh Node is limited by the available wireless backhaul capacity.
* (5G) A fronthaul-side problem does not present as a distinct "bottleneck" indicator today — it is more likely to appear as poor BLER on the affected UE rather than as a throughput ceiling. If RF conditions and interference have already been ruled out (see Poor RF Conditions or Interference) and BLER remains poor with no other explanation, capture the Analytics plot and contact GXC Technical Support to review fronthaul-side logs; this data is not currently exportable to customers.

**Resolution**

The issue is resolved when all of the following conditions are met:

* Backhaul capacity is no longer the limiting factor.
* Throughput for the affected UE returns to the expected level.

**Escalation**

Escalate to GXC Technical Support if a Mesh Node backhaul bottleneck is confirmed and cannot be resolved by redistribution, or if a fronthaul-side issue is suspected per the note above. Provide the following information:

* Mesh Node's vendor-assigned serial number
* Analytics page throughput export for affected UE

</details>

## Network Outage & Equipment Offline Issues

This section covers equipment-level outages — Access Points, Mesh Nodes, Onyx Edge, and FHM going offline — as distinct from the UE-experience symptoms above.

<details>

<summary><strong>Onyx Edge Offline / Service Down</strong></summary>

**Symptoms**

One or more Onyx Edge components are offline or unavailable, resulting in a site-wide service outage.

**Indicators**

* Alerts: One or more of the following alerts may be active:
  * "Onyx Edge (Core) Offline" (Critical)
  * "Onyx Edge (NR) Offline" (Critical)
  * "Onyx Edge (Core/NR) Service Down" (Critical)
  * "HA Peer Down" (Major)
* Status Indicators: Onyx Edge's Online Status is red.
* All APs and UE at the site lose connectivity simultaneously.

**Impact**

An Onyx Edge outage affects all connected APs, FHMs, and Mesh Nodes. As a result, all attached UE lose service simultaneously. This represents the highest-priority equipment outage.

**Possible Causes**

* Loss of management-plane connectivity between the Onyx Edge and the Onyx Portal.
* One or more critical services have stopped or are repeatedly restarting.
* Disk utilization exceeds 70% on Onyx Edge (Core) / >80% on Onyx Edge (NR), causing the file system to enter read-only mode.
* Both Onyx Edge nodes in an HA pool were rebooted simultaneously, resulting in service initialization or port allocation conflicts.
* The MeshMgr process failed after changing the Mesh Node Assignment Subnet in an HA deployment.
* Physical fronthaul link failure while Interface Status incorrectly remains green (known 5G limitation).
* The Onyx Edge has been added to the Onyx Portal, but the physical equipment has not yet been deployed, resulting in a false offline alert.

**Actions**

1. Check the Online Status and Interface Status — In the Onyx Edge summary page, verify the Online Status and Interface Status.\
   Hover over the Interface Status icon to view:
   * The number of AMF-connected APs
   * The N6 IP address
   * The last reporting time
2. Verify N6 reachability — Ping the Onyx Edge's N6 IP address from the next hop connecting to it. This confirms whether the Onyx Edge is reachable from the network before checking anything else.
3. If the N6 address is reachable, check whether the underlying server or VM is up — for a physical server, confirm the server is powered on; for a VM-based deployment, log in to the hypervisor/host and confirm the VM is in a running state.
4. Run network diagnostics — Log in to the Onyx Edge as *agw\_user* and run *network\_check\_summary.sh* to identify connectivity, DNS, or interface issues between the Onyx Edge and the internet/enterprise network.
5. Restart services from the Onyx Portal — In the Onyx Edge details page, click **Actions** > **Restart Services**. All UE connectivity is interrupted while services restart. Service recovery typically takes 2-5 minutes.
6. Reboot the Onyx Edge — If restarting services does not resolve the issue, reboot the Onyx Edge. Allow approximately 5-10 minutes for the system to restart and for associated APs to reconnect.
7. If the Onyx Edge is reachable but still shown offline in the Onyx Portal, collect diagnostic bundle — In the Onyx Edge details page, click **Actions** > **Trigger Debug Information Collection**. After collection completes, click **Actions** > **List Debug Information Files in OE** to retrieve the diagnostic bundle.

   <div data-gb-custom-block data-tag="hint" data-style="info" class="hint hint-info"><p><strong>NOTE:</strong> If the Onyx Edge is completely offline, the <strong>Actions</strong> menu may be unavailable. In this condition, <strong>Restart Services</strong>, <strong>Reboot</strong>, and <strong>Trigger Debug Information Collection</strong> are unavailable. A physical power cycle or on-site intervention is required. If remote or on-site access is unavailable, escalate the issue to GXC Technical Support.</p></div>
8. Check for simultaneous HA pool reboot issue — If both Onyx Edge nodes in an HA pool were rebooted simultaneously, critical services may fail to initialize because of port allocation conflicts. Contact GXC Technical Support before performing additional restart or reboot operations.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The Onyx Edge Online Status, Service Status, and Compute Resource indicators show healthy values.
* All associated APs, FHMs, and Mesh Nodes return to an operational state.

**Escalation**

Escalate to GXC Technical Support if the Onyx Edge remains offline after completing the recovery procedures, or if HA pool service failures continue after recovery attempts. Provide the following information:

* Onyx Edge name
* Onyx Edge software version
* Service Status screenshot
* Compute Resource values
* Diagnostic bundle

</details>

<details>

<summary><strong>Mesh Node Offline</strong></summary>

**Symptoms**

A Mesh Node is offline or unavailable.

**Indicators**

* Alerts: "Mesh Node Offline" alert (Critical) is active.
* Status Indicators: Mesh Node's Online Status is red.
* Excessive alert notification emails are generated because of status fluctuations during periods of high backhaul utilization.

**Dependency**

A Mesh Node outage causes its associated Mesh AP(s) to become unavailable. Always resolve the Mesh Node issue before troubleshooting the Mesh AP — see *Mesh AP Offline*.

**Possible Causes**

* Loss of management-plane connectivity between the Mesh Node and the Onyx Edge.
* CPE-CBSD grant unavailable — The Mesh Node is waiting for a SAS grant.
* Cellular modem failure.
* Watchdog-triggered power cycle following a critical service failure.
* High wireless backhaul utilization causes temporary status fluctuations (known limitation).

**Actions**

1. Check whether the Mesh AP (backhaul UE) is connected — In the Dashboard > Subscribers view, check whether the mesh backhaul UE is shown as connected. Because the Mesh Node itself is offline, this is the minimum check available from the Onyx Portal; none of the CLI-based diagnostics below apply until the Mesh Node is back online.
2. If the backhaul UE is not connected, check its event history — In the Subscriber details page > **Events** tab for the backhaul UE, check whether it is frequently cycling between attached and disconnected states. Frequent cycling indicates a link-level issue rather than a one-time outage.
3. Check the backhaul link quality — During periods of high backhaul utilization, the Mesh AP status may fluctuate (known limitation).\
   Review the backhaul signal quality.
4. Check the Mesh Node CPE status — Connect to the Mesh Node using SSH and run `sudo meshcli cpe_status`. Verify that:
   * The Mesh Node is connected to the intended AP.
   * The reported RSRP is better than -90 dBm.
5. Check the backhaul RSRP in the Onyx Portal — As an alternative to the CLI method above, review the reported RSRP in the Mesh Node details page > **Analytics** tab > **Backhaul Status Details**.
6. Check CPE-CBSD grant status — Review the output of `journalctl -fu magma@cped`. Messages such as "*No channel available*" or "*Grant is not available*" indicate that the Mesh Node is waiting for a SAS grant.
7. Verify the GRE tunnel status — If the Mesh Node Interface Status is red, verify that the GRE tunnel between the Mesh Node and Mesh AP has been successfully established.
8. Run modem recovery commands — Use the following commands as appropriate:
   * `meshcli cpe_reset` — Reboots the modem.
   * `cpe_register` — Forces a registration retry.
   * `cpe_connect` — Attempts to establish the backhaul connection. Use this command only when the modem is Registered and the `cpe_controller` service is not running.
9. Review the Watchdog service status — Critical service failures trigger an automatic power cycle and generate a "Watchdog Powercycle" alert (Minor) that includes the service failure description.
10. Verify Mesh Node-AP Affinity *(if configured)* — If an Allowed Cell is configured, verify that:
    * The preferred cell provides an RSRP better than -90 dBm.
    * Neighboring or undesired cells remain below -110 dBm.
11. Check recovery after a Gateway AP reboot — A Mesh Node typically reconnects within two minutes after the Gateway AP restarts. If recovery takes longer, verify:
    * The GRE tunnel
    * The CPE connection
    * That Mesh AP transmission has not been intentionally disabled

**Resolution**

The issue is resolved when all of the following conditions are met:

* The Mesh Node's Online Status and Interface Status are green.
* The associated Mesh AP's Service Status returns to green.

**Escalation**

Escalate to GXC Technical Support if the Mesh Node remains offline after completing the recovery steps, or if the GRE tunnel cannot be established. Provide the following information:

* Mesh Node's vendor-assigned serial number
* CLI diagnostic output
* Onyx Portal alerts export

Escalate to the SAS Vendor Support if CPE-CBSD grant failures persist. Provide the following information:

* CBSD ID
* Grant request logs (if a CPE-CBSD grant failure has been confirmed)

**Reference**

* *4G Mesh Node Application Note*

</details>

<details>

<summary><strong>Mesh AP Offline</strong></summary>

**Symptoms**

A Mesh AP is offline or is unavailable while its parent Mesh Node is online.

**Indicators**

* Status Indicators: The Mesh AP's Service Status is red, or the Mesh AP appears offline, while the parent Mesh Node's Online Status is green.

**Dependency**

If the parent Mesh Node is also offline, resolve *Mesh Node Offline* first — a Mesh Node outage always takes its Mesh AP down with it.

**Possible Causes**

* GRE tunnel between the Mesh Node and the Mesh AP has not been established.
* Mesh AP transmission has been intentionally or unintentionally disabled.

**Actions**

1. Confirm the parent Mesh Node is online — In the Mesh Node details page, verify that the Mesh Node's Online Status and Interface Status are green. The Mesh AP cannot be online unless its parent Mesh Node is online.
2. The same base checks as [*Access Point Offline*](#access-point-offline) apply to the Mesh AP itself (physical connectivity, power, and portal registration/serial number verification).
3. Verify the GRE tunnel status — If the Mesh Node Interface Status is red, verify that the GRE tunnel between the Mesh Node and Mesh AP has been successfully established.

**Resolution**

The associated Mesh AP's Service Status returns to green.

**Escalation**

Escalate to GXC Technical Support if the Mesh AP remains offline after confirming the parent Mesh Node is online and completing the base AP checks.

</details>

<details>

<summary><strong>Access Point Offline</strong></summary>

**Symptoms**

The AP has no management-plane connectivity to the Onyx Edge — it does not appear online in the Onyx Portal at all.

**Indicators**

* Alerts: Access Point "Management Plane Offline" alert (Major) is active.
* Status Indicators: AP's Online Status is red (or grey).

{% hint style="info" %}
**NOTE:** An AP's Online Status indicates management-plane connectivity and changes to red after approximately 5 minutes of connectivity loss.
{% endhint %}

**Possible Causes**

* The AP is offline, in an alert state, or unable to register with the Onyx Edge.
* The AP was added to the Onyx Portal with an incorrect MAC address or serial number, or was not associated with the correct Onyx Edge.
* The AP has been added to the Onyx Portal, but the physical hardware has not yet been deployed, resulting in a false offline alert.

**Actions**

When an AP is truly offline (no management-plane connectivity), most Onyx Portal-based and CLI-based diagnostics do not apply — there is no *network\_check\_summary.sh* equivalent for a node, and IP reachability can only be checked after logging into the NR or Core, which itself requires the AP to be reachable. The customer-actionable checks are limited to the following two:

1. Verify the AP was added correctly in the Onyx Portal — Confirm the correct MAC address and/or serial number was entered when the AP was added, and that it is associated with the correct Onyx Edge. A mismatched serial number or MAC address will prevent the AP from ever coming online, independent of any physical issue.
2. Check the physical connection to the AP — Verify cabling, switch port, and PoE/PoE++ delivery to the AP. This confirms the physical layer is intact but cannot confirm reachability from the Onyx Edge.

If both checks pass and the AP is still offline, the remaining diagnosis (e.g., confirming actual IP reachability from the network side) requires GXC Technical Support — escalate.

**Resolution**

The issue is resolved when the AP's Online Status returns to green.

**Escalation**

Escalate to GXC Technical Support if the AP remains offline after confirming correct portal registration and physical connectivity. Provide the following information:

* AP's vendor-assigned serial number
* Active alert details
* Onyx Edge diagnostic bundle

</details>

<details>

<summary><strong>Cell Service Down</strong></summary>

**Symptoms**

The AP is online (management-plane connectivity is established) but is not providing cellular service — it is not transmitting, or is transmitting when it should not be. Applies to both 4G and 5G APs.

**Indicators**

* Alerts:
  * "Cell Service Down" alert (Major) is active — The AP cell has not transmitted for at least 15 minutes.
  * "Cell Unwarranted Transmission" alert (Major) is active — The AP is transmitting when it should not be.
  * (5G) "Multiport Cell Service Down" alert (Major) is active — The AP has stopped transmitting because multiport transmission is disabled while transmission is still desired.
* The UE cannot detect the network from the affected AP, even while within the expected coverage area.

{% hint style="info" %}
**NOTE:** An AP's Online Status indicates management-plane connectivity and changes to red after approximately 5 minutes of connectivity loss. Service Status indicates the RF transmission state. An AP can report a green Online Status while Service Status is red. Always verify both indicators. A red Sync Status can also prevent RF transmission even when Online Status and SAS Status are healthy.
{% endhint %}

**Possible Causes**

* Service Status is red because RF transmission is configured as Enabled, but the radio is not transmitting.
* The CBRS SAS grant has expired or could not be obtained.
* Sync Status is red because the AP has lost PTP or GPS synchronization.
* IPSec tunnel failure prevents the AP from re-establishing management-plane connectivity.
* A PLMN or TAC mismatch between the AP and the Onyx Edge configuration prevents the SCTP association from being established.
* (4G) SCTP connection failure in Dual Carrier deployments with SAS enabled.
* (5G) Multiport transmission is disabled while RF transmission is still required.

#### Actions

1. Check for scheduled maintenance — In the Onyx Portal header, click the Notifications icon to verify that no scheduled maintenance is in progress that explains the outage.
2. Check the AP Online and Service Status — In the AP summary page, verify both the Online Status and Service Status. If the Online Status is grey, first resolve the associated Onyx Edge connectivity issue before continuing.
3. Verify the physical-layer indicators:
   * AP's power LED is steady green, confirming PoE or AC power is being supplied correctly.
   * Cell LED behavior:

     * A slow flash indicates that the cell is activated
     * A fast flash indicates active data transmission

     The absence of either can point to a power or hardware issue before deeper investigation is needed.
   * The backhaul Ethernet cable is securely connected.
   * The switch port is active.
4. Verify PLMN and TAC configuration — Verify that the PLMN and TAC configured on the AP match the values configured on the Onyx Edge. A mismatch can prevent the SCTP association between the AP and Onyx Edge from being established, causing the AP to appear offline in the Onyx Portal.
5. Ping the AP from the Onyx Edge — In the Onyx Edge details page, click **Actions** > **Ping**, select the **NR** filter (or **CORE**, if testing the core-side path), enter the AP's management IP address, then click **Ping**.\
   A successful result confirms network-layer connectivity between the Onyx Edge and the AP, helping isolate the issue to the AP, its RAN configuration, or the local switch or VLAN rather than the Onyx Edge.  The result includes Average Response (ms), Error, Host or IP, and packet sent/received/transmitted counts.
6. (CBRS deployments) Check the SAS grant status — In the AP details page > **Overview** tab > SAS Status, verify that the **Registered**, **Granted**, and **Authorized** indicators are all green.
7. Check AP Sync Status — In the AP details page, verify the Sync Status. If the Sync Status is red, verify that the GPS antenna or upstream PTP clock source is connected and operating correctly.
8. Check fronthaul/eCPRI interface status on Onyx Edge — In the Onyx Edge details page, verify the Interface Status. A red status indicates that no APs are AMF-connected or that the N6 interface is unavailable.
9. Check the AP RAN Transmit configuration — In the AP details page > **Overview** tab > **RAN** section, verify that the **Transmit** parameter is set to **Enabled**. If transmission was disabled unintentionally, re-enable it.
10. Check IPSec tunnel status — Verify that the IPSec tunnel is established. If the initial IKE negotiation times out, the AP may not automatically retry. Restart the AP to initiate a new IPSec connection if needed.
11. (4G) Check SCTP connection for Dual Carrier with SAS — If one carrier loses SAS authorization, the SCTP connection may terminate after several minutes. As a workaround, disable SAS and then re-enable SAS for the affected AP through the RAN settings.
12. (5G) Check Multi-Port transmission status — If the "Multiport Cell Service Down" alert (Major) is active, verify whether multi-port transmission was intentionally disabled. If not, re-enable it via the AP's RAN configuration.
13. Reboot the AP — In the AP details page, click **Actions** > **Reboot**. Allow approximately 2-5 minutes for the AP to restart and reconnect.
14. If the AP repeatedly transitions between online and offline rather than remaining persistently offline:

    * Inspect fiber cable ends for contamination or damage
    * Confirm SFP modules are compatible with the AP hardware

    This behavior typically indicates a physical-layer or backhaul issue rather than a configuration or software issue.
15. Collect a diagnostic bundle from the Onyx Edge — In the Onyx Edge details page, click **Actions** > **Trigger Debug Information Collection**. After the collection completes, to retrieve the diagnostic bundle, click **Actions** > **List Debug Information Files in OE**.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The AP's Online Status and Service Status are green.
* The AP is transmitting and serving one or more connected UE.

**Escalation**

Escalate to GXC Technical Support if the AP's Service Status remains red after completing the recovery steps, or if IPSec connectivity cannot be restored. Provide the following information:

* AP's vendor-assigned serial number
* Active alert details
* Onyx Edge diagnostic bundle

Escalate to the SAS Vendor Support if SAS grant failures persist. Provide the following information:

* CBSD ID
* SAS status screenshot
* Grant request and response logs (if SAS grant issue has been confirmed)

#### Reference

* *4G & 5G Firewall Configuration Application Note*

</details>

<details>

<summary><strong>(4G) AP Sync / Timing Not Working</strong></summary>

**Symptoms**

A 4G AP is unable to maintain synchronization with its configured timing source. On 4G, the PTP source is external to the AP, so the customer can additionally check the reachability and status of that external PTP Grandmaster clock.

**Indicators**

* Alerts:&#x20;
  * "Timing Out of Sync" alert (Major) is active.
  * "GPS Signal Not Available" alert (Major) is active.
  * "PHC2SYS TS2PHC Offset Is Too High" or "TS2PHC Offset Is Too High" alerts (Warning) are active.
* Status Indicators:
  * AP's Service Status is red, with no SAS or connectivity issue identified.
  * AP's Sync Status is red or amber.
  * Onyx Edge's Sync Status is red.
* The UE cannot detect or camp on the cell even though the AP is online and SAS authorization is available.

**Possible Causes**

* The external PTP Grandmaster clock is unreachable or unstable.
* The GPS antenna is disconnected, damaged, or does not have an unobstructed view of the sky.
* An unsupported synchronization mode is configured for the deployment (for example, unsupported Dual Carrier/Carrier Aggregation scenarios).
* The AP Management URL is configured with a hostname instead of an IP address, causing DNS resolution failures that appear as synchronization issues.

**Actions**

1. Check AP Sync Status — In the AP summary page or AP details page, review the Sync Status indicator.
2. Verify the sync mode configuration — Verify that the configured synchronization mode is supported for the deployment.
   * PTP over Ethernet in Single Carrier mode is the recommended configuration.
   * PTP over UDP is not supported.
   * When performing handover testing, use GNSS synchronization.
3. Check GPS antenna and cabling — If the "GPS Signal Not Available" alert (Major) is active, inspect the GPS antenna, cabling, and connectors, and confirm the antenna has an unobstructed view of the sky.
4. Check the PTP source — Verify that the external PTP Grandmaster clock is operational and reachable. "PHC2SYS Offset is too High" and "TS2PHC Offset is too High" alerts (Warning) indicate the synchronization offset has exceeded the acceptable threshold.&#x20;
5. Check the AP Management URL configuration — Verify it is configured with the Onyx Edge IP address rather than a hostname.
6. Reboot the AP — If the PTP source has been verified and synchronization has not recovered, reboot the AP to initiate a new PTP synchronization process.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The AP's Sync Status is green.
* The AP resumes RF transmission.
* The UE can successfully detect and attach to the cell.

**Escalation**

Escalate to GXC Technical Support if synchronization cannot be restored after verifying the GPS source, PTP path, and AP configuration. Provide the following information:

* AP serial number
* Sync Status screenshot
* Active alert details

</details>

<details>

<summary><strong>(5G) AP Sync / Timing Not Working</strong></summary>

**Symptoms**

A 5G AP is unable to maintain synchronization with its timing source. On 5G, this source is normally the AP's own internal grandmaster clock (the DU acts as its own grandmaster) rather than an external PTP source, except when GNSS synchronization is used. As a result, the customer cannot check PTP source reachability the way they can on 4G — see *4G AP Sync / Timing Not Working*.

**Indicators**

* Alerts:&#x20;
  * "Timing Out of Sync" alert (Major) is active.
  * "GPS Signal Not Available" alert (Major) is active. This alert can be generated at the AP level or at the Onyx Edge (NR) level.
  * "PHC2SYS TS2PHC Offset Is Too High" or "TS2PHC Offset Is Too High" alerts (Warning) are active.
* Status Indicators:
  * AP's Service Status is red, with no SAS or connectivity issue identified.
  * AP's Sync Status is red or amber.
  * Onyx Edge's Sync Status is red.
* The UE cannot detect or camp on the cell even though the AP is online and SAS authorization is available.

**Possible Causes**

* The GPS antenna is disconnected, damaged, or does not have an unobstructed view of the sky (GNSS-synchronized deployments).
* An unsupported synchronization mode is configured for the deployment (for example, PTP over UDP is not supported; PTP over Ethernet in Single Carrier mode is the recommended and only supported configuration).
* The AP's internal PTP/timing service has stopped.

**Actions**

1. Check AP Sync Status — In the AP summary page or AP details page, review the Sync Status indicator. Verify that the AP has successfully synchronized with its configured PTP or SyncE source.
2. Verify the sync mode configuration — Verify that the configured synchronization mode is supported for the deployment.
   * PTP over Ethernet in Single Carrier mode is the recommended configuration.
   * PTP over UDP is not supported.
   * When performing handover testing, use GNSS synchronization.
3. (GNSS-synchronized deployments) Check the GPS antenna and cabling — If the "GPS Signal Not Available" alert (Major) is active, whether generated by the AP or the Onyx Edge (NR), inspect the GPS antenna, cabling, and connectors, and confirm the antenna has an unobstructed view of the sky.
4. Check whether the AP's internal timing service is running — If the Sync Status shows the timing service itself is down (rather than a GPS or configuration issue), this requires GXC Technical Support to investigate; there is no further customer-side check available today.
5. Reboot the AP — If the synchronization source has been verified and synchronization has not recovered, reboot the AP to initiate a new synchronization process.

**Resolution**

The issue is resolved when all of the following conditions are met:

* The AP's Sync Status is green.
* The AP resumes RF transmission.
* The UE can successfully detect and attach to the cell.

**Escalation**

Escalate to GXC Technical Support if synchronization cannot be restored after verifying the GPS source and AP configuration. Provide the following information:

* AP serial number
* Sync Status screenshot
* Active alert details

</details>

<details>

<summary><strong>General Networking / Firewall Issues</strong></summary>

**Symptoms**

The Onyx equipment cannot communicate with the Onyx Portal or the enterprise network, or is online but specific functionality (logs, speed test) is not working as expected.

**Indicators**

* Status Indicators:&#x20;
  * Equipment's Online Status is red.
  * Onyx Edge's N6 Interface Status is red.
  * Equipment repeatedly transitions between Online and Offline. See the equipment-specific offline sections above for detailed troubleshooting.
* Logs: `network_check_summary.sh` reports connectivity, DNS, routing, or interface failures.
* Equipment is online, but logs do not appear in the Onyx Portal — this can indicate the dedicated log port is blocked, since logs are carried on a separate port from general management traffic.
* Equipment is online, but a Subscriber Speed Test or backhaul speed test does not work — this can indicate the speed test port is blocked.

  &#x20;

**Possible Causes**

* Required firewall ports or traffic flows are blocked between the Onyx equipment and the Onyx Portal. Most management-plane traffic runs on port 443; logs and speed test functionality each use a separate, dedicated port, so a firewall that allows port 443 but blocks the other ports can still result in equipment showing online while logs or speed tests fail.
* The Onyx Edge N6 interface does not have a valid IP address.
* The on-premises security key has expired, preventing management-plane authentication.

**Actions**

1. If logs are not appearing, or a speed test is not working, while the equipment otherwise shows online, contact GXC Technical Support for the current required port list to verify with your network/firewall team.
2. Check Onyx Edge N6 interface status — Hover over the Onyx Edge Interface Status icon and verify that:
   * An N6 IP address is assigned.
   * The Reporting Time is recent.
3. Run comprehensive network diagnostics — Connect to the Onyx Edge using SSH and run `network_check_summary.sh`. Review the output for connectivity, interface, routing, and DNS failures.
4. Check management plane connectivity — Run `checkin_cli.py`. A response of "Success!" confirms that management-plane connectivity to the Onyx Portal is functioning correctly.
5. Check the on-premises security key — If the on-premises security key has expired, contact GXC Technical Support to obtain a replacement. Do not attempt to modify or regenerate the security key manually.

**Resolution**

The issue is resolved when all of the following conditions are met:

* All affected equipment report Online Status as green, and logs and speed test functionality work as expected.
* *network\_check\_summary.sh* completes without reporting connectivity, interface, routing, or DNS failures.

**Escalation**

Escalate to GXC Technical Support if:

* Specific ports have been confirmed open with GXC Technical Support and the issue persists.
* Management-plane connectivity cannot be restored.
* Network diagnostics continue to report failures after completing the troubleshooting steps.

Provide the following information:

* Onyx Edge NR/Core software version
* `network_check_summary.sh` output
* Interface Status screenshot

**Reference**

* *4G & 5G Firewall Configuration Application Note*

</details>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.gxc.io/docs/troubleshooting/onyx-troubleshooting-guide/service-issues.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.