> For the complete documentation index, see [llms.txt](https://docs.gxc.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.gxc.io/docs/network-configuration-and-operations-guides/5g-onyx-portal-4.x-alerts-reference.md).

# 5G Onyx Portal 4.x Alerts Reference

This reference describes Onyx Portal 5G predefined alerts listed in order of type, module, and name in alphabetical order.

<p align="center"><strong>Onyx Portal 5G Alerts</strong></p>

<table><thead><tr><th width="163.166748046875">Type</th><th width="148.5">Module</th><th width="325.3331298828125">Alert Name</th><th width="110">Severity</th></tr></thead><tbody><tr><td>Access Point</td><td>Control Plane</td><td><a href="#neighbor-tables-incomplete">Neighbor Tables Incomplete</a></td><td>Major</td></tr><tr><td>Access Point</td><td>CU-Plane</td><td><a href="#cell-service-down">Cell Service Down</a></td><td>Major</td></tr><tr><td>Access Point</td><td>CU-Plane</td><td><a href="#insufficient-sas-power">Insufficient SAS Power</a></td><td>Major</td></tr><tr><td>Access Point</td><td>CU-Plane</td><td><a href="#multiport-cell-service-down">Multiport Cell Service Down</a></td><td>Major</td></tr><tr><td>Access Point</td><td>Domain Proxy</td><td><a href="#cbsd-lesser-bw-granted">CBSD Lesser BW Granted</a></td><td>Major</td></tr><tr><td>Access Point</td><td>Domain Proxy</td><td><a href="#cbsd-new-grant">CBSD New Grant</a></td><td>Warning</td></tr><tr><td>Access Point</td><td>Domain Proxy</td><td><a href="#cbsd-preferred-frequency-unavailable">CBSD Preferred Frequency Unavailable</a></td><td>Warning</td></tr><tr><td>Access Point</td><td>Domain Proxy</td><td><a href="#cbsd-registration-failure">CBSD Registration Failure</a></td><td>Major</td></tr><tr><td>Access Point</td><td>M-Plane</td><td><a href="#access-point-offline">Offline</a></td><td>Major</td></tr><tr><td>Access Point</td><td>MCU-Plane</td><td><a href="#cell-unwarranted-transmission">Cell Unwarranted Transmission</a></td><td>Major</td></tr><tr><td>Access Point</td><td>MCU-Plane</td><td><a href="#mesh-filter-frequency-mismatch">Mesh Filter-Frequency Mismatch</a></td><td>Major</td></tr><tr><td>Access Point</td><td>S-Plane</td><td><a href="#timing-out-of-sync">Timing Out of Sync</a></td><td>Major</td></tr><tr><td>FHM</td><td>C-Plane</td><td><a href="#packet-loss-detected">Packet Loss Detected</a></td><td>Major</td></tr><tr><td>FHM</td><td>M-Plane</td><td><a href="#fhm-offline">Offline</a></td><td>Major</td></tr><tr><td>Mesh Node</td><td>Domain Proxy</td><td><a href="#cpe-cbsd-new-grant">CPE CBSD New Grant</a></td><td>Warning</td></tr><tr><td>Mesh Node</td><td>Domain Proxy</td><td><a href="#cpe-cbsd-preferred-frequency-or-bandwidth-unavailable">CPE CBSD Preferred Frequency or Bandwidth Unavailable</a></td><td>Warning</td></tr><tr><td>Mesh Node</td><td>Domain Proxy</td><td><a href="#cpe-cbsd-registration-failure">CPE CBSD Registration Failure</a></td><td>Major</td></tr><tr><td>Mesh Node</td><td>Management Plane</td><td><a href="#mesh-node-offline">Offline</a></td><td>Critical</td></tr><tr><td>Mesh Node</td><td>MCU-Plane</td><td><a href="#tunnel-down">Tunnel Down</a></td><td>Major</td></tr><tr><td>Mesh Node</td><td>MCU-Plane</td><td><a href="#watchdog-powercycle">Watchdog Powercycle</a></td><td>Minor</td></tr><tr><td>Mesh Node</td><td>Platform</td><td><a href="#mesh-node-cpu-usage">CPU Usage</a></td><td>Minor</td></tr><tr><td>Mesh Node</td><td>Platform</td><td><a href="#mesh-node-disk-usage">Disk Usage</a></td><td>Major</td></tr><tr><td>Mesh Node</td><td>Platform</td><td><a href="#mesh-node-memory-usage">Memory Usage</a></td><td>Minor</td></tr><tr><td>Mesh Node</td><td>Provisioning Server</td><td><a href="#mesh-node-provisioning-server-connection-failure">Provisioning Server Connection Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (Core)</td><td>C-Plane</td><td><a href="#dhcp-server-failed">DHCP Server Failed</a></td><td>Major</td></tr><tr><td>Onyx Edge (Core)</td><td>C-Plane</td><td><a href="#ng-setup-failure">NG Setup Failure</a></td><td>Major</td></tr><tr><td>Onyx Edge (Core)</td><td>C-Plane</td><td><a href="#sctp-outbound-stalled">SCTP Outbound Stalled</a></td><td>Major</td></tr><tr><td>Onyx Edge (Core)</td><td>C-Plane</td><td><a href="#ue-imsi-imei-validation-failure">UE IMSI-IMEI Validation Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (Core)</td><td>C-Plane</td><td><a href="#ue-registration-failure">UE Registration Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (Core)</td><td>CU Plane</td><td><a href="#ha-peer-down">HA Peer Down</a></td><td>Major</td></tr><tr><td>Onyx Edge (Core)</td><td>M-Plane</td><td><a href="#onyx-edge-core-offline">Offline</a></td><td>Critical</td></tr><tr><td>Onyx Edge (Core)</td><td>MCU-Plane</td><td><a href="#onyx-edge-core-service-down">Service Down</a></td><td>Critical</td></tr><tr><td>Onyx Edge (Core)</td><td>Platform</td><td><a href="#onyx-edge-core-cpu-usage">CPU Usage</a></td><td>Minor</td></tr><tr><td>Onyx Edge (Core)</td><td>Platform</td><td><a href="#onyx-edge-core-disk-usage">Disk Usage</a></td><td>Major</td></tr><tr><td>Onyx Edge (Core)</td><td>Platform</td><td><a href="#onyx-edge-core-memory-usage">Memory Usage</a></td><td>Minor</td></tr><tr><td>Onyx Edge (Core)</td><td>Provisioning Server</td><td><a href="#onyx-edge-core-provisioning-server-connection-failure">Provisioning Server Connection Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>C-Plane</td><td><a href="#f1-setup-failure">F1 Setup Failure</a></td><td>Critical</td></tr><tr><td>Onyx Edge (NR)</td><td>C-Plane</td><td><a href="#rrc-security-mode-failure">RRC Security Mode Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>C-Plane</td><td><a href="#rrc-setup-failure">RRC Setup Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>M-Plane</td><td><a href="#onyx-edge-nr-offline">Offline</a></td><td>Critical</td></tr><tr><td>Onyx Edge (NR)</td><td>MCU-Plane</td><td><a href="#onyx-edge-nr-service-down">Service Down</a></td><td>Critical</td></tr><tr><td>Onyx Edge (NR)</td><td>Platform</td><td><a href="#onyx-edge-nr-cpu-usage">CPU Usage</a></td><td>Minor</td></tr><tr><td>Onyx Edge (NR)</td><td>Platform</td><td><a href="#onyx-edge-nr-disk-usage">Disk Usage</a></td><td>Major</td></tr><tr><td>Onyx Edge (NR)</td><td>Platform</td><td><a href="#onyx-edge-nr-memory-usage">Memory Usage</a></td><td>Minor</td></tr><tr><td>Onyx Edge (NR)</td><td>Provisioning Server</td><td><a href="#onyx-edge-nr-provisioning-server-connection-failure">Provisioning Server Connection Failure</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>S-Plane</td><td><a href="#gps-signal-not-available">GPS Signal Not Available</a></td><td>Major</td></tr><tr><td>Onyx Edge (NR)</td><td>S-Plane</td><td><a href="#phc2sys-offset-is-too-high">PHC2SYS Offset Is Too High</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>S-Plane</td><td><a href="#ts2phc-offset-is-too-high">TS2PHC Offset Is Too High</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>User Plane</td><td><a href="#cell-high-dl-bler">Cell High DL BLER</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>User Plane</td><td><a href="#cell-high-ul-bler">Cell High UL BLER</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>User Plane</td><td><a href="#early-ecpri-packets">Early eCPRI Packets</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>User Plane</td><td><a href="#late-ecpri-packets">Late eCPRI Packets</a></td><td>Warning</td></tr><tr><td>Onyx Edge (NR)</td><td>User Plane</td><td><a href="#ecpri-rx-down">eCPRI Rx Down</a></td><td>Warning</td></tr><tr><td>Orchestrator</td><td>Domain Proxy</td><td><a href="#sas-connection-failure">SAS Connection Failure</a></td><td>Critical</td></tr></tbody></table>

***

## Common Indications

### Alerts Counter Panel Indication

The Onyx Portal Alerts Counter panel presents dedicated counters for Critical, Major, and Minor alerts. All other active alerts are grouped under the "Other" counter.

When an alert is triggered, the corresponding counter's value increases by one.

Clicking the Alerts Counter panel takes you to the **Alerts** > "**Active Alerts**" page.

To view an alert's details, click the alert's name.

### Cell Transmission Down Indications

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Depending on the percentage of affected APs, the icon will be ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FMAz57Ik4b346ClNaZTwC%2Ficon-service-status-amber.png?alt=media\&token=374562e6-68cd-4177-a70f-5919de0b2081) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F0UKJd1nToitCqjeKcDpy%2Ficon-service-status-red.png?alt=media\&token=6bc4e4e9-3874-44d4-9e05-e7323704786f) red.
* Onyx Portal > **Dashboard** > **Subscribers** tile: If the AP results in loss of online subscribers, the connected subscribers count will change.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: The traffic reduces as the AP is no longer serving.
* Onyx Portal > **Dashboard** > **Data Usage** tile: The usage reduces as the AP is no longer serving.
* Onyx Portal > **Dashboard** > **Access Point SAS Status** tile: Authorized AP count may be reduced if this is due to a grant issue. Other counts may increase depending on the issue.
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **Access Point Details** tile > **Service Status** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Folh5kTB9B1MkNFDmL0mA%2Ficon-service-status-red.png?alt=media\&token=2b8cfe45-3ad0-4263-97fd-5e275521159c) red.

***

## Predefined Alerts

{% hint style="info" %}
Predefined alert rules are delivered preconfigured with the Onyx Solution and can only be created, modified, or deleted by GXC.
{% endhint %}

## Access Point Alerts

### Access Point Control Plane Alerts

#### Neighbor Tables Incomplete

**Trigger Condition**

Not all eligible neighbor gNBs could be populated in the AP's neighbor table, and this condition persists for at least 15 minutes while the AP remains connected for management.

**Clear Condition**

All eligible neighbor gNBs are successfully populated in the neighbor table.

**Severity**

Major

***

### Access Point CU-Plane Alerts

#### Cell Service Down

**Trigger Condition**

When an AP cell does not transmit for at least 15 minutes.

**Probable Causes**

The grant is either unavailable or unauthorized, preventing the AP from transmitting.

**Clear Condition**

When the cell starts transmitting again.

**Severity**

Major

**System Impact**

The AP stops transmitting.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* See [Cell Transmission Down Indications](#cell-transmission-down-indications).

**Corrective Actions**

1. Check if any maintenance activity is currently in progress. (Onyx Portal > header > <img src="https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FTEfejR8ZxSUPt2v7wro3%2Fimage.png?alt=media&amp;token=d0f525c5-7be8-4a76-9a25-5ce479425ec9" alt="" data-size="line"> Notifications icon.)
2. Check if SAS grants are available. (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab >![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FIfBRw6SNOgPe14nzqN2V%2Ficon-sas-authorized-green.png?alt=media\&token=15fa2e0d-4ebd-4104-8520-0b0eca747545) SAS Status.)
3. Check if PTP sync is locked in the AP status. (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **Access Point Details** tile > **Sync Status** icon > ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fp3m1OCpXjUqVZA2XRnm3%2Ficon-ipsec-status-green.png?alt=media\&token=54339a76-c2b1-4d1f-868d-3194d1643003) green.)
4. Ensure that the ECPRI status reported by L1 (NR server) is good. (Onyx Portal > **Equipment** > **Onyx Edge (Core + NR)** tab / **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile > **Interface Status** icon > ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FiHpD3enfL8FhYTj6L3zw%2Ficon-interface-status-green.png?alt=media\&token=017804ad-c643-464b-8158-1d9d26d29f65) green.)
5. If the issue persists after completing the above steps, reboot the AP.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### Insufficient SAS Power

**Trigger Condition**

SAS has granted the CBSD(s) associated with this AP a maximum transmit power that is below the AP's configured hardware minimum power, and the AP has force-disabled transmission as a result.

**Probable Causes**

* SAS is unable to grant sufficient power due to spectrum congestion or interference near the deployment location.
* The AP's configured minimum power threshold does not match the actual hardware/deployment requirement.

**Clear Condition**

SAS grants a maximum transmit power at or above the AP's configured hardware minimum power.

**Severity**

Major

**System Impact**

The AP will stop transmitting.

**Indications**

See [Cell Transmission Down Indications](#cell-transmission-down-indications).

**Corrective Actions**

1. Check to see if any maintenance activity is in progress.
2. Check the SAS granted power for the CBSD(s) associated with the AP (Onyx Portal > **Equipment** > **Access Point** tab > AP details page > SAS Status Display).
3. In the AP's DP logs, check the granted power against the expected minimum (Onyx Portal > **Equipment** > **Access Point** tab > AP details page > **Logs** tab > **DP Logs** section).
4. If the granted power is consistently below the minimum, contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

**Related Alerts**

* [Access Point Cell Service Down](#cell-service-down)
* [Access Point Multiport Cell Service Down](#multiport-cell-service-down)

***

#### Multiport Cell Service Down

**Trigger Condition**

When a cell with Multiport enabled is not transmitting for at least 5 minutes.

**Probable Causes**

Grant for all Multiport antennas is unavailable or unauthorized.

**Clear Condition**

When the cell is transmitting again.

**Severity**

Major

**System Impact**

The AP will stop transmitting.

**Indications**

See [Cell Transmission Down Indications](#cell-transmission-down-indications).

**Corrective Actions**

1. Check if any scheduled maintenance activity is in progress.
2. Verify grants are available and authorized for all antennas to transmit:
   * Check SAS status (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > SAS Status display).
   * Check the AP's DP logs for failures on each antenna and take required action (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Logs** tab > **DP Logs** section).
   * If grants are not available or not authorized, adjust the preferences to obtain grants (Onyx Portal > **Equipment** > **Access Point** tab > edit AP's details > **RAN** tab).
   * If grants are still not available for the minimum acceptable preferences, transmission will not be possible until resources become available with SAS again.
3. Verify synchronization and connectivity:
   * Check if PTP sync is locked in the AP's details page.
   * Check if the eCPRI status reported by L1 (NR server) shows as connected.
4. If the issue persists after completing the above steps, reboot the AP.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Access Point Domain Proxy Alerts

#### CBSD Lesser BW Granted

**Trigger Condition**

When a CBSD has been granted less bandwidth than desired, and it continues to be in this state for at least five minutes.

**Clear Condition**

When either of the following conditions occur:

* CBSD gets the desired bandwidth in a grant.
* CBSD loses the grant.

**Severity**

Major

**System Impact**

The network capacity is reduced from what was planned.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Traffic will reduce as bandwidth is reduced.
* Onyx Portal > **Dashboard** > **Data Usage** tile: Usage will not increase at the expected rate due to reduced bandwidth.

**Corrective Actions**

1. Check if any maintenance activity is currently in progress.
2. In the AP's DP logs, check Spectrum Inquiry Response to determine available frequencies (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Logs** tab > **DP Logs** section).
3. If Strict Preference is enabled, disable it, if possible (Onyx Portal > **Equipment** > **Access Point** tab > edit AP's details > **RAN** tab > **Strict Preference** parameter), or change the frequency preference to obtain frequency with higher bandwidth (Onyx Portal > **Equipment** > **Access Point** tab > edit AP's details > **RAN** tab).
4. Contact GXC Technical Support.

**Resolution Verification**

1. All tiles have returned to their pre-incident conditions.
2. In the AP's details, verify the granted bandwidth (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **RAN** tile > **Bandwidth (MHz)** parameter).

**Related Alerts**

Access Point Domain Proxy [CBSD New Grant](#cbsd-new-grant) Alert — Triggered in conjunction with the Access Point Domain Proxy "CBSD Lesser BW Granted" alert.

***

#### CBSD New Grant

**Trigger Condition**

When new grants have been received within the last one hour.

**Clear Condition**

When no new grants have been received for at least one hour.

**Severity**

Warning

**System Impact**

This alert indicates a new grant has been received from SAS. For an AP that is already transmitting, it could change the transmission frequency, bandwidth, power, etc. The operator should verify the new grant will not affect the required KPIs.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon — Depending on grant authorization state and previous transmission state of the AP, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F9Xj1RCcna36ncj4MAnQk%2Ficon-service-status-green.png?alt=media\&token=f1721cd0-5d7e-4754-b9e5-a5c4409a2cee) green or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fj2vEu7tuneLlhAS1KGgh%2Ficon-service-status-amber.png?alt=media\&token=8f0028ea-ef81-4f67-b8ae-296bd82093b0) amber.
* Onyx Portal > **Dashboard** > **Subscribers** tile:
  * If the new grant comes with a lower power level, subscribers at the edge may lose connection.
  * If the AP was not previously transmitting, and the grant got authorized as well, previously unconnected subscribers in the coverage area can get connected.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: The traffic may change depending on AP's previous transmission state, grant authorization, and UE behavior. For example, a reduced bandwidth will not transmit as much data as before.
* Onyx Portal > **Dashboard** > **Data Usage** tile: The usage may change depending on the AP's previous transmission state, grant authorization, and UE behavior.
* Onyx Portal > **Dashboard** > **Access Point SAS Status** tile: Granted/Authorized CBSDs count may change depending on the CBSD's previous state.
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **RAN** tile: SAS Status will move to Granted and generally to Authorized if no previous grants were available.
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **Access Point Details** tile > **Service Status** icon: If the AP was not transmitting previously, the icon can be ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F7CNYA8tJ1uO7yEAdTYK3%2Ficon-service-status-green.png?alt=media\&token=ba0aab84-33aa-46d8-9bc0-31b5c95c4c47) green.

**Corrective Actions**

None.

**Resolution Verification**

None required for initial alert for an AP.

**Related Alerts**

Access Point Domain Proxy [CBSD Lesser BW Granted](#cbsd-lesser-bw-granted) Alert — Triggered if the new grant does not have the desired bandwidth.

**Additional Information**

* This alert simply indicates that a new grant has been received from SAS.
* As each grant change is a separate grant success, this alert can also be used to track grant changes. However, note that the very first grant is also tracked.

***

#### CBSD Preferred Frequency Unavailable

**Trigger Condition**

* In SC/DC mode, none of the frequencies in the preferred list are available with SAS.
* In CA mode, no non-overlapping frequency pair from the preferred list is available with SAS.

**Clear Condition** When any of the following conditions occur:

* Preferred frequencies become available again.
* CBSD is deregistered.

**Severity**

Warning

**System Impact**

* Network radio transmissions may not occur as planned, potentially causing interference from neighboring cells.
* This alert indicates that transmission on preferred frequencies is not fully possible. If Strict Preference is enabled, the AP will stop transmitting.
* In other cases, the AP will continue transmitting on whatever frequency is received via SAS grant.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* In case Strict Preference is disabled (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details), the granted frequency would not be part of the preferred list.
* In case Strict Preference is enabled, see [Cell Transmission Down Indications](#cell-transmission-down-indications).

**Corrective Actions**

1. Change the preference list (Onyx Portal > **Equipment** > **Access Point** tab > edit AP's details > **RAN** tab).
2. The available frequency can be found in the "Spectrum Inquiry Response" log type (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Logs** tab > **DP Logs** section > "Spectrum Inquiry Response" log type). Filter for "From SAS To DP logs".

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### CBSD Registration Failure

**Trigger Condition**

When one or more CBSD registration attempts have failed within the last one hour.

**Clear Condition**

When no CBSD registration failures occur for at least one hour.

**Severity**

Major

**System Impact**

The AP becomes non-functional, as it cannot obtain any grants to transmit.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > SAS Status: Will show Registered in red ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F5VA3M9ZwBJh7YrexuiLG%2Ficon-sas-status-registered-red.png?alt=media\&token=af7c983a-c74d-4245-9652-9943a04f4d1c).
* See [Cell Transmission Down Indications](#cell-transmission-down-indications).

**Corrective Actions**

1. Check the AP's DP logs for the failure reason and take necessary action (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Logs** tab > **DP Logs** section).
2. The most probable cause is a parameter mismatch between the Onyx Portal and the SAS Portal. Verify the CBSD registration on the Federated Wireless Portal has the correct information. Verify the FCC ID, Serial Number, CBSD Type and Category, Location, and Antenna parameters.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Access Point M-Plane Alerts

#### Offline <a href="#access-point-offline" id="access-point-offline"></a>

**Trigger Condition**

When no M-Plane messages have been received for the past 10 minutes (five minutes in alert condition + five minutes for changing the metric). This may occur, for example, if the AP is powered off.

**Clear Condition**

When M-Plane connection is re-established with the AP.

**Severity**

Major

**System Impact**

* Management access to the AP will be down. Configurations, alerts, events, metrics, etc. between the Onyx Portal and the AP will not be in sync.
* User traffic may be unavailable depending on the mode of failure.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Access Point** tile > **Online** icon: This AP is offline. Depending on other APs in the network, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FOpLAHi39XG7cmPa8PVBp%2Ficon-online-status-red.png?alt=media\&token=db230380-fae7-4af9-b9ea-41af4ebd0acc) red or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FCq8PoUs57k0L9CJlclNY%2Ficon-online-status-amber.png?alt=media\&token=d61e8c6b-e22f-4d2b-a700-147f0f2e3169) amber. Hover information shows the offline APs.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Depending on other APs in the network, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fj4VaMAVgYG04xOIyQ8bx%2Ficon-service-status-grey.png?alt=media\&token=16c769cf-8fd0-497f-a5cc-0f43c401fd75) grey or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FVAPdOm8Yoxng44CLRySF%2Ficon-service-status-amber.png?alt=media\&token=82683c65-e89c-4392-bbf8-dde8b06988fc) amber.
* Onyx Portal > **Dashboard** > **Subscribers** tile: If the AP results in loss of online subscribers, the connected subscribers count will change.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: If S1 is down, the traffic will commensurately reduce.
* Onyx Portal > **Dashboard** > **Data Usage** tile: If S1 is down, the usage will commensurately reduce.
* Onyx Portal > **Dashboard** > **Access Point SAS Status** tile: For SAS-enabled APs that are not online for the last four hours:
  * If unregistered, it will not get registered until it is online.
  * Any grants will be relinquished, and the AP will move to Registered state.
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **Access Point Details** tile > **Online** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FQpJ4jTuXNFxzzRXMuque%2Ficon-online-status-red.png?alt=media\&token=c2ec23a8-92e6-4fa9-b18c-0ce78f5ef419) red. All other statuses will turn grey.

**Corrective Actions**

1. Check if any maintenance activity is currently in progress.
2. Check for any power outages at the equipment's location.
3. Verify the AP is associated with the correct Onyx Edge.
4. Verify network connectivity between the AP and Onyx Edge.
5. Contact a local technician for physical AP inspection.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

**Related Alerts**

Onyx Edge (Core) MCU-Plane [Service Down](#onyx-edge-core-service-down) Alert — Triggered if all APs connected to the Onyx Edge are offline.

***

### Access Point MCU-Plane Alerts

#### Cell Unwarranted Transmission

**Trigger Condition**

When even though transmission is disabled, an AP's cell transmits for at least 15 minutes.

**Clear Condition**

When the cell stops transmission.

**Severity**

Major

**System Impact**

* The AP cell may transmit when not expected, potentially causing interference to neighboring radio equipment.
* This may result in a violation of federal spectrum regulations.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FJwRIT1UKs8iq0TNNL1gd%2Ficon-service-status-amber.png?alt=media\&token=cecdcf87-dc11-4cd4-847a-3182ad0fe699) amber.

**Corrective Actions**

1. Check if any maintenance activity is currently in progress.
2. Check the AP's events and alerts and resolve any faults (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Events & Alerts** tab). Any faults will be represented as "unexpected\_fault\_received" event, and new alerts as "enb\_alarm\_raised" event.
3. Check the AP's logs and resolve any errors (Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Logs** tab).
4. If the issue persists after completing the above steps, reboot the AP.
5. If the condition does not clear, contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### Mesh Filter-Frequency Mismatch

**Trigger Condition**

The Mesh Node's filter configuration and the attached AP's frequency are mismatched, and this condition persists for at least five minutes.

**Clear Condition**

The Mesh Node's filter configuration and the AP's frequency are aligned again.

**Severity**

Major

***

### Access Point S-Plane Alerts

#### Timing Out of Sync

**Trigger Condition**

When a 5G AP fails to obtain a PTP lock.

**Clear Condition**

When the AP successfully receives the PTP signal and synchronizes with it.

**Severity**

Major

**System Impact**

The AP will stop transmitting.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > **Overview** tab > **Access Point Details** section > **Sync Status** icon: If PTP is not locked, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FF7FMl7EuB6nZBaQJonfB%2Ficon-ipsec-status-red.png?alt=media\&token=c75a0a18-6fe9-4193-9f96-067174db1471) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert count has returned to its pre-incident state.
* Onyx Portal > **Dashboard**: All dashboard tiles have returned to their pre-incident state.

***

## FHM Alerts

### FHM C-Plane Alerts

#### Packet Loss Detected

**Trigger Condition**

When "Radio Unit Loss of Signal Detection" (RU-los-detect) is triggered on FHM RU ports due to early/late packets at least three times in five minutes.

**Clear Condition**

When RU-los-detect is not triggered more than two times in the last five minutes, and no early/late packets are detected between the FHM and RU.

**Severity**

Major

**System Impact**

* The network is affected due to the RU not being able to function correctly.
* User traffic may be unavailable depending on the mode of failure.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* An alert is raised for the specific FHM port number and the serial number of the RU attached to this port.

**Corrective Actions**

1. Check if any maintenance activity is in progress.
2. Verify network connectivity between the FHM, RU, and Onyx Edge.
3. Verify timing/sync status on all NR components.
4. Contact a local technician for physical FHM inspection.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Confirm that packet loss is no longer occurring between the FHM and RU. (Onyx Portal > **Equipment** > **FHM** tab > check port status.)

***

### FHM M-Plane Alerts

#### Offline <a href="#fhm-offline" id="fhm-offline"></a>

**Trigger Condition**

When no M-plane messages have been received for the past 15 minutes. This can happen, for example, due to the FHM being powered off.

**Clear Condition**

When the M-plane connection is established again with the AP.

**Severity**

Major

**System Impact**

Management access to the FHM will be unavailable. Configurations, alerts, events, and metrics between the Onyx Portal and the FHM will not be in sync. User traffic may be unavailable depending on the mode of failure.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **FHM** tile > **Online** icon: The online status icon will display in ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FZtqu29eqONFb9jPmyNn0%2Ficon-online-status-red.png?alt=media\&token=c9a2e5a7-3b17-455f-bbb5-c9080245e14f) red indicating the FHM is offline. Hover tooltip displays the offline FHMs.
* Onyx Portal > **Equipment** > **FHM** tab > view FHM's details > **Overview** tab > **Online** icon: The **Online** icon will display in ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FcuWxY4m9jp7vurnxlqRB%2Ficon-online-status-red.png?alt=media\&token=0b7df0e1-9045-49b8-b0a7-83301fd5f66b) red, and all other statuses will display in grey.

**Corrective Actions**

1. Check if any maintenance activity is in progress.
2. Check for power outages at the device location.
3. Verify the FHM is associated with the correct Onyx Edge (Onyx Portal > **Equipment** > **FHM** tab > view FHM's details > **FHM Details** section > **Onyx Edge** parameter).
4. Verify network connectivity between the FHM and Onyx Edge. Ping the FHM from Onyx Edge.
5. Contact a local technician for physical FHM inspection.

**Resolution Verification**

* Verify that the alert has cleared.
* Verify that all Dashboard tiles have returned to pre-incident conditions.
* Check FHM details (Onyx Portal > **Equipment** > **FHM** tab > view FHM details > **Overview** tab > **Online** icon). The **Online** icon should display in green.

***

## Mesh Node Alerts

### Mesh Node Domain Proxy Alerts

#### CPE CBSD New Grant

**Trigger Condition**

When using the GM03 Mesh Node, one or more new grants have been received within the last hour.

**Clear Condition**

When no new grants are received within the last one hour.

**Severity**

Warning

**System Impact**

* This alert is triggered upon receiving a new grant from SAS. For a Mesh Node that is already in transmission, it may result in a change in transmission power.
* Operators should verify that the new grant will not affect the required KPIs.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Based on grant authorization state and previous transmission state of the Mesh Node, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FOHNDqXplsIgLtKeBXaQL%2Ficon-service-status-green.png?alt=media\&token=b5ec69f6-5d1c-4a53-8a7a-fdac383601d5) green or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FytQM2MHvSYpVZ5GipljL%2Ficon-service-status-amber.png?alt=media\&token=7bc8c920-e4ae-4828-982f-4a035decd410) amber.
* Onyx Portal > **Dashboard** > **Access Point SAS Status** tile: Granted/Authorized CBSDs count may change depending on the CBSD's previous state.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab: If no previous grants were available, SAS Status will move to Granted state and generally to Authorized state.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Service Status** icon: If the Mesh Node was not transmitting previously, it can change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fx9TB9XIgCo80mg2MZJq0%2Ficon-service-status-green.png?alt=media\&token=5ac9ddcf-fa77-49e0-8961-a56bfbe27c88) green.

**Corrective Actions**

None.

**Resolution Verification**

None required for initial alert.

***

#### CPE CBSD Preferred Frequency or Bandwidth Unavailable

**Trigger Condition**

When using the GM03 Mesh Node, if the configured backhaul frequency or bandwidth to which the Mesh Node is connected becomes unavailable.

**Clear Condition**

When the preferred frequencies or bandwidth become available again.

**Severity**

Warning

**System Impact**

The Mesh Node is transmitting at reduced capacity and power and will not support any data traffic.

**Indications**

The GM03 Mesh Node will be online but does not have a grant.

**Corrective Actions**

1. Change the Mesh Node's backhaul frequency or bandwidth and reconnect.
2. The available frequency can be found in the "Spectrum Inquiry Response" log type (Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Logs** tab > **DP Logs** section > "Spectrum Inquiry Response" log type). Filter for "From SAS To DP logs".

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to their pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### CPE CBSD Registration Failure

**Trigger Condition**

When using the GM03 Mesh Node, if one or more CPE CBSD registration attempts have failed within the last one hour.

**Clear Condition**

When no CPE CBSD registration failures occur for at least one hour.

**Severity**

Major

**System Impact**

The Mesh Node will become non-functional, as it cannot obtain any grants to transmit.

**Indications**

* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > SAS Status will show **Registered** in red ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FzLj4bB4Do1fn85XiYSFv%2Fimage.png?alt=media\&token=f749cbdc-3cae-4990-980d-24faf473eb44).
* The Mesh Node will be reported offline. See the Mesh Node M-Plane [Offline](#mesh-node-offline) alert.

**Corrective Actions**

1. Check the Mesh Node's DP logs for the failure reason and take necessary action (Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Logs** tab > **DP Logs** section).
2. The most probable cause is a parameter mismatch between the Onyx Portal and the SAS Portal. Verify the CPE CBSD registration on the Federated Wireless Portal has the correct information. Verify the FCC ID, Serial Number, CBSD Type and Category, Location, and Antenna parameters.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to their pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Mesh Node M-Plane Alerts

#### Offline <a href="#mesh-node-offline" id="mesh-node-offline"></a>

**Trigger Condition**

When a Mesh Node has remained offline for at least 10 minutes.

**Clear Condition**

When the Mesh Node comes back online.

**Severity**

Critical

**System Impact**

Configurations, alerts, events, metrics, etc. between the Onyx Portal and Mesh Node will not be in sync.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Mesh Node** tile > **Online** icon: Depending on whether other Mesh Nodes are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FaG2Nh6ANp7rfrIv9hWP0%2Ficon-online-status-amber.png?alt=media\&token=b3bf2124-4a80-4725-998f-243aa070aeae) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FU3shYkxRCNaxLKs5WZWL%2Ficon-online-status-red.png?alt=media\&token=31d58312-8a68-4447-a53c-707b962d9745) red.
* Onyx Portal > **Dashboard** > **Mesh Node** tile > **Service Status** icon: Depending on whether other Mesh Nodes are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FeTm7hjL3leOH7dqo0Emx%2Ficon-service-status-amber.png?alt=media\&token=9a27f1e5-9eba-4370-8ff7-d058fd623716) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FDYn8wFL6Teltxq7R1pUp%2Ficon-service-status-grey.png?alt=media\&token=894fec5a-5592-4b06-8e67-23a219afb84e) grey.
* Onyx Portal > **Equipment** > **Mesh Node** tab > Mesh Nodes list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FdNHyf7MRU1e6f2Gx1Kaz%2Ficon-compute-resource-status-red.png?alt=media\&token=5dfd2390-c349-47da-b253-9bf7e1c43fe6) red.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FykxNnTdhG9ThAxt0zMUR%2Ficon-compute-resource-status-red.png?alt=media\&token=403396d5-b350-4d38-8975-69948bcc4b9f) red.

**Corrective Actions**

1. Check the Mesh Node's Internet connectivity.
2. Verify that the subscriber associated with the Mesh Node is connected and active.
3. If Internet connectivity is confirmed, ensure the Mesh Node is properly provisioned. Log on to the Mesh Node and run the `checkin_cli.py` command. Investigate and resolve any errors in the output.
4. Verify power supply to the Mesh Node equipment.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.

***

### Mesh Node MCU-Plane Alerts

#### Tunnel Down

**Trigger Condition**

When the Mesh Tunnel between the Mesh Node and the Onyx Edge remains down for at least five minutes.

**Clear Condition**

When the Mesh Tunnel between the Mesh Node and the Onyx Edge is restored.

**Severity**

Major

**System Impact**

The Mesh AP behind the Mesh Node will not be configured, preventing UE from attaching to it.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Mesh Node** tab > Mesh Nodes list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FzHCm2Xw9fqaBO5ZxAlyD%2Ficon-compute-resource-status-red.png?alt=media\&token=f9547d7e-5625-492b-88a9-d33f58bc3787) red.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FXmwuQOkxa2YCkY98JQuC%2Ficon-compute-resource-status-red.png?alt=media\&token=7c216fbd-b314-4892-9822-0c8f92cf092e) red.

**Corrective Actions**

1. Check the Mesh Node's Internet connectivity.
2. Check if the subscriber associated with the Mesh Node is connected and active.
3. Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.

***

#### Watchdog Powercycle

**Trigger Condition**

When the Mesh Node is power cycled by the Watchdog service within the last 10 minutes.

**Clear Condition**

When the Mesh Node Watchdog service has not triggered a power cycle on the Mesh Node for at least 10 minutes.

**Severity**

Minor

**System Impact**

* The mesh network will be affected until the Mesh Node reconnects and comes back online.
* UE will be unable to attach to the Mesh AP.

**Indications**

Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).

**Corrective Actions**

* Watchdog is triggered due to issues with the "cpe\_controller" service or modem identification. If the alert persists, contact GXC Technical Support.
* Check the alert's details for the cause.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to their pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Mesh Node Platform Alerts

#### CPU Usage <a href="#mesh-node-cpu-usage" id="mesh-node-cpu-usage"></a>

**Trigger Condition**

When a Mesh Node's CPU usage exceeds 70% of capacity in the last five minutes.

**Clear Condition**

When the Mesh Node's CPU usage drops below 70% of capacity.

**Severity**

Minor

**System Impact**

* Delays may occur in signaling and data processing at the Mesh Node.
* If CPU usage continues to rise, unexpected system behavior may occur.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Mesh Node** tab > Mesh Nodes list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FRUxC3pkamaLwjIVFv0Cc%2Ficon-compute-resource-status-red.png?alt=media\&token=5b38d343-8631-4dca-b6b5-a5652b141d9b) red.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FLzx10kYUXsuhzjGpgngW%2Ficon-compute-resource-status-red.png?alt=media\&token=341872cd-21cf-4459-b7fe-2323e9211917) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FKooyzRw3V8hH9uhQijgW%2Ficon-compute-resource-status-green.png?alt=media\&token=f322ccad-d3ef-4216-84d0-d67e384ab7f5) green.

***

#### Disk Usage <a href="#mesh-node-disk-usage" id="mesh-node-disk-usage"></a>

**Trigger Condition**

When a Mesh Node's disk usage exceeds 70% of capacity.

**Clear Condition**

When the Mesh Node's disk usage drops below 70% of capacity.

**Severity**

Major

**System Impact**

Delays in signaling and data processing at the Mesh Node.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Mesh Node** tab > **Mesh Node** list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FZGq9itSvORArJjDX4Ayl%2Ficon-compute-resource-status-red.png?alt=media\&token=eb84ed92-14b9-440a-99a2-fe1808b452ca) red.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FFuLjvcsLqDuc39C61OuL%2Ficon-compute-resource-status-red.png?alt=media\&token=5da8109b-2fb8-41d1-9459-190e538f8c91) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fu0gFIJ722dP8FkBMXs2J%2Ficon-compute-resource-status-green.png?alt=media\&token=7050e286-81fb-4c6e-8e9b-b2826e554242) green.

***

#### Memory Usage <a href="#mesh-node-memory-usage" id="mesh-node-memory-usage"></a>

**Trigger Condition**

When a Mesh Node's memory usage exceeds 70% of capacity.

**Clear Condition**

When the Mesh Node's memory usage drops below 70% of capacity.

**Severity**

Minor

**System Impact**

Delays in signaling and data processing at the Mesh Node.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Mesh Node** tab > Mesh Nodes list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F3ESxDiBr0KD4kRrzSUBq%2Ficon-compute-resource-status-red.png?alt=media\&token=17c30002-f9d0-47ae-8396-5b3700a0ebef) red.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F3NgezXZyfwk7wXO1KlAH%2Ficon-compute-resource-status-red.png?alt=media\&token=4e3aa68d-d0df-4920-8ca2-929a3c8253a5) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to their pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.
* Onyx Portal > **Equipment** > **Mesh Node** tab > view Mesh Node's details > **Overview** tab > **Mesh Node Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fimtud18CpycnvdYwQSas%2Ficon-compute-resource-status-green.png?alt=media\&token=1765a66a-4ae6-4d93-9c19-cf42b671499d) green.

***

### Mesh Node Provisioning Server Alerts

#### Provisioning Server Connection Failure <a href="#mesh-node-provisioning-server-connection-failure" id="mesh-node-provisioning-server-connection-failure"></a>

**Trigger Condition**

When a Mesh Node has not successfully checked in with the Provisioning Server within the past 35 minutes.

**Clear Condition**

When the Mesh Node successfully checks in with the Provisioning Server.

**Severity**

Warning

**System Impact**

* A new Mesh Node will be unable to connect to the Onyx Portal or begin operations.
* For an existing Mesh Node:
  * If the keys have changed, it will be unable to get the updated keys from the Provisioning Server and will be unable to connect to the Onyx Portal.
  * If the keys have not changed, there will be no impact.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Mesh Node** tile > **Online** icon:
  * If the Mesh Node's unique keys have changed, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FioI0gPMznykfImkccoOb%2Ficon-online-status-red.png?alt=media\&token=7317c6b6-2ee7-4cda-b02b-f2688b49430c) red.
  * If the Mesh Node's unique keys have not changed, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FlNIcVQFtsQAByMbPNKHm%2Ficon-online-status-green.png?alt=media\&token=72e44ad1-b16d-42e4-b480-1c2efc50a6e1) green.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to their pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

## Onyx Edge (Core) Alerts

### Onyx Edge (Core) C-Plane Alerts

#### DHCP Server Failed

**Trigger Condition**

When the DHCP server, responsible for assigning IP addresses to UE, has been unavailable for five minutes.

**Clear Condition**

When the DHCP server becomes available.

**Severity**

Major

**System Impact**

* UE may not be able to obtain IP addresses for data connectivity.
* New data sessions will be impacted while existing sessions may continue to function.

**Indication**

Onyx Portal > **Dashboard** > common counter indications.

**Corrective Actions**

Check the DHCP server health in the network.

**Resolution Verification**

Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.

***

#### NG Setup Failure

**Trigger Condition**

When an NG Setup Request failure is detected within the last five minutes.

**Clear Condition**

When no NG Setup Request failures occur within the last five minutes.

**Severity**

Major

**System Impact**

All APs connected to the Onyx Edge (Core) will stop transmission if setup fails.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** icon: If failures are persistent, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FYMxmNSDX4VP1NwNeiVXz%2Ficon-service-status-red.png?alt=media\&token=cc1d601d-cd7c-43ec-a1f8-82d589c8397f) red.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: If failures are persistent, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F8ysSXtAaw4CNhFJpbR33%2Ficon-service-status-red.png?alt=media\&token=b2e06aba-8f05-4881-8dd4-98a4e08f4e60) red.
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by the Onyx Edge will move to **Unknown** state.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Traffic will reduce to the extent of the traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Data Usage** tile: Data Usage will reduce to the extent of the traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Backhaul (SGi) Health** tile: May reduce to the extent of traffic handled by the Onyx Edge.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### SCTP Outbound Stalled

**Trigger Condition**

SCTP inbound ordered chunks have been increasing over the last 15 minutes while SCTP outbound ordered chunks have remained flat.

**Clear Condition**

Outbound SCTP chunks resume normal flow, indicating the core is processing and responding to signaling messages again.

**Severity**

Major

**System Impact**

* The NGAP/NAS signaling layer on the Onyx Edge core is stalled — SCTP packets are arriving from the access network but the core is not sending any responses back.
* New UE procedures (attach, registration, handover, PDU session establishment) will fail because signaling responses are not being sent.
* Existing sessions may continue to pass user-plane traffic for a time, but any control-plane operation (e.g., service request, paging response, session modification) will time out.
* If the condition persists, APs may tear down their SCTP associations and trigger S1/NG Setup failures, leading to a broader service outage.

**Indications**

* Onyx Portal > **Dashboard** > common counter indications
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** — May change to Red if the stall persists and signaling associations are lost
* Onyx Portal > **Dashboard** > **Subscribers** — Active subscriber count may begin to decline as UE procedures fail
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** — Control-plane stall may eventually impact data traffic as sessions cannot be established or modified

**Corrective Actions**

Contact GXC Support. There is no automatic recovery action for this condition.

**Resolution Verification**

From the Dashboard:

* The alerts count has returned to the condition prior to the incident.
* SCTP outbound chunk counters are incrementing normally.
* UE attach/registration procedures are succeeding again.

***

#### UE IMSI-IMEI Validation Failure

**Trigger Condition**

When attach requests for a UE have been rejected in the last 15 minutes because the UE IMEI does not match the configured IMSI-IMEI mapping.

**Clear Condition**

When no attach requests for the UE have failed in the last 15 minutes due to IMSI-IMEI validation.

**Severity**

Warning

**System Impact**

Any of the following:

* Legitimate UE cannot attach to the network due to incorrect configuration.
* Illegal attempt to use the network.

**Indications**

* Onyx Portal > **Dashboard** > common counter indications.
* Onyx Edge Control Plane Transaction Status shows attach failures.
* Onyx Edge events list displays IMEI mismatch failures (Onyx Portal > **Equipment** > **Onyx Edge (Core)** tab > view Onyx Edge's details > **Events & Alerts** tab).

**Corrective Actions**

1. Identify the offending IMSI and IMEI: In the Onyx Edge events view, filter with stream "mme" and look for Attach Reject events to identify the offending IMSI and IMEI. (Onyx Portal > **Equipment** > **Onyx Edge (Core)** tab > view Onyx Edge's details > **Events & Alerts** tab.)
2. Determine the appropriate action:
   * If the IMEI is not authorized for the IMSI, no further action is required in Onyx Portal. The operator can investigate who is attempting to attach with the unauthorized UE.
   * If the IMEI mapping is incorrectly configured, correct it in the subscriber configuration (Onyx Portal > **Subscribers**).

**Resolution Verification**

* Verify that the alert has cleared.
* Confirm that the UE can successfully attach to the network.

***

#### UE Registration Failure

**Trigger Condition**

When registration requests for a UE have failed for reasons other than unknown user or IMEI validation failures in the past 15 minutes.

**Clear Condition**

When no registration requests for a UE have failed for reasons other than unknown user or IMEI validation failures for at least 15 minutes.

**Severity**

Warning

**System Impact**

UE cannot attach to the network.

**Indications**

* Onyx Portal > **Dashboard** > common counter indications.
* Onyx Portal > **Equipment** > **Onyx Edge (Core)** tab > view Onyx Edge's details: Control Plane Transaction Status shows registration failures.

**Corrective Actions**

1. Ensure the subscriber/IMSI is configured in the Onyx Portal (Onyx Portal > **Subscribers** > **Subscriber** tab).
2. Ensure the subscriber's service status is set to Active (Onyx Portal > **Subscribers** > **Subscriber** tab > view subscriber's details > **Subscriber Details** section > **Service** parameter).
3. Contact GXC Technical Support.

**Resolution Verification**

* Verify that the alert has cleared.
* Confirm that the affected UE can attach successfully and pass data traffic.

***

### Onyx Edge (Core) CU-Plane Alerts

#### HA Peer Down

**Trigger Condition**

When, in an Onyx Edge pool, the Primary Onyx Edge detects a peer Onyx Edge is down. A peer Onyx Edge can be marked as "down" for the following reasons:

* NG interface is down
* N6 interface is down
* Both NG and N6 interfaces are down
* The peer Onyx Edge is powered down

The Primary Onyx Edge detects a peer is down within 10 minutes of it going down.

**Clear Condition**

When the Primary Onyx Edge detects that the peer Onyx Edge is operational.

**Severity**

Major

**System Impact**

When a peer Onyx Edge is reported as down, if the NG/N6 interface is also down, it will not be able to handle any UE traffic and is flagged as non-operational. A complete blackout will occur if all the Onyx Edge in a pool become non-operational.

**Indications**

Common Dashboard indications.

**Corrective Actions**

1. Check the Onyx Edge pool's status (Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge Pool** tile > view pool's details > **HA Connectivity Status** icon):
   * ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FQBHBnEOIonV6n1L1UpRC%2Ficon-interface-status-green.png?alt=media\&token=d6fce60f-e836-4299-a4b1-fd2d4ae4a386) green — One Onyx Edge is functioning as the Primary, and the others as Standby.
   * ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F3ujIEqcoYEtUFyCGwXUR%2Ficon-interface-status-amber.png?alt=media\&token=2cd08986-7738-4a85-a4b2-5ee93736f0f5) amber — One Onyx Edge is functioning as the Primary, but one or more of the other Onyx Edge are non-functional.
   * ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FxLQLnX7NvrNF8KnYmctO%2Ficon-service-status-red.png?alt=media\&token=43d4d5a9-8fd0-4792-9ae2-8314a968f57e) red — All Onyx Edge are non-functional, or their status is not available.
   * &#x20;![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FmkURSMVfyvixm51SQVym%2Ficon-service-status-grey.png?alt=media\&token=1ff38e69-050c-4436-98e9-18c7a11f29ee) grey — All Onyx Edge are offline.
2. Check if all the Onyx Edge in the pool are online (Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge Pool** tile > view pool's details > **Associated Onyx Edge** tile > **Online** column > **Online** icon):
   * ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FOPqx90HIHOAK2mWzFW67%2Ficon-online-status-green.png?alt=media\&token=1610ad96-889b-488a-a89c-a97e92a22a92) green — The Onyx Edge has checked in with the Onyx Portal in the last five minutes.
   * ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FQKgsPs4abw9SX69P9NFH%2Ficon-online-status-red.png?alt=media\&token=de13ad48-a3d9-4d46-b0db-ee797bcdf35a) red — The Onyx Edge has not checked in with the Onyx Portal in the last five minutes.
3. If the NG interface is down, check the NG connection between the peer Onyx Edge and APs. In a healthy NG interface, the Onyx Edge should be able to ping each AP (Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Actions** > **Ping**). Verify physical connections, switch configuration, and VLAN assignments for both the Onyx Edge and APs.
4. If the N6 interface is down, check Internet connectivity. In a healthy N6 interface, the Onyx Edge should be able to ping the Internet (8.8.8.8) (Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Actions** > **Ping**). Verify physical connections, switch configuration, and VLAN assignments.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.

***

### Onyx Edge (Core) M-Plane Alerts

#### Offline <a href="#onyx-edge-core-offline" id="onyx-edge-core-offline"></a>

**Trigger Condition**

When an Onyx Edge (Core) remains offline for at least 10 minutes.

**Clear Condition**

When the Onyx Edge (Core) comes back online.

**Severity**

Critical

**System Impact**

Configurations, alerts, events, metrics, etc. between the Onyx Portal and the Onyx Edge (Core) will be out of sync.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Online** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FDt2Fc6N4qm63OnkTXz9z%2Ficon-online-status-amber.png?alt=media\&token=ab8d407e-eb17-4739-9b8a-2940bdb70bb3) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FnAots6ovjHbRjY0sRqqM%2Ficon-online-status-red.png?alt=media\&token=d8b3e245-09f2-4120-953a-b7a6d0b8a265) red.
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fp2wOW6LuDcQZG5IxHlHK%2Ficon-service-status-amber.png?alt=media\&token=6ea9aa26-8f38-4eaf-8e6b-a04df14db603) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FAU0ICqHJXWliJguUhIYi%2Ficon-service-status-grey.png?alt=media\&token=7dfb8ac2-ece9-4402-9b89-0427e693bd49) grey.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Online** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FCx397HCxwdm6z0cSVopb%2Ficon-online-status-amber.png?alt=media\&token=5c955262-41a1-4030-9a3a-fbf461b01ef2) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FR7Mm4Fs9aoUzCPjEAXdO%2Ficon-online-status-grey.png?alt=media\&token=f0d7e2c4-367a-4625-8fd4-3c2e7b1b9092) grey.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fq8lrwkMrcEmDX6XWcLNa%2Ficon-service-status-amber.png?alt=media\&token=f6964b7d-a655-4b2d-90b2-78c21c20433c) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FsI6hOTf0TEeQyJARs3Am%2Ficon-service-status-grey.png?alt=media\&token=4234e891-a688-40ef-8aae-1ff904d64248) grey.
* Onyx Portal > **Dashboard** > **Subscribers** tile: Total subscribers impacted by this Onyx Edge will change to Unknown.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Reduces by the extent of traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Data Usage** tile: Reduces by the extent of traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Backhaul (SGi) Health** tile: Reduces by the extent of traffic handled by the Onyx Edge.

**Corrective Actions**

1. Check the Onyx Edge (Core)'s Internet connectivity on the N6 interface. This cannot be done from the Onyx Portal when the Onyx Edge (Core) is offline. If some other system, such as a remote laptop in the local network is accessible, ping the Onyx Edge (Core) to check if there is a local network issue, and then log on to the Onyx Edge (Core) to verify Internet connectivity.
2. At the Onyx Edge (Core)'s location, verify power and connectivity to the equipment.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Onyx Edge (Core) MCU-Plane Alerts

#### Service Down <a href="#onyx-edge-core-service-down" id="onyx-edge-core-service-down"></a>

**Trigger Condition**

When any of the following conditions occur:

* All APs remain offline for five minutes.
* All APs are S1-disconnected for five minutes.
* Any critical service has restarted more than three times in the last five minutes.
* Any critical service remains down for five minutes.

**Clear Condition**

When the trigger condition is resolved.

**Severity**

Critical

**System Impact**

The Onyx Edge may be offline or may not function as expected, potentially causing UE to lose service.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Depending on how other APs are connected to this and other Onyx Edge, and how they are affected, the icon will change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FHeG6Gta9XQPnFAwBdk7P%2Ficon-service-status-amber.png?alt=media\&token=37db14be-80d0-46f1-9d0b-e4f948497c7f) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FJP01Q9WBDsBHCF6C8XA0%2Ficon-service-status-red.png?alt=media\&token=155e88aa-368c-4e86-82fa-d12d57931032) red.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Online** icon: Depending on if the APs are affected, the icon may change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FYtsbboq6WYPIUKNdNpIq%2Ficon-online-status-amber.png?alt=media\&token=1e4a3f79-d559-4f70-b0b7-c94ca5179e2d) amber.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Depending on if the APs are affected, the icon will change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FhEeVeG775VoNzgKscnON%2Ficon-service-status-amber.png?alt=media\&token=72b02bf2-2da5-470e-9d36-cbf17fe49376) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FdhDnlsQ30i7OKJ4Ug5FQ%2Ficon-service-status-red.png?alt=media\&token=d41bc711-b4ca-43c7-9548-ea0accd75ee0) red.
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by the Onyx Edge may move to **Unregistered** state.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Traffic will decrease proportionately to the volume previously handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Data Usage** tile: May decrease proportionately to the traffic previously handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Backhaul (SGi) Health** tile: May reduce to the extent of traffic handled by the Onyx Edge.

**Corrective Actions**

1. Check if the Onyx Edge is properly provisioned.
2. If an AP is offline and the [Access Point M-Plane Offline](#access-point-offline) alert has triggered:
   * Check for network connectivity issues between the AP and the Onyx Edge.
   * Check if the AP is associated with the correct Onyx Edge.
3. If an AP is S1-disconnected, and the "CBRS AP S1 Disconnected" alert has triggered:
   * Check the AP's IPSec status.
   * Check for any CBRS AP faults and alarm events (**Events and Alerts** tab).
4. If neither of the above alerts has triggered, restart Onyx Edge services (Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge (Core)'s details > **Actions** > **Restart Services**).

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Returns to pre-incident alert counts.
* Onyx Portal > **Dashboard**: All tiles return to their pre-incident states.

**Related Alerts**

* Access Point Management Plane [Offline](#access-point-offline) Alert
* Onyx Edge (Core) Management Plane [Offline](#onyx-edge-core-offline) Alert

***

### Onyx Edge (Core) Platform Alerts

#### CPU Usage <a href="#onyx-edge-core-cpu-usage" id="onyx-edge-core-cpu-usage"></a>

**Trigger Condition**

When an Onyx Edge (Core)'s CPU usage exceeds 70% of capacity in the last five minutes.

**Clear Condition**

When the Onyx Edge (Core)'s CPU usage drops below 70% of capacity in the last five minutes.

**Severity**

Minor

**System Impact**

* Possible delays in signaling and data processing at the Onyx Edge (Core).
* If CPU usage continues to rise, unexpected behaviors may be seen.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge** list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FbiOXFzU9RIPETMZ8Uv9J%2Ficon-compute-resource-status-red.png?alt=media\&token=49618b87-76c8-4d73-85a4-3720e02cda24) red.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FnBdbqRydBng5nFm2Zcns%2Ficon-compute-resource-status-red.png?alt=media\&token=b0d609ee-f18a-48b1-8681-50995e83dab4) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge** list > **Compute Resource** icon: Depending on the usage, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FwIw3uUnzDcOKutqpSgjb%2Ficon-compute-resource-status-amber.png?alt=media\&token=f1453d13-5039-4376-8360-4f7cf9df33eb) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FuPvyQwCsTRIGja1N6i6y%2Ficon-compute-resource-status-green.png?alt=media\&token=c50d8573-4ef7-47e6-9220-301e1b967f19) green.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge (Core)'s details > **Overview** tab > **Onyx Edge Details** tile > **Compute Resource** icon: Depending on the usage, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FtddPlaFl8eKDds8FbOaw%2Ficon-compute-resource-status-amber.png?alt=media\&token=6bbad3bc-864e-4bb8-b80d-c76a64312978) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FfU231Uh6NxoGGAmOjaXG%2Ficon-compute-resource-status-green.png?alt=media\&token=a3770869-ae17-4795-b0bb-72ddd05d4e77) green.

***

#### Disk Usage <a href="#onyx-edge-core-disk-usage" id="onyx-edge-core-disk-usage"></a>

**Trigger Condition**

When an Onyx Edge (Core)'s disk usage exceeds 70% of capacity in the last five minutes.

**Clear Condition**

When the Onyx Edge (Core)'s disk usage drops below 70% of capacity in the last five minutes.

**Severity**

Major

**System Impact**

* If the disk usage continues to rise, it may eventually cause downtime on the Onyx Edge (Core).
* The Onyx Edge (Core) may stop collecting critical logs required for debugging issues.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge** list > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FUhz6tK9Z49fh8hRGzqXk%2Ficon-compute-resource-status-red.png?alt=media\&token=fe13211d-52a4-428d-953f-f12456bcbb80) red.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge (Core)'s details > **Overview** tab > **Onyx Edge Details** tile > **Compute Resource** icon: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F3cHlt9sDSCM2GRk8WluZ%2Ficon-compute-resource-status-red.png?alt=media\&token=8665c615-f422-4d22-814e-7865494ad646) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge** list > **Compute Resource** icon: Depending on the usage, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F6PvYocsZ9YPJ3kHJ2pTc%2Ficon-compute-resource-status-amber.png?alt=media\&token=dc9d03b6-ac94-4076-9ecd-2f97b9f007aa) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fb12PBGCdpZ9w01H6aqtx%2Ficon-compute-resource-status-green.png?alt=media\&token=da0334c7-97e6-4545-871e-740bec76d091) green.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge (Core)'s details > **Overview** tab > **Onyx Edge Details** tile > **Compute Resource** icon: Depending on the usage, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FwpVvTVc4pt506giGwB8h%2Ficon-compute-resource-status-amber.png?alt=media\&token=b7a6de7d-1d9a-44b6-ab6a-505e6ff6cf0a) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FNAf4Px2QZpxC7oVjchQH%2Ficon-compute-resource-status-green.png?alt=media\&token=61bda5a8-469f-4929-9b38-d5d9691f9f23) green.

***

#### Memory Usage <a href="#onyx-edge-core-memory-usage" id="onyx-edge-core-memory-usage"></a>

**Trigger Condition**

When an Onyx Edge (Core)'s memory usage exceeds 70% of capacity.

**Clear Condition**

When the Onyx Edge (Core)'s memory usage drops below 70% of capacity.

**Severity**

Minor

**Indications**

Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge** list > **Compute Resource** icon: Depending on the usage, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FBDVYmL3fMUUmt63HGCUk%2Ficon-compute-resource-status-amber.png?alt=media\&token=467a9c02-cfcd-4f6e-bec6-759ba4a35c84) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fxfm0uJZYfjeLMSop3XmT%2Ficon-compute-resource-status-green.png?alt=media\&token=43301e38-ea2f-44c0-a1da-65f84014f23b) green.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge (Core)'s details > **Overview** tab > **Onyx Edge Details** tile > **Compute Resource** icon: Depending on the usage, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FKmFb8BYR7wnesNDNWbIA%2Ficon-compute-resource-status-amber.png?alt=media\&token=a520f4c1-5c3c-4719-bbc4-c8e1b2239042) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FDziNgO37RDoyAsSqr7fM%2Ficon-compute-resource-status-green.png?alt=media\&token=3007dfe7-3027-4c96-b5f8-071c991260c2) green.

***

### Onyx Edge (Core) Provisioning Server Alerts

#### Provisioning Server Connection Failure <a href="#onyx-edge-core-provisioning-server-connection-failure" id="onyx-edge-core-provisioning-server-connection-failure"></a>

**Trigger Condition**

When an Onyx Edge (Core) has not successfully checked in with the Provisioning Server within the past 35 minutes.

**Clear Condition**

When the Onyx Edge (Core) successfully checks in with the Provisioning Server.

**Severity**

Warning

**System Impact**

* A new Onyx Edge (Core) will be unable to connect to the Onyx Portal or begin operations.
* For an existing Onyx Edge (Core):
  * If the keys have changed, it will be unable to get the updated keys from the Provisioning Server and will be unable to connect to the Onyx Portal.
  * If the keys have not changed, there will be no impact.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Online** icon:
  * If the Onyx Edge (Core)'s unique keys have changed: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FJz7V89fhQ1g3rFBmQg3m%2Ficon-online-status-red.png?alt=media\&token=cb53c767-ebb2-4cfe-ab40-c1e2c5f9dfb5) red.
  * If the Onyx Edge (Core)'s unique keys have not changed: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FvAxfWbJefdHPboh4RMSU%2Ficon-online-status-green.png?alt=media\&token=dc6d0629-a9ee-4bac-af6c-0b250dcafa2c) green.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

## Onyx Edge (NR) Alerts

### Onyx Edge (NR) C-Plane Alerts

#### F1 Setup Failure

**Trigger Condition**

When an F1 Setup Request failure occurs within the last five minutes.

**Clear Condition**

When no F1 Setup Request failures occur within the last five minutes.

**Severity**

Critical

**System Impact**

All APs connected to the Onyx Edge may stop transmission if the setup fails.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** icon: If failures are persistent, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FUai0GWMtErEUsEgui7EN%2Ficon-service-status-red.png?alt=media\&token=5720c24c-981d-49e6-8c0e-ec1346648fa4) red.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: If failures are persistent, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FITze4rXIZ8cHjjLPPpen%2Ficon-service-status-red.png?alt=media\&token=b9057a0a-7670-4d65-97b5-28a100023919) red.
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by the Onyx Edge will move to **Unknown** state.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Traffic will reduce to the extent of the traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Data Usage** tile: Data Usage will reduce to the extent of the traffic handled by the Onyx Edge.

**Corrective Actions**

1. Check if any maintenance activity is currently in progress.
2. Check if any recent configuration changes have been made.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### RRC Security Mode Failure

**Trigger Condition**

When at least one RRC Security Mode failure occurs within the last five minutes.

**Clear Condition**

When no RRC Security Mode failures occur within the last five minutes.

**Severity**

Warning

**System Impact**

Affected UE may experience issues connecting to the system.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by this issue will move to **Unknown** state.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

#### RRC Setup Failure

**Trigger Condition**

When more than 25% of RRC Connection Setup attempts in the last 15 minutes have failed, and this condition has persisted for at least the last three minutes.

**Clear Condition**

When more than 75% of RRC Connection Setup attempts in the last 15 minutes completed successfully.

**Severity**

Warning

**System Impact**

Failures may result from UE being in poor coverage areas or a larger underlying system issue preventing UE from establishing an RRC connection.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by this issue will move to **Unknown** state.

**Corrective Actions**

1. Check if any maintenance activity is currently in progress.
2. Check if any recent configuration changes have been made.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Onyx Edge (NR) M-Plane Alerts

#### Offline <a href="#onyx-edge-nr-offline" id="onyx-edge-nr-offline"></a>

**Trigger Condition**

When an Onyx Edge (NR) remains offline for at least 10 minutes.

**Clear Condition**

When the Onyx Edge (NR) comes back online.

**Severity**

Critical

**System Impact**

Configurations, alerts, events, metrics, etc. between the Onyx Portal and the Onyx Edge (NR) will not be in sync.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Online** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FDGrgz6vy8G2U1pDUHKNu%2Ficon-online-status-amber.png?alt=media\&token=a481e596-e073-460a-a248-cdd50c59ba39) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FTOXrmCJoeBC2ZnCx0bc7%2Ficon-online-status-red.png?alt=media\&token=e9ddf8e5-fc73-4665-8941-fabf15450846) red.
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FMMuswcCVLUpYxed6jOeL%2Ficon-service-status-amber.png?alt=media\&token=b834d395-a9ac-4197-9a42-fd6e7b4db179) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FTPiZAagWyWmIi0hrZR7w%2Ficon-service-status-grey.png?alt=media\&token=8f37bdeb-9d3e-439c-ac9a-132d559f1268) grey.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Online** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fd47ax8RhHu1OCTe73dG8%2Ficon-online-status-amber.png?alt=media\&token=b28e1fbb-106a-44fd-95e8-cf9f227fcd3c) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FOcJ7pFvXribYU5TddBhp%2Ficon-online-status-grey.png?alt=media\&token=c0c757d8-4854-4d73-acc3-c807702e4fac) grey.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Depending on whether other Onyx Edge are offline, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F86yAOaWRq3GGSFXmeUFW%2Ficon-service-status-amber.png?alt=media\&token=ddfdcc1d-ab12-4ed1-8b42-f111292fab19) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FOXA4Sgd3rZ7cLfe3l8AL%2Ficon-service-status-grey.png?alt=media\&token=c522ec5f-d4a4-4d37-bb79-2584e5d65feb) grey.
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by the Onyx Edge will move to **Unknown** state.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Traffic will reduce to the extent of traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Data Usage** tile: Data Usage will reduce to the extent of traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Backhaul (SGi) Health** tile: Will reduce to the extent of traffic handled by the Onyx Edge.

**Corrective Actions**

1. Check the Onyx Edge's Internet connectivity on the SGi/N6 interface.
   * This cannot be done from the Onyx Portal when the Onyx Edge is offline.
   * If another system, such as a remote laptop in the local network is accessible, try pinging the Onyx Edge to check if there is a local network issue.
   * If the Onyx Edge is reachable, log in to verify Internet connectivity.
2. At the Edge location, verify power and connectivity to the equipment.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Onyx Edge (NR) MCU-Plane Alerts

#### Service Down <a href="#onyx-edge-nr-service-down" id="onyx-edge-nr-service-down"></a>

**Trigger Condition**

When any of the following conditions occur:

* F1c connection is down for five minutes.
* NGc connection is down for five minutes.
* Any critical Edge services have unexpectedly restarted more than three times within the last five minutes.
* Any critical Edge services remain down for five minutes.
* gNB services have restarted due to failures in the last five minutes.

**Clear Condition**

When the trigger condition is resolved.

**Severity**

Critical

**System Impact**

The Onyx Edge may be offline or not functioning as expected, potentially preventing UE from receiving service.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Depending on whether other Onyx Edge are also experiencing failures, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F8z9UrIGomP7W2xvfYbtx%2Ficon-service-status-amber.png?alt=media\&token=f23bc113-3a33-4035-860b-80d75bac571a) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FjPvZF9KHCFWEtNSz7ejZ%2Ficon-service-status-red.png?alt=media\&token=366a1659-a1c8-4893-9dfb-119ea9300f91) red.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Online** icon: Depending on whether the APs are affected, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F7KrVwkJGbqsjP1mFtP5f%2Ficon-online-status-amber.png?alt=media\&token=0b0c768d-f434-414d-80c2-1bc7d88ae9c0) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FgUMRPD5vwT4WFd3kJqUe%2Ficon-online-status-red.png?alt=media\&token=a02abb33-b73f-45c2-add5-b01095f62291) red.
* Onyx Portal > **Dashboard** > **Access Point** tile > **Service Status** icon: Depending on whether the APs are affected, ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FgzZ8sxSBpE0T30qZgNdn%2Ficon-service-status-amber.png?alt=media\&token=16672bec-e3f2-4a48-906f-1b460beed803) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FagnvjT5gnSczSg4Hl7Bm%2Ficon-service-status-red.png?alt=media\&token=c9a47f31-0c83-4f37-b209-ac9b1680e3ea) red.
* Onyx Portal > **Dashboard** > **Subscribers** tile: Subscribers impacted by the Onyx Edge may move to **Unregistered** state.
* Onyx Portal > **Dashboard** > **Downlink/Uplink Traffic** tile: Traffic will reduce to the extent of the traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Data Usage** tile: Data Usage will reduce to the extent of the traffic handled by the Onyx Edge.
* Onyx Portal > **Dashboard** > **Backhaul (SGi) Health** tile: May reduce to the extent of traffic handled by the Onyx Edge.

**Corrective Actions**

1. Check if the Onyx Edge (NR) is provisioned properly.
2. Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

**Related Alerts**

* Onyx Edge (NR) Control Plane [F1 Setup Failure](#f1-setup-failure) Alert — Can trigger the Onyx Edge (NR) MCU-Plane "Service Down" alert.
* Onyx Edge (Core) Control Plane [NG Setup Failure](#ng-setup-failure) Alert — Can trigger the Onyx Edge (NR) MCU-Plane "Service Down" alert.

***

### Onyx Edge (NR) Platform Alerts

#### CPU Usage <a href="#onyx-edge-nr-cpu-usage" id="onyx-edge-nr-cpu-usage"></a>

**Trigger Condition**

When an Onyx Edge (NR)'s CPU usage exceeds 80% of capacity in the last five minutes.

**Clear Condition**

When the Onyx Edge (NR)'s CPU usage drops below 80% of capacity in the last five minutes.

**Severity**

Minor

**System Impact**

* Potential delays in signaling and data processing at the Onyx Edge (NR).
* If CPU usage continues to rise, unexpected behaviors may be seen.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge (n)** tile > **Compute Resource** column, and Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile: The Compute Resource icon will change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F5G7q7imwnlje3nVdbXBh%2Ficon-compute-resource-status-red.png?alt=media\&token=63ca7e90-36ab-4e2a-aabd-012cd0beef46) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge (n)** tile > **Compute Resource** column, and Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile: Depending on the usage level, the Compute Resource icon will change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fz1BeHpF2arI0Am80i0jl%2Ficon-compute-resource-status-amber.png?alt=media\&token=f6f91af6-0744-4763-8d62-e8c90c7b06b8) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FlcevYbH112BxZFEAAfCZ%2Ficon-compute-resource-status-green.png?alt=media\&token=2c24e4a3-f963-44d5-86ab-0f7909ae16e0) green.

***

#### Disk Usage <a href="#onyx-edge-nr-disk-usage" id="onyx-edge-nr-disk-usage"></a>

**Trigger Condition**

When an Onyx Edge (NR)'s disk usage exceeds 70% of capacity in the last five minutes.

**Clear Condition**

When the Onyx Edge (NR)'s disk usage drops below 70% of capacity in the last five minutes.

**Severity**

Major

**System Impact**

* If disk usage continues to rise, it may eventually cause downtime on the Onyx Edge.
* The Onyx Edge (NR) may stop collecting critical logs needed for debugging issues.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge (n)** tile > **Compute Resource** column, and Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile: The Compute Resource icon will change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FeyQCXPXr6SuloaAirJH4%2Ficon-compute-resource-status-red.png?alt=media\&token=a7b9727c-208e-4b15-9ef3-98bf1180f042) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge (n)** tile > **Compute Resource** column, and Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile: Depending on the usage level, the Compute Resource icon will change to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FrofSSZ3FhT7HXWTDGBRK%2Ficon-compute-resource-status-amber.png?alt=media\&token=606327de-7260-409e-9b36-69f17481412a) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FMFdbegnUYx1dy9KvbIlc%2Ficon-compute-resource-status-green.png?alt=media\&token=c18772a6-54c6-4667-858d-4881c9cb1ed2) green.

***

#### Memory Usage <a href="#onyx-edge-nr-memory-usage" id="onyx-edge-nr-memory-usage"></a>

**Trigger Condition**

When an Onyx Edge (NR)'s memory usage exceeds 80% of capacity.

**Clear Condition**

When the Onyx Edge (NR)'s memory usage drops below 80% of capacity.

**Severity**

Minor

**System Impact**

If memory usage continues to rise, it could lead to downtime on the Onyx Edge.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge (n)** tile > **Compute Resource** column, and Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile: The Compute Resource icon changes to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FCbk1FxgW6tdNxBx6iz9y%2Ficon-compute-resource-status-red.png?alt=media\&token=6dd48c62-1dce-47a4-9f3f-77965b232120) red.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Equipment** > **Onyx Edge** tab > **Onyx Edge (n)** tile > **Compute Resource** column, and Onyx Portal > **Equipment** > **Onyx Edge** tab > view Onyx Edge's details > **Overview** tab > **Onyx Edge Details** tile: Depending on the usage level, the Compute Resource icon changes to ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FKPnM5uuRSDh2FDcSTmvb%2Ficon-compute-resource-status-amber.png?alt=media\&token=098cd5cd-5501-477b-8ff0-3e51636a4898) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fm8r3ANodzDsycrrtssu5%2Ficon-compute-resource-status-green.png?alt=media\&token=3eb197db-401e-4ec1-8330-2a851f1e61c6) green.

***

### Onyx Edge (NR) Provisioning Server Alerts

#### Provisioning Server Connection Failure <a href="#onyx-edge-nr-provisioning-server-connection-failure" id="onyx-edge-nr-provisioning-server-connection-failure"></a>

**Trigger Condition**

When an Onyx Edge (NR) has not successfully checked in with the Provisioning Server within the past 35 minutes.

**Clear Condition**

When the Onyx Edge (NR) successfully checks in with the Provisioning Server.

**Severity**

Warning

**System Impact**

* A new Onyx Edge (NR) will be unable to connect to the Onyx Portal or begin operations.
* For an existing Onyx Edge (NR):
  * If the keys have changed, it will be unable to get the updated keys from the Provisioning Server and will be unable to connect to the Onyx Portal.
  * If the keys have not changed, there will be no impact.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* Onyx Portal > **Dashboard** > **Onyx Edge** tile > **Online** icon:
  * If the Onyx Edge (NR)'s unique keys have changed: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FAT1Ag4dv8862wKFwW6Ib%2Ficon-online-status-red.png?alt=media\&token=3a4a355d-eca8-4a8a-a370-42bc2b745142) red.
  * If the Onyx Edge (NR)'s unique keys have not changed: ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Fm7BrtKpLZ8XJ2op1JwiJ%2Ficon-online-status-green.png?alt=media\&token=61e8708d-186e-4878-907a-b423460ec590) green.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

***

### Onyx Edge (NR) S-Plane Alerts

#### GPS Signal Not Available

**Trigger Condition**

When the Onyx Edge timing service has a high time difference between external timestamp signals (GPS) and the PTP hardware clock, and is unable to sync the hardware clock to GPS timing.

**Clear Condition**

When the Onyx Edge timing service successfully syncs the hardware clock to GPS timing.

**Severity**

Major

**System Impact**

The Onyx Edge Cell service may not function as expected. UE may be unable to obtain service.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* **Dashboard** > **Onyx Edge** tile > **Sync Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FsOINUJHatWICSbUvzKDQ%2Ficon-ipsec-status-red.png?alt=media\&token=4765e4c6-4027-40b3-bfd1-67d0e1a639c3) red; hover tooltip shows error details.
* **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Will display as amber or red depending on whether other Onyx Edge are also failing.
* **Dashboard** > **Access Point** tile > **Service Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FH0iCpaslMYdAcGfrcQUe%2Ficon-service-status-amber.png?alt=media\&token=25c6d6ec-5bc3-4bbc-909d-be4ab45f791f) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FeVvRPLji4x6wGZeYNpzL%2Ficon-service-status-red.png?alt=media\&token=f7710d5f-1dd8-4aa2-9e1c-bd604b8c3b79) red depending on whether they are affected.
* **Dashboard** > **Subscribers** tile: Total subscribers impacted by this Onyx Edge may become unregistered.

**Corrective Actions**

1. Check if the Onyx Edge (NR) is provisioned properly.
2. Check the physical connections and if GPS mode is selected, verify GPS signal.
3. After the GPS signal is restored, restart all NR services to return to GPS mode.
4. Contact GXC Technical Support.

**Resolution Verification**

* Verify that all Dashboard tiles have returned to pre-incident states.
* Verify that the alert has cleared.

***

#### PHC2SYS Offset Is Too High

**Trigger Condition**

When the Onyx Edge timing service has a high time difference between the system clock and PTP hardware clock, and is unable to sync the system clock to the hardware clock.

**Clear Condition**

When the Onyx Edge timing service successfully syncs the system clock to the hardware clock.

**Severity**

Warning

**System Impact**

The Onyx Edge cell service may not function as expected. UE may be unable to obtain service.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* **Dashboard** > **Onyx Edge** tile > **Sync Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FzxEA5MdTcafxAmGblEod%2Ficon-ipsec-status-red.png?alt=media\&token=7495124f-c891-41be-b8cc-de9f3f76aa02) red; hover tooltip shows error details.
* **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FWxi6plXmg0R0gkDxOyGo%2Ficon-service-status-amber.png?alt=media\&token=d97d2dbb-4324-45d2-a363-b56c138aa7b1) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FpPbPtVKmITyCWSkFdyDy%2Ficon-service-status-red.png?alt=media\&token=a6c26cb2-b800-4468-9e88-b35466dec058) red depending on whether other Onyx Edges are also failing.
* **Dashboard** > **Access Point** tile > **Service Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F0AsxWZk3X3yRs881cyYo%2Ficon-service-status-amber.png?alt=media\&token=0a09a602-742b-4dcd-a2ac-1347b19981e5) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FBinMJMR3wQtcc104zPmN%2Ficon-service-status-red.png?alt=media\&token=ee8af2f6-f961-4647-9186-bd8eafa81152) red depending on whether they are affected.
* **Dashboard** > **Subscribers**: Total subscribers impacted by this Onyx Edge may become unregistered.

**Corrective Actions**

1. Check if the Onyx Edge (NR) is provisioned properly.
2. Check the physical connections and verify GPS signal if GPS mode is selected.
3. Contact GXC Technical Support.

**Resolution Verification**

* Verify that the alert has cleared.
* Verify all Dashboard tiles have returned to pre-incident states.

***

#### TS2PHC Offset Is Too High

**Trigger Condition**

When the Onyx Edge timing service has a high time difference between external timestamp signals (GPS) and the PTP hardware clock and is unable to sync the hardware clock to GPS timing.

**Clear Condition**

When the Onyx Edge timing service successfully syncs the hardware clock to GPS timing.

**Severity**

Warning

**System Impact**

The Onyx Edge cell service may not function as expected. UE may be unable to obtain service.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* **Dashboard** > **Onyx Edge** tile > **Sync Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FAftzxZSQmj70eFXwu5Ew%2Ficon-ipsec-status-red.png?alt=media\&token=3dd0918f-1570-46ed-ab0d-1232ec53ddfc) red; hover tooltip shows error details.
* **Dashboard** > **Onyx Edge** tile > **Service Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FLzG0HrjgfE9WrK31cv4P%2Ficon-service-status-amber.png?alt=media\&token=c8974b46-9d7c-40de-bae4-7c3dbe55ba23) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FaYQcS3Qv6pCAcVre41YO%2Ficon-service-status-red.png?alt=media\&token=ecaa9bd8-518c-4a42-9197-7efe68ce9e61) red depending on whether other Onyx Edges are also failing.
* **Dashboard** > **Access Point** tile > **Service Status** icon: Will display as ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2Flzty6iooADdWeY2L3r93%2Ficon-service-status-amber.png?alt=media\&token=7b9b71d2-aa2f-474b-b2d3-32d804085fea) amber or ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2FoMKQze70iQvYIIOtkKGa%2Ficon-service-status-red.png?alt=media\&token=d1867ec8-70b3-40a0-b745-e437e384c2d7) red depending on whether they are affected.
* **Dashboard** > **Subscribers**: Total subscribers impacted by this Onyx Edge may become unregistered.

**Corrective Actions**

1. Check if the Onyx Edge (NR) is provisioned properly.
2. Check the physical connections and verify GPS signal if GPS mode is selected.
3. Contact GXC Technical Support.

**Resolution Verification**

* Verify that the alert has cleared.
* Confirm that all Dashboard tiles have returned to their pre-incident states.

***

### Onyx Edge (NR) User Plane Alerts

#### Cell High DL BLER

**Trigger Condition**

Downlink BLER (Block Error Rate) for the cell exceeds 50% for at least 15 minutes.

**Clear Condition**

Downlink BLER for the cell falls to or below 50%.

**Severity**

Warning

***

#### Cell High UL BLER

**Trigger Condition**

Uplink BLER for the cell exceeds 50% for at least 15 minutes.

**Clear Condition**

Uplink BLER for the cell falls to or below 50%.

**Severity**

Warning

***

#### Early eCPRI Packets

**Trigger Condition**

More than 50% of eCPRI packets received for the cell in the last 15 minutes are classified as early.

**Clear Condition**

The proportion of early eCPRI packets falls to or below 50%.

**Severity**

Warning

***

#### Late eCPRI Packets

**Trigger Condition**

More than 50% of eCPRI packets received for the cell in the last 15 minutes are classified as late.

**Clear Condition**

The proportion of late eCPRI packets falls to or below 50%.

**Severity**

Warning

***

#### eCPRI Rx Down

**Trigger Condition**

No eCPRI Rx packets are received for the cell for at least 10 minutes, even though the RU's RF Tx is enabled/desired.

**Clear Condition**

eCPRI Rx packets resume being received while Tx remains enabled.

**Severity**

Warning

***

## Orchestrator Alerts

### Orchestrator Domain Proxy Alerts

#### SAS Connection Failure

**Trigger Condition**

When any of the following conditions occur:

* Could not connect to SAS at least three times in the last 15 minutes.
* At least one HTTP error from SAS in the last 15 minutes.

**Clear Condition**

When there are no SAS connection errors and no HTTP errors for at least 15 minutes.

**Severity**

Critical

**System Impact**

This could potentially mean the entire network is down, at least those APs that are dependent on SAS for their grants.

**Indications**

* Onyx Portal > [Alerts Counter Panel Indication](#alerts-counter-panel-indication).
* In case of constant connection errors, the transmit timer could expire and result in the cells going down. See [Cell Transmission Down Indications](#cell-transmission-down-indications).
* Onyx Portal > **Equipment** > **Access Point** tab > view AP's details > SAS status: The SAS state may not progress if not authorized, and the Authorized state may eventually revert to Granted on timer expiry.

**Corrective Actions**

Contact GXC Technical Support.

**Resolution Verification**

* Onyx Portal > Alerts Counter Panel: Alert counts revert to pre-incident levels.
* Onyx Portal > **Dashboard**: All tiles revert to their pre-incident states.

**Additional Information**

* There are two kinds of errors:

  * **HTTP errors** — Should not occur under normal conditions; typically indicate an SAS-side issue or fundamental breakage in Domain Proxy.
  * **Transport-level errors** — May occur due to connectivity issues (unable to reach SAS) or security issues (server-side or client-side authentication failures).

  The error type can be determined from the alert labels.
* SAS Connection Failure alerting generates a single consolidated alert at the network level instead of individual alerts per AP.

## Related Documentation

* *5G Onyx Portal Operations Guide*
* *GXC Glossary*

## Contact GXC

To get in touch with GXC, please visit <https://gxc.io/contact-us/>.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.gxc.io/docs/network-configuration-and-operations-guides/5g-onyx-portal-4.x-alerts-reference.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
