> For the complete documentation index, see [llms.txt](https://docs.gxc.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.gxc.io/docs/network-configuration-and-operations-guides/4g-onyx-portal-4.x-operations/manage-alerts.md). # Manage Alerts Alerts provide notifications about significant events within the network and are key to monitoring and debugging potential issues in your network in a timely fashion. For example, you can configure an alert rule to notify when the CPU percentage on the gateway is running > 75% in the last five minutes. Or, for example, when more than three Onyx Edge services restarted in the last one hour. ## Alert Notifications The Onyx Portal provides alert notifications via the Active Alerts Counters panel and email notifications. ### Active Alert Counters Panel The Active Alert Counters panel in the header area displays the count of active Critical, Major, Minor, and Other alerts. The "Other" category accounts for alerts of Warning, Info, and Notice severities.

**To view active alerts:** {% stepper %} {% step %} Click the Alert Counters panel. The **Active Alerts** page is displayed. For more information, see [*View Active Alerts*](#view-active-alerts). {% endstep %} {% endstepper %} ### Alert Email Notifications The Onyx Portal sends alert notification emails when an alert is triggered, every one hour until the alert gets cleared, and when the alert finally gets cleared. {% hint style="info" %} **NOTE:** The alert notification emails are only sent to receivers configured to receive alert notifications for specified alert severity levels. For more information, see [*Manage Alert Receiver Accounts*](#manage-alert-receiver-accounts). {% endhint %} The alert notification emails use the following format for their subject lines: `Alert Notifications | | ` The body of the alert notification emails include the following details: * **Time Zone** – The timestamp at which the alert was generated or cleared, and the time zone of the venue where the alert was generated or cleared. * **Alert Duration** – The duration for which the alert has remained active. * **Network Name** – The network's name. * **Alert Name** – The alert's name. * **Equipment Name** – The alert's source (AP, Mesh Node, or Onyx Edge equipment's name) linking to the Onyx Portal equipment details page. * Link to the Onyx Portal Alerts page—[*https://portal.gxc.io/#/alert/alerts*](https://portal.gxc.io/%23/alert/alerts). ## View Active Alerts Active alerts are alerts that have been triggered but not yet cleared and still require attention. The Active Alerts view displays all currently active instances of all alerts. The Active Alerts view is auto-refreshed every one minute. **To view active alerts' details:** {% stepper %} {% step %} In the navigation pane, click **Alerts**. The **Alerts** page > **Active Alerts** tab displays the list of active alert instances.

* **Start Time** – The timestamp at which the alert was triggered.\ To sort the list of alerts by start time, click the column header. * **Duration** – The duration for which the alert is active. * **Severity** – The alert's severity level—Critical, Major, Minor, Warning, Info, Notice.\ To sort the list of alerts by severity, click the column header. * **ID** – The alert's ID used to identify the alert. This is a fixed ID that third-party tooling can use to identify the alerts. * **Type** – The alert's type used to identify the problem source or type—Access Point, Mesh Node, Onyx Edge, Orchestrator, Provisioning Server. * **Module** – The associated module. * Management Plane – Relating to equipment configuration and communicating the configuration to the device. Module ID: 01. * Control Plane – Relating to protocol operations of the equipment for control plane. Module ID: 02. * User Plane – Relating to protocol operations of the equipment for user plane. Module ID: 03. * Platform – Relating to functioning of the equipment. Module ID: 04. * CU Plane – Relating to either or both control and user plane. Module ID: 05. * MCU Plane – Relating to either or all of Management, Control, User Plane. Module ID: 06. * Domain Proxy – Relating to Domain Proxy. Module ID: 07. * **Name** – The alert's name.\ To view the alert's details, click the alert's name.\ To sort the list of alerts by alert name, click the column header. * **Device ID** – The associated device's ID. * **Description** – The alert's description. * **Instance ID** – The alert's instance ID, which differentiates between multiple instances of the same alert. * **Mute Alert** – While investigating an active alert, mute or unmute an active alert instance to suppress or enable the hourly alert email notifications for that specific alert instance. To mute an alert instance to stop alert email notifications, turn on the toggle button. To resume the alert email notifications, unmute the alert instance by turning off the toggle button.

NOTE: This mute/unmute configuration is alert instance specific. If necessary, any new or other instances of an alert must be explicitly muted/unmuted.

{% endstep %} {% step %} To view an alert's details, in the alerts list, click the alert's name. The ***\*** dialog box displays the alert's details. {% endstep %} {% step %} To sort the list by alert creation timestamp, click the **Start Time** column header. {% endstep %} {% step %} To sort the list by alert severity level, click the **Severity** column header. {% endstep %} {% endstepper %} ## View Cleared Alerts Cleared alerts are alerts that were previously active but have subsequently been cleared either by user intervention or automatically by the system. {% hint style="info" %} **NOTE:** The cleared alerts history is available only for alerts cleared in the last seven days and lists only the most recent 1000 cleared alerts. {% endhint %} {% hint style="info" %} **NOTE:** The Cleared Alerts view auto-refreshes every one minute. {% endhint %} **To view cleared alerts' details:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Cleared Alerts** tab. The **Alerts** page > **Cleared Alerts** tab displays the list of alerts cleared in the selected time duration.

* **Start Time** – The timestamp at which the alert was triggered.\ To sort the list of alerts by start time, click the column header. * **Duration** – The duration for which the alert was active.\ To sort the list by duration, click the column header. * **Severity** – The alert's severity level—Critical, Major, Minor, Warning, Info, Notice.\ To sort the list by severity, click the column header. * **ID** – The alert's ID used to reference the alert. To search for cleared alerts by alert ID, click

, then in the Search box, enter the alert ID. * **Type** – The alert's type—Onyx Edge, Access Point, Provisioning Server, etc. To search for cleared alerts by alert type, click

, then in the Search box, enter the alert type. * **Module** – The associated module—Platform, Management Plane, MCU Plane, etc. To search for cleared alerts by associated module, click

, then in the Search box, enter the module name. * **Name** – The alert's name. To search for cleared alerts by alert name, click

, then in the Search box, enter the alert name.\ To sort the list by alert name, click the column header. * **Device ID** – The associated device ID. To search for cleared alerts by device ID, click

, then in the Search box, enter the device ID. * **Description** – The alert's description. * **Instance ID** – The alert's instance ID, which differentiates between multiple instances of the same alert. To search for cleared alerts by instance ID, click

, then in the Search box, enter the instance ID. {% endstep %} {% step %} To change the duration for which the cleared alerts are displayed, change the selected from and to timestamps

. {% endstep %} {% step %} To sort the alerts list by alert creation timestamp, click the **Start Time** column header. {% endstep %} {% endstepper %} ## Manage Muting Alerts Muting alerts involves suspending the alert email notifications sent by the Onyx system. Typically, alerts are muted to prevent unnecessary alert email notifications during planned maintenance and other situations where the alerts may not require immediate attention. Alerts can be muted at the following levels: * Active Alert Instance Level – Muting and unmuting active alert instances allows you to suppress and enable alert email notifications for specific active instances of an alert. For more information, see [*Mute/Unmute Specific Instances of Active Alerts*](#muteunmute-specific-instances-of-active-alerts). * Alert Rule Level – Muting and unmuting alerts per alert rule allows to suppress and enable alert email notifications for all alerts generated by individual alert rules. For more information, see [*Mute/Unmute Alerts per Alert Rule*](#muteunmute-alerts-per-alert-rule). ### View Muted Alerts **To view muted alerts:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Muted Alerts** tab. The **Alerts** page > **Muted Alerts** tab displays the list of muted alerts.

(checkbox) – Select/clear all or specific alerts. * **Type** – The alert's type—Access Point, Mesh Node, Onyx Edge, Orchestrator. * **Name** – The alert's name. * **Devices** – The device names for which an alert is muted. * **Actions** – Options to edit and delete a muted alert rule. {% endstep %} {% endstepper %} ### Mute/Unmute Active Alert Instances Muting and unmuting active alert instances allows you to suppress or enable alert email notifications for specific active instances of an alert. You can mute alert email notifications for a specific alert either for all equipment or only for specific equipment. For example, you can mute alert notifications for the "Offline" and "Cell Service Down" alerts for a specific AP or for all APs in the network. #### Mute Active Alerts {% hint style="info" %} **NOTE:** You can mute active alert notifications only if you are a GXC Administrator. {% endhint %} You can either mute: * All alert notifications for a specific alert. * Alert notifications for a specific alert either for all equipment or only for specific equipment. For example, you can mute alert notifications for the "Offline" and "Cell Service Down" alerts for a specific AP or for all APs in the network. #### Create Alert Mute Rules **To create an alert mute rule:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Muted Alerts** tab. The **Alerts** page > **Muted Alerts** tab displays the list of muted alerts. {% endstep %} {% step %} Click **Mute New Alerts**. The **Add Mute Rule** dialog box is displayed.

{% endstep %} {% step %} In the **Select Alert Type** dropdown, select the alert type from **Onyx Edge**, **Access Point**, **Orchestrator**, **Mesh Node**. The list of alerts related to the selected alert type is displayed. {% endstep %} {% step %} In the

(select/clear) column, select the alert that you want to mute. {% endstep %} {% step %} (Applicable only to Onyx Edge, Access Point, and Mesh Node alert types.) To mute the selected alert for specific equipment, in the **Device List** column, click the corresponding **Select Devices** dropdown, and select the specific equipment. To mute alert notifications for all equipment, select **Select All**. {% endstep %} {% step %} Select any other alerts to mute and select the required equipment. {% endstep %} {% step %} To save your edits, click **Submit**. {% endstep %} {% endstepper %} #### Edit Alert Mute Rules {% hint style="info" %} **NOTE:** You can edit alert mute rules only if you are a GXC Administrator. {% endhint %} **To edit alert mute rules:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Muted Alerts** tab. The **Alerts** page > **Muted Alerts** tab displays the list of muted alerts. {% endstep %} {% step %} For the alert mute rule that you want to edit, in the **Actions** column, click the corresponding ⋮ (options) icon, then click **Edit**.

{% endstep %} {% step %} In the **Devices** column, select or clear the devices for which the alert is muted.

{% endstep %} {% step %} To save your edits, click

. {% endstep %} {% endstepper %} #### Delete Alert Mute Rules {% hint style="info" %} **NOTE:** You can delete alert mute rules only if you are a GXC Administrator. {% endhint %} **To delete alert mute rules:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Muted Alerts** tab. The **Alerts** page > **Muted Alerts** tab displays the list of muted alerts. {% endstep %} {% step %} For the alert mute rule(s) that you want to delete, in the

(select/clear) column, select the rule(s), then click **Delete**. Alternately, to delete a particular alert mute rule, in the **Actions** column, click the corresponding ⋮ (options) icon, then click **Delete**.

A confirmation dialog box is displayed.

{% endstep %} {% step %} To confirm the deletion, click **Yes**. {% endstep %} {% endstepper %} ### Mute/Unmute Specific Instances of Active Alerts Muting and unmuting alert instances allows you to suppress or enable alert email notifications for specific instances of an active alert. This allows to silence an alert's individual instance instead of silencing the entire alert providing more granular control over the alerting process. {% hint style="info" %} **NOTE:** This mute/unmute configuration is alert instance specific. If necessary, any new or other instances of an alert must be explicitly muted/unmuted. {% endhint %} #### Mute Specific Instance of an Active Alert **To mute an alert's instance:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Active Alerts** tab. The **Alerts** page > **Active Alerts** tab displays the list of active alert instances.

{% endstep %} {% step %} For the alert instance that you want to mute, in the **Mute Alert** column, turn on the corresponding toggle switch. The **Mute *****\ \***** Alert** dialog box is displayed.

{% endstep %} {% step %} In the **Select a Time Duration** dropdown, select the duration for which the alert instance must remain muted from **Twenty-Four Hours**, **One Week**, **Forever**. {% endstep %} {% step %} To save your changes, click **Submit**. {% endstep %} {% endstepper %} #### Unmute Specific Instance of an Active Alert **To unmute an alert's instance:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Active Alerts** tab. The **Alerts** page > **Active Alerts** tab displays the list of active alert instances. {% endstep %} {% step %} For the alert instance that you want to unmute, in the **Mute Alert** column, turn off the corresponding toggle switch.

{% endstep %} {% step %} To save your change, click **Submit**. {% endstep %} {% endstepper %} ### Mute/Unmute Alerts per Alert Rule Muting and unmuting alerts per alert rule allows you to suppress or enable alert email notifications for any alerts generated by individual alert rules. Muting or unmuting alerts per alert rule enables you to receive notifications only for the most relevant alerts, offering more granular control over the alerting process. #### Mute Alert Notifications per Custom Alert Rule {% hint style="info" %} **NOTE:** You can disable or enable alert notifications for custom alert rules only if you are a GXC Administrator. {% endhint %} **To disable or enable alert notifications for a custom alert rule:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab is displayed. The **Custom Alerts** section lists the custom alert rules. {% hint style="info" %} **NOTE:** The **Custom Alerts** section is displayed only if at least one custom alert rule exists. {% endhint %} {% endstep %} {% step %} For the custom alert rule that you want to disable or enable alert notifications for, in the **Actions** column, click the corresponding ⋮ (options) icon, then click **Edit**. {% endstep %} {% step %} Under **Notifications**: * To disable alert notifications, clear the **Enable Email Notification** checkbox. * To enable alert notifications, select the **Enable Email Notification** checkbox. {% endstep %} {% step %} To save your edit, in the upper-right corner of the page, click **Save Rule**. {% endstep %} {% endstepper %} #### Mute Alert Notifications per Predefined Alert Rule {% hint style="info" %} **NOTE:** You can disable or enable alert notifications for predefined alert rules only if you are a GXC Administrator. {% endhint %} **To disable or enable alert notifications for a predefined alert rule:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab is displayed. The **Predefined Alerts** section lists the predefined alert rules. {% endstep %} {% step %} In the **Predefined Alerts** section, for the predefined alert rule that you want to disable or enable alert notifications for, in the **Notifications** column, click the toggle button to enable or disable notifications. A confirmation dialog box is displayed. {% endstep %} {% step %} To confirm your change, click **Yes**. {% endstep %} {% endstepper %} ## Manage Alert Rules An alert rule specifies the criteria of a condition based on Prometheus Query Language expressions. When the condition becomes true, an alert is triggered, which changes the state of the alert to active, and initiates an alert notification. As long as an alert is active, an alert notification is sent every one hour until the alert gets cleared. When the condition becomes false, the state of the alert changes to resolved, which clears the alert and initiates an alert notification. The Onyx Portal supports alert rules of three types—predefined, custom, and semi-custom. * Predefined alert rules are provided pre-configured with the Onyx Solution product, and can be created, edited, and deleted only by GXC. * Custom alert rules are specific to each network. You can create and manage custom alert rules only if you are a GXC Administrator. * Semi-custom alert rules are predefined by GXC but allow limited configuration changes, such as threshold values. These rules retain their core logic as defined by GXC but can be adjusted by to better fit a specific network's operational needs. ### View Alert Rules **To view available alert rules:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab displays predefined and custom alert rule lists.

* **Predefined Alerts** * **ID** – The alert rule's ID. * **Type** – The alert rule's type—Access Point, Mesh Node, Onyx Edge, Orchestrator, and Provisioning Server. * **Module** – The alert rule's module—Control Plane, CU Plane, Domain Proxy, Platform, Management Plane, MCU Plane. * **Name** – The alert rule's name.\ To sort the list by alert names, click the column header. * **Severity** – The alert rule's severity classification—Critical, Major, Minor, Warning, Info, Notice.\ To sort the list by severity, click the column header. * **Description** – The alert rule's description. *

(view details) – Click to view the alert rule's details, including its PromQL expression and labels. * **Custom Alerts**

NOTE: The Custom Alerts section is displayed only if at least one custom alert rule has been created.

* **Name** – The alert rule's name.\ To sort the list by of alert names, click the column header. * **Severity** – The alert rule's severity classification—Critical, Major, Minor, Warning, Info, Notice.\ To sort the list by severity, click the column header. * **Description** – The alert rule's description. * **Notifications** – Indicates whether email notification receivers are notified when the alert is triggered. For more information, see [*Manage Alert Receiver Accounts*](#manage-alert-receiver-accounts). * **Actions** – Options to: *

(view details) – Click to view the alert rule's details, including its PromQL expression and labels. *

(edit) – (GXC Administrators only) Edit the alert rule. *

(delete) – (GXC Administrators only) Delete the alert rule. {% hint style="info" %} **NOTE:** The **Custom Alerts** section is displayed only if at least one custom alert rule has been created. {% endhint %} {% endstep %} {% endstepper %} ### Predefined Alert Rules This section describes the predefined alert rules. #### Predefined Alert Rules {% hint style="info" %} **NOTE:** The Onyx Portal includes predefined alert rules out of the box. {% endhint %} {% hint style="info" %} **NOTE:** Only GXC can create, modify, and delete predefined alert rules. {% endhint %} {% hint style="info" %} **NOTE:** Whenever a new network is created, the predefined alert rules are auto synchronized, and do not have to be synchronized manually. {% endhint %} {% hint style="info" %} **NOTE:** For detailed information on predefined alert rules, refer to the *4G Onyx Portal Alerts Reference*. {% endhint %} The following table lists the predefined alert rules supported in 4G networks.

4G Predefined Alert Rules

**Access Point > Control Plane Alerts**

Alert	Severity	Trigger and Clear Conditions
Neighbor Tables Incomplete	Major	Trigger Condition: Inability to fill all “Neighbor Frequency Table” entries or “Neighbor Cell Table” entries, as the AP has a respective limit of 8 and 16 for the above. Clear Condition: When all neighbor entries are filled.
S1 Disconnected	Major	Trigger Condition: When an AP has not connected to MME S1 interface for at least 15 minutes. Clear Condition: When the AP is S1-reconnected.

Alert

Severity

Trigger and Clear Conditions

Neighbor Tables Incomplete

Major

Trigger Condition: Inability to fill all “Neighbor Frequency Table” entries or “Neighbor Cell Table” entries, as the AP has a respective limit of 8 and 16 for the above.

Clear Condition: When all neighbor entries are filled.

S1 Disconnected

Major

Trigger Condition: When an AP has not connected to MME S1 interface for at least 15 minutes.

Clear Condition: When the AP is S1-reconnected.

**Access Point > CU-Plane Alerts**


Cell Service Down	Major	Trigger Condition: When a cell is not transmitting even when it is desired to be transmitting for at least 15 minutes. Clear Condition: When the cell starts transmitting as expected.
DAS Associated Access Point Cell Service Down	Major	Trigger Condition: Cell associated with a DAS is not transmitting for at least five minutes. Clear Condition: When the cell starts transmitting again.
IPSec Certificate Expiry	Minor	Trigger Condition: When an AP’s IPSec certificate is expiring in three days or less or has already expired. Clear Condition: When a new IPSec certificate is generated and uploaded to the AP.
IPSec Connection Down	Major	Trigger Condition: When the IPSec tunnel between an AP and Onyx Edge is still not up or has gone down. Clear Condition: When the IPSec tunnel between the AP and Onyx Edge is up.

**Access Point > Domain Proxy Alerts**


CBSD Lesser BW Granted	Major	Trigger Condition: When a CBSD is granted lesser-than-desired bandwidth, and it continues to be in this state for at least five minutes. Clear Condition: When the CBSD gets the desired bandwidth in a grant, or when CBSD loses the grant.
CBSD New Grant	Warning	Trigger Condition: When new grants are received in the last one-hour window. Clear Condition: When there are no new grants in the last one-hour window. As each grant change is a separate grant success, this alert can also be used to track grant changes. However, note that the very first grant is also tracked.
CBSD Preferred Frequency Unavailable	Warning	Trigger Condition: When the preferred frequency list has been exhausted, and other frequencies are being attempted. Clear Condition: When a preferred frequency becomes available, or the CBSD is deregistered.
CBSD Registration Failure	Major	Trigger Condition: When a CBSD registration fails in the last one-hour window. Clear Condition: When there are no failed CBSD registrations in the last one-hour window.

**Access Point > Management Plane Alerts**


Offline	Major	Trigger Condition: When an AP has not connected via TR-069 in the last 15 minutes. Clear Condition: When the AP reconnects via TR-069.
TR-069 Session Stuck	Major	Trigger Condition: When a TR-069 session with an AP has started but not completed for more than 15 minutes, indicating the session is stuck in an error loop. Clear Condition: When the TR-069 session completes successfully.
Unexpected TR-069 Fault	Minor	Trigger Condition: When there is an unexpected TR-069 fault from an AP in the last one hour. Clear Condition: When there are no new TR-069 faults from the AP in the last one-hour.

Offline

Major

Trigger Condition: When an AP has not connected via TR-069 in the last 15 minutes.

Clear Condition: When the AP reconnects via TR-069.

TR-069 Session Stuck

Major

Trigger Condition: When a TR-069 session with an AP has started but not completed for more than 15 minutes, indicating the session is stuck in an error loop.
Clear Condition: When the TR-069 session completes successfully.

Unexpected TR-069 Fault

Minor

Trigger Condition: When there is an unexpected TR-069 fault from an AP in the last one hour.

Clear Condition: When there are no new TR-069 faults from the AP in the last one-hour.

**Access Point > MCU-Plane Alerts**


Cell Unwarranted Transmission	Major	Trigger Condition: When an AP is disabled from transmitting but transmits for at least 15 minutes. Clear Condition: When the AP is in sync with desired transmit status.
Mesh Filter-Frequency Mismatch	Major	Trigger Condition: When a Mesh Node’s filter and frequency of its attached Mesh AP are in an incompatible state for at least five minutes. The frequency for Mesh AP attached to Mesh Node is not compatible with filter setting on the Mesh Node. The filter setting is determined by the frequency of the Gateway AP to which the Mesh Node is attached. Clear Condition: When the Mesh Node filter and the frequency of its attached Mesh AP become compatible.

Cell Unwarranted Transmission

Major

Trigger Condition: When an AP is disabled from transmitting but transmits for at least 15 minutes.

Clear Condition: When the AP is in sync with desired transmit status.

Mesh Filter-Frequency Mismatch

Major

Trigger Condition: When a Mesh Node’s filter and frequency of its attached Mesh AP are in an incompatible state for at least five minutes.

The frequency for Mesh AP attached to Mesh Node is not compatible with filter setting on the Mesh Node. The filter setting is determined by the frequency of the Gateway AP to which the Mesh Node is attached.

Clear Condition: When the Mesh Node filter and the frequency of its attached Mesh AP become compatible.

**DAS > Domain Proxy Alerts**


TP CBSD New Grant	Warning	Trigger Condition: New grants have been received in the last one hour. Clear Condition: No new grants have been received for at least one hour.
TP CBSD Preferred Frequency Unavailable	Warning	Trigger Condition: For DAS TPs, the frequency selected for associated AP is unavailable with SAS. Clear Condition: Any of the following: - Preferred frequency is available again - CBSD is deregistered
TP CBSD Preferred Max Power Unavailable	Warning	Trigger Condition: For DAS TPs, the max power selected for DAS TPs is unavailable with SAS. Clear Condition: Any of the following: - Preferred power is available again - CBSD is deregistered
TP CBSD Registration Failure	Major	Trigger Condition: CBSD registrations have failed within the past hour. Clear Condition: No CBSD registrations have failed for at least an hour.

**Mesh Node > Domain Proxy Alerts**


CPE CBSD New Grant	Warning	Trigger Condition: New grants were received within the last one hour. Clear Condition: No new grants were received in the last one hour.
CPE CBSD Preferred Frequency or Bandwidth Unavailable	Warning	Trigger Condition: The backhaul frequency or bandwidth to which the Mesh Node is connected is unavailable. Clear Condition: When the preferred frequencies or bandwidth are available again.
CPE CBSD Registration Failure	Major	Trigger Condition: CPE CBSD registrations have encountered failures within the last hour. Clear Condition: No CPE CBSD registrations have failed for at least one hour.

CPE CBSD New Grant

Warning

Trigger Condition: New grants were received within the last one hour.

Clear Condition: No new grants were received in the last one hour.

CPE CBSD Preferred Frequency or Bandwidth Unavailable

Warning

Trigger Condition: The backhaul frequency or bandwidth to which the Mesh Node is connected is unavailable.

Clear Condition: When the preferred frequencies or bandwidth are available again.

CPE CBSD Registration Failure

Major

Trigger Condition: CPE CBSD registrations have encountered failures within the last hour.

Clear Condition: No CPE CBSD registrations have failed for at least one hour.

**Mesh Node > Management Plane Alerts**


Offline	Critical	Trigger Condition: When a Mesh Node has remained offline for at least 10 minutes. Clear Condition: When the Mesh Node comes online.

Offline

Critical

Trigger Condition: When a Mesh Node has remained offline for at least 10 minutes.

Clear Condition: When the Mesh Node comes online.

**Mesh Node > MCU-Plane Alerts**


Tunnel Down	Major	Trigger Condition: When the mesh tunnel between a Mesh Node and Onyx Edge is down for at least five minutes. Clear Condition: When the mesh tunnel between the Mesh Node and Onyx Edge comes up.
Watchdog Powercycle	Minor	Trigger Condition: A Mesh Node power cycle is triggered by the watchdog within the last 10 minutes. Clear Condition: No watchdog-triggered power cycles occur within a subsequent 10-minute window.

Tunnel Down

Major

Trigger Condition: When the mesh tunnel between a Mesh Node and Onyx Edge is down for at least five minutes.

Clear Condition: When the mesh tunnel between the Mesh Node and Onyx Edge comes up.

Watchdog Powercycle

Minor

Trigger Condition: A Mesh Node power cycle is triggered by the watchdog within the last 10 minutes.

Clear Condition: No watchdog-triggered power cycles occur within a subsequent 10-minute window.

**Mesh Node > Platform Alerts**


CPU Usage Alert	Minor	Trigger Condition: When a Mesh Node’s CPU utilization in the last five minutes exceeds 70% of capacity. Clear Condition: When the CPU utilization falls below 70% of capacity.
Memory Usage Alert	Minor	Trigger Condition: When a Mesh Node’s RAM utilization exceeds 70% of capacity. Clear Condition: When the RAM utilization falls below 70% of capacity.
Disk Usage Alert	Major	Trigger Condition: When a Mesh Node’s disk utilization in the last five minutes exceeds 70% of capacity. Clear Condition: When the disk usage falls below 70% of capacity.

CPU Usage Alert

Minor

Trigger Condition: When a Mesh Node’s CPU utilization in the last five minutes exceeds 70% of capacity.

Clear Condition: When the CPU utilization falls below 70% of capacity.

Memory Usage Alert

Minor

Trigger Condition: When a Mesh Node’s RAM utilization exceeds 70% of capacity.

Clear Condition: When the RAM utilization falls below 70% of capacity.

Disk Usage Alert

Major

Trigger Condition: When a Mesh Node’s disk utilization in the last five minutes exceeds 70% of capacity.

Clear Condition: When the disk usage falls below 70% of capacity.

**Mesh Node > Provisioning Server Alerts**


Provisioning Server Connection Failure	Warning	Trigger Condition: When a Mesh Node has not successfully checked in with the Provisioning Server within the past 35 minutes. Clear Condition: When the Mesh Node successfully checks in with the Provisioning Server.

Provisioning Server Connection Failure

Warning

Trigger Condition: When a Mesh Node has not successfully checked in with the Provisioning Server within the past 35 minutes.

Clear Condition: When the Mesh Node successfully checks in with the Provisioning Server.

**Onyx Edge (Core) > Control Plane Alerts**


Authentication Failure	Warning	Trigger Condition: When an authentication failure occurs during UE attach. Clear Condition: When there is no authentication failure in the next five-minute window.
DHCP Server Failed	Major	Trigger Condition: When the DHCP server responsible for assigning IP addresses to UE is unavailable for five minutes. Clear Condition: When the DHCP server becomes available.
S1 Setup Failure	Major	Trigger Condition: When an S1 Setup failure occurs. Clear Condition: When there is no S1 Setup failure in the next five-minute window.
SCTP Outbound Stalled	Major	Trigger Condition: SCTP inbound ordered chunks have been increasing over the last 15 minutes while SCTP outbound ordered chunks have remained flat. Clear Condition: Outbound SCTP chunks resume normal flow, indicating the core is processing and responding to signaling messages again.
UE Attach Failure	Warning	Trigger Condition: When UE attach requests fail for reasons other than unknown user/IMEI validations in the last 15 minutes. Clear Condition: When no UE attach requests fail for reasons other than unknown user/IMEI validations for at least 15 minutes.
UE IMSI-IMEI Validation Failure	Warning	Trigger Condition: When a UE attach failure occurs due to IMSI-IMEI validation failure in the last 15 minutes. Clear Condition: When there is no UE attach failure due to IMSI-IMEI validation failure in the last 15 minutes.
UE Session Failure	Warning	Trigger Condition: When a PDU Session Creation failure occurs. Clear Condition: When there are no PDU Session Creation failures in the next five-minute window.

**Onyx Edge (Core) > CU-Plane Alerts**


HA Peer Down	Major	Trigger Condition: When, in an Onyx Edge pool, the Primary ACS Onyx Edge detects a peer Onyx Edge is down. A peer can be marked as “down” for the following reasons: · S1 interface is down · SGi interface is down · Both S1 and SGi interfaces are down · The peer Onyx Edge is powered down The Primary Onyx Edge detects a peer is down within 10 minutes of it going down. Clear Condition: When the Primary ACS Onyx Edge detects that the peer Onyx Edge is operational.
IPSec Service Down	Critical	Trigger Condition: When an Onyx Edge’s IPSec service goes down and is not listening for new IPSec connection requests from APs. Clear Condition: When IPSec service is ready and is listening for connections from APs.
IPSec Service Restarting Frequently	Major	Trigger Condition: When an Onyx Edge’s IPSec service restarts too frequently. Clear Condition: When the IPSec service is stable.

HA Peer Down

Major

Trigger Condition: When, in an Onyx Edge pool, the Primary ACS Onyx Edge detects a peer Onyx Edge is down.

A peer can be marked as “down” for the following reasons:

· S1 interface is down

· SGi interface is down

· Both S1 and SGi interfaces are down

· The peer Onyx Edge is powered down

The Primary Onyx Edge detects a peer is down within 10 minutes of it going down.

Clear Condition: When the Primary ACS Onyx Edge detects that the peer Onyx Edge is operational.

IPSec Service Down

Critical

Trigger Condition: When an Onyx Edge’s IPSec service goes down and is not listening for new IPSec connection requests from APs.

Clear Condition: When IPSec service is ready and is listening for connections from APs.

IPSec Service Restarting Frequently

Major

Trigger Condition: When an Onyx Edge’s IPSec service restarts too frequently.

Clear Condition: When the IPSec service is stable.

**Onyx Edge (Core) > Management Plane Alerts**


Offline	Critical	Trigger Condition: When an Onyx Edge remains offline for more than 10 minutes. Clear Condition: When the Onyx Edge comes online.

Offline

Critical

Trigger Condition: When an Onyx Edge remains offline for more than 10 minutes.

Clear Condition: When the Onyx Edge comes online.

**Onyx Edge (Core) > MCU-Plane Alerts**


Service Down	Critical	Trigger Condition: When critical Onyx Edge services are down, or there is connectivity issue (all APs are S1 disconnected) in the last five minutes. Clear Condition: When critical Onyx Edge services are running correctly, and not all APs are S1-disconnected.

Service Down

Critical

Trigger Condition: When critical Onyx Edge services are down, or there is connectivity issue (all APs are S1 disconnected) in the last five minutes.

Clear Condition: When critical Onyx Edge services are running correctly, and not all APs are S1-disconnected.

**Onyx Edge (Core) > Platform Alerts**


CPU Usage	Minor	Trigger Condition: When an Onyx Edge’s CPU usage exceeds 70% in the last five minutes. Clear Condition: When the CPU usage falls below 70%.
Disk Usage	Major	Trigger Condition: When an Onyx Edge’s disk usage exceeds 70% in the last five minutes. Clear Condition: When disk usage falls below 70%.
Memory Usage	Minor	Trigger Condition: When an Onyx Edge’s RAM usage exceeds 70%. Clear Condition: When the RAM usage falls below 70%.

CPU Usage

Minor

Trigger Condition: When an Onyx Edge’s CPU usage exceeds 70% in the last five minutes.

Clear Condition: When the CPU usage falls below 70%.

Disk Usage

Major

Trigger Condition: When an Onyx Edge’s disk usage exceeds 70% in the last five minutes.

Clear Condition: When disk usage falls below 70%.

Memory Usage

Minor

Trigger Condition: When an Onyx Edge’s RAM usage exceeds 70%.

Clear Condition: When the RAM usage falls below 70%.

**Onyx Edge (Core) > Provisioning Server Alerts**


Provisioning Server Connection Failure	Warning	Trigger Condition: When an Onyx Edge has not successfully checked in with the Provisioning Server within the past 35 minutes. Clear Condition: When the Onyx Edge successfully checks in with the Provisioning Server.

Provisioning Server Connection Failure

Warning

Trigger Condition: When an Onyx Edge has not successfully checked in with the Provisioning Server within the past 35 minutes.

Clear Condition: When the Onyx Edge successfully checks in with the Provisioning Server.

**Orchestrator > Domain Proxy Alerts**


SAS Connection Failure	Critical	Trigger Condition: When unable to connect to SAS for at least three times in the last 15 minutes, or when HTTP-level errors are seen in the last 15 minutes. The latter is most likely due to a misconfigured URL or a change in URL on the SAS side. Clear Condition: When there are no SAS connection errors and no HTTP errors in the last 15 minutes.

SAS Connection Failure

Critical

Trigger Condition: When unable to connect to SAS for at least three times in the last 15 minutes, or when HTTP-level errors are seen in the last 15 minutes. The latter is most likely due to a misconfigured URL or a change in URL on the SAS side.

Clear Condition: When there are no SAS connection errors and no HTTP errors in the last 15 minutes.

### Custom Alert Rules {% hint style="info" %} **NOTE:** Custom alert rules are specific to each network. {% endhint %} {% hint style="info" %} **NOTE:** You can create and manage custom alert rules only if you are a GXC Administrator. {% endhint %} #### View a Custom Alert Rule's Details **To view a custom alert rule's details:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab is displayed. The **Custom Alerts** section lists the existing custom alert rules. {% hint style="info" %} **NOTE:** The **Custom Alerts** section is displayed only if at least one custom alert rule exists. {% endhint %} {% endstep %} {% step %} For the custom alert rule that you want to view details for, click the corresponding

(view details) icon. {% endstep %} {% step %} The **View Alert Rule** dialog box displays the rule's details. * **Alert** – The custom alert's name. * **Description** – Description of what the alert monitors or the condition it checks for. * **Expression** – The expression or condition that triggers the alert. * **Duration** – The duration for which the expression must continuously remain in effect before the alert is triggered. * **Severity** – The alert's severity classification—Notice, Info, Warning, Minor, Major, or Critical. * **Labels** – Metadata associated with the alert in key-value pairs. {% endstep %} {% endstepper %} #### Create Custom Alert Rules {% hint style="info" %} **NOTE:** You can create custom alert rules only if you are a GXC Administrator. {% endhint %} You can create custom alert rules to define conditions for which alerts must be triggered. Before you create a new alert rule, check if an existing alert rule already serves the purpose or can be modified for your requirements. **To create a custom alert rule:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab is displayed. The **Custom Alerts** section lists the custom alert rules. {% hint style="info" %} **NOTE:** The **Custom Alerts** section is displayed only if at least one custom alert rule has already been created. {% endhint %} {% endstep %} {% step %} In the upper-right corner of the page, click **Add New**. The **Add Alert Rule** page is displayed.

{% endstep %} {% step %} Enter the following details: * **Summary**: * **Rule Name** – (Required) The alert rule's name. Alert rule names can be a maximum of 64 characters and can contain alphanumeric and \_ (underscore) and - (hyphen) characters. * **Description** – (Optional) A brief explanation of what the alert monitors or the condition it checks for. * **Conditions**: * **Expression** – (Required) The Prometheus expression. Contact GXC Engineering for the expression to use.\ Link to the Prometheus Query Language (PromQL) documentation: [*https://prometheus.io/docs/prometheus/latest/querying/basics/*](https://prometheus.io/docs/prometheus/latest/querying/basics/) * **Duration** – (Optional) Select the duration for which the condition must continuously remain in effect before the alert is triggered. To trigger an alert immediately when a condition becomes true, enter 0. * **Unit** – (Optional) Select the units for the value specified in **Duration** from **Seconds**, **Minutes**, or **Hours**. * **Severity** – (Optional) Select the severity classification for the alert from: * **Notice** – A normal but important condition, not related to any error situation. * **Info** – General information about a specific condition. * **Warning** – A potential or impending condition requiring analysis and/or corrective action to prevent it from becoming a Major or a Minor condition. * **Minor** – A condition requiring corrective action to prevent it from becoming a Major condition. * **Major** – A condition requiring urgent corrective action to prevent it from becoming a Critical condition. * **Critical** – A significant condition requiring urgent corrective action. * **Notifications** – Manage alert rule notifications. * **Enable e-mail notification** – Select to notify alert receivers when this rule triggers an alert. For information on alert receivers, see [*Manage Alert Receiver Accounts*](#manage-alert-receiver-accounts). * **Labels** – Add labels to provide more information about the alerts. * **Add New Label** – (Optional) Click to create additional labels to provide more information about the alerts. * **Name** – The label's name. The label name must be an alphanumeric string. It cannot contain white spaces or special characters. * **Value** – The label's value. * **Action** – To delete the label, click the ![](https://4071075005-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FZc9hpHmiTCrh1sn4mWXn%2Fuploads%2F5nvdoVJAv3ABoZSdOcXI%2Fimage.png?alt=media\&token=adb4e8c6-2f55-43c7-8217-4aeea489d2d8) (delete) icon. {% endstep %} {% endstepper %} #### Edit Custom Alert Rules {% hint style="info" %} **NOTE:** You can edit custom alert rules only if you are a GXC Administrator. {% endhint %} {% hint style="info" %} **NOTE:** You cannot edit predefined alert rules, except to enable or disable per alert rule email notifications. For more information, see [*Mute Alert Notifications per Predefined Alert Rule*](#mute-alert-notifications-per-predefined-alert-rule). {% endhint %} **To edit a custom alert rule:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab is displayed. The **Custom Alerts** section lists the custom alert rules. {% hint style="info" %} **NOTE:** The **Custom Alerts** section is displayed only if at least one custom alert rule exists. {% endhint %} {% endstep %} {% step %} For the custom alert rule that you want to edit, click the corresponding

(edit) icon. {% endstep %} {% step %} Update the custom alert rule as required. For parameter descriptions, see [*Add Custom Alert Rules section*](#create-custom-alert-rules). {% endstep %} {% step %} To save your edits, in the upper-right corner of the page, click **Save Rule**. {% endstep %} {% endstepper %} #### Delete Custom Alert Rules {% hint style="info" %} **NOTE:** You can delete custom alert rules only if you are a GXC Administrator. Predefined alert rules cannot be deleted. {% endhint %} {% hint style="info" %} **NOTE:** Custom alert rules when deleted are lost for good. Before you delete a custom alert rule, review the rule to ensure it is no longer required before you delete it. {% endhint %} **To delete a custom alert rule:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Alert Rules** tab. The **Alerts** page > **Alert Rules** tab is displayed. The **Custom Alerts** section lists the custom alert rules. {% hint style="info" %} **NOTE:** The **Custom Alerts** section is displayed only if at least one custom alert rule exists. {% endhint %} {% endstep %} {% step %} For the custom alert rule that you want to delete, click the corresponding

(delete) icon. A confirmation dialog box is displayed. {% endstep %} {% step %} To confirm deleting the custom alert rule, click **Yes**. {% endstep %} {% endstepper %} ### Semi-Custom Alert Rules Semi-custom alert rules are predefined by GXC but allow limited configuration changes, such as threshold values. These rules retain their core logic as defined by GXC but can be adjusted to better fit a specific network's operational needs.

Semi-Custom Alert Rules

**Mesh Node > MCU Plane Alerts**\[1]

Alert Name	Severity	Description
Mesh Node <serial number> RSRP is lower than threshold for 5 minutes	Major	Trigger Condition: Triggers when RSRP is lower than threshold for 5 minutes. Clear Condition: Clears when RSRP rises to or above the threshold for 5 minutes.
Mesh Node <serial number> RSRQ is lower than threshold for 5 minutes	Major	Trigger Condition: Triggers when RSRQ is lower than threshold for 5 minutes. Clear Condition: Clears when RSRQ rises to or above the threshold for 5 minutes.
Mesh Node <serial number> SINR is lower than threshold for 5 minutes	Major	Trigger Condition: Triggers when SINR is lower than threshold for 5 minutes. Clear Condition: Clears when SINR rises to or above the threshold for 5 minutes.

Alert Name

Severity

Description

Mesh Node <serial number> RSRP is lower than threshold for 5 minutes

Major

Trigger Condition: Triggers when RSRP is lower than threshold for 5 minutes.

Clear Condition: Clears when RSRP rises to or above the threshold for 5 minutes.

Mesh Node <serial number> RSRQ is lower than threshold for 5 minutes

Major

Trigger Condition: Triggers when RSRQ is lower than threshold for 5 minutes.

Clear Condition: Clears when RSRQ rises to or above the threshold for 5 minutes.

Mesh Node <serial number> SINR is lower than threshold for 5 minutes

Major

Trigger Condition: Triggers when SINR is lower than threshold for 5 minutes.

Clear Condition: Clears when SINR rises to or above the threshold for 5 minutes.

*** \[1] For information on configuring thresholds for Mesh Node semi-custom alerts, see *Add Mesh Nodes*. ## Manage Alert Receiver Accounts You can configure the receivers to be notified when alerts are triggered. The receivers are notified by email in real-time to review the alerts and take action. As per the configuration, a receiver can be notified about all alerts or only alerts of specific severities. {% hint style="info" %} **NOTE:** Alert receiver configurations are specific to the selected network. {% endhint %} ### View Summary Details of All Alert Receiver Accounts **To view the summary details of all alert receiver accounts:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Receivers** tab. The **Alerts** page > **Receivers** tab displays the list of alert receiver accounts configured in the selected network.

* **E-mail** – The alert receiver's email ID. * **Severity of Alert** – The alert severity levels for which the receiver is notified. * **Actions** – Options to view, edit, and delete alert receiver accounts. {% endstep %} {% endstepper %} ### Create Alert Receiver Accounts {% hint style="info" %} **NOTE:** You can add alert receiver accounts only if you are a GXC Administrator, Partner Administrator, Super Administrator, or an Administrator. {% endhint %} **To create an alert receiver account:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Receivers** tab. The **Alerts** page > **Receivers** tab displays the list of alert receiver accounts configured in the selected network. {% endstep %} {% step %} In the upper-right corner of the page, click **Add Receivers**. The **Create Receiver** page is displayed.

{% endstep %} {% step %} Enter the following details: * **E-mail** – (Required) The alert receiver's e-mail ID to send alert notifications. * **Severity Level** – (Required) The alert severity levels for which notifications should be sent to this receiver. Select from **Critical**, **Major**, **Minor**, or **All**. {% endstep %} {% step %} To save the alert receiver account, in the upper-right corner of the page, click **Save Receiver**. The alert receiver account is saved and is added to the list of receivers. {% endstep %} {% endstepper %} ### Edit Alert Receiver Accounts {% hint style="info" %} **NOTE:** You can edit alert receiver accounts only if you are a GXC Administrator, Partner Administrator, Super Administrator, or an Administrator. {% endhint %} **To edit an alert receiver account:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Receivers** tab. The **Alerts** page > **Receivers** tab displays the list of alert receiver accounts configured in the selected network. {% endstep %} {% step %} For the alert receiver account that you want to edit, in the **Actions** column, click the corresponding ⋮ (options) icon, then click **Edit**.

The **Edit Receiver** page is displayed.

{% endstep %} {% step %} Edit the following details field as required: * **E-mail** – (Read only) The alert receiver's email Id. * **Severity Level** – (Required) The alert severity levels for which notifications should be sent for this receiver. Select from **Critical**, **Major**, **Minor**, or **All**. {% endstep %} {% step %} To save your edit, in the upper-right corner of the page, click **Save Receiver**. {% endstep %} {% endstepper %} ### Delete Alert Receiver Accounts {% hint style="info" %} **NOTE:** You can delete alert receiver accounts only if you are a GXC Administrator, Partner Administrator, or a Super Administrator. {% endhint %} **To delete an alert receiver account:** {% stepper %} {% step %} In the navigation pane, click **Alerts** > **Receivers** tab. The **Alerts** page > **Receivers** tab displays the list of alert receiver accounts configured in the selected network. {% endstep %} {% step %} For the alert receiver account that you want to delete, in the **Actions** column, click the corresponding ⋮ (options) icon, then click **Delete**.

A confirmation dialog box is displayed.

{% endstep %} {% step %} To confirm deleting the alert receiver account, click **Yes**. {% endstep %} {% endstepper %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.gxc.io/docs/network-configuration-and-operations-guides/4g-onyx-portal-4.x-operations/manage-alerts.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.