SSP Monitoring
1. Overview
This section describes the SSP monitoring solution from infrastructure to business. Business monitoring is fully based on logs produced by the SSP applications. These logs are collected in a single centralized database. Business activities monitoring and fault detection rely on queries performed on this database.
2. Business Monitoring
The SSP solution for logs collection and visualization is powered by the AWS OpenSearch service (https://opensearch.org/ (https://aws.amazon.com/opensearch-service/ )
2.1. Logs Collection and Storage
Each application generates log files on its host virtual machine. The logs are circular and rotate on ten files of 1 GB. The so-called "Elastic Stack" chain provided by http://elastic.co collects and stores these logs in a centralized database.
Beats daemon (i.e. Filebeat) [https://www.elastic.co/products/beats/filebeat ] is responsible for collecting the logs generated by the applications and shipping them to LogStash.
LogStash application [https://www.elastic.co/products/logstash ] is responsible for parsing, adapting and inserting the logs in the appropriate Elasticsearch database so that they are well-indexed and easy to query.
OpenSearch database cluster [https://opensearch.org/ ] is a noSQL database specialized for storing textual data. It provides a powerful search engine that extracts specific information about SSP activities.
The logs are stored in dedicated indices, depending on their type (i.e. business, technical or business). Old logs are automatically purged after a given retention period. This retention period depends on the type of logs.
Type | ES indices | Description | Retention |
|---|---|---|---|
business | logstash-* | Business logs: used to monitor SSP business activities (i.e. business dashboard in Kibana) | 180 days |
technical | technical-* | Technical logs: used to troubleshoot the applications and generate alerts. | Three days |
nginx-* | NGINX logs: used to monitor the SSP inbound HTTP traffic and generate alerts. | Three days |
2.2. Logs Visualization
Once stored and correctly indexed in the Elasticsearch database, the OpenSearch Dashboards web GUI [https://opensearch.org/docs/2.5/dashboards/quickstart-dashboards/ ] is used to browse and visualize the collected logs.
The OpenSearch Dashboards allows running any query on the data stored by the applications into the Elasticsearch database. It is a “read-only” tool that cannot be used to edit the logs into the database.
Role-based access control to OpenSearch Dashboards is powered by cognito ↔ AWS IAM role ↔ internal role in OpenSearch Dashboards. It ensures that each tenant's users only have access to their own data and dashboards.
The OpenSearch Dashboards UI is accessed through the SSP Console using the URL provided by the NAGRA Cloud Operations team when the tenant is created and delivered to the customer.
OpenSearch Dashboards are delivered with predefined dashboards that show the SSP services' status and performance. Please refer to subsequent sections for predefined dashboards documentation (c.f. Monitoring Dashboards).
3. Infrastructure Monitoring
3.1. Hardware Monitoring
We use the monitoring mechanisms provided by the public cloud provider on a public cloud. I.e. on Amazon’s AWS deployment, AWS CloudWatch [https://aws.amazon.com/cloudwatch ] is used.
3.2. Database Monitoring
Databases are monitored on a public cloud through the dedicated consoles provided by the public cloud providers.