Skip to content

Monitoring

The application includes a fully provisioned and preconfigured monitoring stack that allows you to monitor container performance, resource usage, and logs. This stack is ready to use immediately after installation and is accessible via the monitoring URL.

Monitoring Stack Overview

The monitoring stack consists of the following components:

Component Version Purpose
Grafana 12.4.2 Visualization and dashboards for metrics and logs
Prometheus 3.11.3 Time-series database storing container and host metrics
Loki 3.7.2 Log aggregation system for storing and querying container logs
Grafana Alloy 1.15.1 Collection agent gathering container metrics and logs from the host

Grafana Alloy is the single collection agent for the stack. It reads container metadata from the Docker daemon and resource metrics from the host, delivering metrics to Prometheus and container logs to Loki. No separate metrics exporter or log shipper is deployed alongside it.

Key Features

  • Container Status: View the status of all containers in the application.
  • Resource Usage: Monitor CPU, memory, and network usage for each container.
  • Log Aggregation: Access centralized logs for all containers.
  • Log Search and Error Detection: Search logs for specific terms (e.g., ERROR) to quickly identify issues.

Accessing the Monitoring Dashboard

The monitoring stack is served through the same reverse proxy as the application, under the /monitor path:

Text Only
https://$hostname/monitor/

Replace $hostname with the hostname or IP of your server.

Sign-in credentials

The default user name is grafana. The password is the value of GF_SECURITY_ADMIN_PASSWORD in the deployment configuration, and it is set during installation.

Security Note

For security purposes, it is recommended to change the credentials after the first login.

Preconfigured Dashboards

The monitoring stack comes with preconfigured dashboards in Grafana, ready to use out of the box:

1. Docker Monitoring Dashboard

  • Container Selection: Filter by specific container (e.g., backend, unstructured-worker)
  • Container Status: View uptime and operational status of containers
  • Log Event Metrics: Track total log events, categorized by Info, Warning, and Error counts
  • Time Range Selection: Adjust the time window (e.g., last 30 minutes)
  • Resource Visualizations:
    • CPU Usage: Real-time graph showing CPU utilization percentage over time
    • Memory Usage: Detailed memory consumption tracking in GB
    • Disk I/O: Separate panels for read and write operations showing I/O rates
  • Log Tail: Live stream of recent logs with timestamp, log level, thread ID, and message content
  • Refresh Controls: Manual refresh and auto-refresh timing options

2. Container Metrics Dashboard

  • Displays CPU, memory, and network usage for all containers.
  • Helps identify resource bottlenecks or underutilized containers.

3. Log Dashboard

  • Aggregates logs from all containers.
  • Allows searching for specific terms (e.g., ERROR, WARNING) to troubleshoot issues.

4. Overview Dashboard

  • Provides a high-level summary of container health and application performance.

How to Use the Monitoring Tools

1. View Container Metrics

  • Navigate to the Docker Monitoring Dashboard in Grafana.
  • Select the specific container you want to monitor from the dropdown menu.
  • View the container's status (UP/DOWN) and uptime duration.
  • Analyze CPU, memory, and disk I/O usage through the time-series graphs.
  • Use the time range selector to focus on specific time periods (e.g., last 30 minutes, last hour).

2. Search Logs

  • Use the Log Tail panel to view the most recent log entries.
  • Filter logs by log level (INFO, WARN, ERROR) to focus on specific types of events.
  • Use the search bar to filter logs by keywords or specific service names.
  • Check log counts (Info, Warn, Error) to quickly identify potential issues.

3. Detect Errors

  • Monitor the Error Count metric for unexpected increases.
  • Review WARN and ERROR log entries in the Log Tail section.
  • Identify patterns in error messages, such as repeated warnings about vulnerability alerts.
  • Set up alerts in Grafana (optional) to notify you when specific log patterns are detected.

4. Watch Memory Pressure During Scans

Scan workers shut themselves down when system-wide memory usage stays above 85%, so a scan that stops progressing is often a memory problem rather than a scan problem.

  • Watch total memory usage on the host while a scan is running.
  • Check the unstructured-worker containers for repeated restarts — a rising restart count alongside high memory usage points to memory pressure on the server rather than to a fault in the scan.
  • Confirm that no other workload is running on the server. See Dedicated Server.

Deployment Details

Network Configuration

  • The monitoring components do not publish any ports to the host. Only the reverse proxy is reachable from outside, on ports 80 and 443.
  • All components communicate internally within the Docker network, and Grafana is reached through the reverse proxy under /monitor.

Data Retention

  • Metrics and logs are stored in dedicated Docker volumes (prometheus-data, loki-data), and Grafana keeps its own state in grafana-data. Monitoring data therefore survives a restart of the containers.
  • Container log files on the host are rotated by the Docker logging driver, which is configured per service in the Compose file.

Preconfigured Setup

  • The stack is fully provisioned and preconfigured during installation.
  • No additional setup is required after deployment.

Security Recommendations

  1. Change Credentials:

    • After the first login, update the default Grafana credentials to a secure username and password.

  2. Restrict Access:

    • Ensure that access to the monitoring stack is restricted to authorized users only.

  3. Monitor Logs for Security Issues:

    • Regularly search logs for unusual activity or errors that could indicate security issues.