Learn about Azure Centralized Monitoring – Data Sources, Data Platform and Consumption layers in detail

Azure Centralized Monitoring provides a unified view of the health, performance, and security of Azure resources and workloads. It is typically structured into three main layers: Data Sources, Data Platform, and Consumption Layers. These layers work together to collect, process, analyze, and visualize monitoring data.

Data Sources

The Data Sources layer is where raw telemetry and logs are generated. It includes a variety of sources, such as Azure services, applications, and infrastructure components.

Key Components in Data Sources

1.1 Metrics

Definition:

Numerical data that measures the performance or usage of resources over time.

Examples:

1. CPU utilization, disk IOPS, network throughput for VMs.

2. Request latency and throughput for applications.

Granularity:

Often emitted at short intervals (e.g., every minute) for near-real-time insights.

1.2 Logs

Definition:

Detailed, event-driven records that provide context for system and application behavior.

Examples:

1. Activity Logs: Track Azure resource management operations.

2. Resource Logs: Contain diagnostic data from Azure resources (e.g., HTTP requests to an API).

3. Application Logs: Custom logs generated by applications (e.g., errors, exceptions).

4. Security Logs: Alerts, vulnerabilities, and compliance-related events.

1.3 Traces

Definition:

Fine-grained data about the execution of distributed applications.

Purpose:

Helps identify performance bottlenecks and dependencies.

Examples:

Application Insights traces for distributed microservices.

1.4 Change Tracking

Definition:

Tracks configuration changes in resources and environments.

Examples:

Updates to VM settings or network configurations.

1.5 Dependency Data

Definition:

Information about dependencies between resources, such as network connections or service calls.

Examples:

Application Insights dependency tracking between APIs and databases.

Data Platform

The Data Platform layer processes, stores, and analyzes the raw telemetry data collected from the sources. It serves as the backbone for centralized monitoring.

Key Components in Data Platform

2.1 Data Collection

1. Tools:

   • Azure Monitor Agents:

     • Azure Monitor Agent (AMA): Collects logs and metrics from Azure and non-Azure resources.

     • Log Analytics Agent: Older agent for collecting telemetry (being phased out).

     • Azure Diagnostics Extension: Captures VM-level diagnostics.

   • Application Insights SDKs: Used in applications for collecting telemetry like traces, exceptions, and performance data.

2. Ingestion Capabilities:

   • Supports real-time and batch ingestion.

   • Allows data filtering and transformation during ingestion.

2.2 Data Storage

Purpose:

Provides scalable storage for collected telemetry and logs.

Key Stores:

1. Log Analytics Workspace:

   • Central repository for logs and analytics.

   • Built on Azure Data Explorer for querying large volumes of data.

2. Azure Blob Storage:

   • Stores diagnostic data (e.g., VM diagnostics) for long-term archival.

3. Azure Data Lake:

   • Used for advanced analytics and machine learning.

Retention Policies:

Configurable retention periods for short-term (e.g., 30 days) and long-term (years) storage.

2.3 Data Processing and Enrichment

Purpose:

Extracts meaningful insights from raw data.

Tools and Services:

1. KQL (Kusto Query Language): Used for querying and transforming data in Log Analytics.

2. Azure Functions: Processes and enriches data during ingestion.

3. Azure Stream Analytics: Processes real-time streaming telemetry.

Capabilities:

1. Correlation of logs across multiple resources (e.g., VM and database logs).

2. Aggregation and trend analysis.

2.4 Integration and Forwarding

Purpose:

Shares telemetry with external systems for further processing or compliance.

Tools:

1. Event Hubs: Sends logs and metrics to third-party monitoring tools.

2. Logic Apps: Automates workflows based on telemetry.

3. Azure Data Factory: Moves data to other analytics platforms.

Supported Formats:

JSON, CSV, and custom formats.

Consumption Layer

The Consumption Layer is where data is visualized, queried, and consumed by users or systems to make informed decisions.

Let's explore the Key Components on this layer.

3.1 Dashboards and Visualizations

1. Azure Dashboards: Customizable dashboards in the Azure Portal. Aggregate multiple widgets showing metrics, logs, and health overviews.

2. Workbooks: Interactive dashboards built on Log Analytics queries. Enable drill-down into detailed insights (e.g., application performance analysis).

3. Power BI Integration: Allows creation of rich, interactive visualizations using data from Log Analytics.

3.2 Querying and Analytics

1. Log Analytics: Query logs and telemetry using KQL for deep analysis.

   Examples: Identify VMs with high CPU utilization over the last 24 hours. Detect anomalies in application response times.

2. Application Insights Analytics: Provides specific tools for querying application performance and dependencies. Includes distributed tracing for microservices.

3.3 Alerts and Notifications

1. Azure Alerts: Define alert rules based on metrics or log conditions. Supports dynamic thresholds to automatically adjust to historical baselines.

   Examples:

   • Alert when CPU usage exceeds 80% for 5 minutes.

   • Notify on database query response time exceeding 2 seconds.

2. Notification Channels: Email, SMS, push notifications, or integration with ITSM tools like ServiceNow. Webhooks for custom integrations.

3.4 Automation

Purpose:

Automates responses to events or alerts.

Tools:

1. Azure Logic Apps: Trigger workflows (e.g., create a ticket in ServiceNow).

2. Azure Automation Runbooks: Automate remediations (e.g., restarting a VM).

3. Azure Functions: Execute custom scripts based on telemetry.

3.5 Insights and Recommendations

1. Azure Insights: Built-in insights for common workloads:

   • VM Insights: Tracks performance and dependency mapping for virtual machines.

   • Container Insights: Monitors health and resource usage in Kubernetes clusters.

   • SQL Insights: Provides query performance and database-level diagnostics.

2. Azure Advisor: Offers optimization recommendations based on telemetry (e.g., cost, performance, security).

Summary

This architecture ensures scalable, secure, and actionable centralized monitoring for Azure environments, combining real-time insights with advanced analytics for operational excellence.