Episode #1: Setting Up Azure Observability with Log Analytics and Azure Storage

YouTube Video

Source Code

Welcome to the first episode of Azure Terraformer, where we dive deep into using Terraform on Azure to set up powerful, scalable cloud solutions. Today, we’ll be setting up a basic Azure observability stack using Log Analytics and Azure Storage to store and monitor your activity logs efficiently. This setup is foundational for gaining insights into the health, performance, and usage of your Azure resources, and it’s also incredibly useful for auditing and compliance needs.

Our stack will use Azure Log Analytics to capture and analyze log data, paired with an Azure Storage Account to store those logs for long-term retention. To avoid naming conflicts, we’ll generate random strings to append to our resources, while adhering to standard naming conventions with prefixes from the official Azure resource abbreviation list.

Episode Roadmap

This article will guide you through the following key steps:

1. Provider Configuration - Setting up the Azure provider to manage Azure resources.
2. Core Resources - Creating the foundational resources: a resource group, storage account, and log analytics workspace.
3. Diagnostic Settings - Using the azurerm_subscription data source to set the scope for your Azure Monitor diagnostic settings, allowing Azure Activity Logs to be stored in the Log Analytics workspace and Azure Storage account.

Each section includes placeholders for Terraform code that you’ll fill in as you follow along. Ready? Let’s terraform our way into Azure observability!

---

Step 1: Provider Configuration

To begin, we need to configure the Azure provider in Terraform. This will allow Terraform to interact with your Azure account and manage resources on your behalf. If you haven’t set up authentication for Azure yet, you may want to look into using a Service Principal or Managed Identity for secure access.

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"      
      version = "~> 3.28.0"
    }
    random = {
      source = "hashicorp/random"
      version = "~> 3.4.3"
    }
  }
}

provider "azurerm" {
  features {
  }
}

---

Step 1: Creating Core Resources

We’ll create the essential components: a Resource Group, Storage Account, and Log Analytics Workspace.

1. Resource Group: The container for all resources in our observability stack.
2. Storage Account: Where our Azure Activity Logs will be stored for long-term retention.
3. Log Analytics Workspace: The central location where logs and metrics are collected for analysis.

Naming Convention and Random String

To ensure naming consistency and avoid conflicts, we’ll use a combination of the standard Azure prefixes along with a randomly generated string. For example:

• Log Analytics Workspace - log-<random_string>
• Storage Account - st<random_string>

resource "random_string" "main" {
  length  = 8
  upper   = false
  special = false
}

resource "azurerm_resource_group" "main" {
  name     = "rg-ep1-${random_string.main.result}"
  location = var.location
}

resource "azurerm_storage_account" "main" {
  name                     = "st${random_string.main.result}"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = azurerm_resource_group.main.location
  account_tier             = "Standard"
  account_replication_type = "GRS"
}

resource "azurerm_log_analytics_workspace" "main" {
  name                = "log-ep1-${random_string.main.result}"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
  sku                 = "PerGB2018"
  retention_in_days   = 30
}

---

Step 2: Diagnostic Settings for Azure Activity Logs

With our core resources created, we now need to capture Azure Activity Logs and send them to our Log Analytics Workspace and Storage Account. We’ll use the azurerm_subscription data source to set the scope of the Azure Monitor Diagnostic Setting, which will allow us to monitor activity at the subscription level.

The Diagnostic Setting will capture critical logs and metrics across your subscription, giving you centralized observability in your workspace and storage.

resource "random_string" "activity_logs" {
  length  = 8
  upper   = false
  special = false
}

data "azurerm_subscription" "current" { }

resource "azurerm_monitor_diagnostic_setting" "activity_logs" {
  name                       = "diag-${random_string.activity_logs.result}"
  target_resource_id         = data.azurerm_subscription.current.id
  storage_account_id         = azurerm_storage_account.main.id
  log_analytics_workspace_id = azurerm_log_analytics_workspace.main.id

  enabled_log {
    category = "Administrative"
  }
  enabled_log {
    category = "Security"
  }
  enabled_log {
    category = "ServiceHealth"
  }
  enabled_log {
    category = "Alert"
  }
  enabled_log {
    category = "Recommendation"
  }
  enabled_log {
    category = "Policy"
  }
  enabled_log {
    category = "Autoscale"
  }
  enabled_log {
    category = "ResourceHealth"
  }

}

---

Wrapping Up

With this setup, you’ve taken the first step in establishing a robust observability stack in Azure using Terraform. This configuration will collect critical activity logs at the subscription level, storing them in your Log Analytics Workspace for analysis and your Storage Account for long-term retention.

This completes our first episode of Azure Terraformer!

Next Steps

In future episodes, we’ll dive deeper into building advanced monitoring and alerting configurations, setting up dashboards, and leveraging insights to improve your Azure environment’s performance and resilience.

Happy Azure Terraforming!