Multi-homing Logging with new Azure Monitor Agent

Sending logs from Azure virtual machine/virtual machine scale set to different Azure Log Analytics workspace (as known as multi-homing) is a common requirement in a large cloud environment. In the past Azure only supported configuring multi-homing on Windows virtual machine. Azure Log Analytics agent for Linux didn’t support to configure a secondary log analytics workspace.

In this article, let’s look at the new Azure Monitor Agent and data collection approach in Azure that supports multi-homing scenario. The article is based on Azure Monitor Agent preview that might be subject to change in the future.

Use Case

In a large Azure cloud environment you may be asked by InfoSec team to collect your virtual machine’s log and send to a centralized Azure Log Analytics workspace. Data in that workspace is typically security event log (Windows Security event, Syslog kernel process, privilege activities…). The current Azure Log Analytics agent only supports to add additional Log Analytics workspace for Windows only. That said, Linux virtual machine’s log (e.g. Syslog) can only be sent to a single workspace.

There are couple of benefits of multi-homing:

  • Support InfoSec team to collect security-related event and log to a single Log Analytics workspace so security analysts don’t have to ask for access to the subscription of virtual machines they want to investigate.
  • Off-load security log. It helps system/application team fully focus on maintaining operational logs as well as system/application operation.

Azure Monitor Agent Overview

Azure Monitor Agent is a complete new agent that helps collect guest operating system of virtual machine and virtual machine scale set. The agent doesn’t needs any special configuration like Azure Linux Diagnostics agent or Log Analytics agent which require you to provide Log Analytics Workspace Id and key during extension deployment. The agent also relies on system-assigned managed identity to authenticate and infer VM identity.

In the new data collection approach, Microsoft introduced two components:

  • Data Collection Rule (DCR): a definition template of what data to collect, how data should be processed and where data should be sent to.
  • Data Collection Rule Association (DCRA): is a pointer on a virtual machine/virtual machine scale set to a rule.

The two components are very important. While DCA instructs the agent what it needs to collect and where it needs to send logs to, the DCRA just tells Azure what DCA a virtual machine is associated to.

The diagram above gives you an overview of the new data collection model in Azure Monitor. There are two different DCRs:

  • Operation: this rule is used to collect different log type. For Windows, it defines two log categories: Application and System. For Linux, it defines two Syslog facilities: LOG_CRON, LOG_DAEMON. The destination is the local Log Analytics workspace.
  • Security: this rule is used to collect Windows Security event and LOG_AUTH, LOG_AUTHPRIV as an example. The destination is the centralized Log Analytics workspace SecOps team is managing.

There are two different associations:

  • Association 1: VM/VMSS in subscription A is associated to DCR Operation.
  • Association 2: VM/VMSS in subscription A  is associated to DCR Security.

In this case, Azure Monitor Agent on VM/VMSSs in subscription A will collect pre-defined log type in DCR template and send them to corresponding Log Analytics workspace.

In a nutshell, to achieve multi-homing, what you would need to do with Azure Monitor Agent and Data Collection is to create DCR template and associate to target VM/VMSS.

Deployment from Azure Portal

There are several ways to create and deploy an Azure resource. The simplest way to get familiar with the new data collection concept is to start with Azure Portal. You can find Data Collection Rules under Settings in Azure Monitor service.

On the right side click Add to start adding a new data collection rule. In this demonstration I have 2 subscriptions:

  • Development
  • Shared Service

My goal is to store data collection rule for collecting operational log in Development subscription and security log in a workspace in Shared Service subscription.

Under Basics tab, enter rule name, select subscription and region. The region is very important. Your DCR must be in the same region with the Log Analytics workspace you want to store log.

Under Resources tab, click Add resource and start adding VM and VMSS from different subscriptions.

Under Collect and deliver tab, click Add data source. From the page, under Data source tab, select Windows event logs and check log types you would like to collect.

Under Destination tab, click Add destination and then select Log Analytics workspace you want to store log. Select Azure Monitor Logs, Subscription and Log Analytics workspace. Again, remember you can only see workspaces that are deployed in the same region as your data collection rule.

Go back to Data source tab and add Linux Syslog and its destination.

Under Review + create tab, review your rule setting and click Create.

During the creation time, Azure will perform two things:

  • Enable System-assigned managed identity on target VM/VMSS.
  • Deploy Azure Monitor Agent extension. For Windows it is AzureMonitorWindowsAgent. For Linux it is AzureMonitorLinuxAgent.

For DCR for Security log, the steps are similar to the one we just created above.

Deployment As Code (ARM Template)

Azure Portal is not the only way to get DCR deployed. You can use REST API, Azure ARM template or Az CLI/PowerShell to create DCR. In this section, let’s have a look at how DCR can be created in form of ARM.

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "dcrOperationName": {
      "type": "string",
      "metadata": {
        "description": "Name of Data Collection Rule"
      }
    },
    "workspaceId": {
      "type": "string",
      "metadata": {
        "description": "ID of the Log Analytics workspace"
      }
    }
  },
  "variables": {},
  "resources": [
    {
      "type": "Microsoft.Insights/dataCollectionRules",
      "apiVersion": "2019-11-01-preview",
      "name": "[parameters('dcrOperationName')]",
      "location": "westus",
      "properties": {
        "dataSources": {
          "windowsEventLogs": [
            {
              "streams": ["Microsoft-Event"],
              "scheduledTransferPeriod": "PT5M",
              "xPathQueries": [
                "Application!*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=5)]]",
                "System!*[System[(Level=1 or Level=2 or Level=3 or Level=4 or Level=5)]]"
              ],
              "name": "operationalWindowsLog"
            }
          ],
          "syslog": [
            {
              "streams": ["Microsoft-Syslog"],
              "facilityNames": ["cron", "daemon"],
              "logLevels": ["Debug"],
              "name": "operationalSyslog"
            }
          ]
        },
        "destinations": {
          "logAnalytics": [
            {
              "workspaceResourceId": "[parameters('workspaceId')]",
              "name": "localWorkspace"
            }
          ]
        },
        "dataFlows": [
          {
            "streams": ["Microsoft-Event", "Microsoft-Syslog"],
            "destinations": ["localWorkspace"]
          }
        ]
      }
    }
  ]
}

Once you have data collection rule created you can go to Azure Portal to add VM/VMSS or deploy an association template:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "vmName": {
      "type": "string",
      "metadata": {
        "description": "Name of VM you want to associate DCR"
      }
    },
    "associationName": {
      "type": "string",
      "metadata": {
        "description": "Name of VM you want to associate DCR"
      }
    },
    "dcrId": {
      "type": "string",
      "metadata": {
        "description": "ID of the Data Collection Rule"
      }
    }
  },
  "resources": [
    {
      "type": "Microsoft.Compute/virtualMachines/providers/dataCollectionRuleAssociations",
      "name": "[concat(parameters('vmName'),'/microsoft.insights/', parameters('associationName'))]",
      "apiVersion": "2019-11-01-preview",
      "properties": {
        "description": "Association of Data Collection Rule",
        "dataCollectionRuleId": "[parameters('dcrId')]"
      }
    }
  ]
}

Azure Monitor Agent extension can only be automatically installed if you add resource via Azure Portal. That said, if you are working with ARM template deployment you need to install Azure Monitor Agent VM/VMSS extension manually. Below is the sample code snippet of Azure Monitor Agent for Linux machine:

{
  "type": "Microsoft.Compute/virtualMachines/extensions",
  "name": "[concat(parameters('vmName'), '/AMALinux')]",
  "apiVersion": "2019-07-01",
  "location": "[parameters('location')]",
  "dependOn": ["[parameters('vmName')]"],
  "properties": {
    "publisher": "Microsoft.Azure.Monitor",
    "type": "AzureMonitorLinuxAgent",
    "typeHandlerVersion": "1.0",
    "autoUpgradeMinorVersion": true
  }
}

Deployment with Az CLI or PowerShell

Microsoft already developed Az CLI and PowerShell for working with Azure Monitor Agent and Data Collection Rule. You can find them from the following links:

For Azure PowerShell, find cmdlet from Azure Monitor category:

For Az CLI, find available commands from az monitor data-collection

Azure Monitor Data Collection REST API

Microsoft already published some common operations for Azure Monitor Data Collection Rule and Association:

Troubleshooting

Troubleshooting Azure VM agent is not easy. You may have faced several issues with other agents like Azure Linux Diagnostics Agent or Azure OMS Agent. The new Azure Monitor Agent is not different. Knowing where it is deployed and what it does can help when you want to troubleshoot Azure Monitor Agent and Data Collection.

For Windows, check the following locations:

  • Extension: C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\{version}
  • Extension Log: C:\WindowsAzure\Logs\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent
  • Agent Configuration: C:\WindowsAzure\Resources\{ID}._{vm_name}.AMADataStore
  • DCR Config: C:\WindowsAzure\Resources\{ID}._{vm_name}.AMADataStore\mcs\configchunks
  • Data Store: C:\WindowsAzure\Resources\{ID}._{vm_name}.AMADataStore\Tables

So basically Azure Monitor Agent on Windows machine reads configuration (DCR rule) stored C:\WindowsAzure\Resources\{ID}._{vm_name}.AMADataStore\mcs\configchunks and collect VM log. It stores log in C:\WindowsAzure\Resources\{ID}._{vm_name}.AMADataStore\Tables in format of *.tfs file

There is also a tool Microsoft provides to help you convert from *.tsf to .txt so you can verify data that the agent has collected. The tool is located in C:\Packages\Plugins\Microsoft.Azure.Monitor.AzureMonitorWindowsAgent\1.0.9.0\Monitoring\Agent\table2csv.exe

For Linux, check the following locations:

  • Extension: /var/lib/waagent/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-{version}
  • Extension log: /var/log/azure/Microsoft.Azure.Monitor.AzureMonitorLinuxAgent-{version}
  • Agent Log: /var/log/mdsd.*
  • Config File: /etc/mdsd.d/config-cache/configchunks

Conclusion

Azure Monitor Agent and Data Collection Rule help you simplify data collection approach in Azure Monitor. It introduces a new agent which has several improvements. The multi-homing is now fully supported on Windows and Linux.

You can find some sample templates from this repository

If you have any question about Azure Monitor Agent and Data Collection, feel free to drop a comment in this post or create an issue in the repository.

This entry was posted in Monitoring & Detection, Security Operation and tagged , . Bookmark the permalink.

Leave a Reply