Export virtual machines with ASC monitoring agent issue

There is a recommendation named “Monitoring agent health issues should be resolved on your machine” in Azure Security Center that provides you list of unhealthy resources (virtual machine resource type). There are several reasons that can cause unhealthy monitoring state on your virtual machines.

You may wonder if there is a way to get all the unhealthy virtual machines along with monitoring state without opening Azure Portal? In this article, let’s see how to export all unhealthy virtual machines and corresponding monitoring agent state.

Why is auditing monitoring agent important? Log Analytics agent is the core agent that collects virtual machine log for you. Azure Security Center sensor is integrated into virtual machine agent in order to collect security event, audit log and report to Log Analytics backend as well as Azure Security Center portal. If an agent is not working, you won’t have data in order for troubleshooting, especially for incident investigation.

In Azure Resource Graph, there is a table named securityresources that provides you security state, pricing tier as well as compliance state of your resources that are scanned by Azure Policy. There is a type named microsoft.security/securitystatuses  that provides you policy compliance result (Azure Security Center Recommendation is powered by Azure Policy).

You can do a simple query to like below to check what kind of data:

securityresources
| where type == "microsoft.security/securitystatuses"

The policyAssessments array gives you all policies that are used for specific resource. Below is the sample one:

{
    "resourceDetails": [
        {
            "value": "On",
            "name": "VM Agent installed"
        },
        {
            "value": "Off",
            "name": "Monitoring agent extension installed"
        },
        {
            "value": "FailureDueToVmStopped",
            "name": "Monitoring agent installation status"
        },
        {
            "value": "On",
            "name": "Automatic monitoring agent installation policy"
        },
        {
            "value": "Linux",
            "name": "OS Type"
        },
        {
            "value": "True",
            "name": "Is supported"
        },
        {
            "value": "False",
            "name": "OS disk encrypted"
        },
        {
            "value": "True",
            "name": "Data disk encrypted"
        }
    ],
    "name": "GenericResourceHealthProperties",
    "securityState": "High",
    "securityStateByCategory": [
        {
            "category": "Compute",
            "securityState": "High"
        },
        {
            "category": "Networking",
            "securityState": "High"
        }
    ],
    "policyAssessments": [
        {
            "policyDefinitionId": "",
            "assessmentKey": "8e2b96ff-3de2-289b-b5c1-3b9921a3441e",
            "assessmentResult": "Medium",
            "category": "Compute",
            "policyName": "Monitoring agent health issues should be resolved on your machines",
            "assessmentDetails": []
        },
        {
            "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/86b3d65f-7626-441e-b690-81a8b71cff60",
            "assessmentKey": "c0f5316d-5ac5-9218-b77a-b96e16ccfd66",
            "assessmentResult": "None",
            "category": "Compute",
            "policyName": "Your machines should be restarted to apply system updates",
            "assessmentDetails": []
        },
        {
            "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/0961003e-5a0a-4549-abde-af6a37f2724d",
            "assessmentKey": "d57a4221-a804-52ca-3dea-768284f06bb7",
            "assessmentResult": "High",
            "category": "Compute",
            "policyName": "Disk encryption should be applied on virtual machines",
            "assessmentDetails": []
        },
        {
            "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/e1e5fd5d-3e4c-4ce1-8661-7d1873ae6b15",
            "assessmentKey": "181ac480-f7c4-544b-9865-11b8ffe87f47",
            "assessmentResult": "None",
            "category": "Compute",
            "policyName": "Vulnerabilities in security configuration on your machines should be remediated",
            "assessmentDetails": [
                {
                    "value": "01/01/0001 00:00:00",
                    "name": "Last scan time"
                },
                {
                    "value": "False",
                    "name": "Scan data is valid"
                },
                {
                    "value": "False",
                    "name": "Scan data exists"
                },
                {
                    "value": "On",
                    "name": "PolicyState"
                }
            ]
        },
        {
            "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/86b3d65f-7626-441e-b690-81a8b71cff60",
            "assessmentKey": "4ab6e3c5-74dd-8b35-9ab9-f61b30875b27",
            "assessmentResult": "None",
            "category": "Compute",
            "policyName": "System updates should be installed on your machines",
            "assessmentDetails": [
                {
                    "value": "01/01/0001 00:00:00",
                    "name": "Last scan time"
                },
                {
                    "value": "False",
                    "name": "Scan data is valid"
                },
                {
                    "value": "False",
                    "name": "Scan data exists"
                },
                {
                    "value": "On",
                    "name": "PolicyState"
                }
            ]
        },
        {
            "policyDefinitionId": "/providers/Microsoft.Authorization/policyDefinitions/760a85ff-6162-42b3-8d70-698e268f648c",
            "assessmentKey": "71992a2a-d168-42e0-b10e-6b45fa2ecddb",
            "assessmentResult": "None",
            "category": "Compute",
            "policyName": "Vulnerabilities should be remediated by a Vulnerability Assessment solution"
        }
    ],
    "type": "VirtualMachine"
}

There is a policy named “Monitoring agent health issues should be resolved on your machines” so you simply need to run the following query:

securityresources
| where type == "microsoft.security/securitystatuses"
| mv-expand properties.policyAssessments
| where properties_policyAssessments.policyName ==  "Monitoring agent health issues should be resolved on your machines"

mv-expand() is used to extract each of object in the collection to a separate row. We expand the policyAssessments[] collection before filtering the recommendation name.

Below is the table of status code and description

Status CodeDescription
NoHeartbeatAgent not responsive or missing ID
FailureDueToVmStoppedPower Off State
MonitoringExtensionGeneralFailureInstallation failed - general error
InProgressPending automatic agent installation
WorkspaceNotAccessibleMissing or inaccessible workspace
FailureDueToAgentNotRespondingMissing or invalid Azure VM agent
ExtensionExistingAgentFailureInstallation failed - local agent already installed

To get all virtual machines with monitoring agent status code, use the query below

securityresources
| where type == "microsoft.security/securitystatuses"
| mv-expand properties.policyAssessments
| where properties_policyAssessments.policyName ==  "Monitoring agent health issues should be resolved on your machines"
| mv-expand properties.resourceDetails
| where properties_resourceDetails.value in (
	"NoHeartbeat", //Agent not responsive or missing ID
	"FailureDueToVmStopped", //Power Off State
	"MonitoringExtensionGeneralFailure", //Installation failed - general error
	"InProgress", //Pending automatic agent installation
	"WorkspaceNotAccessible", //Missing or inaccessible workspace
	"FailureDueToAgentNotResponding", //Missing or invalid Azure VM agent
	"ExtensionExistingAgentFailure" //Installation failed - local agent already installed 
)
| where properties_policyAssessments.assessmentResult != "healthy"
| project subscriptionId, 
		  vmName=name, 
		  resourceGroup, 
		  statusCode=properties_resourceDetails.value

Once the query is succesfully executed, you can download result as CSV and start checking and fixing/redeploying the agent.

This entry was posted in Monitoring & Detection and tagged , . Bookmark the permalink.

Leave a Reply