Azure Automation Hybrid worker is used to manage Azure resources in local environment where compliant connectivity is needed. Normally a hybrid worker needs a certificate installed on it so it can be authorized by Azure AD before it can perform any administrative tasks defined in the runbook.
In this article, we will take a look at a real-world scenario where an attacker with compromised Azure VM in a non-production subscription to gain Log Analytics workspace information then trick Azure cloud administrator to laterally move to a production subscription and gain service principal credential.
Below is a common deployment of Azure Log Analytics workspace and Automation Account linked to support managing and operating VM in an Azure environment. There are several features that require a link between Automation Account and Log Analytics workspace. Some of them are Update Management or Change Tracking.
In the deployment, 3 virtual machines are connected to a Log Analytics workspace. An automation account with a hybrid worker is used to host and execute some runbooks to manage VMs or other resource types.
A hybrid worker has a certificate imported so when it needs to authorize Azure AD it would use the certificate. Depending on the RBAC action assigned on the service principal, the scope of access may vary. You may see such a service principal has widely access (Contributor role) in Azure environment.
To successfully move to production subscription, the following conditions need to be met
- A bad actor/insider compromises a virtual machine in production subscription.
- A bad actor/insider has Virtual Machine Contributor in non-production subscription to create a virtual machine.
- A bad actor/insider has READ access at subscription level/or Azure Automation account.
The target obliviously is the service principal’s certificate. With the certificate, the attacker can sign into Azure and preform recon to see how much he can get from the service principal. To get the certificate, there are several ways:
- Export certificate from Azure Automation
- Gain access to the hybrid worker and download the certificate
Since access to Azure Automation is limited (with Contributor role) there is one option left – gain access to the hybrid worker. However, it is difficult to remotely access it. What if the attacker could add a compromised VM to the hybrid worker group and wait for the cloud administrator to drop the certificate to it?
To add a virtual machine to a hybrid worker, you need two things:
- Workspace Key and ID to add a virtual machine to a Log Analytics workspace
- READ access on Azure Automation to get Automation Account key.
Extract Workspace Key from Log Analytics agent configuration
Workspace information including Workspace Key is stored in Log Analytics agent runtime setting that is located at /var/lib/waagent/Microsoft.Enter
After attacker successfully gained access to a virtual machine, he can decode protectedSettings value to get the workspace key using the technique described in the below article.
Get Azure Automation Key
Azure Automation Key can be retrieved with only Read access. You don’t need any special operation like listKey for Azure Storage account.
With to keys combined together the attacker can create another virtual machine in a non-production environment (which normally left unsecure and unattended) and then join to a hybrid worker group in production automation account.
$endpoint="https://eus2-agentservice-prod-1.azure-automation.net/accounts/xxxxxx" $token="+ApZ7mn64oS3j51vJMQF/S" Add-HybridRunbookWorker -GroupName hw-group1 ` -EndPoint $endpoint="https://eus2..` -Token $token
The attacker can name a virtual machine as if it is created for hybrid worker role e.g. hw-prod-vm0002.
Now the attacker would just need to wait for cloud administrator to deploy the certificate to a newly added VM in the hybrid worker. When a certificate is acquired, the attacker could abuse it to laterally move around resources in that subscription. Worst case is the service principal is widely used among all subscriptions.
Below are steps in order
- A virtual machine A is compromised.
- The attacker acquires workspace key by decrypting cipher in the agent setting file.
- The attacker creates a fully controlled VM (VM B) in a non-production subscription
- The attacker joins his VM (VM B) to the same Log Analytics workspace with the compromised VM (VM A)
- The attacker adds his VM (VM B) to the linked automation account’s hybrid worker.
- The attacker waits for an Azure Automation certificate to be installed onto his virtual machine (VM B).
- The attacker acquires the automation account service principal certificate and use it to sign into Azure.
- The attacker starts scanning environment with the acquired certificate.
The following image illustrates the attack vector
Why does this still matter?
Not many cloud administrators are aware of what has happened in the environment they are managing. SecOps may be blind too because they wouldn’t care if a virtual machine is added to a hybrid worker. When a new VM is added to a hybrid worker it can be picked by Azure to execute runbook. Since that VM doesn’t have certificate, the runbook will fail at the very first step when it tries to authorizes Azure AD. In that case, the cloud administrator may investigate and install certificate for the attacker-controlled VM.
There are still conditions needed, this technique has proven its ability to trick cloud administrator without many steps. Moreover, it is difficult to detect if a service principal is compromised especially when you have a DevOps environment with many automation implementations. In a large environment as a SecOps engineer you don’t normally manage or monitor VM deployment in a private network. And you wouldn’t really care about Hybrid worker deployment because it is part of cloud operation tasks.
A good option right now would be system-assigned managed identity to avoid taking care of credential. However if a bad actor could control a hybrid worker by himself then he could still get access token (by calling Azure Service Metadata Instance) from the managed identity and does his evil job.
If you have any question and feedback please feel free to leave it under Comment box. I’d be happy to discuss with you to make cloud stay healthy and secure.