Hunting in Azure using Kusto Query Language to write query against Log Analytics workspace may not be enough for you. Given an example like this article, you would want to extract all attacker IP addressees and use VirusTotal to verify if those are bad known sources.
This article is not going to introduce Azure Sentinel notebooks. Instead it provides a simple guidance and key takeaways for beginners who would like to use and explore Azure Notebooks for hunting.
Azure Sentinel uses Log Analytics workspace to store security data and event. By default design Azure Sentinel connects to a single workspace only. It means you would need to stream data from different sources and services to that workspace. The main query language to be used is Kusto Query Language (KQL). KQL is a good tool. However not all the cases it can be used effectively. Another example when KQL is not good enough is when working with Linux audit log. PROCTITLE value inn Audit log is an encoded hexadecimal so you would have to convert to a human readable language. Unfortunately as of this article KQL doesn’t provide any function to help. The available function is tohex() that doesn’t help in this case.
In many cases you will need another tool to help proceed and formalize log. Moreover, for a sequential hunting steps, there should be a tool to help analyst to interactively execute a query whether it is a KQL query or a Python script) in a specified period of time (e.g. during suspicious event time). Azure Notebooks can be your companion.
To fully understand Azure Notebooks, below are some helpful materials:
Where to run your notebook?
You have two options to run your notebook. Azure Notebooks provides you a Free Compute to run. I’d NOT recommend you to run your notebook using Free Computer tier because it is very slow. You might run in a simple testing case. The only advantage I’d personally see from this option is the ability to connect to Azure to grant access to Azure Notebooks. With this you wouldn’t have to handle any authorization.
Another option is to use a hosted virtual machine to host and run your notebook. This virtual machine must have Jupiter Notebook server to be configured. Moreover it must allows inbound traffic at port 8000 so Azure Notesbook can connect and use your virtual machine.
Once you click Validate you may encounter connection issue when connecting to hosted Jupiter VM. To solve it, simply browser the URL (http://xxx.x.x.x:8000) in browser to spawn the machine then click Validate again.
The very big concern in using a virtual machine to run your notebook is security. In a large environment where security and compliance is the very first place this approach may not be allowed. Currently Azure Notebooks support username and password only. That said, certificate-based is not supported at this moment. For network path you can build a tunneling to forward to port 8000, or place a load balancer in front of the Jupiter notebook so you could restrict access from that load balancer to your Jupiter VM.
For virtual machine SKU, I use Microsoft Data Science Ubuntu described here.
My recommendation is to use the Jupiter link for working with your notebook. Azure Sentinel Notebook looks like a presentation of the Jupiter one and you’d need to deal with high latency when sending command to it.
Authorization from Azure Notebook
Before working with your Azure environment you would need to gain access to Azure subscription. In Azure Notebooks, you can run things like az login to log into Azure.
You can also run any other Azure CLI in Notebooks because Azure CLI is actually written on top of Python. For example acquiring access token using:
This is simply to show you something. This doesn’t mean to recommend you to use Azure CLI for your work.
Start with msticpy and kqlmagic
You must use Python 3.6+ in order to use kqlmagic (>=0.1.9x)
!pip install Kqlmagic --no-cache-dir --upgrade %reload_ext Kqlmagic
There are a few ways to connect to your Log Analytics workspace before you can do the query. Below is an example to connect using service principal with password:
TENANT_ID='xxxxx-xxxxxx-xxxxxxxxx-xxxxxx' APP_ID='xxxxx-xxxxxx-xxxxxxxxx-xxxxx' CLIENT_SECRET='xxxxx-xxxxxx-xxxxxxxxx-xxxxx' WORKSPACE_ID='xxxxx-xxxxxx-xxxxxxxxx-xxxxx' %kql loganalytics://tenant=TENANT_ID;clientid=CLIENT_ID;clientsecret=APP_ID;workspace=WORKSPACE_ID;alias='azsecdb'
Make sure the APP_ID is the application ID of your service principal. It is NOT ObjectId.
While msticpy provides number of helpful utilities and functions for SecOps hunter kqlmagic provides a simplest way to interact with Log Analytics query using KQL. With kqlmagic you can dynamically pass value to a KQL query too.
Resources to get started
There wouldn’t be any better place to find helpful resources than the one below:
- Azure Notebooks environment setup
- Basic setup and test kqlmagic
- Use Jupyter notebooks to hunt for security threats
- Threat Hunting in the cloud with Azure Notebooks: supercharge your hunting skills using Jupyter and KQL
- Azure Sentinel Notebooks GitHub
- Visual Studio Code — the swiss army knife for threat hunting with Azure Sentinel