Author: socinabox

If you’re a user of Microsoft Sentinel, you’re likely familiar with it’s Threat Hunting feature which lets you run hundreds of KQL queries in a matter of seconds.

Unfortunately the Threat Hunting in the M365 Defender portal doesn’t have this feature, so you’re stuck running your hunting queries one at a time.

So I’ve created a proof of concept script that provides some threat hunting automation by taking the 400+ threat hunting queries in the Microsoft Sentinel Github repository and feeding them into the M365 Defender ThreatHunting api.

Requirements

• a unix shell from which you can run python and git commands
• python3, git command, and the python modules you see at the top of the scripts attached below
• admin access to Azure to set up an app registration
• operational use of the M365 Defender portal (https://security.microsoft.com)

Caveats

There are some known performance limitations to using Defender advanced threat hunting, so although the script may seem to be working, there could be timeouts happening in the background if Defender decides you’re using too many resources. This script doesn’t have any built in error checking so re-running the script or validating the queries within the Defender portal may be required.

The Setup Procedure:

1. Create an app registration in Azure. Configuring an app registration is out of the scope of this article, but there are plenty of examples on how to do this. What’s important are the permissions you’ll need to allow:
   • ThreatHunting.Read.All
   • APIConnectors.ReadWrite.All
2. From the app registration created in step #1, copy the tenantID, clientID and the secret key, and paste them into the script I’ve provided below.
3. Create a directory named ‘queries’. This will be used to store the .yaml files from Github. These files contain the hunting queries.
4. In the same directory, download the Github repository using this command:
   • git clone https://github.com/Azure/Azure-Sentinel
5. Now it’s time to create and run the first of 2 scripts. This first script should require no changes. Its purpose is to scan the Azure-Sentinel subdirectories (that you downloaded in step #4) for .yaml files and copy them all to the ‘queries’ directory.
   • Here’s the script, name it something like get_yaml.py and then run it like: “python3 get_yaml.py“

# python script name: "get_yaml.py"
import os
import shutil

# Set the directory paths
source_dir = 'Azure-Sentinel/Hunting Queries/Microsoft 365 Defender/'
target_dir = 'queries'

# Recursively search the source directory for YAML files
for root, dirs, files in os.walk(source_dir):
    for file in files:
        if file.endswith('.yaml'):
            # Create the target directory if it doesn't already exist
            os.makedirs(target_dir, exist_ok=True)

            # Copy the file to the target directory
            source_file = os.path.join(root, file)
            target_file = os.path.join(target_dir, file)
            shutil.copy2(source_file, target_file)

            # Print a message to indicate that the file has been copied
            print(f'Copied {source_file} to {target_file}')

6. Edit this next script (below) and insert the tenantID, clientID and secret from the Azure app registration, as mentioned above in step #1. Name the script something like threathunt.py and run it with the python (or python3) command: “python3 threathunt.py”.

• Note that the script below expects the .yaml files to be in a folder named “queries” (from when you ran the first script above).
• After running the script below, look in the queries folder for any new files with a .json extension.
• If the query ran successfully and generated results, a .json file will be created with the same filename as the matching .yaml file (eg. test.yaml > test.json)

import requests
import os
import yaml
import json
from azure.identity import ClientSecretCredential

# Replace the values below with your Azure AD tenant ID, client ID, and client secret
client_id = "YOUR APP REGISTRATION CLIENT ID"
client_secret = "YOUR APP REGISTRATION SECRET KEY"
tenant_id = "YOUR TENANT ID"
scope = "https://graph.microsoft.com/.default"

authority = f"https://login.microsoftonline.com/"

# Create a credential object using the client ID and client secret
credential = ClientSecretCredential(
    authority=authority,
    tenant_id=tenant_id,
    client_id=client_id,
    client_secret=client_secret
)

# Use the credential object to obtain an access token
access_token = credential.get_token("https://graph.microsoft.com/.default").token

# Print the access token
print(access_token)

# Define the headers for the API call
headers = {
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/json",
    "Accept": "application/json"
}

# Define the API endpoint
url = "https://graph.microsoft.com/v1.0/security/microsoft.graph.security/runHuntingQuery"

# Define the path to the queries directory
directory_path = "queries"

# Loop through all YAML files in the queries directory
for file_name in os.listdir(directory_path):
    if file_name.endswith(".yaml") or file_name.endswith(".yml"):
        # Read the YAML file
        with open(os.path.join(directory_path, file_name)) as file:
            query_data = yaml.load(file, Loader=yaml.FullLoader)
            # Extract the query field from the YAML data
            query = query_data.get("query")
            if query:
                print(query)
                # Define the request body
                body = {
                    "query": query
                }
                # Send the API request
                response = requests.post(url, headers=headers, json=body)
                # Parse the response as JSON
                json_response = response.json()

                if json_response.get('results') and len(json_response['results']) > 0:
                  #print(json.dumps(json_response, indent=4))
                  output_file_name = os.path.splitext(file_name)[0] + ".json"
                    # Write the JSON response to the output file
                  with open(os.path.join(directory_path, output_file_name), "w") as output_file:
                      output_file.write(json.dumps(json_response, indent=4))

Once you have the results from your threat hunt you could:

• Log into the M365 Defender portal and re-run the hunting queries that generated data. All of these YAML queries from github should be in the Threat Hunting > Queries bookmarks. If not you can manually copy/paste the query from inside the YAML file directly into Query Editor and play around with it.
• Import the data into Excel or Power BI and play around with the results.
• Create another python script that does something with the resulting .json files like aggregate the fields and look for commonalities/anomalies.

Happy Hunting!