A SIEM with 140 sources and 1,000 analytic rules does not have 20,000 integrations, which is what it would take for each rule to read the 20 or so raw sources in its schema directly. Each source has one parser into a common schema such as ASIM or the Splunk CIM, and each rule is written once against the schema. In Microsoft Sentinel a rule that queries _Im_Authentication covers Entra ID, Okta, AWS CloudTrail and every other authentication source with a parser, including sources added after the rule was written. The integration count is one parser per source, and the rules add none. (A standard Sentinel workspace allows 512 enabled scheduled rules; 1,000 needs a dedicated cluster, which allows 1,024, or a second workspace.)

What works: SIEM correlation rules and agents both reading normalized data

Figure 1. With normalization: 140 sources, 7 schemas, 1,000 rules and 1,000 agents

Left to right: 140 raw sources grouped by the ASIM schema they map to, the seven schemas in the middle, then the two consumer fleets, 1,000 analytic rules (navy) and 1,000 agents (blue). Each source has one line into its schema, and every rule and agent reads only the schemas. The agents are grouped into four specialist types, one per schema family. Incidents from the rules start an orchestrator (one per incident), which hands work to the specialists, and everything they propose goes through the enforcer bar on the far right before it can change a system. That is the SOAR playbook split in three. Moving dots are events. Adding agents adds dots on the right and no lines on the left.

Agents are consumers of the same data, and the same arithmetic applies. Without a normalized layer, every agent that investigates sign-ins has to read every raw authentication format, either through source-specific tools or by working out each format from the raw record on every call. With 140 sources spread across seven schemas, that is about 20 source-specific integrations per agent, and about 40,000 across 1,000 rules and 1,000 agents. Each new agent adds 20 more. Each field rename at a source breaks every rule and agent that reads that source directly, and agents that interpret raw formats themselves spend tokens doing it and can reach different conclusions from the same record.

What doesn't work: SIEM rules and agents reading every source without normalization

Figure 2. Without normalization: every rule and agent wired to every source

The same 140 sources and 2,000 consumers with the schema layer removed. Every consumer now needs a line to each source in its schema; the drawing shows a sample of the 40,000. Red dots are the rules and agents that read a FortiGate field directly; the field was just renamed, so they are now broken.

With the normalized layer, the ASIM parsers are KQL functions, so the function an analytic rule calls is the same function an agent’s query tool calls. Adding the 1,001st agent requires no new parser and no new integration. Adding a source requires one parser, and every existing rule and agent covers it. A field rename at a source is fixed once, in its parser. The number of agents is then limited by model capacity and query limits, not by integration work. The Log Analytics query API, for example, runs 5 concurrent queries per identity and queues up to 200 before returning HTTP 429, so query load is planned across agent identities the way rule load is planned across workspaces.

A common objection is that MCP already handles this. MCP standardizes how an agent finds and calls a tool: the transport, the tool description, and a JSON schema for each tool’s inputs and, optionally, its outputs. Each server author chooses that schema. The protocol has no shared model for a sign-in, a firewall session or a lab result, so an MCP tool that queries a raw table returns that table’s field names, and thirty authentication sources behind thirty MCP tools are still thirty formats. In the architecture later in this post, MCP is the tool layer that carries the data to agents; ASIM or FHIR defines its shape.

The work that makes this possible is existing SIEM work: onboarding sources, writing and testing parsers, maintaining schema coverage, and monitoring connector health. That work does not change when the consumers are agents. It becomes the precondition for running them at scale.

Each part of a SIEM pipeline has a counterpart in a multi-agent pipeline. The table lists them side by side.

The normalization, transport, ordering and cost rows are where SIEM experience transfers almost directly. The detection and response rows are where the work changes.

A SIEM rule depends on field names. If a custom application changes user_id to uid, the parser still runs, the user field comes back empty, and the analytic rule stops matching. Nothing errors. The detection goes quiet and the gap is found later in a coverage review, or after an incident.

In an agent pipeline the same boundary is handled with a schema contract. Every event that enters the pipeline is validated against a typed model, and every verdict an agent returns is validated against another. A Python sketch:

from typing import Literal
from pydantic import BaseModel, Field

class SignInEvent(BaseModel):
    user_principal_name: str
    ip_address: str
    result_code: int
    app_display_name: str | None = None

class TriageVerdict(BaseModel):
    verdict: Literal["benign", "suspicious", "malicious"]
    confidence: float = Field(ge=0, le=1)
    evidence: list[str]
    proposed_actions: list[str]

When an event fails validation, the pipeline has a choice a SIEM parser does not have. A mapping step can pass the raw record and the contract to a model and ask which source field corresponds to user_principal_name. The model’s answer is a proposal, not a fix. It is validated against the contract, written to a log, and used to open a ticket to correct the parser. The model does not remove the need for normalization. It gives the pipeline a fallback path when a source drifts, and it makes the drift visible instead of silent.

The skills that apply here are the ones SIEM engineers already use: knowing the target schema (ASIM or CIM), writing field-level data quality checks, and testing parsers against real samples before a source goes live.

An analytic rule is cheap, fast and deterministic. It runs in milliseconds, costs nothing per evaluation beyond the query, and gives the same answer every time. A model call takes seconds, costs tokens and can return different answers for the same input. Replacing every rule with an agent would raise cost and lower predictability for detections that already work.

A practical design keeps the analytic rules. Rules filter and score the high-volume stream. Agents take over at three points the rules handle poorly:

• Triage of incidents the rules have already raised, where an analyst would normally pull context from several tables before deciding.
• Free-text input, such as a phishing report written by a user, that no field-based rule can parse.
• Sources that fail normalization, where the mapping step described above applies.

Each specialist has one job, its own system prompt, and only the tools that job needs. An identity specialist can query SigninLogs and AuditLogs. An endpoint specialist can query DeviceProcessEvents. An email specialist can query EmailEvents and UrlClickEvents. The scoping is the same least-privilege work done today for a Logic App’s managed identity or a Splunk service account.

A SOAR playbook encodes every branch in advance. The agent version splits that work across three roles:

• The orchestrator reads the incident, decides which specialists are needed, and starts them. It does not take actions itself.
• The specialists investigate in parallel and each return a typed verdict with evidence and proposed actions.
• The enforcer checks every proposed action before anything runs.

The enforcer should be code, not a model. It applies fixed rules: an allowlist of permitted actions per incident type, a list of protected accounts (break-glass accounts, the Entra Connect sync account, executives under a separate process), a blast-radius limit such as no more than five accounts disabled per incident, and a human approval requirement for destructive actions such as soft-deleting mail or isolating a server. These are the approval gates and change-control rules that already sit inside mature SOAR playbooks, moved out of each playbook into one place that every agent passes through.

The enforcer matters more in security than in most agent deployments because log content is attacker-controlled. A user agent string, an email body, a file name or a command line can contain text written to instruct a model. A specialist that reads a phishing email is reading the attacker’s words. Treat every field that came from a log as untrusted input, keep it out of system prompts, and rely on the enforcer, not the specialist’s judgment, to stop an action the attacker’s text talked the model into proposing.

Each agent should also have its own identity, for example a Microsoft Entra Agent ID, so that its queries and actions appear in sign-in and audit logs under a name that can be monitored, scoped by Conditional Access, and disabled. The agent pipeline becomes a log source for the SIEM it supports.

Detection-as-code practice carries over with one change. A rule test checks that a sample log produces a match. An agent test checks that a replayed incident produces the expected verdict and the expected proposed actions, within a confidence range, across repeated runs, because the output is not deterministic. Build the evaluation set from closed incidents with known outcomes, and rerun it whenever a prompt, a model version or a tool changes.

Operational metrics also carry over. Ingestion volume per day becomes tokens per incident. Rule latency becomes time to verdict. False positive rate stays as it is. Add two that SIEM pipelines do not need: the rate at which the enforcer rejects proposed actions, and the rate at which schema validation fails on agent output. A rising rejection rate means a prompt or a model change has made a specialist less reliable, and it should be visible on a dashboard before it causes a missed action or a wrong one.

Skills that transfer with little change:

• Normalization to a common schema and data quality checks at ingestion.
• Queue design, partitioning and ordering.
• Enrichment as an asynchronous lookup.
• Least-privilege service identities for automation.
• Approval gates and change control in response automation.
• Detection testing against known samples.
• Cost management by tier and volume.

Skills to add:

• Writing system prompts as specifications, with the output schema defined first.
• Building and maintaining evaluation sets.
• Choosing models by task, with smaller models for mapping and classification and larger ones for investigation.
• Defending against prompt injection from log content.
• Designing an enforcer as code, separate from the agents it controls.

Hospital networks have the same structure. Patient data arrives from EHR registration, lab systems sending HL7 v2 results, admit, discharge and transfer feeds, pharmacy and e-prescribing systems, radiology and pathology, and claims. HL7 FHIR plays the role ASIM plays in Sentinel: each system is mapped once into FHIR resources such as Patient, Observation, Encounter, MedicationRequest, Condition, Procedure and DiagnosticReport, with standard codes (LOINC, RxNorm, ICD-10). FHIR R4 is the version required for patient access APIs in certified US EHR technology under the ONC Cures Act Final Rule.

The consumers are clinical decision support rules, such as sepsis screening or drug-interaction checks, and agents for prior authorization, care-gap outreach and coding review. The arithmetic is the same as for detections. With FHIR in place, a new agent calls the FHIR API and needs no interface work for any of the source systems. Without it, each agent decodes each feed itself. A source change shows the cost: if a lab system starts sending glucose in mmol/L instead of mg/dL, the FHIR mapping is corrected once, while every rule and agent reading the raw value directly sees results about 18 times too low. In security that is a missed detection; in healthcare it is a patient safety issue.

In Microsoft terms the design has five layers, and the order matters. Sources are onboarded and normalized first; the rule fleet reads the normalized store directly; agents reach it only through a tool layer; an agent runtime hosts the agent types and their concurrent runs; and an enforcer stands between proposed actions and anything that changes a system. Identity and observability apply to every layer.

• Normalized data: Microsoft Sentinel’s analytics tier and data lake with ASIM parsers. This is where the 1,000 rules run.
• Tool layer: the Sentinel MCP server, with custom tools built from saved KQL queries that call the ASIM functions. Agents call tools and never read raw tables.
• Agent runtime: Microsoft Foundry Agent Service for the fleet, which Microsoft documents as having no limit on the number of agents in a project, with rate limits applied at the model deployment. Security Copilot hosts a few agent types embedded in Defender and Entra and is billed in Security Compute Units.
• Actions and oversight: Logic Apps playbooks with approval steps, a protected-account watchlist and blast-radius limits, then Defender and Entra response actions.
• Identity and observability: an Entra Agent ID per agent group, OpenTelemetry traces to Application Insights, and agent activity written back to Sentinel as a log source.

Figure 3. Security architecture on Microsoft: Sentinel, ASIM, MCP, Foundry

Left to right. Six source types feed the normalized store (the lime cylinder: Sentinel analytics tier and data lake with the seven ASIM schemas). The rule fleet reads it directly, and the dashed navy line shows incidents from rules starting agent work. Agents reach the data only through the Sentinel MCP server in the middle. Inside the agent runtime, an orchestrator routes each incident to specialist groups (identity, endpoint, network, cloud and audit), each running many concurrent copies. Proposed actions pass through the enforcer before response actions run; the bracket under the runtime marks orchestrator, specialists and enforcer as the three parts a SOAR playbook used to hold. The two bands at the bottom apply to every layer; the grey dashed line is agent activity logged back to Sentinel.

The healthcare version uses the same layers. The Azure Health Data Services FHIR service is the normalized store, with $convert-data turning HL7 v2 and C-CDA into FHIR resources and the MedTech service mapping device data to Observation. A FHIR MCP server using SMART Backend Services authentication is the tool layer, alongside reference servers for ICD-10 codes, the NPI Registry and CMS coverage policies. Foundry hosts the agents; Microsoft publishes a prior authorization multi-agent accelerator with four hosted agents as a Foundry template. A clinician approves any order, payer submission or record write-back.

Figure 4. Healthcare architecture on Microsoft: FHIR service, MCP, Foundry

Same layout as the security diagram. Clinical systems on the left feed the FHIR service, which holds seven FHIR resource types. Decision support rules read FHIR directly; agents reach it through the FHIR MCP server. The orchestrator routes work to care-gap, clinical review, medication, and prior auth and coding specialists. Proposed actions stop at clinician review, which plays the enforcer role, and approved changes are written back through the FHIR API. FHIR service audit logs record every agent read and write.

The accompanying page draws 140 raw sources grouped by ASIM schema, the seven schemas in the middle, and two consumer fleets on the right: 1,000 correlation rules and 1,000 agents. With normalization selected, each source has one line to its schema and every consumer reads only the schema. With normalization removed, every consumer needs a line to every source in its schema, and the drawing becomes a dense mesh. Sliders change the number of sources and agents, and a counts table gives both designs side by side: integrations to maintain, work to add one agent, work to add one source, and consumers broken when a FortiGate source renames a field. Adding 1,000 agents to the normalized design adds no integrations; adding them without normalization adds about 20,000. A second button switches the same diagram to the healthcare example, with FHIR resources as the middle layer and decision support rules and clinical agents as the consumers. An Architecture view draws each example as Microsoft services: Sentinel with ASIM, the Sentinel MCP server and Foundry Agent Service for security; the Azure Health Data Services FHIR service, a FHIR MCP server and Foundry Agent Service for healthcare.

The visual is at https://broglock.github.io/siem-to-agent-normalization/ and the source, including this post, is at https://github.com/broglock/siem-to-agent-normalization.

A second page maps 18 security agent use cases to the data each one reads. The columns are the seven normalized schemas from the visual (Authentication, Process Event, Network Session, DNS, Web Session, Audit Event and File Event), which agents reach through custom Sentinel MCP tools, and six Security Copilot plugin sources: Microsoft Entra, Defender XDR, Intune, Purview, Defender Threat Intelligence and Defender External Attack Surface Management. A solid dot marks the main source for a use case and a ring marks a context source. The bottom row counts the agents that read each source.

Figure 5. Security agent use cases mapped to normalized schemas and Security Copilot plugins

Rows are agent use cases grouped by specialist family: identity, endpoint, network, cloud and audit, and cross-cutting. The blue columns are the seven normalized schemas, reached through custom Sentinel MCP tools; the navy columns are Security Copilot plugin sources. A solid dot is the main source for that agent and a ring is a context source. A lime tag marks a use case Microsoft already ships as a Security Copilot agent.

Five of the use cases overlap agents Microsoft already ships in Security Copilot: Conditional Access Optimization, Vulnerability Remediation, Phishing Triage, Purview DLP alert triage and Threat Intelligence Briefing. Building your own version is still useful when it needs to read non-Microsoft sources through the normalized schemas. Selecting a row on the page shows its trigger rule, the tools it needs and the actions it proposes. The matrix is at https://broglock.github.io/siem-to-agent-normalization/use-case-matrix.html.

• Service limits for Microsoft SentinelMicrosoft Learn
• Azure Monitor service limitsMicrosoft Learn
• Microsoft Sentinel MCP tools overviewMicrosoft Learn
• Foundry Agent Service quotas and limitsMicrosoft Learn
• Automate prior authorization with AI agentsMicrosoft Foundry Blog
• Microsoft Security Copilot plugins overviewMicrosoft Learn
• Security Copilot agents overviewMicrosoft Learn
• HL7 FHIR R4HL7
• ONC Cures Act Final RuleHealthIT.gov
• Interactive visualGitHub Pages
• Project repository: post and visualGitHub

My approach is to do the normalization work once, ahead of time, and store the result in the lake. Scheduled lake jobs read every source table and write two tables:

• HuntEntity holds one row per entity, per hour, per source and role (an IP, a user, a host, a URL, a hash), with its type, its role (source, destination, actor, target, device), a lowercase join value, a count, first and last seen times, and a few sample record keys. Rolling up by the hour keeps the table small enough to search a full year cheaply.
• HuntEvent holds sign-ins, network sessions and other events with ASIM field names such as SrcIpAddr, TargetUsername and EventResult, for sources mapped well enough to trust. It is rolled up by the hour too: repeated events with the same user, addresses, result and action become one row with a count.

Every row in both tables names the source table and hour and keeps a few sample record keys that point back to the original records. The agent searches the index, then pivots to the raw rows when it needs the full detail.

HOW TO READ IT

On the left, the agent must query each table with that table’s own field name, and two of the five hide the IP in free text. On the right, scheduled jobs (the line on the far right) feed every source into the index ahead of time, and the agent asks a single question of it.

No single method covers hundreds of sources, so the index uses four, in order of trust. Each source uses the best one available to it, and each index row records which one produced it.

HOW TO READ IT

Top to bottom is most trusted to least. The middle column shows the kinds of sources each method suits; the right column is the tier the index records, which the agent uses to weigh a result. The dashed band catches whatever the other three miss.

The result is that every source is searchable on day one, and the mapping gets better as sources are reviewed.

Everything runs inside the Sentinel data lake. There are no new data connectors, DCRs or ingestion changes: the index is built from data already there.

The index builders are of two kinds. KQL jobs run every approved mapping: ASIM logic, registry rules and approved message templates. A lake notebook handles the sources nobody has reviewed yet. The notebook is Python for Spark (PySpark), written as a Jupyter notebook in Visual Studio Code with the Microsoft Sentinel extension, that reads lake tables and writes its results back to the lake. KQL jobs and notebooks can both run on a schedule, so once a source is approved nobody runs anything by hand.

Three scheduled notebooks do the work that would otherwise be manual:

• Notebook 01 runs daily. It reads only new tables, changed schemas and the lines no template matched, profiles each field (how often it is filled and what its values look like) and proposes ASIM names, roles and message templates for an analyst to review. Sources with approved, high-confidence mappings are skipped.
• Notebook 02 runs hourly on sources nobody has reviewed yet. It indexes them through the value classifier into the same two tables the KQL jobs write, so they are searchable from day one. Once a source is approved, its mapping moves into a KQL job and the notebook stops reading it.
• Notebook 03 runs daily. It builds the coverage report from counts the index jobs record as they run (rows read, rows with an entity, rows matching a template) and flags any source whose extraction rate drops, which usually means the vendor changed its log format. It reads no source data.

A mapping is decided once. After approval, the only recurring work for a source is applying its mapping to each new hour of events in a KQL job; nothing already indexed is read again. Analysts approve mappings and check results. Finding new fields, building the index and catching format changes all run on a schedule.

HOW TO READ IT

The top row runs left to right. Source tables of every shape feed the index builders. Approved groups run in one KQL job per source table; only groups still under review run in the lake notebook. Both write the two index tables that the agent and analysts query: HuntEntity holds hourly entity counts plus each job’s row counts, and HuntEvent holds hourly counts of Tier 1 events with ASIM field names. Every index row carries sample record keys, so a hit is one pivot from the raw events. The bottom row is the control loop: notebook 01 reads only new or changed tables and drifting groups and proposes mappings, analysts approve them in the registry, the registry generates the builders, and notebook 03 measures coverage from the job counts. The note on the right is the drift rule that sends an approved group back to notebook 01.

The example below uses sample data from five sources with very different shapes: Windows security events, FortiGate firewall logs over CEF, Linux sshd syslog, a JSON audit table from an in-house app, and a VPN appliance that writes raw key=value text. The agent asks one question: where has 203.0.113.45 appeared?

One query over the index found the address in all five, each time labelled as the source, with a failed or denied result. Asked for the same user’s sign-ins, the index returns four failures from that address within about a minute, then successes from an internal address 16 minutes later. Without the index, the agent would need five different queries and would have to know each table’s field names.

Once the index is running, a new log source needs a mapping review, not a new pipeline. The new table is searchable through the classifier from its first indexing run, before anyone has looked at it.

HOW TO READ IT

The top row runs left to right, then the flow turns down at the registry and runs right to left. Review starts from the parser if Microsoft ships one, otherwise from the fields and templates notebook 01 proposes. After deployment, four checks run: coverage numbers in the coverage table, a hand check of sample rows, a known test event found through the agent, and a pivot back to the raw row. Any failure sends the source back to review.

The daily workflow can be seen in two places. Both use synthetic data only: documentation IP ranges, private 10.x addresses and contoso.com names.

• Walkthrough demo: an 18-step guided tour that runs in the browser with no server. Notebook 01 checks new and changed sources and proposes fields for a new badge reader table, an analyst reviews them, an admin approves the change, the generated lake job is deployed and passes four validation checks, a vendor format change is fixed, and a hunt for 203.0.113.45 runs across ten source groups.
• Demo video: about three minutes, recorded from the Lake Hunt app in demo mode. It covers the architecture, onboarding the badge reader, approval, validation and the same hunt.

At 1 TB a day of lake ingestion, the index adds about $440 to $490 a month on top of the lake’s own cost for the raw data. That lake cost is about $6,080 a month at this volume: $4,500 to ingest it ($0.05 per GB ingestion plus $0.10 per GB data processing) and $1,580 to store it for a year. The index figure covers the hourly KQL jobs, the three notebooks and one year of storage for both index tables. Most of it is the KQL jobs, which read each new hour of data once. The notebooks cost under $150 a month because they only touch new, changed and unreviewed sources. The figures use Microsoft’s published Sentinel data lake prices in US dollars for East US. The full calculation and its assumptions are in the project’s GitHub repository, along with a walkthrough demo of the daily workflow.

• The index covers common hunting fields: IPs, users, hosts, URLs, hashes, results and event types. Fields that only matter to one product stay in the raw table, one pivot away.
• Index freshness follows the job schedule, hourly in this design. Real-time detection stays in the analytics tier.
• Lake KQL jobs have tenant limits on count and concurrency, so sources are batched into shared jobs rather than one job per source.
• The design has been deployed and run in a test tenant with the Sentinel data lake onboarded. Behaviour at production volume still needs confirming, including notebook run times and how the job limits apply at 1 TB a day.

The point of the design is that the cost of normalization is paid once per source, not once per hunt, and that an AI agent gets one consistent set of field names across every log source in the lake.

• Normalization and the Advanced Security Information Model (ASIM)Microsoft Learn
• Jupyter notebooks and the Microsoft Sentinel data lakeMicrosoft Learn
• Create and manage Jupyter notebook jobsMicrosoft Learn
• Plan costs and understand Microsoft Sentinel pricing and billingMicrosoft Learn
• Project repository: design, solution kit, cost model and walkthrough demoGitHub
• Lake Hunt walkthrough demoGitHub Pages
• Lake Hunt demo videoGitHub