Azure Data Explorer (Kusto)¶

Scope¶

Azure Data Explorer as the telemetry, log, and time-series analytics engine, and its relationship to Fabric Eventhouse and Azure Monitor Log Analytics. Covers the engine architecture (storage and compute separation, immutable extents, the multilevel cache, the row store), cluster SKUs and the compute-optimized versus storage-optimized decision, optimized autoscale, the caching policy versus retention policy distinction that most teams get wrong, ingestion paths (streaming versus queued, the batching policy defaults), update policies and materialized views and when to use which, KQL and the narrow T-SQL surface, the free cluster, and cross-service query with Azure Monitor in both directions. For Fabric Eventhouse and KQL databases inside Fabric, this file supplies the engine mechanics and providers/azure/fabric.md supplies the capacity and governance model. For the wider Azure analytics stack see providers/azure/analytics.md.

Azure Data Explorer has no announced retirement date and no Microsoft lifecycle retirement entry; its overview documentation was revised in July 2026 and is actively maintained. Fabric Eventhouse is the strategic destination for new work, but ADX is not deprecated. Verified against Microsoft Learn on 2026-07-26.

Checklist¶

Engine Architecture and Data Layout¶

[Critical] Is the storage and compute separation understood as the basis for every cost and performance decision? "Azure Data Explorer separates storage and compute resources. Persistent data resides in Azure Blob Storage, while compute resources might store temporary data or act as a cache." Cluster sizing is therefore a cache and CPU decision, not a capacity decision — the data does not have to fit on the nodes.
[Critical] Is the extent model understood, given that it explains both the immutability guarantees and the maintenance behaviour? ADX "partitions all ingested data into extents, or data shards… an extent can start as small as a single record," and each extent is encoded and indexed independently. Extents are immutable — "never modified. Data modification happens by creating new extents and transactionally swapping old extents with the new ones." The recommended ceiling is 10,000,000 extents per cluster; a cluster approaching it usually has an ingestion-batching problem rather than a data-volume problem.
[Recommended] Is the two-cluster shape understood when reasoning about ingestion failures? A deployment comprises an engine cluster and a data management cluster, with production requiring at least two nodes each; the DM cluster is "fully managed by the Azure Data Explorer service." Queued ingestion failures usually live in the DM cluster, not the engine.
[Recommended] Is it known that the multilevel cache operates on compressed data, decompressed only at query time? This is why the hot cache holds far more data than a naive GB calculation predicts, and why cache sizing is best done empirically against the cache utilization metric rather than arithmetically.
[Optional] Are the object limits designed around at scale — 10,000 databases per cluster and 10,000 tables per database?

Caching Policy vs Retention Policy¶

[Critical] Is the difference between caching and retention understood, since conflating them is the most common ADX design error? Microsoft states it directly: "Caching policy is independent of retention policy: Caching policy defines how to prioritize resources… Retention policy defines the extent of the queryable data in a table/database (specifically, SoftDeletePeriod)." Retention decides how long data is queryable at all; caching decides how much of that window sits on SSD. Cold data is neither deleted nor unqueryable — just slower.
[Critical] Is the worked example internalised, because it is the one that corrects the misconception? With SoftDeletePeriod = 56d and a hot cache policy of 28d, "the last 28 days of data is stored on the SSD and the additional 28 days of data is stored in Azure blob storage. You can run queries on the full 56 days of data." Shortening the cache policy reduces cost and slows old queries; it does not shorten the queryable window.
[Critical] Is the retention default known before assuming data ages out? SoftDeletePeriod "defaults to 1,000 years" (the default object carries 365000.00:00:00), and new databases and tables have no retention policy defined at all. Data does not expire unless someone configures it to. Recoverability defaults to Enabled, meaning soft-deleted data remains recoverable for 14 days and this is "not possible to configure" — which matters for any erasure obligation.
[Recommended] Is the cache policy expressed with the correct keywords? The settable inputs are hot and hot_window — hot = 30d implicitly sets both DataHotSpan and IndexHotSpan, and HotWindows is the returned array populated by repeated hot_window = datetime(a) .. datetime(b) entries. DataHotSpan, IndexHotSpan, and HotWindows are output fields, not command keywords.
[Recommended] Is the 95/5 cache split understood when diagnosing a query that appears to have evicted the hot cache? "The cache uses 95% of the local SSD disk for hot data… the remaining 5% is used for data that isn't categorized as hot. This design ensures that queries loading lots of cold data won't evict hot data." Cache membership is evaluated against extent ingestion time, not event time — a late-arriving backfill lands in the hot cache regardless of the timestamps inside it.
[Recommended] Is cache convergence latency accounted for in change windows? "It can take up to an hour to fully update the cluster disk cache" after a policy change.
[Optional] Is query_datascope used to keep exploratory queries off cold storage where cost predictability matters — set query_datascope='hotcache' at session level or T datascope=hotcache per table?

Cluster Sizing and Scaling¶

[Critical] Is the compute-optimized versus storage-optimized choice made against the hot-data-to-CPU ratio rather than by price? Compute optimized SKUs offer "high core to cache ratio and the lowest cost per core" on local SSD; storage optimized SKUs offer "1 terabyte (TB) to 4 TB per engine node," and Microsoft warns that "certain SKUs use premium storage (PS)… slower than local SSDs, so SKUs with PS are more expensive and are less performant." Paying more for a slower SKU is an easy mistake to make from a price list.
[Critical] Is the cluster type chosen deliberately? Production clusters carry an SLA and require at least two engine and two data management nodes; Dev/Test clusters are single-node with no SLA, and on them "Azure Data Explorer markup isn't charged." A Dev/Test cluster is not a cheap production cluster.
[Recommended] Is optimized autoscale left enabled, and are its inputs understood? It is the default and is driven by CPU, cache utilization factor, and ingestion utilization with both predictive and reactive logic. Scale-in requires a one-day evaluation period, so a cluster does not shrink promptly after a load drop.
[Recommended] Is the vertical-scaling disruption window scheduled rather than discovered? A SKU change causes "one to three-minute service disruption." Microsoft's guidance is to scale up to a larger SKU and then scale out by adding nodes as needed.
[Optional] Is it noted that per-SKU cache sizes are not published in the documentation and are visible only in the portal, and that region availability defers to the ListSkus API? Any GB-of-cache-per-SKU figure from a third party should be treated as unsourced.

Ingestion¶

[Critical] Is queued ingestion the default choice, with streaming reserved for genuine low-latency needs? Microsoft's own guidance: "For most scenarios, we recommend using queued ingestion as it is the more performant option." Queued ingestion is "optimized for high ingestion throughput" with a 6 GB per-request data limit and at-least-once semantics; it buffers reliably for 7 days and then drops, and retries for 48 hours with exponential backoff.
[Critical] Are the ingestion batching policy defaults known, since they set the floor on end-to-end latency for queued ingestion? "Default values are as follows: 5 minutes maximum delay time, 500 items, total size of 1 GB." A batch seals when the first condition is met — Size, Count, Time, or FlushImmediately. (The commonly cited "1,000 items" is wrong.) Tunable ranges are 1–25,000 items, 100–4,096 MB, and 10 seconds–30 minutes; a low-latency configuration uses a 20–30 second timespan.
[Critical] If streaming ingestion is enabled, is the always-on cost understood? Enabling it "reduces the storage available for hot cache" even when it is unused. Crossover guidance is to move to queued above 4 GB per hour per table; limits are 6 concurrent requests per core and a 4 MB request size; schema updates take "up to five minutes"; and cursor updates can lag "up to 60 seconds." Enabling is two-step — cluster Configurations → Streaming ingestion → On, then .alter table T policy streamingingestion enable — and disabling "may take a few hours."
[Recommended] Is bulk load sized toward the documented optimum of roughly 1 GB uncompressed per batch, with the 64 MB per-record/value ceiling respected?
[Optional] Is the reported streaming latency stated as a range rather than a single figure? Two Microsoft pages disagree — one says "less than a second," another "less than a few seconds."

Update Policies and Materialized Views¶

[Critical] Is the choice between an update policy and a materialized view made on the aggregation question? Microsoft states it plainly: "Materialized views are suitable for aggregations, while update policies aren't. Update policies run separately for each ingestion batch, and therefore can only perform aggregations within the same ingestion batch." Timing differs too: update policies run during ingestion and block visibility until complete; materialized views materialize periodically in the background, post ingestion.
[Critical] Is IsTransactional set deliberately on update policies? It defaults to false. With IsTransactional: true, "if an update policy fails, data isn't ingested to the source or target table" — safer for correctness, and a single-point failure for the whole ingestion path. Note the standing hazard either way: "an incorrect query can prevent data ingestion into the source table."
[Critical] Is the replace data-loss hazard on update-policy chains known? "Data in derived tables is replaced in the same way as in the source table. Data may be lost in all tables with an update policy relationship if the replace command is invoked. Consider using .set-or-append."
[Critical] Is it understood that deletions do not propagate into a materialized view? "Records that are removed from the source table… have no impact on the materialized view," so "the materialized view might include records that aren't present in the source table." For any workload with an erasure or correction requirement, this is a correctness gap, not a performance detail.
[Recommended] Is materialized_view("V") used rather than referencing the view by name where staleness is acceptable? Querying by name merges the materialized part with the unprocessed delta and is always fresh but slower; the function reads only the materialized part and is faster but possibly stale. Microsoft's guidance: "Always use the materialized_view() function when applicable."
[Recommended] Are the materialized view constraints checked before designing one? Single fact table; a single summarize as the last operator; no now()-dependent filters; no sort, top, top-nested, partition, or serialize; no composite aggregations such as sum(a)/sum(b); no cross-cluster or external tables; cannot be created in follower databases; continuous export from a view is unsupported; and the recommended source ingestion ceiling is 1–2 GB/sec. Twenty-one aggregation functions are supported.
[Optional] Is backfill planned as an operation rather than a flag? It must run async, the view is unqueryable until it completes, it is not supported on cold-cache data, and Microsoft does not recommend it "when the number of records in the source table exceeds number-of-nodes X 200 million."

Query Surface: KQL and T-SQL¶

[Critical] Is it known that "KQL is case-sensitive for everything"? This catches every team arriving from T-SQL, and it applies to table names, column names, and function names alike.
[Recommended] Is the dot-prefix convention understood as a security boundary rather than a syntax quirk? "Management commands are distinguished from queries by having the first character… be the dot (.)… This distinction prevents many kinds of security attacks, simply because it prevents embedding management commands inside queries." It is a genuine argument for ADX in an environment worried about injection.
[Recommended] Is the T-SQL surface scoped correctly if it is being relied on for tool compatibility? "Only Data Query Language (DQL) commands are supported," invoked by starting the query with an empty -- comment line, and explain emits the equivalent KQL. Unsupported: CREATE, INSERT, DROP, ALTER, ANY/ALL/EXISTS, WITHIN GROUP, TOP PERCENT, correlated subqueries, recursive CTEs, AT TIME ZONE, and cursors. It is a compatibility shim for BI tools, not a migration target.
[Optional] Is the free cluster used for evaluation and training? It provides roughly 100 GB uncompressed (20 GB compressed), up to 10 databases, 100 tables per database, 200 columns per table, and 5 materialized views per database, for "a period of one year" (auto-renewable), with no SLA and no Azure subscription or credit card required. Absent: external tables, continuous export, workload groups, purge, follower clusters, partitioning policy, Python and R plugins, CMK, VNet, managed identities, autoscale, Azure Monitor, and ARM templates. Microsoft reserves the right to "change or discontinue the Free Cluster service at any time without notice," and the free cluster's retention and hot-cache periods are not documented.

Relationship to Fabric Eventhouse¶

[Critical] Is the ADX-to-Eventhouse relationship described from the documented evidence rather than the slogan? Microsoft never uses the phrase "Eventhouse is ADX inside Fabric." What is documented: the Kusto documentation set is literally shared, with pages marked "Applies to: ✅ Microsoft Fabric ✅ Azure Data Explorer" and identical commands, policy objects, and KQL; a Fabric database shortcut "behaves similar to a follower database" in ADX; and the ADX-to-Eventhouse migration preserves the source cluster's query and ingestion endpoint URIs, which is only possible with protocol and engine compatibility. Cite those rather than asserting engine identity.
[Critical] Is the storage and billing model difference understood before choosing between them? Eventhouse splits storage into OneLake Cache Storage — "premium storage… when you set the cache policy, you affect this storage tier… comparable to the Azure ADLS premium tier" — and OneLake Standard Storage, which "is used to persist and store all queryable data… when you set the retention policy, you affect this storage tier." Both are billed separately from Capacity Units. Eventhouse bills compute on UpTime (active seconds × vCores) with intrinsic autoscale and automatic suspension when idle; ADX bills provisioned cluster nodes.
[Recommended] Are the Eventhouse capability gaps checked before migrating a cluster? Not supported in Eventhouse: purge, cluster-level policies, system-assigned managed identities, firewall rules. Must be removed before migrating: follower and leader clusters, CMK, VNet-injected clusters, and private or managed private endpoints. Not migrated automatically: data connections, .export and continuous export, and database pretty names. The purge and CMK gaps are usually the decisive ones in a regulated environment.
[Recommended] Is the retired Synapse Data Explorer migration path distinguished from the ADX one? Azure Synapse Data Explorer (Preview) was retired on 7 October 2025 — "after this date, workloads running on Synapse Data Explorer will be deleted, and the associated application data will be lost." Its migration to Eventhouse is REST-API-based, irreversible, preserves the source endpoint URIs for 90 days, and cannot migrate purge, cluster-level policies, data connections, follower/leader clusters, CMK, VNet-injected or private-endpoint clusters, or firewall rules.
[Optional] Where the same data must serve both KQL and Direct Lake reporting, is Eventhouse OneLake availability used rather than an export pipeline? It creates a logical Delta copy at no extra storage cost, but while it is enabled you cannot rename tables, alter a column type, apply row-level security, or delete, truncate, or purge data — and the resulting OneLake table is read-only. See providers/azure/fabric.md.

Azure Monitor Log Analytics Interop¶

[Critical] Is the relationship stated in Microsoft's own words rather than overclaimed? The documented wording is "Azure Monitor Logs is based on Azure Data Explorer and uses the same Kusto Query Language (KQL) to write log queries," and elsewhere that other services are "built on its query language." Microsoft does not say Log Analytics runs on the same engine. Use "is based on."
[Critical] Is the KQL divergence known before porting queries between the two? Azure Monitor is a subset plus extensions, and the absences are precisely the ADX internals primitives: not supported — Alias, query parameters, cluster(), database(), extent_id(), extent_tags(), cursor_*(), current_principal(), cross-cluster join, the Python plugin, and the sql_request plugin. Exclusive to Azure Monitor — workspace(), app(), resource().
[Critical] If cross-service query is being designed in, is the absence of time-filter pushdown accounted for? "Identifying the Timestamp column in a cluster isn't supported. The Log Analytics Query API doesn't pass the time filter" — so a Log-Analytics-initiated adx() query does not prune by time at the ADX side, and a query that looks cheap in the portal can scan far more than intended.
[Recommended] Are the cross-service query limitations reviewed against the deployment's constraints? Cross-service query is not supported in Government clouds and not supported for ADX clusters configured with IP restrictions or Private Link — which rules it out for most network-isolated deployments. It supports "data retrieval only," "the only commands cross-service queries support are .show commands," mv-expand is capped at 2,000 records, external_table() is unsupported, cross-tenant is unsupported, and the Azure Resource Graph portion returns only the first 1,000 records.
[Recommended] Are the correct endpoints used in each direction? Log Analytics to ADX uses adx('https://<cluster>.kusto.windows.net/<database>').Table (with an adx('cluster/database') shorthand) and requires hint.remote= on joins. ADX to Log Analytics or Application Insights adds the workspace as a connection and then uses ordinary cluster(...).database(...); the recommended cross-cluster form uses https://adx.monitor.azure.com/..., with adx.monitor.azure.us/ and adx.monitor.azure.cn/ in sovereign clouds. "The database should have the same name as the resource… names are case sensitive."
[Optional] Is TLS 1.2 or later enforced on all clients? Both cross-service directions have required it since 1 July 2025.

Why This Matters¶

The caching-versus-retention confusion is the defining ADX design error and it produces failures in both directions. A team that believes the cache policy governs how long data is queryable shortens it for cost reasons and is then surprised that queries still work — fine. A team that believes the retention policy governs cache residency lengthens it expecting faster queries, pays for storage it does not need, and gets no performance change. Worse, the retention default is 1,000 years and new tables have no retention policy at all, so an organisation that never thought about it is accumulating queryable data indefinitely — which is a cost problem, and in any regulated context a data-minimisation problem with an audit trail showing nobody made a decision.

Ingestion batching is where ADX latency expectations go wrong. The queued path — Microsoft's own default recommendation for most scenarios — seals a batch on the first of 5 minutes, 500 items, or 1 GB. A team that instruments an application, watches events arrive minutes later, and concludes ADX is slow has simply met the batching policy. Conversely, a team that enables streaming ingestion to fix that inherits a permanent reduction in hot cache capacity whether or not streaming is used, a 4 MB request ceiling, and a documented crossover back to queued above 4 GB per hour per table. Both paths are correct for their workload and neither is correct for both; the batching policy is the knob, not the ingestion mode.

Materialized views carry a correctness caveat that reads like a performance footnote and is not one: deletions in the source table do not propagate, so "the materialized view might include records that aren't present in the source table." Any workload subject to erasure requests, late corrections, or regulatory restatement cannot treat a materialized view as a derived-and-therefore-consistent artefact. The same class of hazard applies to update policy chains, where invoking replace on a source table can lose data across every table in the relationship — which is why Microsoft's guidance is to prefer .set-or-append.

On the Fabric question, precision matters more than the usual amount because the market shorthand overstates what Microsoft documents. Eventhouse and ADX genuinely share the Kusto engine surface — the documentation set is one set with dual product markers, the commands and policies are identical, database shortcuts behave like ADX follower databases, and migration preserves endpoint URIs. But Microsoft never writes "Eventhouse is ADX inside Fabric," and the differences that exist are exactly the ones a regulated deployment cares about: Eventhouse does not support purge, cluster-level policies, system-assigned managed identities, or firewall rules, and CMK, VNet-injected clusters, and private endpoints must be removed before migrating. An architecture that depends on data purge for erasure compliance, or on VNet injection for network isolation, does not have an Eventhouse path today. Saying that plainly is more useful than describing the two as equivalent.

Common Decisions (ADR Triggers)¶

Azure Data Explorer vs Fabric Eventhouse -- Eventhouse for new work where the estate is already in Fabric, the OneLake logical copy is wanted for Direct Lake reporting, and UpTime-based autoscaling suits a bursty workload vs ADX where purge, cluster-level policies, VNet injection, private endpoints, CMK, follower clusters, or firewall rules are required, since none of those are supported in Eventhouse today
ADX vs Log Analytics for telemetry -- Log Analytics where the data is Azure platform and agent telemetry, the Azure Monitor alerting and workbook ecosystem is the consumer, and the KQL subset suffices vs ADX where ingestion volume, retention control, cluster-level tuning, cross-cluster query, update policies, materialized views, or the plugin surface is needed. Note that cross-service query between them is read-only, .show-only, does not push down time filters, and does not work with Private Link or IP restrictions
Compute-optimized vs storage-optimized SKU -- compute-optimized for the highest core-to-cache ratio and lowest cost per core on local SSD, where the hot window is small relative to CPU demand vs storage-optimized for 1–4 TB per engine node where a large hot window dominates, accepting that premium-storage SKUs are both more expensive and less performant than local SSD
Production vs Dev/Test cluster -- production for any workload with an availability expectation, given the SLA and the two-plus-two node minimum vs Dev/Test single-node clusters for evaluation only, where the ADX markup is not charged and there is no SLA at all
Hot cache window sizing -- size the hot window to the interactive query range and let older data serve from blob storage, since retention is unaffected and cold queries still succeed vs cache everything, which is defensible only where every query is latency-critical and is usually a large avoidable cost
Retention policy: explicit vs default -- set SoftDeletePeriod explicitly per table against the actual business and regulatory requirement vs accept the 1,000-year default, which is almost never the intended answer and leaves no record of a decision having been made. Note that Recoverability: Enabled retains soft-deleted data for a further 14 days and cannot be configured
Streaming vs queued ingestion -- queued as the default per Microsoft's own recommendation, tuning the batching policy timespan down to 20–30 seconds where latency matters vs streaming for genuine sub-batch latency on small per-table volumes, accepting the permanent hot-cache reduction, the 4 MB request ceiling, and the documented crossover back to queued above 4 GB/hour per table
Update policy vs materialized view -- update policy for row-level transformation, routing, and enrichment at ingestion time, optionally transactional so a failure blocks ingestion vs materialized view for aggregation across batches, accepting background materialization lag and the fact that source deletions do not propagate
Materialized view freshness -- reference the view by name for always-fresh results that merge the unprocessed delta vs materialized_view() for faster reads of the materialized part only, accepting staleness; Microsoft recommends the function wherever staleness is tolerable
Query interface: KQL vs the T-SQL shim -- KQL for anything written by the team, since it is the full surface and the dot-prefix convention is a genuine injection defence vs the T-SQL shim only for third-party BI tools that cannot speak KQL, recognising it is DQL-only with no DDL, no correlated subqueries, and no recursive CTEs

Reference Architectures¶

What is Azure Data Explorer? -- positioning, workloads, and the KQL-based service family
Azure Data Explorer architecture -- storage and compute separation, extents, the multilevel cache, and the row store
Extents (data shards) overview -- extent immutability, swapping, and the per-cluster ceiling
Cache policy -- hot versus cold storage, the 95/5 split, and the independence from retention
Retention policy -- SoftDeletePeriod, the 1,000-year default, and Recoverability
Hot windows for infrequent queries -- caching specific historical date ranges without extending the hot span
Select the correct compute SKU -- cluster types, compute-optimized versus storage-optimized, and the premium-storage caveat
Manage cluster horizontal scaling -- optimized autoscale inputs and the scale-in evaluation period
Manage cluster vertical scaling -- SKU changes and the service disruption window
Data ingestion overview -- streaming versus queued ingestion and Microsoft's default recommendation
Streaming ingestion -- enablement, limits, the hot-cache cost, and the crossover to queued
Ingestion batching policy -- the 5 minutes / 500 items / 1 GB defaults, tunable ranges, and seal triggers
Update policy overview -- ingestion-time transformation, IsTransactional, limitations, and the replace data-loss hazard
Materialized views overview -- the materialized part and delta, the materialized_view() function, and update-policy comparison
Materialized views limitations -- supported aggregations, query constraints, and non-propagating deletions
Kusto Query Language overview -- KQL structure, case sensitivity, and the dot-prefix management-command boundary
T-SQL support in Azure Data Explorer -- the DQL-only surface, invocation, and unsupported constructs
Start for free with Azure Data Explorer -- free cluster entitlement, quotas, and excluded features
Follower databases -- read-only attached databases, the model Fabric database shortcuts resemble
Migrate Azure Data Explorer to Fabric Eventhouse -- the feature-support matrix and what must be removed before migrating
Migrate Synapse Data Explorer to Fabric Eventhouse -- the retired-component path, endpoint URI preservation, and irreversibility
Eventhouse overview -- UpTime billing, intrinsic autoscale, and suspension behaviour
Real-Time Intelligence consumption -- OneLake Cache versus Standard storage tiers and their governing policies
Log queries in Azure Monitor -- the "based on Azure Data Explorer" statement and the KQL divergence list
Query Azure Data Explorer from Azure Monitor -- the adx() operator, join hints, and cross-service query limitations
Query Azure Monitor data from Azure Data Explorer -- proxy cluster connections, endpoint forms, and sovereign-cloud hosts