Query Engines (Trino, Presto, Starburst, Dremio, Spark SQL, DuckDB)¶

Scope¶

The compute layer that reads data-lake tables. This file covers the open and independent query engines — Trino and Presto, the commercial distributions built on them (Starburst), Dremio, Spark SQL, Flink SQL for streaming, and DuckDB for single-node and embedded analytics — plus query federation across heterogeneous sources.

It covers the architectural model (coordinator/worker MPP vs stage-materializing batch vs in-process single-node), memory management and what happens when a query does not fit, fault-tolerant execution, pushdown behaviour and why federation succeeds or fails on it, concurrency and workload isolation, caching and acceleration layers, the cost-model differences that make one engine cheaper than another for the same work, and — most importantly — when a separate lake engine beats the warehouse's own SQL and when it does not.

For the table formats these engines read see general/open-table-formats.md. For the warehouse-vs-lake-vs-lakehouse platform decision see general/data-analytics.md. For managed instances of this layer see providers/aws/athena.md (managed Trino-lineage SQL), providers/gcp/bigquery.md, providers/databricks/data-platform.md, and providers/snowflake/data-platform.md.

Engine versions, connector capabilities, and published pricing all move quickly. Verify version-dependent claims — particularly Iceberg spec support and pushdown coverage per connector — against current documentation before committing to a design.

Checklist¶

Engine Selection: Lake Engine vs Warehouse¶

[Critical] Has the question "why not just use the warehouse's own SQL?" been answered explicitly? A separate lake query engine is justified by one of a small number of specific needs: querying data in open formats without loading it, federating across sources the warehouse cannot reach, decoupling cost from bytes scanned, avoiding vendor lock-in on the compute layer, or supporting engines and languages the warehouse does not. If none of those apply, the warehouse's SQL is simpler, faster, and better governed.
[Critical] Is the workload profile characterized before choosing — interactive and low-latency, or long-running and large? This is the single most predictive input. Pipelined MPP engines (Trino, Presto, Dremio) stream results through operators without materializing intermediate stages, which gives excellent interactive latency but historically means a query that exceeds memory fails rather than degrading. Stage-materializing engines (Spark SQL) checkpoint between stages, which costs latency but survives very large shuffles and node loss. Choosing an interactive engine for a 6-hour ETL job, or a batch engine for a BI dashboard, is the most common and most expensive mismatch in this layer.
[Critical] Is the concurrency requirement quantified, and has it been checked against the engine's isolation model? A single Trino cluster shares memory and CPU across all concurrent queries, so one large query degrades everyone. Warehouses with independent compute (Snowflake virtual warehouses, Databricks SQL multi-cluster, BigQuery reservations) isolate workloads structurally. If the requirement is "200 concurrent BI users with predictable latency," that is a warehouse-shaped requirement, and meeting it with one Trino cluster will not work.
[Critical] Is exactly one engine designated as the writer of record per table, with the others reading? Multi-engine write to the same open table is where per-engine differences in delete encoding, mutation mode, and maintenance assumptions turn into corruption or silent no-ops. See general/open-table-formats.md.
[Recommended] Has the total cost been modelled on the actual workload shape rather than on list rates? The cost models are structurally different — per-byte-scanned, per-slot-hour, per-credit, per-DBU, or plain cluster-hours — and which is cheapest inverts depending on utilization. A self-managed engine billed only in cluster-hours is cheap at high sustained utilization and expensive when idle; a per-byte engine is the reverse.
[Recommended] Is the operational cost of self-managing an engine counted honestly? Trino is a distributed system with memory tuning, connector configuration, version upgrades, autoscaling, and a coordinator that is a single point of failure. The engineering time is real and is routinely omitted from comparisons against a managed service.
[Optional] Has the single-node option been considered before reaching for a cluster? A large fraction of real analytical datasets fit comfortably on one modern machine, and DuckDB will answer queries over them faster than any distributed engine can schedule them.

Trino and Presto: Architecture and Execution¶

[Critical] Is the coordinator/worker split understood, along with the coordinator's role as a single point of failure and a scaling bottleneck? The coordinator parses SQL, plans, schedules, and returns results; workers execute tasks and exchange data. Query planning, result assembly, and (without an exchange manager) result buffering all happen on the coordinator, so it must be sized for planning load and concurrency rather than for data volume.
[Critical] Is the connector model understood as the thing that makes Trino a federation engine rather than a lake engine specifically? Everything Trino reads — object storage tables, relational databases, Kafka, search indexes — arrives through a connector implementing the same SPI. Capability therefore varies per connector, not per engine, and "Trino supports X" is never a complete answer.
[Critical] Is the split/stage/task/driver model understood well enough to read a query plan? A query becomes stages, stages become tasks distributed across workers, tasks process splits (addressable chunks of input) through pipelines of operators. The number of splits available determines achievable parallelism — which is why a table stored as one enormous non-splittable file cannot use the cluster, and why thousands of tiny files cause scheduling overhead that dwarfs the work.
[Recommended] Is the distinction between Trino and PrestoDB understood, given both descend from the same Facebook project? The original developers forked to PrestoSQL and renamed it Trino in December 2020; PrestoDB continues under the Presto Foundation, hosted at the Linux Foundation and formed in 2019 by Facebook, Uber, Twitter, and Alibaba, with a governing board and a technical steering committee. They are separate projects with diverging features. Documentation, connector behaviour, and support ecosystems are not interchangeable, and most commercial activity and community momentum sits with Trino.
[Optional] Is EXPLAIN/EXPLAIN ANALYZE used routinely on expensive queries, rather than treating the engine as a black box? Almost every serious Trino performance problem is visible in the plan as a missing pushdown, an absent dynamic filter, or a broadcast join that should have been partitioned.

Memory, Failure, and Long-Running Queries¶

[Critical] Are the memory limits configured deliberately (query.max-memory across the cluster, query.max-memory-per-node, query.max-total-memory) and is the resulting failure mode accepted? A query exceeding its per-node limit is killed. On an untuned cluster this presents as "big queries randomly fail," and the fix is usually a combination of limits, query rewriting, and choosing the right engine — not simply raising the ceiling until the coordinator dies.
[Critical] Is it known that spill-to-disk is legacy functionality in Trino, and that the documented recommendation is fault-tolerant execution instead? Trino's own documentation states that spill to disk "and implementation are a legacy functionality of Trino" and directs users to consider fault-tolerant execution with the task retry policy and a configured exchange manager. Designs that plan to "just enable spilling" for large joins are building on a deprecated path. Where spilling is used, it covers joins, aggregations, ORDER BY, and window functions — with the documented caveat that window-function spilling does not work in all cases, such as a single very large window.
[Critical] For batch or ETL-shaped work on Trino, is fault-tolerant execution configured with the correct retry policy? retry-policy=QUERY retries the whole query and is recommended when the workload is many small queries; retry-policy=TASK retries individual tasks and is recommended for large batch queries, because the cluster retries smaller units rather than the whole query. Getting this backwards is costly in both directions.
[Critical] If retry-policy=TASK is used, is an exchange manager configured with external storage, and is the latency trade-off accepted? TASK retries require an exchange manager, which spools intermediate data to S3, Azure Blob Storage, GCS, or HDFS (local filesystem is documented as non-production only). Trino documents that the TASK policy "can result in higher latency for short-running queries executed in high volume," and the recommended practice is separate clusters for task-policy batch work and for short interactive queries. One cluster tuned for both will serve neither well.
[Recommended] With retry-policy=QUERY, is the result-set fault-tolerance limit understood? By default, queries whose result sets exceed a buffer threshold are not fault-tolerant; the buffer (exchange.deduplication-buffer-size) can be raised at the cost of coordinator memory, but an exchange manager with external storage is the recommended answer for larger results.
[Recommended] Is there a query timeout and a maximum-scan guard so a runaway query cannot occupy the cluster indefinitely? On a shared cluster this is the difference between one bad query and a platform-wide incident.
[Optional] Is autoscaling configured with an understanding that scaling down a Trino cluster kills the tasks on the removed workers unless fault-tolerant execution is enabled? This makes graceful scale-down and FTE the same decision.

Pushdown and Federation¶

[Critical] Is pushdown understood as the property that determines whether federation is efficient or catastrophic? When predicates, projections, aggregates, joins, and limits push down, the remote source does the filtering and returns little data. When they do not, the engine pulls the source data across the network and filters locally — turning a selective query into a repeated full extract, at cost to the engine, the network, and the operational source it is reading from.
[Critical] Has pushdown been verified with EXPLAIN for the specific connector and the specific query shapes in scope, rather than assumed from the connector's feature list? Pushdown support differs sharply between connector families: JDBC connectors to relational databases generally support predicate, projection, and some aggregate and join pushdown; object-storage connectors (Iceberg, Delta Lake, Hive) rely instead on partition pruning and file-level statistics. A query that pushes down beautifully with an equality predicate may not push down at all once a function wraps the column.
[Critical] Is federation scoped to exploration, low-volume lookups, and reference-data joins rather than production pipelines? A scheduled federated join against an operational database is an ingestion pipeline written in the least efficient available form, and it puts analytical load on a transactional system. The durable answer is to land the data in the lake. See patterns/data-pipeline.md.
[Recommended] Is dynamic filtering enabled and confirmed to be firing on the large fact-to-dimension joins? Dynamic filtering builds a filter from the small side of a join at runtime and applies it to the large side's scan, which can eliminate most of the scan. On star-schema queries this is frequently the largest single performance factor, and it is easy to defeat with an unhelpful join order or types that do not match.
[Recommended] When federating across sources, is the join ordering controlled so the engine is not asked to join two large remote tables in memory? The general rule is to reduce at the source and join late and small; the failure case is an engine dutifully materializing two multi-hundred-gigabyte remote tables to perform a join that the source could have done.
[Recommended] Are per-source concurrency and rate limits set so analytical federation cannot saturate an operational database? A Trino cluster can trivially open more connections than a production OLTP system tolerates.
[Optional] Is it recognized that every major platform now offers some federation (Athena Federated Query on Lambda connectors, BigQuery federated queries, Redshift federated query, Databricks Lakehouse Federation, Snowflake external tables), and that the pushdown lesson applies identically to all of them? The mechanism differs; the failure mode does not.

Concurrency and Workload Isolation¶

[Critical] Are resource groups configured on a shared Trino cluster to bound concurrency and memory per workload class? Without them, one team's ad-hoc query competes directly with the BI dashboards, and the engine has no notion of priority. Resource groups can queue and cap; what they cannot do is give a workload a genuinely independent pool of hardware, which is the thing warehouses provide natively.
[Critical] For meaningful workload isolation, is the design multiple clusters by workload class rather than one cluster with clever configuration? This is the standard Trino answer: a short-query interactive cluster, a batch cluster with retry-policy=TASK and an exchange manager, and possibly a separate cluster per large tenant. Trying to make one cluster serve interactive and batch simultaneously is the most common source of disappointment with Trino.
[Recommended] If multiple clusters are used, is a routing layer in place so clients see one endpoint? Trino Gateway is an open-source project in the Trino organization that provides routing, load balancing, and query queueing across multiple Trino clusters. Without it, cluster topology leaks into every client's configuration.
[Recommended] Is the coordinator sized and monitored separately from workers, given planning and result handling are concentrated there? High-concurrency workloads become coordinator-bound before they become worker-bound.
[Optional] Is queueing behaviour explained to users, so a queued query is understood as a capacity signal rather than a hung client? Silent queueing is a common support burden.

Caching and Acceleration¶

[Critical] Is the caching strategy explicit, given that a lake engine reading object storage pays a network round trip for every byte on every query? The available layers are different in kind: file system caching (Trino's Alluxio-based local caching of object-storage data on worker disks), result caching, metadata and statistics caching, and materialization (materialized views, or Dremio Reflections). Each addresses a different bottleneck and they are not substitutes.
[Critical] Is materialization used for the repeated, predictable, expensive queries rather than trying to make every ad-hoc query fast? The 80/20 is almost always a handful of dashboard queries. Pre-aggregating those into small tables — or letting the engine maintain the equivalent — is more effective and cheaper than any amount of cluster tuning. See patterns/lakehouse-medallion.md.
[Recommended] If Dremio is in scope, are Reflections understood as optimizer-transparent materializations rather than as views users must reference? Dremio maintains raw, aggregation, and starflake Reflections; the optimizer detects when a Reflection can satisfy a query, generates a plan using it, compares that plan's cost against querying the source directly, and picks the cheaper one — without the query naming the Reflection. This is the main architectural distinction from conventional materialized views, and Dremio also offers autonomous Reflection management.
[Recommended] Is the distinction between open-source and commercial features clear before a design depends on one? File system caching, dynamic filtering, fault-tolerant execution, and resource groups are in open-source Trino. Autonomous indexing and acceleration layers in commercial distributions (Starburst Warp Speed, Starburst materialized and cached views) are not. Treat all vendor performance figures as vendor claims until reproduced on your own workload.
[Recommended] Is cache invalidation reasoned about for mutable tables? A cached file that has been rewritten by compaction, or a cached statistic that predates a large merge, produces stale or slow results. Open table formats make this tractable because the snapshot identifies the file set — but only if the cache is snapshot-aware.
[Optional] Are table and column statistics collected and kept current, given the cost-based optimizer depends on them? Missing statistics are a common cause of a catastrophically bad join order that no amount of hardware fixes.

Single-Node and Embedded Analytics¶

[Critical] Has the dataset actually been measured before a distributed engine is chosen? DuckDB is an in-process, vectorized, columnar OLAP engine that runs inside the calling process with no server, and it handles datasets far larger than most teams expect — including larger-than-memory workloads through spilling. For datasets in the tens or low hundreds of gigabytes, it frequently beats a distributed cluster outright, because it has no scheduling, no shuffle, and no network.
[Recommended] Is DuckDB used in the places where its model is strictly better — local development against production-shaped data, CI test fixtures, notebook analysis, embedded analytics inside an application, and as a fast local reader for Parquet and Iceberg via its extensions (httpfs/S3, iceberg, delta, postgres_scanner)? See general/local-development-environments.md.
[Recommended] Is the single-node ceiling acknowledged explicitly rather than discovered? One process means one machine's memory and cores, no horizontal scaling, no multi-user concurrency model, and no built-in governance layer. It is a superb tool and a poor multi-tenant platform.
[Optional] If DuckLake is under consideration, is its actual proposition understood? DuckLake is both a lakehouse format specification and a DuckDB extension, reaching v1.0 in April 2026 with a stated backward-compatibility guarantee. Its architectural claim is that lakehouse metadata belongs in an ACID SQL database (PostgreSQL, SQLite, or DuckDB) rather than in metadata files alongside the Parquet data, which is what lets it claim arbitrarily many snapshots without frequent compaction. It is a genuinely different point in the design space from Iceberg and Delta — and a much younger one, with a correspondingly smaller engine ecosystem. See general/open-table-formats.md.
[Optional] Where a managed hybrid of local and cloud DuckDB is wanted, has MotherDuck been evaluated on published terms, and is the "small data" argument being weighed on its merits rather than as vendor marketing? The widely cited "Big Data is Dead" essay is published by MotherDuck and written by its chief executive; the underlying observation about typical query and dataset sizes is worth taking seriously, and the source is not neutral.

Cost Models¶

[Critical] Is it understood that the cost models are structurally different, so the cheapest engine depends on utilization rather than on rate cards? The families are: per byte scanned (Athena on demand, BigQuery on demand) — cost tracks data layout, zero cost when idle, unbounded worst case per query; per compute-time unit (BigQuery slot-hours, Snowflake credits, Databricks DBUs, Athena provisioned capacity DPU-hours, Redshift Serverless RPU-hours) — cost tracks time and capacity, predictable, wasted when idle; and plain cluster-hours (self-managed Trino, Spark, or Dremio on EC2/EKS/GKE) — cost is entirely decoupled from bytes scanned and queries, so a heavy scanning workload can be dramatically cheaper, and an idle cluster is pure waste.
[Critical] For a heavy, sustained scanning workload, has the crossover against per-byte pricing been computed? This is the strongest quantitative argument for a self-managed engine: because cluster-hour cost does not increase with bytes scanned, a workload that scans very large volumes repeatedly can be an order of magnitude cheaper on a self-managed cluster — as long as utilization stays high enough to amortize it and the engineering cost is included.
[Critical] Are per-query and per-workload cost guardrails in place on any per-byte engine, before users are given access? A per-byte engine has no natural blast-radius limit: a single careless query scans everything. See providers/aws/athena.md.
[Recommended] Are idle-cost and startup-latency characteristics matched to the access pattern? Serverless and per-byte models cost nothing when idle but can have cold-start latency; provisioned clusters answer instantly and bill continuously. Bursty daytime-only analytics and 24/7 pipelines want opposite answers, and auto-suspend or scheduled scaling is what reconciles them.
[Recommended] Is minimum-billing granularity accounted for in workloads made of very many small queries or very short jobs? Per-second billing with a per-query or per-session minimum behaves quite differently from per-second billing without one, and short-job-heavy workloads are where the difference shows.
[Recommended] Are committed-use or reserved-capacity discounts evaluated only for the genuinely predictable share of the workload, leaving the variable share on demand? Committing the peak is how organizations end up paying for capacity they use a few hours a week.
[Optional] Is cost attributed per team or per workload (separate clusters, workgroups, warehouses, or reservations with tags) so optimization effort can be aimed where the money actually goes? Query spend is almost always concentrated in a handful of queries and dashboards.

Why This Matters¶

The engine layer is the one place in a lakehouse where the same work can differ in cost by an order of magnitude with no change to the data. Because storage is separated from compute, the identical Iceberg table can be served by a per-byte serverless engine, a per-slot warehouse, or a self-managed cluster billed only in instance-hours — and which of those is cheapest depends entirely on utilization and scan volume. Organizations that pick the engine first and then discover their workload shape usually find they are on the wrong side of that curve, paying per byte for a workload that scans enormous volumes repeatedly, or paying for an idle cluster to serve a few analysts a few hours a day.

The mismatch that causes the most operational pain, though, is architectural rather than financial. Pipelined MPP engines are built for interactive latency: they stream tuples through operators without materializing stages, which is why Trino answers a dashboard query in a second and why a query that outgrows memory has historically just died. Batch engines materialize between stages, which is why Spark survives a 10 TB shuffle and a lost worker, and why it is a poor choice behind a BI tool. Teams that standardize on one engine for everything spend the following year discovering the edge they chose against — either fragile large queries or unacceptable interactive latency. The standard resolution is not a better-tuned single cluster; it is separate clusters, or separate engines, per workload class. Trino's own documentation reaches this conclusion from the other direction when it recommends dedicated task-retry clusters for batch work, separate from clusters serving high volumes of short queries.

Spill-to-disk deserves specific mention because so much older guidance recommends it. Trino now describes spilling as legacy functionality and points to fault-tolerant execution with a task retry policy and an exchange manager instead. Any design whose plan for large joins is "enable spilling" is building on a deprecated path, and the modern equivalent has a different shape: it needs external object storage for the exchange manager and it adds latency to short queries, which is precisely why it belongs on a separate cluster.

Federation is where expectations and reality diverge most sharply, and pushdown is the whole story. A federated query with working pushdown is genuinely elegant: the remote database filters and aggregates, and a few thousand rows come back. The same query with a predicate the connector cannot translate pulls the entire remote table across the network, on every execution, while adding analytical load to a production transactional system. Both look like one line of SQL. The difference is visible only in the query plan, which is why "verify with EXPLAIN" is a hard requirement rather than a nicety, and why federation belongs in exploration and reference-data lookups rather than in scheduled pipelines.

Finally, the distributed default deserves to be challenged more often than it is. A very large share of real analytical datasets fit on one machine, and a single-node vectorized engine with no scheduler, no shuffle, and no network will beat a cluster on them — often while being simpler to operate, cheaper, and easier to test. The right instinct is to measure the data before designing for scale, and to reserve distributed engines for the workloads that genuinely need them. The corollary is equally important: single-node engines are excellent tools and poor multi-tenant platforms, so the answer is usually both, in different places, rather than one everywhere.

Common Decisions (ADR Triggers)¶

Separate lake query engine vs the warehouse's own SQL — a lake engine for open-format access without loading, federation across sources the warehouse cannot reach, decoupling cost from bytes scanned, and compute-layer portability vs the warehouse's SQL for lower operational overhead, stronger native governance, and better concurrency isolation. If no specific driver applies, the warehouse wins on simplicity.
Trino vs Spark SQL as the primary engine — Trino for interactive and ad-hoc SQL, federation, and BI-facing latency vs Spark SQL for long-running large-shuffle ETL, non-SQL transformations, and workloads that must survive node loss mid-query. Most mature platforms run both and route by workload class rather than choosing one.
Self-managed Trino vs a managed service — self-managed for cost decoupled from bytes scanned at high utilization, full connector and version control, and no per-query pricing vs managed (Athena, Starburst Galaxy, Dremio Cloud, EMR) for no cluster operations, faster time to value, and someone else owning upgrades and availability. Include engineering time in the comparison or it is not a comparison.
Trino vs Starburst — open-source Trino for zero licence cost and full control vs Starburst Enterprise or Galaxy for support, additional acceleration and governance features, and connector coverage beyond the open-source set. Validate any vendor performance claims against your own workload before pricing the difference in.
Dremio vs Trino — Dremio for optimizer-transparent Reflections that accelerate BI without users changing queries, an integrated semantic layer, and an Arrow-native stack vs Trino for the broadest open connector ecosystem, the largest community, and no vendor dependency on the acceleration layer.
Retry policy and cluster topology on Trino — retry-policy=QUERY on an interactive cluster serving many small queries vs retry-policy=TASK with an exchange manager on a dedicated batch cluster, accepting added latency for short queries. This should be two clusters, not one compromise.
Acceleration strategy — file system caching (reduces object-storage round trips, transparent, bounded by worker disk) vs materialized views or Reflections (largest win on repeated queries, adds storage and refresh cost and staleness) vs pre-aggregated gold tables built by the pipeline (most portable and predictable, most explicit engineering). Usually gold tables for known dashboards, caching for everything else.
Concurrency architecture — one cluster with resource groups (simplest, no true isolation, one heavy query degrades all) vs multiple clusters by workload class behind Trino Gateway (real isolation, more infrastructure) vs a warehouse with independent compute (structural isolation, per-unit compute pricing).
Federation vs ingestion — federated query for exploration, low-volume lookups, and reference-data joins vs landing the data in the lake for anything recurring or high-volume. Decide per source and per query shape, on verified pushdown behaviour.
Single-node vs distributed — DuckDB (or an equivalent) for datasets that fit one machine, local development, CI, and embedded analytics vs a distributed engine for genuinely large data, high multi-user concurrency, and centrally governed access. Measure the data before deciding.

Reference Architectures¶

Trino on Kubernetes over an Iceberg lake¶

Object storage holding Iceberg tables -> an Iceberg REST catalog or Glue Data Catalog -> two Trino deployments on EKS/GKE: an interactive cluster (retry-policy=QUERY, aggressive file system caching on local NVMe, resource groups per team) and a batch cluster (retry-policy=TASK, exchange manager spooling to object storage, larger workers) -> Trino Gateway presents a single endpoint and routes by client or query characteristics -> BI tools and notebooks connect over JDBC to the gateway. Cost is cluster-hours only, so heavy scanning does not increase the bill; utilization management (scheduled scale-down, autoscaling with FTE enabled) is what makes it economical. See general/open-table-formats.md, general/container-orchestration.md.

Managed serverless SQL over a lake¶

S3 with partitioned Parquet or Iceberg -> Glue Data Catalog -> Athena workgroups per team with per-query and per-workgroup scan limits, pinned engine version, and lifecycle-managed result buckets -> Lake Formation enforces column and row access -> repeated dashboard queries served from small pre-aggregated tables rather than detail scans. No infrastructure, per-byte cost, guardrails mandatory. See providers/aws/athena.md, providers/aws/lake-formation.md.

Dual-engine lakehouse¶

One set of Iceberg or Delta tables in object storage -> Spark (Databricks, EMR, or Dataproc) as the writer of record for all pipelines, using stage materialization and fault tolerance for large transformations -> Trino or the platform's SQL warehouse as the read-side engine for interactive and BI queries -> maintenance (compaction, snapshot expiry, orphan cleanup) scheduled on the Spark side. This is the most common mature shape, and it works because write and read paths have genuinely different requirements. See providers/databricks/data-platform.md, patterns/lakehouse-medallion.md.

Federated exploration layer¶

Trino with connectors to the lake catalog, two operational PostgreSQL databases (read replicas only), a Kafka cluster, and a warehouse -> analysts join reference data from operational sources against lake facts for exploration -> per-catalog concurrency limits protect the operational databases -> anything that becomes a recurring query is promoted into an ingestion pipeline and materialized into the lake, with the federated version retired. Federation here is deliberately a discovery tool with a defined graduation path, not a production data path. See patterns/data-pipeline.md.

Single-node and embedded¶

Parquet or Iceberg data in object storage -> DuckDB in a notebook, a CI job, or inside an application process, reading directly via the httpfs/S3 and iceberg extensions -> the same SQL runs against a small local fixture in development and against the real dataset in analysis, with no cluster in either case. Used alongside a distributed engine for the governed multi-user path rather than instead of it. See general/local-development-environments.md.

Reference Links¶

Trino Concepts -- coordinator, workers, connectors, catalogs, splits, stages, tasks, and drivers
Trino Use Cases -- what Trino is and is not designed for
Trino Fault-Tolerant Execution -- QUERY vs TASK retry policies, exchange manager storage options, and the documented latency trade-offs
Trino Spill to Disk -- the legacy status of spilling, supported operations, and the recommendation to use fault-tolerant execution instead
Trino Resource Management Properties -- query.max-memory, query.max-memory-per-node, query.max-total-memory, and related limits
Trino Resource Groups -- concurrency and memory limits per workload class, with selectors and queueing
Trino Dynamic Filtering -- runtime filter construction for join-heavy workloads and its limitations
Trino Pushdown -- predicate, projection, aggregation, join, and limit pushdown, and which connector families support each
Trino File System Caching -- Alluxio-based local caching of object-storage data on workers
Trino Iceberg Connector -- supported spec versions, DML, and table maintenance procedures
Trino Delta Lake Connector -- read/write support, DML, and maintenance procedures
Trino Hudi Connector -- read support and current limitations
Trino PostgreSQL Connector -- a representative JDBC connector's pushdown and type-mapping behaviour
Trino Release Notes -- release cadence and per-version connector and engine changes
Trino Ecosystem -- clients, drivers, and related projects
Trino Gateway -- routing, load balancing, and queueing across multiple Trino clusters
PrestoDB -- the PrestoDB project, distinct from Trino
Presto Foundation formation -- Linux Foundation hosting and founding members
Starburst Documentation -- Starburst Enterprise and Galaxy features beyond open-source Trino
Starburst Galaxy -- the managed SaaS offering and its published packaging
Dremio Query Acceleration -- Reflections (raw, aggregation, starflake) and how the optimizer selects them
Dremio Autonomous Reflections -- automatic Reflection creation and maintenance
Spark SQL Performance Tuning -- adaptive query execution, join strategies, and partition coalescing
Spark SQL Programming Guide -- Spark SQL, DataFrames, and Datasets
Flink Table API and SQL -- streaming SQL, continuous queries, and unified batch/stream semantics
DuckDB Iceberg Extension -- reading and writing Iceberg from DuckDB, including catalog attachment
DuckDB Workload Tuning -- memory limits, threads, and larger-than-memory processing
DuckLake -- the lakehouse format that keeps metadata in a SQL database, and the ducklake DuckDB extension
MotherDuck Pricing -- published units for managed DuckDB
"Big Data is Dead" -- the widely cited argument that typical analytical datasets are small; published by MotherDuck and written by its chief executive, so read as informed advocacy rather than neutral analysis
BigQuery Pricing -- on-demand per-byte pricing versus capacity-based slot pricing and commitments
Amazon Redshift Pricing -- Redshift Serverless RPU-hours and provisioned node pricing
Databricks Pricing -- DBU rates by workload and warehouse type
Snowflake Compute Cost -- credit consumption, warehouse sizing, and per-second billing with a minimum