The last year of DuckDB enhancements feel like the shift from in-process execution engine (which it is phenomenal at) to an engine that can serve as the foundation of a cloud data warehouse. I know the founders were reticent about not wanting to build that, but I have a feeling it is in the works.
jbmsf•Aug 17, 2026
I've been using MotherDuck as a data warehouse for some time now. No regrets.
ethagnawl•Aug 17, 2026
UPDATE: Removed uninformed comment about the relationship between DuckDB and MotherDuck.
Tomte•Aug 17, 2026
Wait, what? I‘ve always assumed MotherDuck is the company by the main developers of DuckDB? They are unrelated?
ethagnawl•Aug 17, 2026
I was wrong. Apologies.
> They decided to work together, and DuckLabs would become a co-founder of the new endeavor.
Was hoping to see procedural functionality like PL/pgSQL... regardless, an astonishing project overall.
srameshc•Aug 17, 2026
I <3 DuckDB. It has become one of my go to tools for storing, data processing , integrations and now even graph. More importantly it's fun to use because it is so portable. Looking forward to v2.
fragebogen•Aug 17, 2026
Ditto! Very happy with the upcoming async support! Now it'll be a nice little db for serving http traffic as well!
d3Xt3r•Aug 17, 2026
What advantages does it have over SQLite in your use cases? Can you give any examples?
briHass•Aug 17, 2026
Not OP, but for me, the lack of essentially any type system in SQLite makes it a total no-go for storing data long-term or that more than one application needs to access. Date/time being an especially painful footgun in SQLite.
I view SQLite as something a single application can use for storing state/settings/misc operational data instead of directly writing files, especially if the data being stored is relational or needs ACID. As soon as the data itself has meaning and structure per se, you're better off with something that can help enforce and describe the data: rich datatypes, foreign keys that aren't optional, etc.
est•Aug 17, 2026
This is cool
What about the runtime size? I care this because I intend to run a stripped WASM version of DuckDB in browser.
zcw100•Aug 17, 2026
What do you plan on stripping and what's your target? The Emscripten based build is ~10Mb. I have a component build so I'd be interesting on how you'd like to break it up.
datadrivenangel•Aug 17, 2026
They're mostly using optional extensions for this new stuff, so the binaries are still small, like ~20mb?
jtbaker•Aug 17, 2026
DuckDB is one of the things I've been most excited about in a long time. Introduced it to projects at 3 companies since 2023, greatly lowering resource requirements and running it in a variety of environments. Just having the ability to do out of core bigger than memory data processing on lower end consumer grade hardware is remarkable.
Thanks to the team for everything!
throwaw12•Aug 17, 2026
Curious to learn more about how people are using it?
Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?
arealaccount•Aug 17, 2026
We use DuckDB WASM with parquet to build dashboards in-browser. It's cool to be able to write SQL directly in a browser and not have to rely on REST/Graphql/etc to access the data layer.
jayct•Aug 17, 2026
curious if you're using something mostly-out-of-the-box to layer on visualizations for your dashboards?
relatively new to duckdb, love it so far, looking at alternatives for downstream visualization. so far just exporting datasets and piping into python scripts.
mediaman•Aug 17, 2026
I do something similar and just use echarts. Very happy with it.
jtbaker•Aug 17, 2026
For a schema-first (vs. code first) approach (which I think would be a sweet spot for agent driven dashboarding), I'd suggest looking at https://vega.github.io/vega-lite/ or https://vega.github.io/vega/. A little higher level than full D3 but gives you a little higher level approach.
drums8787•Aug 17, 2026
We use WASM DuckDB as the target for an in-browser agentic feature. Generated SQL runs against the user's individual tables that then feed in-browser dashboards.
Excellent performance.
jtbaker•Aug 17, 2026
I've got a couple of different use cases:
- ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously.
- A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consumable format and upload it to a bucket as a .db file.
- A SvelteKit app that used the node duckdb bindings to attach to the .db on the bucket and explore the results through a suite of BI tools. These tables have millions of rows, and would be pretty heavy to store in PG. The DuckDB version works really, really well.
tccole•Aug 17, 2026
Hell yeah; a fellow sveltekit fan.
staticautomatic•Aug 17, 2026
Similar here. Lots of places where we replaced Pandas with DuckDB for transformations. Also have scriptable custom dashboards running on top of BigQuery data pre-aggregated and extracted to parquet on GCS. It's way faster and the only limiting factor is your viz library. It was pretty easy to build and the only big gotcha I encountered was finding, somewhat counter-intuitively, that it's often best minimize partitioning.
jtbaker•Aug 17, 2026
> the only big gotcha I encountered was finding, somewhat counter-intuitively, that it's often best minimize partitioning.
For parquet, I think with partitioning, it's really important to be mindful of the ordering of the data within the parquet file and also the query patterns of the main use cases. A little hard to generalize well to every pattern I guess.
peesem•Aug 17, 2026
maybe a niche use case but i've found it's perfect to store & query random trivia/gameshow questions based on filters for my personal clones of things like Family Feud and Jeopardy
tccole•Aug 17, 2026
Yes.
I have used it with WASM for some web applications for web use. I have also used with locally for querying 100 gigs of data. And I have used it in the cloud as the serverless gold layer for Apache superset.
arpinum•Aug 17, 2026
ETL from DynamoDB into Ducklake
phqb•Aug 17, 2026
I'm using duckdb/duckdb-go as query engine for my Go services: moving hot data from Postgres to Parquet files on S3 or to Iceberg; querying cold data on Iceberg, ... instead of using different Go libraries.
malshe•Aug 17, 2026
I use it locally with parquet files
AlfeG•Aug 18, 2026
Realtime full MSSQL database mirroring into DuckDb to do a complex reporting. Everything is in-process. DuckDb database mapped to temp storage and recreated on app restart. Still order of magnitude faster then doing a direct query over MSSQL Server (2ms vs 40+ seconds on same query).
Some devs in team still cannot believe that there is no cheating, that it's possibe, that some 60Mb DB can do queries faster then MSSQL Server with just around 250Mb+ of memory overhead.
(.Net 10 + DuckDB.NET package)
markus_zhang•Aug 18, 2026
I actually used it in an interview. I downloaded the csv/parquet file and load it into DuckDB.
In the future, I also plan to use it for testing production data pipelines: imagine you have a streaming cdc pipeline running in development env, and at the end you can dump every parquet into DuckDB as a verification — the end result should be the same. I could also use the same Database for testing, but I like DuckDB somehow.
fifilura•Aug 17, 2026
My favourite is AWS Athena (backed by Trino).
"If we use this we get indefinite RAM indefinite CPU and do not need to host a server".
I had an impression that DuckDB was not great at distributing work to other machines, but good at doing it locally? Am I wrong?
ericpauley•Aug 17, 2026
Athena + Clickhouse has been an absolute game changer for us. Perfect combo for OLAP + deeper filtering that we can’t necessarily pre-index for.
abirch•Aug 17, 2026
DuckDB out of the box may not be great. But you have DuckLake, Quack, and even DeepSeek made their own distributed DB based on DuckDB: https://github.com/deepseek-ai/smallpond
jtbaker•Aug 17, 2026
I don't think DuckDB itself can coordinate work across multiple nodes. But you could put it behind an HTTP layer and scale horizontally based on resource utilization?
sv123•Aug 17, 2026
DuckDB is so cool, game changer when it comes to local data processing.
hnlb53nrpg•Aug 17, 2026
Same problem, different day
anentropic•Aug 17, 2026
Please document the new "extensible PEG-based parser" for extension authors
badatnames•Aug 17, 2026
Looks like an awesome release, but the smell of AI from that post is horrid.
Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.
drannex•Aug 17, 2026
I don't really get an AI smell on this, in fact I see multiple parts that an AI would have corrected - grammatical issues, personal writing quirks, etc.
Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.
ostwilkens•Aug 17, 2026
"A major version bump is not something we do lightly, and it is not just ceremony"
For me, this sentence pops out like a sore thumb
jeffbee•Aug 17, 2026
"We reimplemented ICU" U+1F631 FACE SCREAMING IN FEAR
amluto•Aug 17, 2026
I’m currently contemplating that MySQL apparently cannot do an INSTANT change of the collation of an unindexed column, even though, AFAICT, it has no effect whatsoever on the on-disk format or any data structure at all except for the metadata saying what the column type is.
I do not enjoy dealing with text encodings and collations in databases.
logancbrown•Aug 17, 2026
Funny to think one of my favorite software projects this decade is basically "lets make it easy to host your own OLAP database".
amluto•Aug 17, 2026
If I could have a pet feature added to DuckDB, it would be some form of native ordered table. In a database like Clickhouse or any of the dedicated time series DBMSes or log stores, there’s a built-in concept that a table might have an order, and the database will optimize based on the order. But, for databases that are logically just bags of rows (traditional DBMSes and also DuckDB [0]), you either need an index or you need to rely on full table scans or at least scans of big blocks. DuckDB does the latter really well, but I think it would be quite nice for some workflows to have explicit ordering. Also, I bet compression could work a lot better with ordering hints.
All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try.
[0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.
I’m sort of amazing that nothing in that thread mentioned ClickHouse, even just as a product that has this capability and from which one might take syntactical inspiration.
Super excited about Quack (partially due to the name). I use duckdb for both analytics and runtime, but I do have to serve/handle/manage a giant, multi-GiB duckdb file as effectively a runtime artifact[1]. I'm aware that this isn't the _perfect_ database for this, but the mix of it being fast, having spatial support, sane coding interfaces, great dbt integration, and me being able to do everything between "run a giant several hundred step dbt pipeline" to "query the output of said pipeline" to "read/query a csv on disk" with the exact same tool is just so nice. If I could centrally manage said asset more akin to a traditional database, I'd be very happy.
I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I really like way too many things about duckdb to replace it.
I built a platform for some midsize companies in a specific vertical that is basically a data warehouse with some LLM-driven dashboarding and query tools on top. Typical data size 5-150gb. So I built a service layer around duckdb, where each tenant gets their own duckdb. I'm also in the boat of knowing that duckdb is not the perfect solution for this (the classic use case is running it against local data on a laptop), but there's so much I like about it, and it's really nice that each tenant can have total separation with custom schemas and that it's straightforward to pile data into object storage and form your own lake.
And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.
tccole•Aug 17, 2026
I feel like familiarity and ease of use and “good enough” beat out the perfect db for the job in many occasions.
allan_s•Aug 17, 2026
As somebody new to this and with a use case very similar to yours , what would have been a more suitable solution for this ?
The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
mediaman•Aug 17, 2026
Clickhouse has a more intentionally built ingestion system. Duckdb has concurrency limits so you can't have a writer and a reader on the same file if they're not the same process (multiple readers is fine).
But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).
With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".
jamesblonde•Aug 17, 2026
DuckDB's sweet spot is for 'smallish' lakehouses. So, ingestion should not be via duckdb, but rather something like Python/DLTHub for small scale or Spark for large scale or Kafka/Debezium/Flink for streaming data.
MatthausK•Aug 18, 2026
The CEO/Co-Founder of dltHub/dlt here.
For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.
Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.
boc•Aug 17, 2026
Ducklake supports postgres for the catalog, so you get the postgres concurrency benefits + duckdb engine to read the parquet files in the bucket.
8note•Aug 17, 2026
another variant:
i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it
is your setup running into problems that makes you need something more?
smw•Aug 18, 2026
2GB is so tiny it's almost irrelevant when talking about analytics?
otter-in-a-suit•Aug 18, 2026
There's a few options.
Clickhouse, as I mentioned, can be a good final layer, as can postgres.
You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.
duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).
All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.
For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.
ljosifov•Aug 18, 2026
Similar. Noticed DuckDB ever since an old article 'what db should I use' for local small data warehousing. The author was blown away that DuckDB seemed super naturally quick. It was I think columnar store + compression facilitated that? It made duckdb load compressed + on the fly decompress = faster than even reading the uncompressed data. Had forgotten most of it. But was used to mmap-ed files + columnar storing of Kdb. Was pleased that the author was clued to notice the power of that.
Then more recently I was given a somewhat random task to organise a motley collection of web scrapes, historic data, realtime data, data to be fetched on demand dispersed in semi-random collections. DuckDB as backing store + Claude Code that I discovered in Nov-2025 (with suitable skills and schemas) = a data agent where I could tell CC *in English* complicated data requests!? And CC would write glue shell and python code, write SQL and run it against DuckDB that housed most of the data, fetch new data, munge join filter, and present it to the user as "your data is in data slash blah". It seemed a miracle unfolded in front of my eyes! So yeah - fond of DuckDB. :-)
Hey this is neat and I just found out you're from BA, Argentina. I would love you be part of dataengine.ar and maybe talk about arc?
ignaciovdk•Aug 17, 2026
Hey hey. I love to. I'm from Argentina, and I say that Arc is proudly make from Latam, but I live in Costa Rica.
dmix•Aug 17, 2026
> to "read/query a csv on disk"
I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.
grvdrm•Aug 18, 2026
Nice post and you have me checking out your broader site and product!
drannex•Aug 17, 2026
Really looking forward to that new Async system, especially when reading/querying against thousands of parquet files. This is going to monumentally affect me and my work - I have to query against millions of massive parquet files and the speed has already been rather wonderful, but if those metrics are to be even 100% in range, this is going to make life so much better.
DuckDB is seriously an incredible utility.
giovannibonetti•Aug 17, 2026
Disappointed, since I was expecting they would rewrite the implementation from C++ to Zig. I bet that would increase the number of positive pull requests they get, since most developers prefer to stay away from C++ nowadays.
brunoborges•Aug 17, 2026
How does DuckDB compares with PostgreSQL / MariaDB ?
tuvix•Aug 17, 2026
DuckDB is an in-process (now I guess less so with Quack) OLAP database for analytical workloads. PostgreSQL or MariaDB are OLTP row-oriented databases that are great for application/transaction-focused workloads but are less great when you want to query across a giant amount of data.
tomjakubowski•Aug 17, 2026
DuckDB is much like SQLite, but built for OLAP workloads: it's in-process, with a single file format on a disk, and (unlike SQLite) the data is stored in columns for better OLAP performance.
Like SQLite, concurrency options are limited compared to client-server databases like Postgres or MariaDB. DuckDB 2.0 will be adding a client-server mode with the Quack protocol which can allow for greater concurrency.
But you can also use DuckDB as a multitool to connect to and query all kinds of other data sources from one connection. Being able to pull in data from Postgres, CSVs or parquets on a file system or S3, and JSON returned by a web API, and then query across all of it in one place, can be quite handy for ad-hoc data analysis and exploration.
thejosh•Aug 17, 2026
I've been working on a demo database project, and have been really impressed by the UI. So glad they decided to put more effort into it, it has made building a "follow along" tutorial really nice.
dzonga•Aug 17, 2026
well done to the duckDB team - one of the features I'm waiting for is real time materialized views.
d33•Aug 17, 2026
Are there improvements in how memory_limit works? I often had DuckDB get OOM killed because it went beyond its limit. It's definitely one of the reasons why I usually have an AI tune the environment for my datasets.
noodlesUK•Aug 17, 2026
I look forward to DuckDB being the engine that underpins the next gen of analytical data tooling. DuckLake already looks amazing and with the Quack protocol seems like it will be a great natural fit for lots more types of tooling, such as sensor data etc.
orthoxerox•Aug 17, 2026
DuckDB keeps getting better and better. I wonder when something like Apache Gluten will pick it up as a backend.
dartharva•Aug 17, 2026
Would be really cool if they were to add statistical functions too. I'd jump at the chance of getting to use this over pandas
biophysboy•Aug 17, 2026
You might know this already, but you can query pandas/polars/arrow tables directly w/ duckdb and use whatever stats packages you feel like alongside it in the same python script. I feel like they do a decent job sticking to the simpler statistical fans that make sense in sql.
dartharva•Aug 18, 2026
Sorry, I meant duckdb-cli, not duckdb in Python.
1egg0myegg0•Aug 18, 2026
Oh, then you might like these DuckDB community extensions that provide stats functions!
which statistical functions do you want? i’m curious because i love duckdb and use it for a variety of projects but always want to learn more about how to use tools better.
dartharva•Aug 18, 2026
Basic regression and interpolation for starters; perhaps some classification and outlier detection too. Disclaimer: I self-studied applied DuckDB for my job, I don't know if these already exist.
formerly_proven•Aug 17, 2026
In terms of project trajectory this is also an interesting contrast. DuckDB is "SQLite, but for OLAP".
DuckDB 1.0 was in 2024. DuckDB 2.0 (new API, new storage format, new ...) is in 2026.
SQLite has been 3.x since 2004.
remywang•Aug 17, 2026
If you like DuckDB, please consider funding DB research [1]!
With some of these changes, it appears to be encroaching on clickhouse territory. Or are they still very different products?
te_chris•Aug 17, 2026
Well, I can tell you this is making me actively reconsider dropping duckdb for ch as we go from prototype to prod
encoderer•Aug 17, 2026
Definitely encroaching.
Our last product (SaaS observability) uses Clickhouse.
Our next product (self-hosted observability) uses DuckDB.
adhamsalama•Aug 18, 2026
I was building an OpenTelemetry observability platform in a single executable file (in Golang) and was using ClickHouse as the database, and replaced it with an embedded DuckDB, so now it's truly a single executable file platform.
I built a browser tool for querying local Parquet, CSV, JSON, Excel, Arrow, Avro, DBF, and SQLite files with DuckDB-WASM.
Most probably after DuckDB v2.0 release I will revamp my tool as well.
rrgok•Aug 17, 2026
Sometimes I prefer DuckDB query language to MySQL or Postgres. It is not even about performance, it just feels right. I just connect to my DBS from DuckDB just to use its query language. And besides it gives you a single language frontend.
onnimonni•Aug 17, 2026
This! For example I tried to ask PosgreSQL to Add support for QUALIFY keyword since it makes window queries much more simpler to use. Postgres didnt want it. DuckDB maintainers are always thinking about the DX and all nice syntactic sugar.
hans_castorp•Aug 17, 2026
The Postgres devs rejected it, because it wasn't part of the standard back then, and apparently work was going on, to include it into the standard.
Now that it's included in the standard, there is work to implement it:
How stable is DuckDB these days? I remember it was very buggy compared to Clickhouse.
nikolay•Aug 17, 2026
It's sad that almost no migration framework supports DuckDB, and overall support is highly limited, but it's a great product I've been using daily for 6 months without encountering a single issue. I hope v2 brings DuckDB more attention and increases third-party support!
barrrrald•Aug 17, 2026
We've built our whole platform around DuckDB at Hex. Our product truly could not exist without it.
therealdrag0•Aug 17, 2026
Hate to bring it up, but 10,000 commits in less than 6 months is a lot. Is AI a major contribute here?
Is AI use for accelerated development of a beloved tool like DuckDB enough to quiet lingering doubters?
nh2•Aug 18, 2026
If you merge PRs that have commit mesages like this, it's easy to arrive at 10000 commits in 6 months:
rename to NodePointer instead
format
Revert "format"
Revert "rename to NodePointer instead"
rename to OptionalNodePtr
woops
update comment
slot renames
more renames
They seem to squash PRs using the pr description as commit message. So this sounds like a good approach to me.
mastermage•Aug 18, 2026
i love woops
luizfelberti•Aug 17, 2026
> A repository is a name, a URL prefix, and one or more RSA public keys that are trusted to sign the extensions served from it.
Is it too late to beg Hannes and Mark to let us have something like minisign instead of RSA? :)
Very excited for 2.0, congrats folks!
dangoodmanUT•Aug 17, 2026
It's funny to me that we still don't have incremental materialized views. All of the parts are there (export state, agg_state (forget fn name), finalize). I wonder if they're avoiding an explicit war with clickhouse or something. I do recall they mentioned they want to add this to ducklake.
Incremental MVs are ClickHouse's best feature. If DDB adds this, the last moat is distributed query execution.
ghshephard•Aug 17, 2026
Have you run into scenarios where a simple view doesn't accomplish what you require? I always feel like views do everything I want - because the speed is so great, the full recompute isn't that big a deal.
Maybe it's a bigger deal when you have multiple users/and or more repeated queries against something that's really expensive?
Pedro_Ribeiro•Aug 18, 2026
I've had some use cases with low latency requirements and frequent data updates. If you want to serve data on a frontend fast it's often not feasible to recompute everything every time
<3 duckdb run realtime analytics pipeline using a (moderately popular) stream processing engine I built on top of DuckDB. Looking forward to what duckdb provides in terms of perf out of the box!
DuckDB has been a fantastic engine to build on (in python), and processes thousands of events per second, day in an day out, without issue
andyferris•Aug 18, 2026
I was curious to see they are advertising OLTP-like transactional processing speed. It would be super convenient to have one DB for OLTP and OLAP purposes!
Has anybody here tried using it that way?
(though I don't see any way to deal with write skew and other transactional guarantees - no SERIALIZABLE optimisitic concurrency, no SELECT FOR UPDATE pessimistic concurrency, etc)
gw32•Aug 18, 2026
> The VARIANT type shipped in DuckDB v1.5, and the way to think about it is JSON on steroids. Basically, imagine if JSON were fast. [...] DuckDB automatically detects the common structure hidden in your semi-structured data and “shreds” it, so it compresses well in storage
I am really looking forward to this hitting v2.0. I can't stand uncompressed JSON - so space-inefficient. But heterogenous JSON in parquet files is such a pain because of schema differences causing fields to be silently dropped. Having DuckDB solve this is exactly what I've been looking for.
jinjin2•Aug 18, 2026
I’m mostly using Exasol these days (the concurrency and smooth scaling to multi-node is just too seductive), but with the introduction of Quack I might take another look at DuckDB. I’ll have to see how well it handles many agents reading and writing to it concurrently.
ivolimmen•Aug 18, 2026
I am a crew member of Joy Of Coding (https://joyofcoding.org) where we invited Hannes to do a talk. He is a great speaker. Seeing this we will need to invite him another time!
mastermage•Aug 18, 2026
I love DuckDB genuinely more than sqlite even though they do completely different things but DuckDB has like for me the perfect mix between simplicity, embedded capabilities and expressiveness. (and actual Time and Date Types).
MILP•Aug 18, 2026
Love DuckDB. It’s so fast and portable! I mainly use the query engine as part of my ETL process for creating SQLite database. I’d love to pitch it at work, but we’re heavily invested in BigQuery, which makes it a bit difficult.
rubenfiszel•Aug 18, 2026
We have bet early on DuckDB and Ducklake for Windmill and couldn't be happier. The focus on server/client mode is interesting, it opens the way for orchestrators like ours to have "lite" workers/jobs for duckdb that connect to one central bigger beefy nodes and improve the overall efficieny. I'm very curious if benchmark shows that there are performance benefits to do so thanks to co-location and overall less cpu cycles wastes.
hermitcrab•Aug 18, 2026
I'm looking for a lightweight client-server database where I can connect 3 or 4 GUI clients to a single database and concurrently edit the database. Low transaction volumes (probably a few edits per minute). Would DuckDb + Quack be suitable?
43 Comments
> They decided to work together, and DuckLabs would become a co-founder of the new endeavor.
https://motherduck.com/about-us/
I view SQLite as something a single application can use for storing state/settings/misc operational data instead of directly writing files, especially if the data being stored is relational or needs ACID. As soon as the data itself has meaning and structure per se, you're better off with something that can help enforce and describe the data: rich datatypes, foreign keys that aren't optional, etc.
What about the runtime size? I care this because I intend to run a stripped WASM version of DuckDB in browser.
Thanks to the team for everything!
Are they downloading parquet files and running analyses locally, or are they connecting to Iceberg-like data lake and leveraging DuckDBs query engine capabilities or have you exposed an interface (REST, UI) to query your data?
relatively new to duckdb, love it so far, looking at alternatives for downstream visualization. so far just exporting datasets and piping into python scripts.
Excellent performance.
- ETL pipelines running on K8s nodes. Using their streaming processing engine means I can run smaller pods/nodes if needed, for datasets that may have required large dataframe-like transformations that may have buffered a big dataset into memory previously.
- A CLI distributed to an internal team to do a postprocessing step on a large modeling dataset - to get it into a consumable format and upload it to a bucket as a .db file.
- A SvelteKit app that used the node duckdb bindings to attach to the .db on the bucket and explore the results through a suite of BI tools. These tables have millions of rows, and would be pretty heavy to store in PG. The DuckDB version works really, really well.
For parquet, I think with partitioning, it's really important to be mindful of the ordering of the data within the parquet file and also the query patterns of the main use cases. A little hard to generalize well to every pattern I guess.
I have used it with WASM for some web applications for web use. I have also used with locally for querying 100 gigs of data. And I have used it in the cloud as the serverless gold layer for Apache superset.
Some devs in team still cannot believe that there is no cheating, that it's possibe, that some 60Mb DB can do queries faster then MSSQL Server with just around 250Mb+ of memory overhead.
(.Net 10 + DuckDB.NET package)
In the future, I also plan to use it for testing production data pipelines: imagine you have a streaming cdc pipeline running in development env, and at the end you can dump every parquet into DuckDB as a verification — the end result should be the same. I could also use the same Database for testing, but I like DuckDB somehow.
"If we use this we get indefinite RAM indefinite CPU and do not need to host a server".
I had an impression that DuckDB was not great at distributing work to other machines, but good at doing it locally? Am I wrong?
Here is a wild idea: is it really so hard to edit out sentences structured and punctuated like this - it's so painfully obvious and distracts from the content. The effect is real.
Writing similar to this: is quite a normal way of writing for technical articles - especially when you are trying to make the point clear and well organized.
For me, this sentence pops out like a sore thumb
I do not enjoy dealing with text encodings and collations in databases.
All that being said, I’m quite excited about DuckDB 2.0. I want to give the improved VARIANT support a try.
[0] Documentation on DuckDB’s native format is rather sparse AFAICT. But the DDL has nothing resembling an ordered table.
https://github.com/duckdb/duckdb/discussions/8444
I’m sort of amazing that nothing in that thread mentioned ClickHouse, even just as a product that has this capability and from which one might take syntactical inspiration.
I made a dry run extension a few months ago (https://github.com/aleda145/duckdb-dryrun), will be so nice to build it just once and know that it will always work.
Also urge anyone to make an extension, the template makes it quite smooth: https://github.com/duckdb/extension-template
I've partially solved this with separate databases for different steps in the data pipeline(s) and have even experimented with Clickhouse as a complete alternative, but I really like way too many things about duckdb to replace it.
[1]: If you care: https://skaldmaps.com/blog/2026/07/zip-codes-are-a-bad-spati...
And now with things like quack the sharp points around concurrency are relaxing, and it feels like the compromises of using it this way are disappearing.
The guy who first built the architecture made the same decision as yours (I.e one local duckdb for each tenant to work as a copy of big query/their data warehouse) and I dont know what the state of the art for this kind of use cases ?
But that's not too hard to work around. You can either have a single process that owns both writing and reading that file, or you can do a data lake where you post updates as parquet files into object storage, and duckdb handles the catalog. The Quack protocol also basically fixes this (though still in beta).
With Clickhouse, you can of course still have tenant separation, but you have to do it by managing users within Clickhouse that map to users/tenants of your main app, so that you can restrict SQL access by tenant to only their data store. Not a huge deal but I just like the Unix "it's just a file" simplicity of "Tenant A gets to run arbitrary SQL against their separate read-only, no-ATTACH duckdb file".
For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.
Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.
i put duckdb on a lambda and pointed it at s3 for the data. my data was closer to 2GB but the queries were quick and nearly free with superset pointed at it
is your setup running into problems that makes you need something more?
Clickhouse, as I mentioned, can be a good final layer, as can postgres.
You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.
duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).
All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.
For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.
Then more recently I was given a somewhat random task to organise a motley collection of web scrapes, historic data, realtime data, data to be fetched on demand dispersed in semi-random collections. DuckDB as backing store + Claude Code that I discovered in Nov-2025 (with suitable skills and schemas) = a data agent where I could tell CC *in English* complicated data requests!? And CC would write glue shell and python code, write SQL and run it against DuckDB that housed most of the data, fetch new data, munge join filter, and present it to the user as "your data is in data slash blah". It seemed a miracle unfolded in front of my eyes! So yeah - fond of DuckDB. :-)
Latter I read this https://openai.com/index/inside-our-in-house-data-agent/ and thought "but of course".
I discovered DuckDB looking for a way to analyze Nginx access.log's and it's an amazing tool. I believe it should be a standard tool like ripgrep for devs.
DuckDB is seriously an incredible utility.
Like SQLite, concurrency options are limited compared to client-server databases like Postgres or MariaDB. DuckDB 2.0 will be adding a client-server mode with the Quack protocol which can allow for greater concurrency.
But you can also use DuckDB as a multitool to connect to and query all kinds of other data sources from one connection. Being able to pull in data from Postgres, CSVs or parquets on a file system or S3, and JSON returned by a web API, and then query across all of it in one place, can be quite handy for ad-hoc data analysis and exploration.
https://duckdb.org/community_extensions/extensions/stats_duc...
https://duckdb.org/community_extensions/extensions/stochasti...
DuckDB 1.0 was in 2024. DuckDB 2.0 (new API, new storage format, new ...) is in 2026.
SQLite has been 3.x since 2004.
[1]: https://news.ycombinator.com/item?id=49336147
Our last product (SaaS observability) uses Clickhouse.
Our next product (self-hosted observability) uses DuckDB.
https://github.com/adhamsalama/nabatshy
I built a browser tool for querying local Parquet, CSV, JSON, Excel, Arrow, Avro, DBF, and SQLite files with DuckDB-WASM.
Most probably after DuckDB v2.0 release I will revamp my tool as well.
Now that it's included in the standard, there is work to implement it:
https://hackorum.dev/topics/51960#message-1125429
Is AI use for accelerated development of a beloved tool like DuckDB enough to quiet lingering doubters?
If every Ctrl+S is a commit, it'll go up fast.
"woops"!
Is it too late to beg Hannes and Mark to let us have something like minisign instead of RSA? :)
Very excited for 2.0, congrats folks!
Incremental MVs are ClickHouse's best feature. If DDB adds this, the last moat is distributed query execution.
Maybe it's a bigger deal when you have multiple users/and or more repeated queries against something that's really expensive?
https://github.com/turbolytics/sql-flow
DuckDB has been a fantastic engine to build on (in python), and processes thousands of events per second, day in an day out, without issue
Has anybody here tried using it that way?
(though I don't see any way to deal with write skew and other transactional guarantees - no SERIALIZABLE optimisitic concurrency, no SELECT FOR UPDATE pessimistic concurrency, etc)
I am really looking forward to this hitting v2.0. I can't stand uncompressed JSON - so space-inefficient. But heterogenous JSON in parquet files is such a pain because of schema differences causing fields to be silently dropped. Having DuckDB solve this is exactly what I've been looking for.