impact of parquet layout on #powerbi direct lake performance fully automated testing using #dbt and gitthub runners djouallah.github.io/fabric-dbt-b...

I spent some time testing writing to Iceberg REST catalogs across vendors. Same code, seven catalogs. Code's public 👇 github.com/djouallah/te... #ApacheIceberg #DataEngineering #onelake #Lakehouse #Databricks #Snowflake #s3 #cloudflare #Microsoftfabric #duckdb

Porting TPC-DI to #duckdb and #deltlake github.com/djouallah/tp... #sql #dbt #dataengineering

First successful end to end run: #dbt + #DuckDB + #OneLake Iceberg catalog Storage has won !!!

This is (AFAIK) the first ever #apacheiceberg table written by #duckdb using #onelake catalog , thanks to opus 4.7 for finding the bug and providing a PR

Sneak peek at what’s coming with the #OneLake Iceberg Catalog 👀 This is what the power of open standards looks like write using #lakesail read using #duckdb with table name like a real Database :) #Lakehouse #C++ #Rust #MicrosoftFabric #ApacheIceberg

analytics as code, everything inside github #vscode #dbt #duckdb #apacheiceberg #wasm #claude djouallah.github.io/analytics-as...

Seriously impressive performance from #Polars with delta_sink to #onelake 🚀 Fully reproducible test: github.com/djouallah/Fa...

a sql script that export #deltatable metadata from a #ducklkake db never thought that #sql can do that !!!! github.com/djouallah/Fa...

#duckdb #apacheiceberg is becoming good enough to be useful the script will download zip files from the web, extract them, create 4 tables and generate a summary table for reporting all in pure #SQL github.com/djouallah/Fa...

Explaining how Python engines read and write #DeltaTable is not for the faint of heart. The theory is everything will depends on the delta kernet rust for read and write, but we are not there yet github.com/djouallah/Fa... #duckdb #delta_rs #datafusion #chdb #daft #polars #rust #lakesail

running #duckdb at 10 TB scale using #tpch like benchmark #delta #onelake #singlenode #MicrosoftFabric datamonkeysite.com/2025/10/19/r...

you are looking at #duckdb running tpch 1 TB with only 16 cores it used to crash even with 64 pip install duckdb --upgrade is an act of faith basically

Put together a small python package duckrun :) point it at a folder of SQL/Python files, define a pipeline, and it will create Delta tables in #OneLake with #DuckDB and #delta_rs github.com/djouallah/du...

actually #Microsoftfabric Datawarehouse automatically expose an Iceberg rest Catalog thanks to #duckdb UI extension, you can see proper catalog

2 months ago, I got access to a beta release of #onelake #Apacheiceberg REST Catalog, first thing I run it with #duckdb 😀

storage format should not be tied to #SQL logic, #duckdb got it so right !!! but a bit sad that #deltalake is left behind :(

first #apacheiceberg table written by #duckdb

good news #duckdb added support for reading and writing geometry data type Bad news : other Fabric engines don't support it yet, so it is not very useful for now :(

#duckdb is the goat github.com/djouallah/Fa...

Writing #ApacheIceberg in Azure is not particularly hard, but you do need a catalog (essentially a database). For simple tests, you can use an in-memory DB #ADLS #opentableformat #PyIceberg.

I hope this is a fair subjective assessment of the current state of #Python data processing engines

The first solar farm I worked on , 8 years ago :) #PowerBI #Python djouallah.github.io/AEMO-POWERBI/

#Powerbi compression is insane !!!

how to connect to #onelake using duckdb UI step 1 : install azure cli in your laptop step 2 : login to your account step 3 : load credential in Duckdb CREATE or replace SECRET onelake( TYPE azure, PROVIDER credential_chain, CHAIN 'cli', ACCOUNT_NAME 'onelake' ); enjoy

this is a very nice problem to have in #PowerBI :)

We don’t know how it happened, but one day in 2025, the test completed without errors on a single node. Nothing was ever the same after that

worst thing about iceberg is solved !!! #duckdb

ok, maybe my laptop is horrible, but reading the data using mounted storage from #onelake is faster than reading the same data stored in my laptop, WTF !!!