Guides
DuckDB + Iceberg on AWS Glue

DuckDB + Apache Iceberg on AWS Glue

This guide covers how to read and write Apache Iceberg tables in AWS Glue from DuckDB: the setup, the SQL, the IAM permissions, and the limits we hit running this in production. Everything in the first part is plain open-source DuckDB (we run 1.5.x). The last section shows the same thing on Thunderduck, where the setup is done for you.

1. Load the extensions

INSTALL iceberg;  LOAD iceberg;
INSTALL httpfs;   LOAD httpfs;
INSTALL aws;      LOAD aws;

aws is the easy one to forget. It provides PROVIDER credential_chain in the next step. DuckDB will autoload it on first use, but that is a network download in the middle of your first query. Install it up front, or bake it into your image.

2. Create an S3 secret, then attach Glue

CREATE SECRET glue_creds (
    TYPE S3,
    PROVIDER credential_chain,
    REGION 'us-east-1'
);
 
ATTACH ':' AS glue (TYPE ICEBERG, ENDPOINT_TYPE 'GLUE');
  • Order matters. The secret must exist before the ATTACH. Otherwise the attach fails with Could not find a valid storage secret (s3 or aws).
  • credential_chain uses whatever AWS identity the process already has: an ECS task role, an EKS service account, or a profile on your laptop. You don't paste any keys.
  • ':' means "the default Glue catalog of the account these credentials belong to". The same literal works in any account, so there is nothing to configure per environment.

glue is now an attached catalog. Each Glue database is a schema inside it.

3. Find and read tables

SHOW TABLES FROM glue.analytics;
 
DESCRIBE glue.analytics.orders;
 
SELECT order_date, count(*) AS orders, sum(amount) AS revenue
FROM   glue.analytics.orders
WHERE  order_date >= DATE '2026-09-01'
GROUP  BY 1
ORDER  BY 1;

DuckDB reads the table's current snapshot. It uses partition values and per-file column statistics to skip data files that cannot match the WHERE clause, so filters on the partition or sort column are much cheaper than filters on other columns.

To look at the Iceberg structure itself:

-- Snapshot history: one row per commit
SELECT * FROM iceberg_snapshots('glue.analytics.orders');
 
-- Manifests and data files of the current snapshot
SELECT * FROM iceberg_metadata('glue.analytics.orders');

4. Create a table: the location trap

This looks right and fails:

CREATE TABLE glue.analytics.orders_daily AS
SELECT date_trunc('day', created_at) AS day, count(*) AS orders
FROM   glue.analytics.orders
GROUP  BY 1;
400 InvalidInputException: Location information cannot be null
while creating an iceberg table

Glue's Iceberg endpoint requires a storage location for every new table. DuckDB does not send one (duckdb-iceberg#549 (opens in a new tab)). Glue also does not fall back to the database's LocationUri; we tried it on a database with a location set and got the same error.

The fix is to give the table its location explicitly:

CREATE TABLE glue.analytics.orders_daily
    WITH ('location' = 's3://my-lake/analytics/orders_daily/')
AS
SELECT date_trunc('day', created_at) AS day, count(*) AS orders
FROM   glue.analytics.orders
GROUP  BY 1;

WITH ('location' = …) comes between the table name and AS, and needs DuckDB 1.5 or later. Use one prefix per table. Two tables writing to one prefix will interleave their files.

5. IAM permissions

For reading, the role needs these Glue actions on the catalog, the database and its tables:

glue:GetCatalog   glue:GetCatalogs
glue:GetDatabase  glue:GetDatabases
glue:GetTable     glue:GetTables
glue:GetPartition glue:GetPartitions

It also needs s3:GetObject and s3:ListBucket on the table locations.

For CREATE TABLE, add glue:CreateTable and glue:UpdateTable, plus s3:PutObject on the new table's prefix. To check a role without running anything, use aws iam simulate-principal-policy. It reports implicitDeny for each action the role is missing.

If Lake Formation governs the database, its grants apply as well, and IAM alone will not be enough.

Known limits

  • DROP TABLE does not work through the Glue endpoint. The drop request fails even with full permissions. Remove the table with aws glue delete-table, then delete its files with aws s3 rm --recursive.
  • A missing database looks empty. SHOW TABLES FROM glue.typo_db returns no rows instead of an error. If a schema seems empty, check its name first.
  • No information_schema on an Iceberg catalog. Use SHOW TABLES and DESCRIBE, as above.
  • Glue database names can only contain lowercase letters, digits and _. No dots and no dashes.
  • Identifiers are case-insensitive in DuckDB, quoted or not. "Orders" and orders are the same table.
  • Every write is a new Iceberg snapshot. Many small writes leave many small files and a long snapshot history. Compact the table and expire old snapshots on a schedule.

The same thing on Thunderduck

Thunderduck runs DuckDB with exactly this setup. You create an Iceberg on Glue catalog by giving it a name, and the platform handles the rest:

  • Extensions, secret and ATTACH are already in place for every query. You don't create any credentials.
  • The Glue database is created for you, named after the catalog, with its own prefix in the warehouse bucket.
  • Table locations are added automatically. CREATE TABLE … AS SELECT works without a WITH clause. A location you type yourself is accepted only if it is inside the catalog's own prefix.
  • Compaction, sorting and snapshot expiry run from the LakeOps page instead of your own scheduled jobs.
-- catalog "lake" on Thunderduck: no ATTACH, no secret, no location
CREATE TABLE lake.default.orders_daily AS
SELECT date_trunc('day', created_at) AS day, count(*) AS orders
FROM   sales_pg.public.orders
GROUP  BY 1;
 
SELECT * FROM lake.default.orders_daily ORDER BY day DESC LIMIT 7;

Each query runs in its own DuckDB worker. That worker is sized from the query plan and can only reach your own catalogs. Here, the source table is read from PostgreSQL in place, and the result is written to Glue as an Iceberg table.

Migrating from BigQueryOverview & addressing