Reading files

Files in an object-storage catalog (S3, GCS, Azure Blob Storage or Cloudflare R2) are read with DuckDB's file functions. The first argument is a path that starts with the catalog name. Thunderduck turns it into the real bucket address and uses that catalog's own credentials, so no s3:// URLs and no keys appear in your SQL.

SELECT * FROM read_parquet('raw_files/events/2026/*.parquet');

Here raw_files is the catalog's name. To connect a bucket, see S3, GCS, Azure & R2.

Function	Reads
`read_parquet`	Parquet files
`read_csv`	CSV and other delimited text
`read_json`, `read_ndjson`	JSON and newline-delimited JSON
`read_xlsx`	Excel workbooks (`.xlsx`)
`read_text`	Whole files as text, one row per file
`glob`	The list of files matching a pattern

Lance datasets ('raw_files/x.lance') and vector search are covered in Lance datasets.

Paths, wildcards and lists

All of these functions take the same kind of path:

One file: 'raw_files/exports/customers.csv'
A wildcard: * matches within one folder and ** across folders, for example 'raw_files/events/**/*.parquet'. [...] matches one character from a set.
A list of files: ['raw_files/a.csv', 'raw_files/b.csv']. Every entry must start with a catalog name.

A few things are refused, so a query can only read your own catalogs:

a path that doesn't start with one of your catalog names, including full URLs (s3://…, https://…) and server paths (/…)
.. segments, and ? or # anywhere in the path (use * instead of the ? wildcard)
a computed path such as 'raw_files/' || name

The full list is in Accepted SQL.

read_parquet

SELECT order_date, sum(amount) AS revenue
FROM   read_parquet('raw_files/orders/*.parquet')
GROUP  BY 1;

Useful options:

Option	What it does
`hive_partitioning = true`	Turns `key=value` folders into columns (on by default when the path looks partitioned)
`union_by_name = true`	Combines files whose columns differ, matching them by name
`filename = true`	Adds a `filename` column showing which file each row came from

Parquet stores column statistics, so a filter on a column that the files are sorted or partitioned by reads only the files and row groups that can match.

read_csv

SELECT * FROM read_csv('raw_files/exports/customers.csv');

The delimiter, header and column types are detected automatically. Override them when the guess is wrong:

SELECT *
FROM   read_csv('raw_files/exports/customers.csv',
                header  = true,
                delim   = ';',
                columns = {'id': 'BIGINT', 'name': 'VARCHAR', 'signed_up': 'DATE'});

Option	What it does
`header`	Whether the first line holds column names
`delim`	The separator, for example `','`, `';'` or `'\t'`
`columns`	Column names and types, skipping detection
`dateformat`, `timestampformat`	How dates and timestamps are written, for example `'%d/%m/%Y'`
`ignore_errors = true`	Skips lines that can't be parsed instead of failing
`all_varchar = true`	Reads every column as text

read_csv_auto is the same function with detection forced on.

read_json and read_ndjson

SELECT * FROM read_json('raw_files/logs/*.json');
SELECT * FROM read_ndjson('raw_files/events/*.jsonl');

read_json accepts a JSON array of objects or one object per line, and works out which from the file. read_ndjson is for newline-delimited files, one object per line, which is the common format for logs and event streams.

Nested objects come back as STRUCT columns and arrays as LIST columns, which you can reach with dot notation and unnest:

SELECT event.user.id AS user_id, unnest(event.tags) AS tag
FROM   read_ndjson('raw_files/events/*.jsonl') AS event;

Option	What it does
`format`	`'auto'`, `'array'` or `'newline_delimited'`
`columns`	Column names and types, skipping detection
`maximum_object_size`	Raise it for very large single objects

read_xlsx

Excel workbooks are read with read_xlsx. There is no read_excel function.

SELECT * FROM read_xlsx('raw_files/finance/budget.xlsx');
SELECT * FROM read_xlsx('raw_files/finance/budget.xlsx', sheet = 'Q2', range = 'A1:F200');

Without sheet, the first sheet is read.

Option	What it does
`sheet`	The sheet to read, by name
`range`	A cell range such as `'A1:F200'`, to skip titles or notes around the table
`header`	Whether the first row holds column names
`all_varchar = true`	Reads every column as text
`ignore_errors = true`	Turns cells that don't match the column type into `NULL`
`stop_at_empty`	Stops at the first empty row (on by default unless you give a `range`)
`empty_as_varchar`	Types columns that have no values as text, instead of `DOUBLE`

Excel has no integer type, so whole numbers arrive as DOUBLE. Cast them before joining on them:

SELECT CAST(id AS BIGINT) AS id, name
FROM   read_xlsx('raw_files/finance/budget.xlsx', sheet = 'Q2');

read_text

read_text returns each file as one row: filename, content, size and last_modified. It's useful for files that aren't tables, or to parse a format yourself.

SELECT filename, size, last_modified
FROM   read_text('raw_files/contracts/*.txt');

glob

glob lists the files that match a pattern, one row per file, in a column named file. Use it to check what a wildcard will pick up before reading it:

SELECT file FROM glob('raw_files/events/2026/**/*.parquet');

The paths come back as full storage URLs (s3://…), not catalog paths.

Keeping a result

Every read happens at query time. To keep a copy, for example to avoid re-parsing a large CSV, write it to a table:

CREATE TABLE lake.default.customers AS
SELECT * FROM read_csv('raw_files/exports/customers.csv');

See Getting data in for when to copy and when to keep querying in place.

Getting data in Exporting files