DuckDB
Read files in S3, GCS or Azure in place, and join databases in one model, through a DuckDB connection
A DuckDB connection runs DuckDB inside Credible and attaches the sources you give it. Use it to:
- Read files in object storage in place: CSV, Parquet and JSON in Amazon S3, Google Cloud Storage or Azure, as a single file, a folder glob (
path/*.parquet) or a whole subtree (path/**). - Join across databases: attach PostgreSQL, BigQuery and Snowflake side by side and join their tables in one model.
A DuckDB connection needs at least one attached database or bucket. For files that ship inside a package, you don't need a connection at all: see embedded data.
Add Cloud Files (S3)
The quickest way to read files from S3:
- In the connection form, choose DuckDB
- Click Cloud files (S3)
- Enter an Access Key ID and Secret Access Key that can read the bucket
- Optional: Region (default
us-east-1), a Custom Endpoint for an S3-compatible store such as MinIO (host and port, nohttps://), and a Session Token for temporary STS credentials
Then read files by their URL:
source: orders is my_files.table('s3://my-bucket/exports/orders.parquet')
source: events is my_files.table('s3://my-bucket/events/*.csv')Add an Attached Database
Click Attached database, give it a Database Name, and choose its type. The name is how a model refers to it, so it follows the same rules as a connection name, and it cannot be the same as the connection's own name.
| Type | What it needs |
|---|---|
| PostgreSQL | The same fields as a PostgreSQL connection |
| BigQuery | A service account JSON key. The service account also needs the BigQuery Read Session User role |
| Snowflake | The same fields as a Snowflake connection |
| Amazon S3 | Access key ID and secret access key, as in Cloud files |
| Google Cloud Storage (GCS) | An HMAC Access Key ID and Secret Access Key |
| Azure Blob Storage / ADLS Gen2 | A service principal (tenant ID, client ID, client secret, account name and file URL), or a SAS URL |
A table in an attached database is read through the DuckDB connection, prefixed with the attached name:
source: customers is my_duckdb.table('crm.public.customers')GCS HMAC Keys
DuckDB reads GCS with HMAC keys, not a service account JSON key. To create one:
- In the Google Cloud Console, open Cloud Storage > Settings > Interoperability
- Under Access keys for service accounts, click Create a key for a service account
- Choose a service account with Storage Object Viewer on the bucket
- Copy the access key and secret
Read files with a gs:// URL, such as my_duckdb.table('gs://my-bucket/path/*.parquet').
Azure
Choose an Auth Type:
- Service Principal (SPN): an app registration with Storage Blob Data Reader on the storage account. Enter its Tenant ID, Client ID and Client Secret, the Account Name, and the File URL to read, such as
https://account.blob.core.windows.net/container/path/**. - SAS Token: paste the full SAS URL, token included. The token needs read and list permissions (
sp=rl).
Both URLs can point at a single file, a glob (*.parquet) or a subtree (**). The https://, abfss:// and az:// forms all work.
From the CLI
To add the connection with cred add connection, put an entry like this in the JSON file:
[
{
"name": "my_files",
"type": "duckdb",
"duckdbConnection": {
"attachedDatabases": [
{
"name": "cloud_files",
"type": "s3",
"s3Connection": {
"accessKeyId": "...",
"secretAccessKey": "...",
"region": "us-east-1"
}
}
]
}
}
]Each attached database has a name, a type and an object named after the type:
type | Object and keys |
|---|---|
postgres, bigquery, snowflake | postgresConnection, bigqueryConnection or snowflakeConnection, with the same keys as that type's own connection |
s3 | s3Connection: accessKeyId, secretAccessKey, and optionally region, endpoint and sessionToken |
gcs | gcsConnection: keyId and secret, the HMAC key's access key ID and secret |
azure | azureConnection: authType service_principal with tenantId, clientId, clientSecret, accountName and fileUrl, or authType sas_token with sasUrl |
Naming Rules
- The connection cannot be named
duckdb - An attached database cannot share the connection's name, or another attached database's name
Troubleshooting
| Problem | Cause | Fix |
|---|---|---|
| The form will not save | No attached database or bucket | Add one with Cloud files (S3) or Attached database |
HTTP 403 reading an S3 or GCS file | The key cannot read that bucket or prefix | Grant read and list access on the bucket |
BigQuery Permission denied on read sessions | The service account lacks BigQuery Read Session User | Add the role in Google Cloud IAM |
No files found that match the pattern | The URL or glob matches nothing | Check the path. Use ** to read a subtree |