DuckDB

Read files in S3, GCS or Azure in place, and join databases in one model, through a DuckDB connection

A DuckDB connection runs DuckDB inside Credible and attaches the sources you give it. Use it to:

  • Read files in object storage in place: CSV, Parquet and JSON in Amazon S3, Google Cloud Storage or Azure, as a single file, a folder glob (path/*.parquet) or a whole subtree (path/**).
  • Join across databases: attach PostgreSQL, BigQuery and Snowflake side by side and join their tables in one model.

A DuckDB connection needs at least one attached database or bucket. For files that ship inside a package, you don't need a connection at all: see embedded data.

Add Cloud Files (S3)

The quickest way to read files from S3:

  1. In the connection form, choose DuckDB
  2. Click Cloud files (S3)
  3. Enter an Access Key ID and Secret Access Key that can read the bucket
  4. Optional: Region (default us-east-1), a Custom Endpoint for an S3-compatible store such as MinIO (host and port, no https://), and a Session Token for temporary STS credentials

Then read files by their URL:

source: orders is my_files.table('s3://my-bucket/exports/orders.parquet')
source: events is my_files.table('s3://my-bucket/events/*.csv')

Add an Attached Database

Click Attached database, give it a Database Name, and choose its type. The name is how a model refers to it, so it follows the same rules as a connection name, and it cannot be the same as the connection's own name.

TypeWhat it needs
PostgreSQLThe same fields as a PostgreSQL connection
BigQueryA service account JSON key. The service account also needs the BigQuery Read Session User role
SnowflakeThe same fields as a Snowflake connection
Amazon S3Access key ID and secret access key, as in Cloud files
Google Cloud Storage (GCS)An HMAC Access Key ID and Secret Access Key
Azure Blob Storage / ADLS Gen2A service principal (tenant ID, client ID, client secret, account name and file URL), or a SAS URL

A table in an attached database is read through the DuckDB connection, prefixed with the attached name:

source: customers is my_duckdb.table('crm.public.customers')

GCS HMAC Keys

DuckDB reads GCS with HMAC keys, not a service account JSON key. To create one:

  1. In the Google Cloud Console, open Cloud Storage > Settings > Interoperability
  2. Under Access keys for service accounts, click Create a key for a service account
  3. Choose a service account with Storage Object Viewer on the bucket
  4. Copy the access key and secret

Read files with a gs:// URL, such as my_duckdb.table('gs://my-bucket/path/*.parquet').

Azure

Choose an Auth Type:

  • Service Principal (SPN): an app registration with Storage Blob Data Reader on the storage account. Enter its Tenant ID, Client ID and Client Secret, the Account Name, and the File URL to read, such as https://account.blob.core.windows.net/container/path/**.
  • SAS Token: paste the full SAS URL, token included. The token needs read and list permissions (sp=rl).

Both URLs can point at a single file, a glob (*.parquet) or a subtree (**). The https://, abfss:// and az:// forms all work.

From the CLI

To add the connection with cred add connection, put an entry like this in the JSON file:

[
  {
    "name": "my_files",
    "type": "duckdb",
    "duckdbConnection": {
      "attachedDatabases": [
        {
          "name": "cloud_files",
          "type": "s3",
          "s3Connection": {
            "accessKeyId": "...",
            "secretAccessKey": "...",
            "region": "us-east-1"
          }
        }
      ]
    }
  }
]

Each attached database has a name, a type and an object named after the type:

typeObject and keys
postgres, bigquery, snowflakepostgresConnection, bigqueryConnection or snowflakeConnection, with the same keys as that type's own connection
s3s3Connection: accessKeyId, secretAccessKey, and optionally region, endpoint and sessionToken
gcsgcsConnection: keyId and secret, the HMAC key's access key ID and secret
azureazureConnection: authType service_principal with tenantId, clientId, clientSecret, accountName and fileUrl, or authType sas_token with sasUrl

Naming Rules

  • The connection cannot be named duckdb
  • An attached database cannot share the connection's name, or another attached database's name

Troubleshooting

ProblemCauseFix
The form will not saveNo attached database or bucketAdd one with Cloud files (S3) or Attached database
HTTP 403 reading an S3 or GCS fileThe key cannot read that bucket or prefixGrant read and list access on the bucket
BigQuery Permission denied on read sessionsThe service account lacks BigQuery Read Session UserAdd the role in Google Cloud IAM
No files found that match the patternThe URL or glob matches nothingCheck the path. Use ** to read a subtree

Next Steps

On this page