DuckLake

Connect a DuckLake lakehouse (a PostgreSQL catalog plus S3 or GCS storage) to Credible

DuckLake is a lakehouse format that keeps table metadata in a SQL catalog database and the data itself as files in object storage. To connect a DuckLake you already run, give Credible both: the catalog for metadata, and the storage for data.

You do not need DuckLake to persist tables: the engine keeps its own. Connect DuckLake when you want to read a lake you already have.

Prerequisites

  • A catalog database holding the DuckLake metadata. PostgreSQL is supported
  • An object storage location for the data: Amazon S3 or Google Cloud Storage (GCS)
  • Credentials that can read both

Credible attaches a DuckLake connection read-only, so read access is enough.

Step 1: Catalog (PostgreSQL)

Have the catalog database's details ready: Host, Port (default 5432), Database Name, Username and Password, or a single connection string (postgresql://username:password@hostname:port/database).

The catalog form shows the same fields as a PostgreSQL connection, but Database TLS and Statement timeout are not applied to a DuckLake catalog.

Step 2: Storage (S3 or GCS)

Every DuckLake connection needs a Bucket URL: the data path the lake was created with, such as s3://my-bucket/path or gs://my-bucket/path.

For Amazon S3:

FieldNotes
Access Key ID and Secret Access KeyRequired. Read and list access on the bucket and prefix
RegionOptional, default us-east-1
Custom EndpointOptional, for an S3-compatible store such as MinIO. Host and port, no https://
Session TokenOptional, for temporary STS credentials

For Google Cloud Storage, enter an HMAC Access Key ID and Secret Access Key for a service account with Storage Object Viewer on the bucket. Create one under Cloud Storage > Settings > Interoperability in the Google Cloud Console.

Step 3: Connect in Credible

  1. In the connection form, choose DuckLake
  2. Under Catalog, choose PostgreSQL and enter the details from Step 1
  3. Under Storage, choose S3 or GCS, enter the Bucket URL, then the credentials from Step 2
  4. Click Test Connection, which checks that both the catalog and the storage are reachable
  5. Click Connect

From the CLI

To add the connection with cred add connection, put an entry like this in the JSON file:

[
  {
    "name": "lake",
    "type": "ducklake",
    "ducklakeConnection": {
      "catalog": {
        "postgresConnection": {
          "host": "catalog.example.com",
          "port": 5432,
          "databaseName": "ducklake",
          "userName": "credible_readonly",
          "password": "..."
        }
      },
      "storage": {
        "bucketUrl": "s3://my-bucket/path",
        "s3Connection": {
          "accessKeyId": "...",
          "secretAccessKey": "...",
          "region": "us-east-1"
        }
      }
    }
  }
]

For GCS storage, use gcsConnection with keyId and secret in place of s3Connection.

Troubleshooting

ProblemCauseFix
The catalog connection failsWrong PostgreSQL host, port or credentials, or the catalog database isn't reachable from CredibleCheck the catalog details. See Network Access
Storage access is deniedThe bucket URL is wrong, or the key can't list and read objects under that prefixCheck the bucket URL and the key's access. For an S3-compatible store, set Custom Endpoint
Temporary credentials expiredSTS credentials from a Session Token are short-livedGenerate new ones, or use a long-lived key

Next Steps

On this page