Files in Cloud Storage

Query CSV, Parquet and JSON files in Amazon S3, Google Cloud Storage, Azure or any S3-compatible store, in place, without loading them into a database

Credible reads CSV, Parquet and JSON files where they already are, in Amazon S3, Google Cloud Storage, Azure Blob Storage / ADLS Gen2, or any S3-compatible store. Nothing is copied into a database first: you point a model at a file, a folder or a whole bucket path, and query it.

Files are read through a DuckDB connection, so one model can also join them with tables from PostgreSQL, BigQuery or Snowflake.

Connect a Bucket

  1. Click Add Connection in your environment, and choose DuckDB as the Database Type
  2. Add the storage:
    • Amazon S3 or an S3-compatible store: click Cloud files (S3)
    • Google Cloud Storage or Azure: click Attached database, then choose the type
  3. Enter the credentials (see below), test, and click Connect
StorageCredentials
Amazon S3Access Key ID and Secret Access Key with read and list access. Optional Region (default us-east-1) and Session Token
S3-compatibleThe store's access key and secret, plus its Custom Endpoint as host and port, without https://
Google Cloud StorageAn HMAC Access Key ID and Secret Access Key. See GCS HMAC Keys
AzureA service principal (Service Principal (SPN)), or a SAS URL (SAS Token). See Azure

To add the same connection with cred add connection, write it as a DuckDB connection with an attached s3, gcs or azure database. See DuckDB for the JSON.

Read Files in a Model

Refer to a file by its URL, through the connection's name:

source: orders is my_files.table('s3://my-bucket/exports/orders.parquet')
source: events is my_files.table('s3://my-bucket/events/*.csv')
source: logs is my_files.table('gs://my-bucket/logs/**/*.json')
  • A single file: s3://bucket/path/file.parquet
  • Every matching file in a folder: s3://bucket/path/*.csv
  • A whole subtree: s3://bucket/path/**

Files matched by one pattern are read as a single table, so they need the same columns.

Files on Your Computer

To model a file on your computer, you don't need a bucket or a connection:

  • Upload a File in the Credible App: CSV, TSV, Parquet, JSON or Excel (.xlsx), up to 32 MB each
  • Embedded Data: ship CSV, Parquet or Excel files inside a package

Next Steps

On this page