Files in Cloud Storage
Query CSV, Parquet and JSON files in Amazon S3, Google Cloud Storage, Azure or any S3-compatible store, in place, without loading them into a database
Credible reads CSV, Parquet and JSON files where they already are, in Amazon S3, Google Cloud Storage, Azure Blob Storage / ADLS Gen2, or any S3-compatible store. Nothing is copied into a database first: you point a model at a file, a folder or a whole bucket path, and query it.
Files are read through a DuckDB connection, so one model can also join them with tables from PostgreSQL, BigQuery or Snowflake.
Connect a Bucket
- Click Add Connection in your environment, and choose DuckDB as the Database Type
- Add the storage:
- Amazon S3 or an S3-compatible store: click Cloud files (S3)
- Google Cloud Storage or Azure: click Attached database, then choose the type
- Enter the credentials (see below), test, and click Connect
| Storage | Credentials |
|---|---|
| Amazon S3 | Access Key ID and Secret Access Key with read and list access. Optional Region (default us-east-1) and Session Token |
| S3-compatible | The store's access key and secret, plus its Custom Endpoint as host and port, without https:// |
| Google Cloud Storage | An HMAC Access Key ID and Secret Access Key. See GCS HMAC Keys |
| Azure | A service principal (Service Principal (SPN)), or a SAS URL (SAS Token). See Azure |
To add the same connection with cred add connection, write it as a DuckDB connection with an attached s3, gcs or azure database. See DuckDB for the JSON.
Read Files in a Model
Refer to a file by its URL, through the connection's name:
source: orders is my_files.table('s3://my-bucket/exports/orders.parquet')
source: events is my_files.table('s3://my-bucket/events/*.csv')
source: logs is my_files.table('gs://my-bucket/logs/**/*.json')- A single file:
s3://bucket/path/file.parquet - Every matching file in a folder:
s3://bucket/path/*.csv - A whole subtree:
s3://bucket/path/**
Files matched by one pattern are read as a single table, so they need the same columns.
Files on Your Computer
To model a file on your computer, you don't need a bucket or a connection:
- Upload a File in the Credible App: CSV, TSV, Parquet, JSON or Excel (
.xlsx), up to 32 MB each - Embedded Data: ship CSV, Parquet or Excel files inside a package