Embedded Data

Ship small CSV, Parquet, JSON or Excel files inside a package and query them with the package's own DuckDB connection, with no database connection

Data files placed in a package are published with it and versioned with it, and the package's built-in DuckDB connection, named duckdb, reads them in place. No database connection is needed.

Use embedded data for:

  • Small lookup tables, such as country codes or category mappings.
  • Sample data for demos and tests.
  • Standalone models that don't need a database.

Keep the files small. cred publish warns about any data file over 1 MB, and a whole package, data included, can't be over 50 MB. For larger data, keep it in a database or object storage and connect it; a DuckDB connection can read files in cloud storage. Embedded files can't be joined to tables on another connection, because each lives on its own connection.

Supported Files

FormatExtensions
CSV.csv
Parquet.parquet
JSON and newline-delimited JSON.json, .ndjson
Excel.xlsx

Upload a File takes a few more formats, such as TSV and gzipped files.

Add Files to a Package

Put the files anywhere in the package directory; a data/ folder keeps them together:

my-package/
├── publisher.json
├── ecommerce.malloy
└── data/
    ├── country_codes.csv
    ├── product_categories.parquet
    └── budget.xlsx

Query Them in a Model

Reference a file by its path relative to the package root with duckdb.table():

source: country_codes is duckdb.table('data/country_codes.csv') extend {
  dimension:
    country_code is code
    country_name is name
}

source: categories is duckdb.table('data/product_categories.parquet')

duckdb.table('data/budget.xlsx') reads the first sheet of a workbook. To pick a sheet, use a SQL source:

source: q2_budget is duckdb.sql("""
  SELECT * FROM read_xlsx('data/budget.xlsx', sheet = 'Q2')
""")

Publish the package as usual and the files go with it; see Publishing and Versions.

Next Steps

On this page