Embedded Data
Ship small CSV, Parquet, JSON or Excel files inside a package and query them with the package's own DuckDB connection, with no database connection
Data files placed in a package are published with it and versioned with it, and the package's built-in DuckDB connection, named duckdb, reads them in place. No database connection is needed.
Use embedded data for:
- Small lookup tables, such as country codes or category mappings.
- Sample data for demos and tests.
- Standalone models that don't need a database.
Keep the files small. cred publish warns about any data file over 1 MB, and a whole package, data included, can't be over 50 MB. For larger data, keep it in a database or object storage and connect it; a DuckDB connection can read files in cloud storage. Embedded files can't be joined to tables on another connection, because each lives on its own connection.
Supported Files
| Format | Extensions |
|---|---|
| CSV | .csv |
| Parquet | .parquet |
| JSON and newline-delimited JSON | .json, .ndjson |
| Excel | .xlsx |
Upload a File takes a few more formats, such as TSV and gzipped files.
Add Files to a Package
Put the files anywhere in the package directory; a data/ folder keeps them together:
my-package/
├── publisher.json
├── ecommerce.malloy
└── data/
├── country_codes.csv
├── product_categories.parquet
└── budget.xlsxQuery Them in a Model
Reference a file by its path relative to the package root with duckdb.table():
source: country_codes is duckdb.table('data/country_codes.csv') extend {
dimension:
country_code is code
country_name is name
}
source: categories is duckdb.table('data/product_categories.parquet')duckdb.table('data/budget.xlsx') reads the first sheet of a workbook. To pick a sheet, use a SQL source:
source: q2_budget is duckdb.sql("""
SELECT * FROM read_xlsx('data/budget.xlsx', sheet = 'Q2')
""")Publish the package as usual and the files go with it; see Publishing and Versions.