CI/CD

Set up the credible-cicd-template GitHub Actions pipeline that bumps package versions and publishes changed packages to a Credible environment on every merge to main

If you keep your Malloy packages in a GitHub repository you own, Credible's template repository, credibledata/credible-cicd-template (opens in a new tab), adds GitHub Actions workflows that publish each changed package to a Credible environment when changes merge to main.

What the Pipeline Does

On every push to main that changes a file under packages/:

  1. Detect. It finds the packages whose files changed.
  2. Version. For each changed package, it raises the patch version in publisher.json (1.0.0 becomes 1.0.1; prerelease and build tags are kept) and commits the change back to main as a GitHub App. A published version can't be replaced, so every change needs a new one. If you raised the version yourself in the same change, it publishes your version instead.
  3. Publish. It installs the CLI and, for each package, runs cred set-access-token, cred set environment, and cred publish --yes, with --set-latest unless you turn that off.

A version that already exists is skipped with a warning. A package that fails to publish is retried, then reported, and the run fails; the other packages still publish. Deleting a whole package from the repository publishes nothing.

With --set-latest, a new version becomes latest as soon as it is published, before its materialized tables and indexes are built. Set SET_LATEST to false to leave that to the package's auto-promote setting instead. To roll back, make an earlier version latest in the Credible App; nothing in CI needs to run. See Publishing and Versions.

Prerequisites

  • A GitHub organization where you can create a GitHub App, and admin access to the repository.
  • A Credible organization and environment, and a Credible organization admin to create the group and API key.

Step 1: Create the Repository

On the template repository (opens in a new tab), choose Use this template and create a repository in your organization. It contains the workflows in .github/workflows/ and the scripts they run in scripts/. Put each package in its own folder under packages/:

your-repo/
├── .github/workflows/    # from the template
├── scripts/              # from the template
└── packages/
    ├── sales/
    │   ├── publisher.json
    │   └── sales.malloy
    └── finance/
        └── ...

Each package needs a publisher.json with at least a name and a version:

{
  "name": "sales",
  "version": "0.0.0",
  "description": "Sales data model"
}

Step 2: Create a GitHub App

The pipeline commits version bumps to main as a GitHub App, so it can push past branch protection.

  1. In your GitHub organization, go to Settings > Developer settings > GitHub Apps > New GitHub App.
  2. Give it a name that is unique across GitHub, such as <your-company>-credible-cicd. The workflows authenticate with the App ID and private key, not the name.
  3. Enter any valid Homepage URL, such as the repository's URL, and clear Active under Webhook.
  4. Under Repository permissions, grant Contents: Read and write, Pull requests: Read and write, and Metadata: Read-only.
  5. Under Where can this GitHub App be installed?, choose Only on this account, then create the App.
  6. Note the App ID. Under Private keys, generate a key and keep the downloaded .pem file.
  7. Under Install App, install it on Only select repositories and pick this repository. The App can write repository contents, so don't install it on all repositories.

Step 3: Create a Credible API Key

The publish step authenticates as a Credible group through a group access token (API key). An API key acts with every role its group holds, so use a group that exists only for this pipeline:

cred add group ci-publish
cred add permission group:ci-publish modeler --environment <environment>
cred add group-access-token ci-publish github-actions

The Modeler role on the environment lets the group publish packages there. The last command prints the key once; store it only as a GitHub secret. See REST APIs for what a key can reach.

Step 4: Add Secrets and Variables

In the repository, go to Settings > Secrets and variables > Actions.

Secrets:

SecretValue
CICD_BOT_APP_IDThe GitHub App's ID
CICD_BOT_APP_PRIVATE_KEYThe whole .pem file, including the BEGIN and END lines
JWT_ACCESS_TOKENThe Credible API key from Step 3

Variables:

VariableValue
CRED_ORGYour Credible organization name
CRED_ENVThe Credible environment to publish to. Older repositories may use CRED_PROJECT, which is still read
SET_LATESTOptional. false publishes without --set-latest. Anything else, or unset, publishes with it

Step 5: Protect main

Add a branch protection rule for main:

  • Require a pull request before merging, and status checks to pass.
  • Under Allow specified actors to bypass required pull requests, add your GitHub App.
  • Under Restrict who can push to matching branches, add your GitHub App.

The two App entries let the pipeline push its version bump to a branch that otherwise refuses direct pushes. Without them the bump can't land and nothing publishes.

Publishing to More Than One Environment

The template publishes to one environment. To publish to a staging environment and then production, call the template's publish-packages.yml workflow once per environment from deploy.yaml, and pass each call a GitHub environment:

jobs:
  publish-staging:
    needs: bump
    if: needs.bump.outputs.bumped != ''
    uses: ./.github/workflows/publish-packages.yml
    with:
      environment: staging
      packages: ${{ needs.bump.outputs.bumped }}
      cred_org: ${{ vars.CRED_ORG }}
      cred_env: ${{ vars.CRED_ENV_STAGING }}
    secrets:
      jwt_access_token: ${{ secrets.JWT_ACCESS_TOKEN_STAGING }}

The environment input binds the publish job to a GitHub Environment (opens in a new tab), so its protection rules apply: add a required reviewer to production and the production publish waits for approval while staging goes ahead. Give each environment its own CRED_ENV_* variable and its own API key secret, from a group with access to that environment only, so a staging key can't publish to production.

Troubleshooting

ProblemCauseFix
The workflow doesn't runThe change didn't touch a file under packages/ on main, or Actions is turned off for the repositoryChange a file under packages/, and check that Actions is enabled
The version bump can't be pushedThe App isn't installed on the repository with Contents: Read and write, isn't in both branch protection lists, or the private key secret is incompleteCheck the App's installation and permissions, the branch protection rule, and CICD_BOT_APP_PRIVATE_KEY
Publishing failsCRED_ORG or CRED_ENV is wrong, the API key was revoked or has expired, or its group lacks Modeler on the environmentCheck the variables, the key and the group's role. The publish step's log shows the CLI's error

If the cause isn't in the logs, contact support with the run's URL and the error.

How This Fits With Git-Backed Environments

This pipeline is for packages built in your editor and kept in a repository you own. A git-backed environment is a different path, for packages built in the Credible App: publishing a draft opens a pull request in the environment's repository, and merging it publishes. Credible sets up the repository on request, and an environment admin turns it on.

The two can share an environment, but each package should have one owner. After a package has been published through a git-backed environment, cred publish over it is refused unless an admin turns on direct uploads over git-published packages in the environment's settings.

Next Steps

On this page