Unlocked packages and GitHub Actions

2021-03-01

CI/CD, GitHub Actions, SFDX, Salesforce, Unlocked packages

In this post I’d like to discuss the CI/CD pipeline setup I made for this Salesforce project spaghetti-cmd-loader where I use Unlocked packages and GitHub Actions. I also explain which branching strategy I picked, the package release process and few things I always like to consider when designing this kind of pipelines.

If you are already familiar with these topics and are only interested in the GitHub Actions setup you can jump to CI/CD pipeline.

Quick intro

spaghetti-cmd-loader is a Salesforce application I wrote last year (2020) that allows users to load Custom Metadata Type records from a CSV file. My aim at that time was to provide a more user friendly and nice looking tool compared to the older CustomMetadataLoader.

Salesforce setup

I’m using my personal Developer Edition Org as DevHub to create new Scratch Orgs and to maintain the package’s version history. I also registered the namespace spaghettiCMD to prevent deployment issues that might arise due to conflicting API names. More on how to register a namespace in this Help article.

Branching strategy

My branching strategy is based on GitFlow however I use only two main branches:

main, to release new package versions
develop, to integrate features, test and validate package versions before releasing them

and only one kind of individual branch:

feature, to develop new features or bugfixes

This is a visual representation of my strategy:

Release process

The release process is pretty straightforward and can be summarized in four steps:

The content of feature branches must be validated before it can be merged into the develop branch.
Every time new changes are merged into the develop branch a new package version is created. This version cannot be installed in Production Organizations.
Every time new changes are merged into the main branch the latest package version is promoted so it can be installed in Production Organization.
After a version is promoted a new release cycle begins by updating the version’s number inside the sfdx-project.json file and pushing it on the develop branch.

Validate features

To validate the changes made in feature branches every time a Pull Request (from: feature/xyz to: develop) is opened or updated the entire content of the branch is deployed into a new Scratch Org. If the deployment is successful then all Apex test classes are executed to ensure that the overall code coverage is at least 75%. If any of these two steps (deploy and/or tests) fail then the Pull Request cannot be merged.

Preview version

A new package version is generated every time a push is made on the develop branch (this also applies when merging a Pull Request). When creating a new version the overall code coverage is calculated and checked again to ensure that is at least 75%. If the creation is successful then two things happen:

The version is deployed to a dedicated environment like a Sandbox or another Developer Org where it can be tested (keep in mind that this is a preview version and cannot be deployed to a real Production org)
The version’s Id (the one that looks like this 04txxxxxxxxx) is stored in the repository within the content of the sfdx-project.json file

Another thing to keep in mind is that there’s a limit on the number of preview versions you can generate within 24 hours which depends on your DevHub’s edition.

Promote preview version

A version is promoted when the develop branch is merged into the main branch. The package version that gets promoted is the latest one stored in the sfdx-project.json file. After that a new git tag is generated to mark the release point in the main branch. Promoting a package’s version allows for deployment in real Production Organizations. Keep in mind that a preview version cannot be promoted if it does not meet the minimum code coverage requirement (75%). This is why I constantly check for this value during the release process.

Release cycle

To begin a new release cycle the package version’s number has to be incremented. Version’s number follows the Semantic Versioning and it is stored inside the sfdx-project.json file under the packageDirectories property:

...
"packageDirectories": [
        {
            "path": "cmd-loader",
            "default": true,
            "package": "Spaghetti CMD",
            "versionName": "ver 1.2", 
            "versionNumber": "1.2.1.NEXT"
        }
    ],
...

Salesforce prevents you to promote the same MAJOR.MINOR version twice so at least the MINOR number has to increment. I decided not to automate this step yet to have more control on version numbers.

CI/CD pipeline

Time to dive into how I automate everything I discussed until now using GitHub Actions.

GitHub Actions is available with GitHub Free, GitHub Pro, GitHub Free for organizations, GitHub Team, GitHub Enterprise Cloud, GitHub Enterprise Server, GitHub One, and GitHub AE. GitHub Actions is not available for private repositories owned by accounts using legacy per-repository plans.

GitHub Docs:docs.github.com/en/actions/guides/about-continuous-integration

Since I’m using my personal GitHub Free account I have a quota of 2000 minutes per month which is more than enough. You can check your quota by logging in with your account and then go to Settings > Billing & plan. The setup for GitHub Actions is quite easy, the configuration files are written in YAML and must be placed in this specific folder within your repository .github/workflows. Every file identifies a specific workflow in your process. I created three main workflows:

.github
└── workflows
    ├── pr-develop.yml
    ├── push-develop.yml
    └── push-main.yml

The names are self explanatory but anyhow this is a quick overview of what they do:

pr-develop, runs for Pull Requests on the develop branch and validates changes that are part of feature branches
push-develop, runs on every push on the develop branch and creates a new package preview version
push-main, runs on every push on the main branch and promotes the latest package version

Pull Requests workflow

pr-develop.yml

At the beginning of the file I specify on which event this workflow will run:

name: Pull Request to Develop

on:
  pull_request:
    branches: [ develop ]

Next I define the single job that will execute as part of this workflow:

jobs:
  validate_pull_request:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2

validate_pull_request is the job’s name
runs-on: ubuntu-latest specifies the Operating System my virtual machine will run on
- uses: actions/checkout@v2 is the helper module that retrieves the branch content

In the first two steps of this job I install all the tools I need:

Salesforce CLI, pretty obvious why ;)
jq, to parse the JSON output of the CLI. When implementing these automation always rely on the JSON output of the CLI and never on the human readable one!

- name: Install Salesforce CLI
  run: |
    wget https://developer.salesforce.com/media/salesforce-cli/sfdx-linux-amd64.tar.xz
    mkdir sfdx-cli
    tar xJf sfdx-linux-amd64.tar.xz -C sfdx-cli --strip-components 1
    ./sfdx-cli/install
  
- name: Install jq
  run: |
    sudo apt-get install jq

I now proceed to authenticate the CLI with my DevHub org.
I’m using the sfdxurl flow as authentication mechanism since it’s much simpler compared to the JWT one where you have to configure a Connected App in your DevHub, generate an SSL certificate and store the encrypted private key in the repository. To obtain the authentication URL for your DevHub run this command from your laptop:

$ sfdx force:org:display --targetusername=YOUR_DEVHUB_USERNAME --verbose 

=== Org Description
KEY            VALUE
──────────────────────────
...
Sfdx Auth Url  force://PlatformCLI:: - redacted -
...

Store the value for Sfdx Auth Url as a secret in your repository by following this simple guide and give it a name (in my case DEVHUB_SFDX_URL).

- name: Populate auth file
  shell: bash
  run: 'echo ${{ secrets.DEVHUB_SFDX_URL }} > ./DEVHUB_SFDX_URL.txt'
  
- name: Authenticate Dev Hub
  run: 'sfdx force:auth:sfdxurl:store -f ./DEVHUB_SFDX_URL.txt -a devhub -d'

The first step retrieves the encrypted secret, decrypts it and stores it in a text file. The second one references the file to authenticate with the DevHub. Once the CLI is authenticated with the DevHub I create a new Scratch Org and push the content of the repository in it.

- name: Create scratch org
  run: 'sfdx force:org:create -f config/project-scratch-def.json -a ci_scratch -s -d 1'
    
- name: Push source to scratch org
  run: 'sfdx force:source:push'

If the deployment is successful I run all test classes to check that the code coverage requirement is met:

- name: Check code coverage
  run: |
    sfdx force:apex:test:run --codecoverage --resultformat json --synchronous --testlevel RunLocalTests --wait 10 > tests.json
    coverage=$(jq .result.summary.orgWideCoverage tests.json | grep -Eo "[[:digit:]]+")
    test $coverage -ge 75

Finally at the end I delete the Scratch Org:

- name: Delete scratch org
  if: always()
  run: 'sfdx force:org:delete -p -u ci_scratch'

I always execute this step also in case of errors because there is a limit on the number of active Scratch Org. This limit depends on the edition of your DevHub and you can check it by running this command:

$ sfdx force:limits:api:display --targetusername=YOUR_DEVHUB_USERNAME
Name                                         Remaining  Max
───────────────────────────────────────────  ─────────  ─────────
ActiveScratchOrgs                            3          3
DailyScratchOrgs                             6          6

ActiveScratchOrgs is the limit of active Scratch Org related to your DevHub. This limit is static and does not reset.
DailyScratchOrgs is the maximum amount of Scratch Org you can create in 24h. This limit is dynamic and resets every day.

Develop branch workflow

push-develop.yml

At the beginning of the file I specify on which event this workflow will run:

name: Create pre-release version

on:
  push:
    branches: [ develop ]

The next steps are the same as explained before (job’s name, operating system, DevHub authentication, …) so I’ll skip them and jump directly to the package version creation:

- name: Create new version
  run: |
    sfdx force:package:version:create -x -p "Spaghetti CMD" -w 60 --codecoverage
    new_version_id=$(grep -o "04t[[:alnum:]]\{15\}" sfdx-project.json | tail -n1)
    echo "version_id=${new_version_id}" >> $GITHUB_ENV

The force:package:version:create command automatically stores the new version id within the content of sfdx-project.json file. With the next line I extract this value and assign it to a local variable named new_version_id. I then generate an environment variable named version_id which hold the same value as new_version_id so I can reference it in the next steps.

If the version is created successfully I proceed to check the code coverage and then install it in my DevHub so I can test it.

- name: Check code coverage
  run: |
    test $(sfdx force:package:version:report -p "$version_id" --json | jq .result.HasPassedCodeCoverageCheck) = 'true'
  
- name: Install new version in Dev Hub
  run: |
    sfdx force:package:install -p "$version_id" -u devhub --wait 10 --publishwait 10

Here you can use any other environment like a Sandbox or another Developer Edition as long as it’s not a real Production Org since the version is still in a preview state.
As last step I update the README.md file by replacing the previous package version’s Id with the newly created one and commit both the sfdx-config.json file and the README.md file.

- name: Store new version id
  run: |
    sed -i -e "s/04t[[:alnum:]]\{15\}/${version_id}/" README.md
    git config user.name "release[bot]"
    git config user.email "<>"
    git add README.md
    git add sfdx-project.json
    git commit -m "Updating new pre-release version"
    git push

I update the README.md file too because it contains links and instructions to install the package that reference the version id. For example the Sandbox installation URL:

https://test.salesforce.com/packaging/installPackage.apexp?p0=04t1t000003HUFbAAO

Main branch workflow

push-main.yml

At the beginning of the file I specify on which event this workflow will run:

name: Publish new release

on:
  push:
    branches: [ main ]

Then again I define the job’s name, operating system, install the Salesforce CLI and authenticate the DevHub. After that I promote the latest version’s Id stored in the sfdx-project.json file:

- name: Promote latest version
  run: |
    version_id=$(grep -o "04t[[:alnum:]]\{15\}" sfdx-project.json | tail -n1)
    sfdx force:package:version:promote -p "$version_id" --noprompt

If that’s successful I create a new tag on the main branch to mark the release point:

- name: Tag new release
  run: |
    tag_name=$(jq ".packageDirectories[0].versionName" sfdx-project.json | tr -d '"'| tr -d ' ')
    pkg_name=$(jq ".packageDirectories[0].package" sfdx-project.json | tr -d '"')
    git config user.name "release[bot]"
    git config user.email "<>"
    git tag -a "$tag_name" -m "$pkg_name $tag_name"
    git push origin "$tag_name"

With the first two lines I extract the package’s version number and name. I’m then using the version’s number as tag’s name and the package’s name as message for the tag. Having the tag name equals to the package version’s name allows me to quickly identify in GIT at which point in the branch history a specific package was released.

$ git log --oneline
ed6fa85 (HEAD -> main, origin/main, origin/HEAD) renamed action
e441f7b (tag: ver1.2) Merge pull request #30 from maaaaarco/develop
41f54f6 Updating new pre-release version
7ed40ea Merge pull request #29 from maaaaarco/feature/winter-21

e441f7b is where I released the package version 1.2. This could be useful in future if I decide to extend my branching strategy to support hotfix branches as defined in GitFlow.