How to Set up a CI pipeline with Azure Pipelines and Nx

Share on facebook
Share on google
Share on twitter
Share on linkedin

It goes without saying that having a CI pipeline for your Angular apps is a must. Setting one up for regular Angular apps is fairly straightforward but when you have an Nx monorepo there are certain other challenges that you have to overcome to successfully orchestrate a “build once, deploy many” pipeline.

This post will show you, how to create a CI pipeline with Azure Pipelines for an Nx monorepo that not only allows smart builds with the affected commands but also allows you to tag the affected apps to trigger build pipelines that are relevant.

I will show you how to do this in three different ways: seial, parallel/binning and with Distributed Task Execution (DTE).

Serial pipeline

First, let’s see how we can create a pipeline that can run lint, test, build, and e2e using the affected commands in a serial manner:

trigger:
  - main
pr:
  - main

variables:
  CI: 'true'
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(System.PullRequest.PullRequestId) # You can use $(System.PullRequest.PullRequestNumber if your pipeline is triggered by a PR from GitHub ONLY)
    TARGET_BRANCH: $[replace(variables['System.PullRequest.TargetBranch'],'refs/heads/','origin/')]
    BASE_SHA: $(git merge-base $(TARGET_BRANCH) HEAD)
  ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(Build.SourceBranchName)
    BASE_SHA: $(git rev-parse HEAD~1)
  HEAD_SHA: $(git rev-parse HEAD)
  YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

jobs:
  - job: main
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      # Set Azure Devops CLI default settings
      - bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject)
        displayName: 'Set default Azure DevOps organization and project'

      # Get last successfull commit from Azure Devops CLI
      - bash: |
          LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
          if [ -z "$LAST_SHA" ]
          then
            echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA"
          else
            echo "Last successful commit SHA: $LAST_SHA"
            echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
          fi
        displayName: 'Get last successful commit SHA'
        condition: ne(variables['Build.Reason'], 'PullRequest')
        env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

      # Required for nx affected if we're on a branch
      - script: git branch --track main origin/main

      - task: Cache@2
        inputs:
          key: '"yarn" | "$(Agent.OS)" | yarn.lock'
          restoreKeys: |
            "yarn" | "$(Agent.OS)"
            "yarn"
          path: $(YARN_CACHE_FOLDER)
        displayName: Cache Yarn packages

      - script: yarn --frozen-lockfile
        displayName: 'Install dependencies'

      - script: npx nx format:check --base=$(BASE_SHA)
        displayName: 'Format check'
      - script: npx nx affected --base=$(BASE_SHA) -t lint --parallel=3
        displayName: 'Lint'
      - script: npx nx affected --base=$(BASE_SHA) -t test --parallel=3
        displayName: 'Test'
      - script: npx nx affected --base=$(BASE_SHA) -t build --parallel=3
        displayName: 'Build'

      # archive and publish artifacts
      - task: ArchiveFiles@2
        displayName: Archive Build Artifacts
        condition: always()
        continueOnError: true
        inputs:
          rootFolderOrFile: '$(System.DefaultWorkingDirectory)/dist'
          includeRootFolder: false
          archiveType: 'zip'
          archiveFile: $(Build.ArtifactStagingDirectory)/build/$(Build.BuildId).zip
      - task: PublishBuildArtifacts@1
        displayName: Publish Build Artifacts
        condition: always()
        continueOnError: true
        inputs:
          pathtoPublish: $(Build.ArtifactStagingDirectory)/build/$(Build.BuildId).zip
          artifactName: 'drop'

      # Tag build to trigger release pipelines
      - script: |
          projects=`npx nx show projects --affected --base=$(BASE_SHA) --head=$(HEAD_SHA)`
          echo "Touched projects:"
          echo $projects

          for project in ${projects//,/ }
          do
            echo "##vso[build.addbuildtag]$project"
            echo "Creating tag for: $project"
          done
        displayName: 'Tag build'


This pipeline is basing the affected commands on the last successful build when not a pull request build. We are using the Azure Pipelines CLI to look up the SHA for the last successful build to use as the BASE_SHA.

We use the cache task to cache the yarn install results so yarn install will run much faster as most of the dependencies are already retrieved from the cache.

Then we do the format check with nx format:check to check that everything is formatted correctly. Note, this should already have been solved before committing by setting Githooks.

Then we run the affected commands of lint, test and build using the corresponding affected commands.

After this, the build is archived and published and we add tags for all the affected projects so we can later create a release pipeline for each tag to release a given app a build with the app tag is available.

Running the pipeline

Let’s run the pipeline:

Note, that this workflow can be optimized further by splitting this up into multiple jobs, to run the different commands on separate CI agents. Let’s see how to do this…

Binning/Parallel pipeline

The binning/parallel pipeline looks mostly the same but we have split the task execution up in separate jobs utilizing parallel CI agents to minimize the overall CI execution time. This process is called binning, where each agents gets assigned a certain task:

trigger:
  - main
pr:
  - main

variables:
  CI: 'true'
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(System.PullRequest.PullRequestId) # You can use $(System.PullRequest.PullRequestNumber if your pipeline is triggered by a PR from GitHub ONLY)
    TARGET_BRANCH: $[replace(variables['System.PullRequest.TargetBranch'],'refs/heads/','origin/')]
    BASE_SHA: $(git merge-base $(TARGET_BRANCH) HEAD)
  ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(Build.SourceBranchName)
    BASE_SHA: $(git rev-parse HEAD~1)
  HEAD_SHA: $(git rev-parse HEAD)
  YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

jobs:
  - job: Build
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      # Set Azure Devops CLI default settings
      - bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject)
        displayName: 'Set default Azure DevOps organization and project'

      # Get last successfull commit from Azure Devops CLI
      - bash: |
          LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
          if [ -z "$LAST_SHA" ]
          then
            echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA"
          else
            echo "Last successful commit SHA: $LAST_SHA"
            echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
          fi
        displayName: 'Get last successful commit SHA'
        condition: ne(variables['Build.Reason'], 'PullRequest')
        env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

      # Required for nx affected if we're on a branch
      - script: git branch --track main origin/main

      - task: Cache@2
        inputs:
          key: '"yarn" | "$(Agent.OS)" | yarn.lock'
          restoreKeys: |
            "yarn" | "$(Agent.OS)"
            "yarn"
          path: $(YARN_CACHE_FOLDER)
        displayName: Cache Yarn packages

      - script: yarn --frozen-lockfile
        displayName: 'Install dependencies'

      - script: npx nx affected --base=$(BASE_SHA) -t build --parallel=3
        displayName: 'Build'

      # archive and publish artifacts
      - task: ArchiveFiles@2
        displayName: Archive Build Artifacts
        condition: always()
        continueOnError: true
        inputs:
          rootFolderOrFile: '$(System.DefaultWorkingDirectory)/dist'
          includeRootFolder: false
          archiveType: 'zip'
          archiveFile: $(Build.ArtifactStagingDirectory)/build/$(Build.BuildId).zip
      - task: PublishBuildArtifacts@1
        displayName: Publish Build Artifacts
        condition: always()
        continueOnError: true
        inputs:
          pathtoPublish: $(Build.ArtifactStagingDirectory)/build/$(Build.BuildId).zip
          artifactName: 'drop'

      # Tag build to trigger release pipelines
      - script: |
          projects=`npx nx show projects --affected --base=$(BASE_SHA) --head=$(HEAD_SHA)`
          echo "Touched projects:"
          echo $projects

          for project in ${projects//,/ }
          do
            echo "##vso[build.addbuildtag]$project"
            echo "Creating tag for: $project"
          done
        displayName: 'Tag build'

  - job: Test
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      # Set Azure Devops CLI default settings
      - bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject)
        displayName: 'Set default Azure DevOps organization and project'

      # Get last successfull commit from Azure Devops CLI
      - bash: |
          LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
          if [ -z "$LAST_SHA" ]
          then
            echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA"
          else
            echo "Last successful commit SHA: $LAST_SHA"
            echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
          fi
        displayName: 'Get last successful commit SHA'
        condition: ne(variables['Build.Reason'], 'PullRequest')
        env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

      # Required for nx affected if we're on a branch
      - script: git branch --track main origin/main

      - task: Cache@2
        inputs:
          key: '"yarn" | "$(Agent.OS)" | yarn.lock'
          restoreKeys: |
            "yarn" | "$(Agent.OS)"
            "yarn"
          path: $(YARN_CACHE_FOLDER)
        displayName: Cache Yarn packages

      - script: yarn --frozen-lockfile
        displayName: 'Install dependencies'

      - script: npx nx affected --base=$(BASE_SHA) -t test --parallel=3
        displayName: 'Test'

  - job: Lint
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      # Set Azure Devops CLI default settings
      - bash: az devops configure --defaults organization=$(System.TeamFoundationCollectionUri) project=$(System.TeamProject)
        displayName: 'Set default Azure DevOps organization and project'

      # Get last successfull commit from Azure Devops CLI
      - bash: |
          LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
          if [ -z "$LAST_SHA" ]
          then
            echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA"
          else
            echo "Last successful commit SHA: $LAST_SHA"
            echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
          fi
        displayName: 'Get last successful commit SHA'
        condition: ne(variables['Build.Reason'], 'PullRequest')
        env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

      # Required for nx affected if we're on a branch
      - script: git branch --track main origin/main

      - task: Cache@2
        inputs:
          key: '"yarn" | "$(Agent.OS)" | yarn.lock'
          restoreKeys: |
            "yarn" | "$(Agent.OS)"
            "yarn"
          path: $(YARN_CACHE_FOLDER)
        displayName: Cache Yarn packages

      - script: yarn --frozen-lockfile
        displayName: 'Install dependencies'

      - script: npx nx format:check --base=$(BASE_SHA)
        displayName: 'Format check'
      - script: npx nx affected --base=$(BASE_SHA) -t lint --parallel=3
        displayName: 'Lint'

Running the pipeline

Let’s run the pipeline:

Pipeline with DTE

The distributed task execution pipeline helps to reduce the CI execution time by optimizing the parallelization of the tasks. The parallel pipeline we just covered, is not utilizing parallelization optimally as the work is not equally loaded on all of the agents (thus we see different durations for agents), eg. build takes longer than lint. In such case, Nx’s DTE would parallelize the work equally on the nx-cloud agents.

This is taken from Nx’s Distributed Task Execution page.
trigger:
  - main
pr:
  - main

variables:
  CI: 'true'
  ${{ if eq(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(System.PullRequest.PullRequestId) # You can use $(System.PullRequest.PullRequestNumber if your pipeline is triggered by a PR from GitHub ONLY)
    TARGET_BRANCH: $[replace(variables['System.PullRequest.TargetBranch'],'refs/heads/','origin/')]
    BASE_SHA: $(git merge-base $(TARGET_BRANCH) HEAD)
  ${{ if ne(variables['Build.Reason'], 'PullRequest') }}:
    NX_BRANCH: $(Build.SourceBranchName)
    BASE_SHA: $(git rev-parse HEAD~1)
  HEAD_SHA: $(git rev-parse HEAD)
  YARN_CACHE_FOLDER: $(Pipeline.Workspace)/.yarn

jobs:
  - job: agents
    strategy:
      parallel: 3
    displayName: Nx Cloud Agent
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - task: Cache@2
        inputs:
          key: '"yarn" | "$(Agent.OS)" | yarn.lock'
          restoreKeys: |
            "yarn" | "$(Agent.OS)"
            "yarn"
          path: $(YARN_CACHE_FOLDER)
        displayName: Cache Yarn packages

      - script: yarn --frozen-lockfile
        displayName: 'Install dependencies'
      - script: npx nx-cloud start-agent
        displayName: Start Nx-Cloud agent

  - job: main
    displayName: Nx Cloud Main
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      # Get last successfull commit from Azure Devops CLI
      - bash: |
          LAST_SHA=$(az pipelines build list --branch $(Build.SourceBranchName) --definition-ids $(System.DefinitionId) --result succeeded --top 1 --query "[0].triggerInfo.\"ci.sourceSha\"")
          if [ -z "$LAST_SHA" ]
          then
            echo "Last successful commit not found. Using fallback 'HEAD~1': $BASE_SHA"
          else
            echo "Last successful commit SHA: $LAST_SHA"
            echo "##vso[task.setvariable variable=BASE_SHA]$LAST_SHA"
          fi
        displayName: 'Get last successful commit SHA'
        condition: ne(variables['Build.Reason'], 'PullRequest')
        env:
          AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

      - script: git branch --track main origin/main
      - task: Cache@2
        inputs:
          key: '"yarn" | "$(Agent.OS)" | yarn.lock'
          restoreKeys: |
            "yarn" | "$(Agent.OS)"
            "yarn"
          path: $(YARN_CACHE_FOLDER)
        displayName: Cache Yarn packages

      - script: yarn --frozen-lockfile
      - script: yarn nx-cloud start-ci-run --stop-agents-after="build" --agent-count=3
        displayName: Start CI run
      - script: yarn nx-cloud record -- yarn nx format:check --base=$(BASE_SHA) --head=$(HEAD_SHA)
        displayName: Check format
      - script: yarn nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) --target=lint --parallel=3
        displayName: Run lint
      - script: yarn nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) --target=test --parallel=3 --ci --code-coverage
        displayName: Run test
      - script: yarn nx affected --base=$(BASE_SHA) --head=$(HEAD_SHA) --target=build --parallel=3
        displayName: Run build
        # archive and publish artifacts
      - task: ArchiveFiles@2
        displayName: Archive Build Artifacts
        condition: always()
        continueOnError: true
        inputs:
          rootFolderOrFile: '$(System.DefaultWorkingDirectory)/dist'
          includeRootFolder: false
          archiveType: 'zip'
          archiveFile: $(Build.ArtifactStagingDirectory)/build/$(Build.BuildId).zip
      - task: PublishBuildArtifacts@1
        displayName: Publish Build Artifacts
        condition: always()
        continueOnError: true
        inputs:
          pathtoPublish: $(Build.ArtifactStagingDirectory)/build/$(Build.BuildId).zip
          artifactName: 'drop'

      - script: |
          projects=`npx nx show projects --affected --base=$(BASE_SHA) --head=$(HEAD_SHA)`
          echo "Touched projects:"
          echo $projects

          for project in ${projects//,/ }
          do
            echo "##vso[build.addbuildtag]$project"
            echo "Creating tag for: $project"
          done
        displayName: 'Tag build'

      - script: yarn nx-cloud stop-all-agents
        condition: always()
        displayName: Stop all Nx-Cloud agents

We start 3 parallel agents on Azure Pipelines and have them run as Nx cloud agents.

In the main job we prepare for DTE by running: nx-cloud start-ci-run --agent-count=3. Then we run our affected commands and they will execute on our Nx-Cloud agents. Finally, we will tag our build with affected apps and stop all Nx-Cloud agents.

Running the pipeline

Let’s run the pipeline:

Comparing the build pipelines

We saw the results from the 3 approaches, from the serial pipeline (6m 27s) to binning (4m 56s) to DTE (7m 47s).

Oddly enough, we see that the DTE pipeline takes the longest to execute (with no distributed cache). This can be because of some overhead with distributing the tasks to the different agents. This is done for a fairly small project so it’s expected the optimized parallelization will outperform the overhead from distribution with DTE. Also when using Nx Cloud distributed caching the differences in execution time would, on average, be much smaller as the task results can be fetched from the cache.

Firebase hosting release pipeline

Now we have the build pipelines in place we need to release these builds using build pipelines.

For each app, we will create a release pipeline that subscribes to the corresponding tag for the app it will deploy.

And we make sure to get the build artifact:

Note, that we can also repeat this for pull request branches and trigger the build on a pull request to deploy feature sites.

We do this for each app we want to release and now we have a build once, deploy-many pipeline setup allowing us to easily scale to multiple different apps while keeping the same build pipeline.

We can then eg. deploy the site using the Firebase SDK to Firebase hosting:

Conclusion

We saw how to create the CI pipeline and release pipeline with Azure Pipelines with three different build pipeline variations ranking: serial, binning/parallel, and DTE. If you don’t want to depend on Nx-Cloud and you can afford the CI agent resources I recommend you go with the binning approach otherwise DTE will give you optimal parallization.

Also, if you are interested in how to set this up with Github Actions you can see it in this post.

Do you want to become an Angular architect? Check out Angular Architect Accelerator.

Related Posts and Comments

How to Set Up Git Hooks in an Nx Repo

Git hooks can be used to automate tasks in your development workflow. The earlier a bug is discovered, the cheaper it is to fix (and the less impact it has). Therefore it can be helpful to run tasks such as linting, formatting, and tests when you are e.g. committing and pushing your code, so any

Read More »

The Stages of an Angular Architecture with Nx

Long gone are the times when the frontend was just a dumb static website. Frontend apps have gotten increasingly complex since the rise of single-page application frameworks like Angular. It comes with the price of increased complexity and the ever-changing frontend landscape requires you to have an architecture that allows you to scale and adapt

Read More »

The Best Way to Use Signals in Angular Apps

Since Angular 16, Angular now has experimental support for signals and there is a lot of confusion in the community about whether this is going to replace RxJS or how it should be used in an app in combination with RxJS. This blog post sheds some light on what I think is the best way

Read More »

High ROI Testing with Cypress Component Testing

Testing is one of the most struggled topics in Angular development and many developers are either giving up testing altogether or applying inefficient testing practices consuming all their precious time while giving few results in return. This blog post will change all this as we will cover how I overcame these struggles the hard way

Read More »

Supabase and Angular: A Powerful Combination for Building Web Applications

Supabase is a cloud-based backend as a service (BaaS) platform that provides developers with a set of tools and services for building scalable and secure web applications.It’s much like Firebase but Supabase provides a PostgreSQL database which solves some of the inconveniences with a NoSQL database such as Firestore.For that reason, Supabase has now become

Read More »