Nx is a build system with best-in-class support for monorepos. Nx helps establish structured code sharing and ownership management to keep your workspace maintainable. Furthermore, Nx also provides a modern integrated dev experience with a dedicated VS Code plugin, interactive visualizations, and Github integrations. This article will focus on the devops aspect of your workflow and how Nx can help improve and optimize your devops pipeline.
Drawing from their years of experience in the industry, Christian and Lars discuss how to set up a devops pipeline with Nx, including how to create a more efficient workflow and address security concerns when using services like Nx Cloud.
If you prefer a video version of this article, watch the entire discussion on Youtube.
TLDR:
- Setup a basic CI pipeline with Nx
- Setup a pipeline with ‘nx affected’ commands
- Distributed Computation via Nx Cloud
- Security concerns when using Nx Cloud
- Setup a CI workflow with Distributed Task Execution (DTE) via Nx Cloud
Setup a basic CI pipeline with Nx
For this blog post, we’ll use Github Actions as our CI pipeline. Create a `ci.yml` file in a `.github/workflows` directory. Github Actions will look for a `.github` directory in your project and use its contents to set up a pipeline for the repo.
Github uses events (`on` keyword in the snippet below) to trigger a workflow. For example, a workflow that needs to be executed when pushing to the `main` branch or opening a PR against the `main` branch will look like this:
To use Nx’s functionalities, we’ll need to create scripts for each npm command in our `package.json` to call their respective Nx commands.
The `target` property in the build, test, lint, and e2e command refers to the target in your application’s `project.json`. This is similar to Angular CLI’s `architect` property that can be found in your `angular.json`.
Basic optimizations
The above workflow can be optimized by splitting the single job into separate jobs. Separating each task into its own job allows Github to execute the commands in parallel instead of sequentially. This can reduce the time it takes for your CI to run significantly.
The following snippet shows the updated `ci.yml` with a dedicated job for each task:
Setup a pipeline with ‘nx affected’ commands
The `nx affected` commands reduce the amount of work to be executed by only running the specified command on targets that are affected by the change. This can be done locally or in the CI depending on your setup.
Lars’s Pro Tip: Run `nx graph` to explore your workspace dependencies. You can also run `nx affected:graph` to view a graph of the dependencies that are affected by your current changes. Read more about graphs in Nx’s docs.
By default, Github only fetches the latest commit when checking out the code in the pipeline. To use `nx-affected` in our workflow, we will need Github to fetch more than just the latest commit. We can set the `fetch-depth` in our checkout step to `0` to tell Github to fetch all the branches and commits and make them available in the Github runner. This is required by `nrwl/nx-set-shas` to run the `nx affected` commands.
`nrwl/nx-set-shas` compares the latest successful build to the commit that triggered the workflow to check which part of the workspace is affected by the changes. It will then run the commands only on the target that is affected by the code changes, skipping the unaffected parts of your project.
The default build command we had, in the beginning, can be rewritten to the following to use Nx’s `affected` commands:
We’ll then need to update the `package.json` to include a new script to execute the Nx command:
Lars’s Pro Tip: Run a full build on a merge instead of just on the affected targets to check if the entire project still builds correctly. There could be some differences between the state of the PR and after everything is merged. Building everything will ensure that the changes introduced in the merge didn’t have any breaking changes. You can use an `if` statement to separate the two flows. The code snippet below shows a build command with two separate flows – one when triggered by a PR and one when triggered by a merge.
Unit tests, lint, and end-to-end follow the same pattern. Instead of running the commands on the entire project, you can use Nx’s `affected` command to only run the commands against the affected targets. The code snippet below shows the complete `ci.yml` using Nx’s `affected` command:
And the code snippet below is their corresponding scripts in the `package.json`:
Lars’s Pro Tip: Stop ignoring lint warnings by setting `max-warnings` in your lint command to `0`. This will treat warnings as errors, stopping the CI when your lint command generates any warnings.
If you notice in the `ci.yml`, the lint job is slightly different from the rest. Nx has a special command called `workspace-lint` to make sure the workspace is set up correctly and every source code file is part of an Nx workspace. This should be executed before our lint command to ensure that our workspace is valid.
Distributed Computation via Nx Cloud
By default, Nx automatically caches the Nx commands results locally to save time when the same command is run again. For example, if you run the following command to test your application:
yarn nx run-many –target=test –-all –parallel
Nx will test your application and cache the result of the test. If you run the same command again, the results will be instant as Nx will detect that there is no change between the previous and current run. Instead of rerunning the tests, Nx returns the previous test results.
Nx will print the following to let you know that it used the cached results:
existing outputs match the cache, left as is
If Nx detects that a file has changed since the last cached command was run, Nx will only run the command against the changed file and the affected projects, skipping the unaffected ones.
Taking this a step further, you can use Nx Cloud to cache the command output in the cloud across multiple machines. In practice, when a developer runs the build on their local machine with Nx Cloud enabled, it will save the output in the cloud. When another developer runs the same build command on their machine while the source code remains unchanged, Nx Cloud will skip the build process and immediately return the cached build output from the previous developer’s build run. This can save you a lot of time especially if you have long-running processes.
Setup caching in Nx Cloud
If your project wasn’t initially setup to use Nx Cloud, run the following command to connect your project to Nx Cloud and enable you to use their features:
yarn nx connect-to-nx-cloud
The command above should update your project’s configurations to include the required properties to connect to Nx Cloud.
To enable Nx Cloud to cache your command outputs, you will need to set the following config in your project’s `nx.json`:
{
"tasksRunnerOptions": {
"default": {
"runner": "@nrwl/nx-cloud",
"options": {
"accessToken": "<ACCESS_TOKEN>",
"cacheableOperations": ["build", "test", "lint", "e2e"]
}
}
}
}
Security concerns when using Nx Cloud
Using a service to run your CI in the cloud always introduces security concerns. How safe is Nx Cloud? Would using Nx Cloud create a vulnerability in my process?
Nx Cloud provides you with an option to encrypt your data both during transfer and in the cloud. These data would only be decrypted once it reaches your machines using an Nx Cloud key.
Nx Cloud Enterprise offers an additional option to run Nx Cloud on premise. This allows you to decide where the Nx Cloud instance should be running and where your data will be stored.
Lars’s Pro Tip: Enabling Nx Cloud automatically generates a read-write access token for the project and includes it in your project’s configuration. This is especially dangerous for open-source or public repos. The access token would be visible as part of your source and code and could potentially be used to inject malicious code into your Nx Cloud workflow. Make sure to replace the default token with a new read-only access token. Alternatively, store the read-write access token in Github Secrets so it is not part of the public source code.
You can read more about various security scenarios and how to address them in Nx’s official docs.
Setup a CI workflow with Distributed Task Execution (DTE) via Nx Cloud
DTE is a smarter and better CI management provided by Nx Cloud. Instead of creating separate jobs for build, test, lint, and end-to-end test and running them sequentially or in parallel, Nx will act as an orchestrator and distribute the tasks equally between the available worker agents.
Nx first checks how many Github runners or agents are currently available. It then assigns the tasks to the idle agents ensuring that the workload is balanced throughout all the workers you have in your CI.
Distributing the tasks efficiently is key when working with a huge number of CI processes. Each process takes a varying amount of time to complete and making sure that you are utilizing all available resources efficiently can be a challenge. For example, end-to-end tasks typically take longer compared to lint tasks. With Nx Cloud’s DTE, the end-to-end task and lint task will be assigned to separate workers. Once the worker with the lint task is done, Nx assigns a new task to it while the other worker is still working through the end-to-end task.
Setup DTE in Github Actions
This is an Nx Cloud feature, so make sure your project is connected to Nx Cloud to use DTE.
A typical CI workflow file using DTE is as follows:
Breaking down the DTE-specific parts of the workflow file:
- number-of-agents – the number of available agents your CI has
- init-commands – command to connect to Nx Cloud and kick off the process
- parallel-commands – commands that are run on the coordinator. By default, Nx will lint the workspace configurations to make sure your workspace is valid and apply prettier to your project.
- parallel-commands-on-agents – list of tasks to run on the available agents. The `–parallel` flag controls how many of the tasks can be assigned to a single agent.
Conclusion
Nx is a powerful tool that goes above and beyond in creating an efficient workflow for your projects. Although this post uses Nx with Github Actions to showcase how Nx can be integrated into your existing workflow, the same techniques can also be used with other CIs such as Azure Devops or Circle CI. Implement the techniques that are applicable to your project and save time and resources on your CI process.
Further reading
- Check out the Energy Insight project Lars uses to showcase some of the Nx features covered in this post,
- Learn more about Nx and its features from the Nx website.
