Continuous Integration of Hyperledger Composer applications with Gitlab CI

In my previous article, “Hyperledger Fabric test network on AWS using Ansible” , I introduced a simple way to provision VM instances in the cloud using Ansible with the necessary software to create a Hyperledger Fabric and Composer test environment. Assuming that this environment is now ready to install and offer Business Network Archives (BNAs), we want to continuously test, build, and deliver the code changes to this environment.

Attention: This article is from 2018. Since 2019 Hyperledger Composer is deprecated. It should not be used anymore (especially in a production environment). Instead the project suggests to look into Fabric 1.4+ and 2.0 . Also the article refers to an old version of Hyperledger Fabric and Node.js 8.12.

Gitlab CI is an excellent tool for defining such pipelines, so I’d like to present what a rudimentary pipeline implementation might look like.

Conception of a pipeline for Hyperledger Composer BNAs

The first step is the conception of the pipeline to be implemented. A classical and simple sequence for an application implemented with JavaScript, which is to be tested, installed and built, can look like this:

1. Dependency and Code Security Checks with RetireJS and NPM Audit
2. Installing the NPM Dependencies from package.json
3. Verification of compliance with the coding guidelines (linting)
4. Execution of Unit Tests
5. Execution of BDD tests
6. Building the BNA and installing/updating on the target environment
7. Increasing the minor version in a CI commit as well as Git tagging

We use the demo application on Github from one of my earlier articles “Implementing a Blockchain Application with Hyperledger Fabric and Composer” as a guideline for the practical implementation of the pipeline. In the application’s repository, the branch “gitlab-ci-pipeline” was created, which demonstrates the implementation of this pipeline. However, before we move step by step closer to the pipeline, we should see if it can be implemented without problems.

The package.json of the demo application already contains some scripts that we need for the individual pipeline steps (jobs). NPM Audit , the Dependency Security Checker recently integrated by Node, is also available without further ado. In order to scan the dependencies even deeper, we install and use RetireJS in addition. We can call the functionality of Composer via the local installation in ./node_modules/.bin/composer. For the creation of the Git version bump commit and tag we can use NPM, which provides us with “npm version patch” . For the required download of the Composer PeerAdmin-Card, which is needed for the deployment or the access to the network with the PeerAdmin role, wget is available in the Node-Docker-Image. We should use a specific version of Node (let’s choose 8.12.0) in code (forced by .nvmrc in the repository), so we also use the same Docker image in the Gitlab build because it comes with the required NPM tools in the right version.

Another useful feature of Gitlab CI is that jobs can be parallelized with the help of stages. The NPM Audit can run in parallel with the installation of the NPM dependencies. Afterwards, in the next stage, linting, different tests, and the RetireJS checks can also be run in parallel without affecting each other. As a result, the entire pipeline runtime is minimized, but job dependencies are still maintained. If all jobs were successful before deployment, the deployment can complete the pipeline run with subsequent tagging. We therefore call the stages as follows: “Security & Setup”, “Test” and “Deploy & Release”.

Implementation of the pipeline

It has been ensured that the desired pipeline can be implemented without plumbering some special scripts. The concept in the form of steps and stages is ready and we already know that many jobs can be parallelized. Therefore we start at the very beginning, with the basic scaffold of the pipeline definition for Gitlab CI and create .gitlab-ci.yml in the Gitlab repository of our application.

The caching feature of Gitlab CI is an excellent tool for caching the usually very heavy node_modules of NPM. At the end of the pipeline, Gitlab will ensure that the specified paths are updated to the cache. In the following jobs and future pipeline runs, the directory structures will be restored from the cache, saving us precious time and nerves, as all dependencies are installed by the presence of node_modules.

The file should initially look like this:

1image: node:8.12.0 # we use the same image through the whole build
2
3cache:
4  paths:
5    - node_modules/
6
7variables:
8  COMPOSER_CMD: "./node_modules/.bin/composer"
9
10stages:
11  - security-and-setup
12  - test
13  - deploy-and-release
14 
15# all job definitions will be listed on the base level, for example: 
16# my-first-job:
17#   stage: security-and-setup
18#   script: npm run some-task
19#   (only "script" is required but there are a lot of more features: https://docs.gitlab.com/ee/ci/yaml/#jobs)

We start with the implementation of the first stage and create two jobs as planned: NPM Audit Security Check and NPM Install. After the above definition of the stages, we start with them:

1install-dependencies:
2  stage: security-and-setup
3  script: npm install --progress=false
4
5npm-audit-security-check:
6  stage: security-and-setup
7  script: npm audit

These two jobs are easy to read and understand. The dependencies are installed and checked for known vulnerabilities in parallel. At the end of install-dependencies, node_modules was created or updated.

The next step is the implementation of the test stage with jobs for RetireJS check, linting, unit tests, and BDD tests. Unfortunately, JavaScript developers don’t learn many new things here either:

1retire-security-check:
2  stage: test
3  script: ./node_modules/.bin/retire -p
4  dependencies:
5    - install-dependencies
6
7lint:
8  stage: test
9  script: npm run lint
10  dependencies:
11    - install-dependencies
12
13unit-tests:
14  stage: test
15  script: npm run test:unit
16  dependencies:
17    - install-dependencies
18
19bdd-tests:
20  stage: test
21  script: npm run test:bdd
22  dependencies:
23    - install-dependencies

The test jobs are dependent on the “install-dependencies” job, since the installed dependencies are also required to execute them. The unit and BDD tests and linting jobs will start running in parallel as soon as the dependencies are installed.

So far everything seems very familiar if you are occasionally on the road in the JavaScript and NPM universe. The next step is to connect to the Hyperledger test network, create the version patch commit and tag, build the business network archive, and install it on the network.

1patch-deploy-publish:
2  stage: release-and-deploy
3  before_script:
4    - wget ${COMPOSER_DEV_CARD_LOCATION}
5    - ${COMPOSER_CMD} card import --file fabric-dev-peer-admin.card
6    - ${COMPOSER_CMD} card list
7  script:
8    - git config user.name "Gitlab CI" && git config user.email "some-product-team@codecentric.de"
9    - NEW_APP_VERSION="$(npm version patch -m '[skip ci] patch version upgrade')"
10    - npm run archive:create
11    - ${COMPOSER_CMD} network install -a dist/engine-supplychain.bna --card PeerAdmin@hlfv1
12    - ${COMPOSER_CMD} network start -n engine-supplychain -V ${NEW_APP_VERSION/v/} --networkAdmin admin --networkAdminEnrollSecret adminpw --card PeerAdmin@hlfv11 --file local-network-admin.card || ${COMPOSER_CMD} network upgrade -c PeerAdmin@hlfv1 -n engine-supplychain -V ${NEW_APP_VERSION/v/}
13  after_script:
14    - git push https://mygitlabname:${PRIVATE_ACCESS_TOKEN_MYGITLABNAME}@gitlab.com/${CI_PROJECT_PATH}.git HEAD:master --follow-tags
15  only:
16    - master
17  artifacts:
18    paths:
19      - dist/engine-supplychain.bna
20      - local-network-admin.card
21    expire_in: 2 week

A little more is happening here than in the previous jobs. First, it was decided that releasing and deployment would only be done on the master branch. On other branches, possibly because feature branches are used, only the first two stages will be executed. For this job to work, it is important to create two special build variables in Gitlab for the project. These should be marked as “protected” if our master branch is protected so that temporary branches cannot use these secrets. A complexity in this job that needs some explanation is the different handling of BNAs for the first time as opposed to BNAs that are already deployed. If a BNA has already been started in one version, composer upgrade must be used to switch to the newly installed version. Initially composer start must be used to start the installed BNA. This is resolved with the OR-operator that uses the update command if the start command does not yield a successful exit code.

Under COMPOSER_DEV_CARD_LOCATION a path to download the PeerAdmin@hlfv1 Composer-Card must be saved. The card contains connection and authentication data against the Hyperledger test network we provisioned. The map was created by the Ansible Playbook when setting up the environment, downloaded to the Ansible host and imported into the local card store. There we can find them and make them available e.g. via an Amazon S3 bucket. For use in a real project, you should think about how to protect the download link of the card with authentication and how the pipeline script can authenticate itself. To make the variable PRIVATE_ACCESS_TOKEN_MYGITLABNAME available in the job, a private access token must be generated in Gitlab for our (or another) user. We store the token in this protected variable. Git uses the token in this job to execute the push on behalf of the token creator. After the push, the version tag and commit are available on the master branch. However, the pipeline is not executed again because the string “skip ci” Gitlab indicates that the commit should be skipped.

When the job has completed successfully, the built BNA and the Composer card of the local network admins are kept as an artifact for two weeks.

Continuous Integration and Hyperledger – The result

As presented, Gitlab CI can create a powerful pipeline in YAML format in just 70 lines of code. The features of Gitlab CI leave no wish unfulfilled for this use case and work on any current installation – whether (even in the free plan) on Gitlab.com or your own on-premise installations. The interaction with Composer turns out to be problem-free, because Composer-CLI is an NPM module that is already required in the dev-dependencies of the project. Since the Composer card bundles all the details needed to communicate with a Composer network from the outside, no annoying scripting is required to implement the deployment. In case the deployment should go wrong, the Git-push of version tag and commit will still be performed. We can take a look at the resulting pipeline in Gitlab. Since the demo repository is not hosted on Gitlab and no test network has been provisioned for it, here is an example.

The pipeline runs in about five minutes, which is quick. The security check results are displayed with a warning sign because they failed. However, in the pipeline definition from which the screenshot was taken, these jobs were annotated with allow_failure: true to provoke a successful build despite partly vulnerable dependencies. At this point, we have an executable test environment with Hyperledger Fabric and Composer Playground, which we can continuously update with tested changes and features. How the code changed during the installation of the Gitlab CI pipeline can be seen in this pull request in the comparison view on the repository .

As explained in previous article , setting up, operating and deciding on a production environment with Hyperledger is still associated with many questions and challenges. The industry is still a long way from reaching a consensus on best practices, tools, and legal frameworks. When the interaction of these factors has matured a bit, there will of course be a blog article on this topic as well.