In classical Monolith-based environments there are normally not so many separate release artifacts, so it’s relatively easy to maintain the Jenkins build jobs manually. But in a Microservice architecture the number of deployment units increases and an automated solution is needed here. Also in enterprise environments with a lot of build jobs it doesn’t really make sense to create every job manually. Furthermore, in a lot of jenkins installations we see often Continuous Delivery Pipelines with separate jobs for building, releasing, deploying and testing the applications.
A Continuous Delivery pipeline for a monolithic application looks often likes this:
- build-application
- release-application
- deploy-application-to-dev
- run-tests
- deploy-application-to-qa
- …
Goals
In this blog post I will show you a way how you can achieve these goals:
- Automatically generate a jenkins job, when a user checks in a new repository/artifact into the SCM
- Reduce the number of jenkins jobs to 1 job for each release artifact, which is able to build, release and deploy the application/microservice
Build and Release
Before we dive into the magic of the Job DSL Plugin I will show you my preferred way of releasing an artifact with Maven. Maybe some of you are also not really satisfied with the Maven Release Plugin . I see two problems here:
- Two many redundant steps (e.g. 3 full clean/compile/test cycles!!)
- Instability – the plugin modifies the pom.xml in the SCM, so there are often manual steps needed to revert the changes if something fails in the release build
For further informations I can recommend the blog posts from Axel Fontaine. He also shows an alternative to release your artifacts in a clean and simple way, which we are also using in this blog post.
The continuous build is very simple:
- Jenkins triggers build process on SCM commit
- Execute normal build process:
1mvn clean package
The release build is also very simple:
- User triggers release build
- Replace SNAPSHOT-Version in pom.xml
- Execute normal build process
- Deploy artifact to artifact repository
- Tag version in SCM
1mvn build-helper:parse-version versions:set -DnewVersion=${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}-${BUILD_NUMBER}
1mvn clean package
1mvn deploy
1mvn scm:tag
Generate a single job
To generate the jobs automatically we need the Job DSL Plugin in Jenkins. You can install it through the normal Jenkins plugin mechanism. I can definitely recommend this plugin. I’m using it in some projects for about a year to generate hundreds of jobs and we didn’t have any big problems yet. It’s also very simple to use. I recommend to start with this detailed tutorial here . In our setup the seed job id called “job-generator” and contains for the first step this DSL script:
1job(type: Maven) { 2 name("batch-boot-demo") 3 triggers { scm("*/5 * * * *") } 4 scm { 5 git { 6 remote { 7 url("https://github.com/codecentric/spring-samples") 8 } 9 createTag(false) 10 } 11 } 12 rootPOM("batch-boot-demo/pom.xml") 13 goals("clean package") 14 wrappers { 15 preBuildCleanup() 16 release { 17 preBuildSteps { 18 maven { 19 mavenInstallation("Maven 3.0.4") 20 rootPOM("${projectName}/pom.xml") 21 goals("build-helper:parse-version") 22 goals("versions:set") 23 property("newVersion", "\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.incrementalVersion}-\${BUILD_NUMBER}") 24 } 25 } 26 postSuccessfulBuildSteps { 27 maven { 28 rootPOM("${projectName}/pom.xml") 29 goals("deploy") 30 } 31 maven { 32 rootPOM("${projectName}/pom.xml") 33 goals("scm:tag") 34 } 35 downstreamParameterized { 36 trigger("deploy-application") { 37 predefinedProp("STAGE", "development") 38 } 39 } 40 } 41 } 42 } 43 publishers { 44 groovyPostBuild("manager.addShortText(manager.build.getEnvironment(manager.listener)[\'POM_VERSION\'])") 45 } 46}
The above DSL script contains all the discussed build and release steps from the first section of the blog post. To integrate the release step into the existing build job, we are using the Jenkins Release Plugin , which adds a Release Button (see screenshot) to the build job UI. The “groovyPostBuild”-element adds the version number to the build history overview (see screenshot), so you can directly see it, when you open the job view. To play around with the plugin I prepared a Docker image which contains all the needed stuff here . Please follow the setup instructions on Github. Alternatively you can also use your own Jenkins and install the plugins by yourself (see a list here ).
Deployment
Actually, the above job is able to build and release artifacts, but the deploy step is missing. Because we don’t want to add separate deploy jobs for each artifact, we use the Promoted Builds Plugin . This plugin introduces the notion of a “promotion”. A “promoted” build is a successful build that passes additional criteria. In our scenario we manually promote builds, when they are deployed to a specific stage. These promoted builds (released artifacts) get a nice star in a specific colour in the build history view (see screenshot below) for every deployment stage.
Add the following DSL snippet to the existing script (see also job-dsl-example.groovy ):
1promotions { 2 promotion("Development") { 3 icon("star-red") 4 conditions { 5 manual('') 6 } 7 actions { 8 downstreamParameterized { 9 trigger("deploy-application","SUCCESS",false,["buildStepFailure": "FAILURE","failure":"FAILURE","unstable":"UNSTABLE"]) { 10 predefinedProp("ENVIRONMENT","dev.microservice.com") 11 predefinedProp("APPLICATION_NAME", "\${PROMOTED_JOB_FULL_NAME}") 12 predefinedProp("BUILD_ID","\${PROMOTED_NUMBER}") 13 } 14 } 15 } 16 } 17 promotion("QA") { 18 icon("star-yellow") 19 conditions { 20 manual('') 21 upstream("Development") 22 } 23 actions { 24 downstreamParameterized { 25 trigger("deploy-application","SUCCESS",false,["buildStepFailure": "FAILURE","failure":"FAILURE","unstable":"UNSTABLE"]) { 26 predefinedProp("ENVIRONMENT","qa.microservice.com") 27 predefinedProp("APPLICATION_NAME", "\${PROMOTED_JOB_FULL_NAME}") 28 predefinedProp("BUILD_ID","\${PROMOTED_NUMBER}") 29 } 30 } 31 } 32 } 33 promotion("Production") { 34 icon("star-green") 35 conditions { 36 manual('prod_admin') 37 upstream("QA") 38 } 39 actions { 40 downstreamParameterized { 41 trigger("deploy-application","SUCCESS",false,["buildStepFailure": "FAILURE","failure":"FAILURE","unstable":"UNSTABLE"]) { 42 predefinedProp("ENVIRONMENT","prod.microservice.com") 43 predefinedProp("APPLICATION_NAME", "\${PROMOTED_JOB_FULL_NAME}") 44 predefinedProp("BUILD_ID","\${PROMOTED_NUMBER}") 45 } 46 } 47 } 48 } 49}
Through the element “manual(‘user’)” it’s possible to restrict the execution of a promotion to a specific user or group. Especially in production this does make sense 😉 When the promotion is manually approved the promotion actions get executed. In our scenario we only trigger the downstream deploy job. After the successful execution the build gets a coloured star. With the “upstream(‘promotionName’)”-element you can make promotions dependent on another promotion, e.g. a deployment to production is only allowed, when the artifact was already deployed to development and qa. This all works great, but there is also a bad message for you. The Promoted Builds Plugin is actually not officially supported by the Job DSL Plugin. My colleague Thomas has implemented some really great stuff , but the Pull Request is not merged until today. But there is an alpha release available , which can be used to generate the promotions. Hopefully we hear some better news from it soon. An alternative solution is to create a template job with the defined promotions which is then referenced with the using-element in the Job DSL definition.
Generate multiple jobs
In the step above the seed job generates only one jenkins job (“batch-boot-demo”). So let’s generate multiple jobs from the Job DSL template. In our example we are using an repository on Github with some Spring Boot projects. Thus it’s simple to get the contents of the repository over the Github API .
1def repository = 'codecentric/spring-samples' 2def contentApi = new URL("https://api.github.com/repos/${repository}/contents") 3def projects = new groovy.json.JsonSlurper().parse(contentApi.newReader()) 4projects.each { 5 def projectName = it.name 6 job(type: Maven) { 7 name("${projectName}") 8 triggers { scm("*/5 * * * *") } 9 10 [...] 11 } 12}
Alternatively you can detect your projects with a simple shell script (e.g. interact directly with the SCM-API) and write the contents into a file:
1codecentric:spring-samples
And then read it inside the DSL script:
1readFileFromWorkspace("projects.txt").eachLine { 2 3 def repository = it.split(":")[0] 4 def projectName = it.split(":")[1] 5 6 [...] 7 8}
Now it’s very important to create a SCM trigger for the seed job, because it should run at any time a project is checked in or deleted. The Job DSL Plugin is even so intelligent that it deletes a Jenkins job if the corresponding repository/project was removed.
What do you think about it? Are you already using the Job DSL Plugin ?
Was this post helpful?
More articles
fromDennis Schulte
5.4.2015 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Dennis Schulte
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.