Automated Wildfly deployments with Shell scripting

Continuous Delivery is hot nowadays and many companies jump in by providing (expensive) tools to assist this process. In this blog post I’m hoping to show that to build a Continuous Delivery pipeline, it’s not always necessary to use 3rd party tools, but a lot can already be achieved by writing some simple shell scripts. There are some caveats naturally, which we’ll cover along the way. But first let’s start by listing our requirements and our limitations.

<< disclaimer:="" I'm="" no="" bash="" guru,="" so="" probably="" lots="" of="" stuff="" can="" be="" improved,="" but="" currently="" seems="" to="" working="" for="" us="">>

Setting the Stage

Our requirements were, at least initially, quite simple:

• Being able to deploy fully automatically from Jenkins
• Use the same deploy process on all environments

Simple, right? Our list of limitations however was a bit longer:

• Infrastructure is outsourced to another party
• We (the developers) have full control over CI and TEST environment, but no control over ACCP and PROD
• Deployments to ACCP and PROD cannot be done automatically (for now) and need some form of permissions
• Multiple application (modules) are installed onto a single Wildfly instance
• Limited downtime. Preferrably when one module is (re-)installed others need to keep running

Design Decisions

By the limitations imposed on us, we quickly came to the choice to create a self-contained shell script (script with the binary that needs to be deployed attached to it). It would be possible to let Jenkins execute this script automatically by using a Maven profile in combination with the ant-run plugin. But the script can also be handed over to the 3rd party responsible for our deployments to acceptance and production environments.

Another decision was made to use the HTTP API for Wildfly deployments, basically by this nice post from Arun Gupta: http://blog.arungupta.me/deploy-to-wildfly-using-curl-tech-tip-10/.
The advantage it has over just placing files in the ‘deployment’ directory, is that this API can be controlled by a username / password. So if later we would like to automate the deployment to acceptance and production environments from Jenkins, we could make a parametrized build and in this way let someone provide the password necessary for the deployment.

Creating the Self-Contained Shell Script

To create a shell script with the container attached to it is actually quite easy. We are using another script for this, which is our ‘compress’ script, and is being called from Maven during a build.

The important line is the following:

1cat deploy_script.sh our_assembly.war > target/our_assembly-deploy.sh

It just attaches the bytes from the WAR file to our prepared deploy script and creates a new file.

To extract the WAR file during installation, we need the following in our prepared deploy script:

1function ExtractArchive {
2    # Find the line inside this file, where the archive starts
3    ARCHIVE=`awk '/^__ARCHIVE_BELOW__/ {print NR + 1; exit 0; }' $0`
4 
5    # Grab the archive part, and extract it into the temp directory
6    tail -n+${ARCHIVE} $0 > ${TMPDIR}/our_assembly.war
7}
8 
9__ARCHIVE_BELOW__

It’s important to note that the “__ARCHIVE_BELOW__” must always be the last line in your script, because the ExtractArchive function will put everything below that line back into the WAR file. Here, the WAR file is extracted into a temp directory we created earlier, and stored the location into the TMPDIR variable.

The Deployment

The deployment initially seemed easy, requiring only these two lines:

1echo "Upload new war from $TMPDIR/our_assembly.war"
2BYTES_VALUE=`curl -F "file=@${TMPDIR}/our_assembly.war" --digest ${WF_MANAGEMENT_URL}/add-content | perl -pe 's/^.*"BYTES_VALUE"\s*:\s*"(.*)".*$/$1/'`
3echo ""
4 
5JSON_STRING_START='{"content":[{"hash": {"BYTES_VALUE" : "'
6JSON_STRING_END='"}}], "address": [{"deployment":"our_assembly.war"}], "runtime-name":"our_assembly.war", "operation":"add", "enabled":"true"}'
7JSON_STRING="${JSON_STRING_START}${BYTES_VALUE}${JSON_STRING_END}"
8 
9echo "Deploy new war"
10RESULT=`curl -S -H "Content-Type: application/json" -d "${JSON_STRING}" --digest ${WF_MANAGEMENT_URL} | sed -ne "s/.*outcome\" *: *\"\([a-zA-Z]\+\).*/\1/p"`
11echo "Deployment result: ${RESULT}"

First the WAR module is uploaded to Wildfly, which as result will generate a unique byte-string to reference the content. In the second step the content is added as deployment and enabled (deployed) as well. The WF_MANAGEMENT_URL is a variable pointing to the correct Wildfly instance. It should be something like; http://${ADMIN_USER}:${ADMIN_PASSWD}@yourhost:9990/management

Undeploying is even easier:

1echo "Undeploy old war"
2curl -S -H "content-Type: application/json" -d '{"operation":"undeploy", "address":[{"deployment":"our_assembly.war"}]}' --digest ${WF_MANAGEMENT_URL}
3echo ""
4 
5echo "Remove old war"
6curl -S -H "content-Type: application/json" -d '{"operation":"remove", "address":[{"deployment":"our_assembly.war"}]}' --digest ${WF_MANAGEMENT_URL}
7echo ""

Then a difficult request came: rollbacks
This would have been easy, had we just copied the WAR file to the deployment directory of Wildfly. But using the HTTP API, there is no way to download the currently deployed WAR file from Wildfly, to be able to restore it later on.
Also, the API contains various methods for deployments and replacing content (operations such as “replace-deployment” and “full-replace-deployment” Wildfly Model Reference). But none of them support any form of rollback. If the new deployment failed, the old assembly is already removed.

To solve this issue we created our own solution, based on the fact that the WAR file uploaded to Wildfly must be unique, but you can give it a ‘runtime-name’ that doesn’t have to be unique.
Because we had modules in production already, one deployment would keep the ‘regular’ name, the alternative or second deployment would get a prefix, and we can interchange them for each deployment.

This became the full solution:

1function ExecuteDeployment {
2    echo "Checking current deployment, whether 'blue' or 'green' is running"
3    BLUE=`curl -S -H "Content-Type: application/json" -d '{"operation":"read-attribute", "name":"runtime-name", "address":[{"deployment":"our_assembly.war"}]}' --digest ${WF_MANAGEMENT_URL} | sed -ne "s/.*outcome\" *: *\"\([a-zA-Z]\+\).*/\1/p"`
4 
5    if [ "${BLUE}" == "success" ]; then
6        OLD_DEPLOY=our_assembly.war
7        NEW_DEPLOY=ALT_our_assembly.war
8        mv ${TMPDIR}/our_assembly.war ${TMPDIR}/${NEW_DEPLOY}
9        echo "BLUE deployment active, new WAR name will be; ${NEW_DEPLOY}"
10    else
11        OLD_DEPLOY=ALT_our_assembly.war
12        NEW_DEPLOY=our_assembly.war
13        echo "GREEN deployment active, new WAR name will be; ${NEW_DEPLOY}"
14    fi
15 
16    echo "Undeploy old war"
17    curl -S -H "content-Type: application/json" -d '{"operation":"undeploy", "address":[{"deployment":"'${OLD_DEPLOY}'"}]}' --digest ${WF_MANAGEMENT_URL}
18    echo ""
19 
20    echo "Upload new war from $TMPDIR/$NEW_DEPLOY"
21    BYTES_VALUE=`curl -F "file=@${TMPDIR}/${NEW_DEPLOY}" --digest ${WF_MANAGEMENT_URL}/add-content | perl -pe 's/^.*"BYTES_VALUE"\s*:\s*"(.*)".*$/$1/'`
22    echo ""
23 
24    JSON_STRING_START='{"content":[{"hash": {"BYTES_VALUE" : "'
25    JSON_STRING_END='"}}], "address": [{"deployment":"'${NEW_DEPLOY}'"}], "runtime-name":"'${WAR_FILE}'", "operation":"add", "enabled":"true"}'
26    JSON_STRING="${JSON_STRING_START}${BYTES_VALUE}${JSON_STRING_END}"
27 
28    echo "Deploy new war"
29    RESULT=`curl -S -H "Content-Type: application/json" -d "${JSON_STRING}" --digest ${WF_MANAGEMENT_URL} | sed -ne "s/.*outcome\" *: *\"\([a-zA-Z]\+\).*/\1/p"`
30    echo "Deployment result: ${RESULT}"
31    echo ""
32 
33    if [ "${RESULT}" == "success" ]; then
34        echo "Remove old war"
35        curl -S -H "content-Type: application/json" -d '{"operation":"remove", "address":[{"deployment":"'${OLD_DEPLOY}'"}]}' --digest ${WF_MANAGEMENT_URL}
36        echo ""
37    else
38    echo "Deployment failed! Try reverting to old deployment"
39        curl -S -H "content-Type: application/json" -d '{"operation":"undeploy", "address":[{"deployment":"'${NEW_DEPLOY}'"}]}' --digest ${WF_MANAGEMENT_URL}
40        curl -S -H "content-Type: application/json" -d '{"operation":"remove", "address":[{"deployment":"'${NEW_DEPLOY}'"}]}' --digest ${WF_MANAGEMENT_URL}
41        curl -S -H "content-Type: application/json" -d '{"operation":"deploy", "address":[{"deployment":"'${OLD_DEPLOY}'"}]}' --digest ${WF_MANAGEMENT_URL}
42        echo ""
43    fi
44}

The function will first undeploy the existing WAR file (after checking what the existing one is). Then it will deploy the new WAR file. If this new deployment fails, it will revert to the previous WAR file, otherwise remove it completely.

Memory Issues

At some point, after adding more and more modules to Wildfly, we stumbled upon a memory issue. Because we tried to limit the Wildfly restarts, the JVM permgen would fill-up after each deployment, where at some point we would get an out-of-memory error. While this is not really a deployment issue but more an architectural problem, we wanted to make sure deployments would not fail when Wildfly risked to run out of memory.

Luckily, the Wildfly HTTP API also allows to retrieve MBean information, which led to the following solution:

1# Get the current memory usage and parse to get percentage
2    local MEMORY=`curl -S -H "Content-Type: application/json" -d '{"operation":"read-attribute", "name":"non-heap-memory-usage", "address":[{"core-service":"platform-mbean"}, {"type":"memory"}]}' --digest ${WF_MANAGEMENT_URL}`
3    local MAX=`echo ${MEMORY} | sed -ne "s/.*max\" *: *\([0-9]\+\).*/\1/p"`
4    local USED=`echo ${MEMORY} | sed -ne "s/.*used\" *: *\([0-9]\+\).*/\1/p"`
5    local CURRENT_MEM=$(echo "${USED} / ${MAX}" | bc -l)
6    echo ""
7    echo "Current non-heap memory percentage in use: ${USED} / ${MAX} = ${CURRENT_MEM}"
8    echo ""
9 
10    if (( $(bc <<< "${CURRENT_MEM} < ${WF_MEMORY_TRESHOLD}") == 1 )) ; then
11        # Restart not necessary, return
12        return
13    else
14        if [ "${ENVIRONMENT}" != "TEST" ] ; then
15            # Ask user for permission
16            echo ""
17            read -t 60 -p "Wildfly restart necessary. To confirm enter the text 'restart' and press [ENTER]: " CONFIRMATION
18            CONFIRMATION=${CONFIRMATION^^} # To upper case
19            if [ "${CONFIRMATION}" != "RESTART" ]; then
20                echo ""
21                echo "No confirmation for restart, aborting installation!!!"
22                Cleanup
23                exit 1
24            fi
25        fi
26    fi

What we do is read the non-heap-memory, and calculate the usage percentage. If this comes above a threshold of e.g. 0.9 (90%), Wildfly must first be restarted before the deployment can continue.
On development environments (ENVIRONMENT variable set to TEST), the restart is allowed to proceed automatically. But on acceptance and production environments this restart must first be acknowledged by the person executing the deployment. Because for now this script is executed manually, we can just ask for confirmation on the command-line.

Learning Points

Writing this script has certainly not been easy, and some things are still not optimal or could be improved. But over time, we managed to make deployments more reliable. And when there is now a change in requirements, we should be able to cater for that quite easily.

When doing an exercise like this, I found it helpful to think about the actions you would manually do, and then step by step start automating. Testing the script along the way.
Also I found that with shell scripts, it is far easier to prevent errors by checking for preconditions, then it is trying to recover from errors. For example make sure that the directories you expect are existing, that Wildfly is running, that the script is executed by the correct user, etc etc.

Hope to hear for any improvements or questions you might have. Good luck coding.