How to automate OpenCms module import

In one of our projects, we have been using OpenCms for a long time as the system for editorial work regarding website content. Additionally, we continuously develop and extend a backend application that takes care of business logic like web service integration. Being an agile development team, we do of course want to have an agile testing environment as well. Thus, the first task in sprint 1 of the current project was to set up a continuous integration environment. The related user story was called “As a tester, I want to test on a daily basis in order to find defects as early as possible.” I’m now going to tell you how we realized the “daily basis” part through automated deployments.

In the “real” target environment, both OpenCms and the underlying database are clustered. So for example, as soon as you deploy a module, the OpenCms cluster logic kicks in and replicates resources to the cluster nodes. In order to reduce the technical complexity for the CI environment, we decided to use a single-node setup. At the same time, we keep in mind that cluster-related bugs can only be found after the modules/applications have been deployed to the clustered target environment.

In order to be able to test on a daily basis, and in order to test the latest versions of all components, a daily deployment of all Cms modules and the backend applications is inevitable. We have been using Hudson for several months now, which provides us with automated build and packaging jobs. Essentialy, we were only missing a link between Hudson builds and deployments on the test server.

Here’s what needs to be done:

1. After a successful build, the build artifact must be uploaded to the test server.
2. The uploaded artifacts must be deployed daily at a given time.
3. After the deployment, an email with log messages has to be sent out.

The required artifact upload to the test server was pretty easy to implement: Hudson tests and builds the artifacts using Maven, so we could simply plug into the “deploy” phase in order to call an Ant SCP target. Upload done.

Now for the actual deployment – we have been using the OpenCms Shell features for years now in order to synchronize modules between Eclipse and OpenCms, or for example to import entire sites with test content. So in order to import a module to OpenCms, we use the Cms Shell commands “loginUser”, “deleteModule” and “importModule”. Here’s an example:

1loginUser Admin admin
2deleteModule de.codecentric.example
3importModule /opt/somewhere/autodeploy/de.codecentric.example.zip
4exit

You would of course put real login information, a real module name and a real path to a module ZIP in there.

Because we develop many modules in parallel, we also have to deploy many modules at once. We decided to write a Bash script that would generate a temporary Cms Shell script which can then be executed by the Cms Shell. Take a look at the commented script:

First, set a couple of variables to configure the script:

1#!/bin/bash
2HOME_DIR=/opt/somewhere
3PACKAGE_DIR=$HOME_DIR/autodeploy
4JBOSS_BIN_DIR=$HOME_DIR/jboss/bin
5JBOSS_SERVER_DIR=$HOME_DIR/jboss/server/opencms
6OPENCMS_BASE=$JBOSS_SERVER_DIR/deploy/cms.war/WEB-INF
7 
8LOG_FILE=$PACKAGE_DIR/log
9LOG_MAILTO=receiver@example.net
10LOG_MAILFROM=autodeploy@yourhost
11 
12CMS_USER=Admin
13CMS_PASSWORD=admin
14 
15JAVA=/opt/jdk1.6.0_12/bin/java

Here’s a little convenience function that prints out which modules were found in the artifact directory:

1function list {
2    if [ 0 != $# ]; then
3        echo "The following modules were found:"
4        i=1
5 
6        for package in $@
7        do
8                echo "  $i. `basename "$package"`"
9                ((i=$i+1))
10        done
11    else
12    echo -e "No modules were found.\n"
13    fi
14}

What follows is essentialy “the heart” of the script: the “deploy” function. It takes a list of Cms module ZIP file paths as a parameter. Only if this parameter is not empty, a Cms Shell script will be generated. Everything following the “exit” statement has been copied from cmsshell.sh (see WEB-INF in the OpenCms WAR) and has of course been adapted to the specific situation on our test server. Take it at as an example 😉 As soon as the classpath is filled with the required entries, the Cms Shell is started. It executes the script we have generated, which means that the modules will now be deployed. One last remark about this: we have added a “purgeJspRepository” to the Cms Shell script in order to make absolutely sure that no JSP fragments will remain in the Flex Cache.

1function deploy {
2    if [ 0 != $# ]; then
3    cd $PACKAGE_DIR
4 
5    # Add login statement to the CMS script
6    echo "loginUser $CMS_USER $CMS_PASSWORD" > script
7 
8    # Add import statements to the CMS script
9    for zipfile in $@
10    do
11        echo "deleteModule \"`basename $zipfile .zip`\"" >> script
12        echo "importModule \"$zipfile\"" >> script
13    done
14 
15    # Purge JSP Repo
16    echo "purgeJSPRepository" >> script
17 
18    # Add exit statement to the CMS script
19    echo "exit" >> script
20 
21    for JAR in $JBOSS_SERVER_DIR/lib/*.jar; do
22        TOMCAT_CLASSPATH="$TOMCAT_CLASSPATH:$JAR"
23    done
24 
25    OPENCMS_CLASSPATH=""
26    for JAR in $JBOSS_SERVER_DIR/deploy/web.war/WEB-INF/lib/*.jar; do
27        OPENCMS_CLASSPATH="$OPENCMS_CLASSPATH:$JAR"
28    done
29 
30    $JAVA -classpath "$OPENCMS_CLASSPATH:$TOMCAT_CLASSPATH:classes" org.opencms.main.CmsShell -base="$OPENCMS_BASE" -script=script >> $LOG_FILE
31    fi
32}

The rest of the bash script consists of a couple of helper functions: they start and stop (in our case) JBoss, send out the logfile mail, and clean up after the deployment:

1function stopJBOSS {
2    $JBOSS_BIN_DIR/shutdown.sh
3    tail -f $JBOSS_SERVER_DIR/log/server.log > $PACKAGE_DIR/jboss_down.log &
4    until tail -n 1 $JBOSS_SERVER_DIR/log/server.log | grep "Shutdown complete"; do sleep 0.01; done
5    killall tail
6}
7 
8function startJBOSS {
9    rm $JBOSS_SERVER_DIR/tmp/* -r
10    rm $JBOSS_SERVER_DIR/work/* -r
11    $JBOSS_BIN_DIR/startup.sh
12    tail -f $JBOSS_SERVER_DIR/log/server.log > $PACKAGE_DIR/jboss_up.log &
13    until tail -n 1 $JBOSS_SERVER_DIR/log/server.log | grep "Started"; do sleep 0.01; done
14    killall tail
15}
16 
17function mailLog {
18    mailsubject="Auto-Deploy-Log `date "+%d.%m.%Y, %H:%M"`" 
19    cat $LOG_FILE $PACKAGE_DIR/jboss_down.log $PACKAGE_DIR/jboss_up.log | mail -s "$mailsubject" $LOG_MAILTO -a "From: $LOG_MAILFROM"
20}
21 
22function cleanup {
23    rm $PACKAGE_DIR/*
24}

In here, please note that startup.sh and shutdown.sh are not usual parts of the JBoss distribution – they are selfmade wrapper scripts.

The only part missing from the bash script now is the couple of lines calling the functions:

1echo -e "Looking for modules in: $PACKAGE_DIR"
2 
3packages=`find $PACKAGE_DIR -name *.zip`
4 
5# Do some work! :)
6stopJBOSS
7list $packages
8deploy $packages
9startJBOSS
10mailLog
11cleanup

As you can see, all ZIP files inside PACKAGE_DIR are regarded as OpenCms modules. The version of the script currently being used by us does some more work: WAR files are deployed, rulesets are updated, and soon we will also run SQL scripts along with each deployment. Feel free to change or extend the script at will!

This script enables us to precisely fulfill the story “As a tester, I want to test on a daily basis in order to find defects as early as possible.”. We are even able to run the script manually during the day in order to quickly deploy a bugfix and retest it. Of course (needless to say this) after every deployment we automatically run our Robot tests. All in all, we have created a very valuable and helpful continuous integration environment that saves us a lot of manual work and that optimally supports our job of developing high-quality software.

A quick note towards the end of this article: originally, our deployment script was only used with OpenCms 7.5.2 and JBoss 4.2.3. In the meantime, another codecentric team has successfully adapted the script and happily uses it with OpenCms on BEA WebLogic.

For the future, we are thinking about – at least partially – automating deployments on the first of three stages in the “real” target environment. We will see if OCEE (information about OCEE available here ) and the OpenCms Shell like each other. If anyone has experience with this: you are very welcome to leave comments 🙂