In times of business process engines, ESBs and SOAs you might think that the good old batch processing is forgotten. But insurance companies and banks have large amounts of data to be moved and here the batch is still the first option to choose. For some customers we have implemented Java-based batch processes and have made good experiences. In many projects we found complex and individual implementations. So, let’s have a look at some standard technology. It’s worth it.
Our partner springsource offers an Open Source Batch Framework in addition to the already very popular components such as the springframework, Spring MVC and Spring WebFlow. Unfortunately, there are many outdated web articles from the early days of the framework.
Technically, the framework is well documented . At this point I would like to show a practical example and not mentioning individual details. Using Spring Batch makes it possible to implement a working solution in a short time and only a little effort.
Example from a Car-Insurance-Company
A customer switches to another insurance company and tells his current claim categories to the new insurer. To check this, the new insurer contacts the GDV (association of german insurers). The GDV provides the so called VWB-service, the system to verify this specified categories. The basis for the communication are text files with a fixed-length record structure.
Basic Configuration
The configuration for the processing of incoming VWB-messages with Spring Batch Version 1.x looks like this:
1<bean id="vwbIncomingJob" parent="simpleJob"> 2 <property name="name" value="vwbIncoming" /> 3 <property name="restartable" value="true" /> 4 <property name="steps"> 5 <list> 6 <bean parent="skipLimitStep"> 7 <property name="streams"> 8 <list> 9 <ref bean="fileItemReader" /> 10 </list> 11 </property> 12 <property name="itemReader" ref="itemReader" /> 13 <property name="itemWriter" ref="itemWriter" /> 14 <property name="skippableExceptionClasses" value="java.lang.Exception" /> 15 <property name="fatalExceptionClasses"> 16 <value> 17 org.springframework.beans.factory.BeanCreationNotAllowedException, 18 java.lang.IllegalStateException, 19 org.springframework.jndi.JndiLookupFailureException 20 </value> 21 </property> 22 <property name="skipLimit" value="${job.vwbIncoming.skipLimit}" /> 23 <property name="commitInterval" value="${job.vwbIncoming.commitInterval}" /> 24 <property name="listeners"> 25 <list> 26 <ref bean="inputFile"/> 27 <ref bean="logFileFail" /> 28 <ref bean="logFileComplete" /> 29 <ref bean="itemLoggerListener"/> 30 </list> 31 </property> 32 </bean> 33 <bean parent="taskletStep"> 34 <property name="tasklet" ref="mailTasklet" /> 35 </bean> 36 </list> 37 </property> 38</bean>
A Spring Batch job consists in most cases of 1-n steps. In this example a special SkipLimitStep is used, where you can configure exactly which types of exceptions will be accepted or will cancel the job directly. This is usually very helpful because not all records can be interpreted correctly and a restart is not necessary, when only a few items are wrong.
The configuration also shows that the individual resources (e.g. input file, log files) are injected in the step as a listener. The purpose here is to use a Spring Batch component, which is responsible for clean creation and processing of files. In addition it is also possible to use wildcards for file names.
Example:
1<bean id="inputFile" class="org.springframework.batch.core.resource.StepExecutionResourceProxy"> 2 <property name="filePattern" value="file:${jboss.server.data.dir}${job.vwbIncoming.incoming.path}//%file.name%"/> 3</bean>
The job is divided into the following tasks:
1. ItemReader: reading the file and transform the records into an XML document
Configuration:
1<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader" > 2 <property name="comments"> 3 <list> 4 <value>#</value> 5 <value>**</value> 6 <value>KONTROLLE</value> 7 </list> 8 </property> 9 <property name="lineTokenizer" ref="flatFileTokenizer"/> 10 <property name="resource" ref="inputFile"/> 11 <property name="fieldSetMapper" ref="vwbDokumentFieldSetMapper"/> 12</bean> 13 14<bean id="flatFileTokenizer" class="org.springframework.batch.item.file.transform.PrefixMatchingCompositeLineTokenizer"> 15 <property name="tokenizers"> 16 <map> 17 <entry key="10" value-ref="recordType10" /> 18 <entry key="20" value-ref="recordType20" /> 19 <entry key="21" value-ref="recordType21" /> 20 [...] 21 </map> 22 </property> 23</bean> 24 25<bean id="recordType10" class="org.springframework.batch.item.file.transform.FixedLengthTokenizer"> 26 <property name="names" value="recordType, operationNumber, date, vuGstNr, vsnr, requestType, vehicleIdentificationNumber, [...]" /> 27 <property name="columns" value="1-2, 3-20, 7-14, 21-28, 29-48, 49-50, 51-67, [...]"/> 28</bean>
This process can be configured exclusively in the XML file. The FlatFileItemReader receives a reference to the input file and delivers each line to a LineTokenizer. The standard implementation of PrefixMatchingCompositeLineTokenizer transforms the data into FieldSets, comparable to an array or a database ResultSet, where each field can be accessed by an index. The GDV provides each record with a record type as a prefix, so that the LineTokenizer always knows exactly which fields to be mapped. Different implementations for example using dynamic record lengths are available. The FieldSetMapper is the only place where you have to implement some code. The implementation of the method public Object mapLine(FieldSet fieldSet) creates from a FieldSet the target object. This example is using a generic implementation that creates a Java object, which is later transformed using XStream into an XML document.
2. ItemWriter: Processing and persistence of the items in the target system
From the perspective of Spring Batch not much is happening here and is not supposed to! The goal should always be to delegate to a business service which is responsible for the processing. This design rule leads to the advantage of better testability and reusability by other components – for example in online processing. In the first stage the document is attached only to the target contract. So manual processing is required after the batch.
3. Tasklet: E-Mail the log files
Of course, as with any clean implemented software component there must be a monitoring. Conceivable are different approaches. Spring Batch offers a listener interface for almost any place in the job. The VWB sample log entries are written per item, which provides information about the success/failure type of processing. In the last step the MailTasklet sends the appropriate log files to the responsible persons.
1<bean id="vwbIncomingTasklet" class="com.codecentric.example.batch.tasklet.MailTasklet"> 2 <property name="mailTo"> 3 <list> 4 <value>${job.vwbIncoming.receiver1}</value> 5 <value>${job.vwbIncoming.receiver2}</value> 6 <value>${job.vwbIncoming.receiver3}</value> 7 </list> 8 </property> 9 <property name="mailSubject" value="${job.vwbIncoming.betreff}" /> 10 <property name="mailText" value="${job.vwbIncoming.body}" /> 11 <property name="mailFrom" value="${jobs.mailtemplate.sender}" /> 12 <property name="attachments"> 13 <map> 14 <entry key="vwbIncomingSuccesful" value-ref="logFileComplete" /> 15 <entry key="vwbIncomingFailure" value-ref="logFileFail" /> 16 </map> 17 </property> 18</bean>
Testing
As expected from Spring the testability of components is very easy. The job configuration can be tested with all needed dependencies using the well-known testing framework components of Spring. Here is an example providing a basis for a test:
1@ContextConfiguration(locations={"classpath:/jobs/vwbIncoming.xml"})
2public class VwbIncomingJobITest extends AbstractJUnit4SpringContextTests {
3
4 /** The Job-Executor */
5 @Autowired
6 private JobLauncher jobLauncher;
7
8 /** The job */
9 @Autowired
10 @Qualifier("vwbIncomingJob")
11 private Job job;
12
13 /** The Service for the processing of incoming documents */
14 private BusinessService businessServiceMock;
15
16 /** The inputfile */
17 private static final String INPUT_FILE = "src/test/resources/vwbIncoming/vwbTest.txt";
18
19 private JobParametersBuilder builder;
20
21 @Before
22 public void setUp() {
23 businessServiceMock= (BusinessService ) applicationContext.getBean("businessServiceMock");
24 builder = new JobParametersBuilder();
25 Resource inputFile = new FileSystemResource(INPUT_FILE);
26 builder.addString("file.name", inputFile.getFilename());
27 }
28
29 @Test
30 public void testLaunchVwbIncomingJob() throws Exception {
31 expect(businessServiceMock.verarbeiteVwbDokument(isA(VwbDocument.class))).andReturn(Boolean.TRUE);
32 replay(businessServiceMock);
33 JobExecution jobExecution = jobLauncher.run(job, builder.toJobParameters());
34 verify(businessServiceMock);
35 assertTrue(jobExecution.getStatus().equals(BatchStatus.COMPLETED));
36 }
37
38 [...]
39
40}
Conclusion
The example shown is mostly based on the old version of the framework. Currently 2.1 is released and offers useful innovations, including a simplification of the configuration. In one of my next blog entries I will discuss the differences in detail. Another interesting topic in this context is the use of Spring Integration, where we would be back in the ESB world 😉 I would be grateful for any feedback and suggestions for topics relating to Spring Batch 🙂
Was this post helpful?
More articles
fromDennis Schulte
5.4.2015 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
More articles in this subject area
Discover exciting further topics and let the codecentric world inspire you.
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Dennis Schulte
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.