This is the first post in a series about transactions in Spring Batch, you find the second one here , it’s about restarting a batch, cursor based reading and listeners, and the third one here , it’s about skip and retry.
Transactions are important in almost any application, but handling transactions in batch applications is something a little more tricky. In standard online applications you usually have one transaction for one user action, and as a developer you normally just have to assure that your code picks up an existing transaction or creates a new one when there’s none (propagation type REQUIRED). That’s it. Developers of batch applications have much more headaches with transactions. Of course you cannot have just one transaction for the whole batch, the database couldn’t cope with that, so there have to be commits somewhere in between. A failed batch then doesn’t mean you get the unchanged data back, and when you throw in features like restarting a failed batch, retrying or skipping failing items, you automatically get a complicated transaction behaviour. Spring Batch offers the functionality just mentioned, but how does it do that?
Spring Batch is a great framework, and there is a lot of documentation and some good books, but after reading a lot about Spring Batch I still wasn’t sure about everything regarding transactions, so in the end all that helped to understand everything was looking into the code and a lot of debugging. So, this is no introduction to Spring Batch, I’m gonna focus just on transactions, and I assume that you’re familiar with transactions in Spring (transaction managers, transaction attributes). And since I have to restrict myself a little bit, I will just talk about one-threaded chunk oriented processing.
Chunk oriented steps
Let’s start with a picture that will follow us throughout this and the following blog posts, only changed in little details every now and then to focus on a certain subject.
It’s already telling a lot about Spring Batch and its transactional behaviour. In chunk-oriented processing we have ItemReaders reading items, one after the other, always delivering the next one item. When there are no more items, the reader delivers null. Then we have optional ItemProcessors taking one item and delivering one item, that may be of another type. Finally we have ItemWriters taking a list of items and writing them somewhere.
The batch is separated in chunks, and each chunk is running in its own transaction. The chunk size actually is determined by a CompletionPolicy, as you can see in the illustration at (1): when the CompletionPolicy is fulfilled, Spring Batch stops reading items and starts with the processing. By default, if you use the commit-interval attribute on chunk, you get a SimpleCompletionPolicy that is completed when the number of items you specified in the attribute is read. If you want something more sophisticated you can specify your own CompletionPolicy in the attribute chunk-completion-policy.
This is all quite straight forward, if there’s a RuntimeException being thrown in one of the participating components, the transaction for the chunk is rolled back and the batch fails. Every already committed chunk of course stays in the processed state.
Business data and batch job data
As you might know already, Spring Batch brings a set of database table definitions. These tables are used to store data about the jobs and steps and the different job and step execution contexts. This persistence layer is useful for some kind of history on the one hand, and for restarting jobs on the other hand. If you’re thinking of putting these tables in a different database than your business data: don’t. The data stored there is about the state of the job and the steps, with numbers of processed items, start time, end time, a state identifier (COMPLETED, FAILED and so on) and much more. In addition there is a map for each step (the step execution context) and job (the job execution context) which can be filled by any batch programmer. Changes in this data have to be in line with the transaction running on our business data, so if we have two databases we’ll need for sure a JtaTransactionManager handling different DataSources, suffering in performance as well. So, if you have a choice, put those tables near to your business data. In the following diagram you can see where in the processing step and job data is persisted. As you can see, it doesn’t happen only inside the chunk transaction, for good reasons: we want to have step and job data persisted in the case of a failure, too.
Note that I use little numbers for indicating items that are explained in a text box. The numbers stay in following versions of the diagram while the text box may disappear due to readability. It’s always possible to look up the explanation in a previous version of the diagram.
A failed batch
Until now, the diagram just includes successful processing. Let’s take a look at the diagram including a possible failure.
If you didn’t configure skip or retry functionality (we’ll get to that in the next blog posts) and there’s an uncaught RuntimeException somewhere in an element executed inside the chunk, the transaction is rolled back, the step is marked as FAILED and the whole job will fail. Persisting step data in a separate transaction at (5) makes sure that the failure state gets into the database.
When I say that an uncaught RuntimeException causes the rollback, then it’s not quite true for every case. We have the option to set no-rollback-exceptions:
1<batch:tasklet> 2 <batch:chunk ... /> 3 <batch:no-rollback-exception-classes> 4 <batch:include class="de.codecentric.MyRuntimeException"/> 5 </batch:no-rollback-exception-classes> 6</batch:tasklet>
Transaction attributes
One more thing for today: if you don’t configure transaction attributes explicitly, you get the defaults. Transaction attributes are propagation type, isolation level and timeout, for example. You may specify those attributes as shown here:
1<batch:tasklet> 2 <batch:transaction-attributes isolation="READ_COMMITTED" propagation="REQUIRES_NEW" timeout="200"/> 3 <batch:chunk reader="myItemReader" writer="myItemWriter" commit-interval="20"/> 4</batch:tasklet>
If you don’t specify them, you’ll get the propagation type REQUIRED and the isolation level DEFAULT, which means that the default of the actual database is used. Normally you don’t want to change the propagation type, but it makes sense to think about the isolation level and check the batch job: am I fine with non-repeatable reads? Am I fine with phantom reads? And: what other applications are accessing and changing the database, do they corrupt the data I’m working on in a way that causes trouble? Is there a possibility to get locks? For more information on the different isolation levels check this wikipedia article.
Conclusion
In this first article on transactions in Spring Batch I explained the basic reader-processor-writer cycle in chunk oriented steps and where the transactions come into play. We saw what happens when a step fails, how to set transaction attributes and no-rollback-exception-classes and how job and step metadata is updated.
Next on the list will be restart, retry and skip functionality: what are the preconditions? How does the transaction management work with these features? Click here for the next blog post in this series about restart, cursor based reading and listeners, and here for the third post about skip and retry.
Was this post helpful?
More articles
fromTobias Flohre
16.2.2022 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Tobias Flohre
Senior Software Developer
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.