This is the third post in a series about transactions in Spring Batch, you find the first one here , it’s about the basics, and the second one here , it’s about restart, cursor based reading and listeners.
Today’s topics are skip and retry functionality, and how they behave regarding transactions. With the skip functionality you may specify certain exception types and a maximum number of skipped items, and whenever one of those skippable exceptions is thrown, the batch job doesn’t fail but skip the item and goes on with the next one. Only when the maximum number of skipped items is reached, the batch job will fail. However, whenever there’s a skip, we still want to roll back the transaction, but only for that one skipped item. Normally we have more than one item in a chunk, so how does Spring Batch accomplish that? With the retry functionality you may specify certain retryable exceptions and a maximum number of retries, and whenever one of those retryable exceptions is thrown, the batch job doesn’t fail but retries to process or write the item. Same question here, we still need a rollback for the failed item if a try fails, and a rollback includes all items in the chunk. Let’s see.
Skip
As you might know, there are two ways of specifying skip behaviour in Spring Batch. They don’t make a difference regarding transactions. The convenient standard way would be specifying a skip-limit on the chunk and nesting skippable-exception-classes inside the chunk:
1<batch:tasklet> 2 <batch:chunk reader="myItemReader" writer="myItemWriter" commit-interval="20" skip-limit="15"> 3 <batch:skippable-exception-classes> 4 <batch:include class="de.codecentric.MySkippableException" /> 5 </batch:skippable-exception-classes> 6 </batch:chunk> 7</batch:tasklet>
And if you need a more sophisticated skip-checking, you may implement the SkipPolicy interface and plug your own policy into your chunk. skip-limit and skippable-exception-classes are ignored then:
1<batch:tasklet> 2 <batch:chunk reader="myItemReader" writer="myItemWriter" commit-interval="20" skip-policy="mySkipPolicy"/> 3</batch:tasklet>
Let’s get to the transactions now, again with the illustration. First we’ll have a look at a skip in an ItemProcessor.
So, if you get a skippable exception (or your SkipPolicy says it’s a skip), the transaction will be rolled back. Spring Batch caches the items that have been read in, so now the item that led to the failure in the ItemProcessor is excluded from that cache. Spring Batch starts a new transaction and uses the now reduced cached items as input for the process phase. If you configured a SkipListener, its onSkipInProcess method will be called with the skipped item right before committing the chunk. If you configured a skip-limit that number is checked on every skippable exception, and when the number is reached, the step fails.
What does that mean? It means that you might get into trouble if you have a transactional reader or do the mistake of doing anything else than reading during the reading phase. A transactional reader for example is a queue, you consume one message from a queue, and if the transaction is rolled back, the message is put back in the queue. With the caching mechanism shown in the illustration, messages would be processed twice. The Spring Batch guys added the possibility to mark the reader as transactional by setting the attribute reader-transactional-queue on the chunk to true. Done that the illustration would look different, because items would be re-read.
Even if you don’t have a transactional reader you might get into trouble. For example, if you define a ItemReadListener to protocol items being read somewhere in a transactional resource, then those protocols get rolled back as well, even though all but one item are processed successful.
It gets even more complicated when we have a skip during writing. Since the writer is just called once with all items, the framework does not know which item caused the skippable exception. It has to find out. And the only way to find out is to split the chunk into small chunks containing just one item. Let’s have a look at the slightly more complicated diagram.
We now get a second loop, indicated with the red colour. It starts with a skippable exception in our normal chunk, leading to a rollback (the yellow line). Now the framework has to find out, which item caused the failure. For each item in the cached list of read items it starts an own transaction. The item is processed by the ItemProcessor and then written by the ItemWriter. If there is no error, the mini-chunk with one item is committed, and the iteration goes on with the next item. We expect at least one skippable exception, and when that happens, the transaction is rolled back and the item is marked as skipped item. As soon as our iteration is complete, we continue with normal chunk processing.
I think I don’t need to mention that the problems with transactional readers apply here as well. In addition, it is possible to mark the processor as non-transactional by setting the attribute processor-transactional on the chunk to false (its default is true). If you do that, Spring Batch caches processed items and doesn’t re-execute the ItemProcessor on a write failure. You just can do that if there is no writing interaction with a transactional resource in the processing phase, otherwise processings get rolled back on a write failure but are not re-executed.
One more thing: what about skipping during reading? I didn’t do a diagram for that, because it’s quite simple: when a skippable exception occurs during reading, we just increase the skip count and keep the exception for a later call on the onSkipInRead method of the SkipListener, if configured. There’s no rollback.
Retry
As with the skip functionality, there are two ways of specifying retry behaviour in Spring Batch. The convenient standard way would be specifying a retry-limit on the chunk and nesting retryable-exception-classes inside the chunk:
1<batch:tasklet> 2 <batch:chunk reader="myItemReader" writer="myItemWriter" commit-interval="20" retry-limit="15"> 3 <batch:retryable-exception-classes> 4 <batch:include class="de.codecentric.MyRetryableException" /> 5 </batch:retryable-exception-classes> 6 </batch:chunk> 7</batch:tasklet>
As with skipping, you may specify your own RetryPolicy and plug it into the chunk:
1<batch:tasklet> 2 <batch:chunk reader="myItemReader" writer="myItemWriter" commit-interval="20" retry-policy="myRetryPolicy"/> 3</batch:tasklet>
Let’s take a look at the diagram for retrying.
Whenever during processing or writing a retryable exception occurs, the chunk is rolled back. Spring Batch checks if the maximum number of retries is exceeded, and if that’s the case, the step fails. If that’s not the case, all items that have been read before are input for the next process phase. Basically, all limitations that apply to skipping items apply here as well. And we can apply modifications to the transactional behaviour via using reader-transactional-queue and processor-transactional in the same manner.
One important thing: at the time of writing (Spring Batch 2.1.8) there is a bug with a failure during writing. If there’s a retryable exception during writing only the first item gets reprocessed, all other items in the cached list of read item are not reprocessed (https://jira.springsource.org/browse/BATCH-1761).
Conclusion
Spring Batch is a great framework offering functionality for complex processings like skipping or retrying failed items, but you still need to understand what Spring Batch does to avoid problems. In this article we saw potential stumbling blocks when using skip and retry functionality.
Was this post helpful?
More articles
fromTobias Flohre
16.2.2022 Your job at codecentric?
Jobs
Agile Developer und Consultant (w/d/m)
Alle Standorte
Gemeinsam bessere Projekte umsetzen.
Wir helfen deinem Unternehmen.
Du stehst vor einer großen IT-Herausforderung? Wir sorgen für eine maßgeschneiderte Unterstützung. Informiere dich jetzt.
Hilf uns, noch besser zu werden.
Wir sind immer auf der Suche nach neuen Talenten. Auch für dich ist die passende Stelle dabei.
Blog author
Tobias Flohre
Senior Software Developer
Do you still have questions? Just send me a message.
Do you still have questions? Just send me a message.