Spring Batch read multiple queries and write into single file

The core processing in Spring Batch framework can be group into 3 sections in each step:

1. Reader

2. Processor

3. Writer

A batch job may consist one or more steps, and each step must define a reader and writer. Spring batch has a simple interface called Tasklet which can be used to do single operation example a web service call.

However, in some scenario people may want to query a few set of data from a database before writes them one shot into a single output file. Because reader and writer are compulsory for each step, if we never specify the writer in a job step, the system will complain during runtime. In this article, we will discuss a trick to overcome the drawback.

1. Define Batch Job with two steps.

In the car_job.xml, we define three steps, first two steps will read in one set of data each from the database and write into a flat file. The third step will read in both file and combined into one single fine. The commit-interval basically means that once the number of items read equals the commit interval, the entire chunk of data will be written out via the ItemWriter, and then the transaction is committed.

2. Read multiple queries

In Spring Batch framework, there are few ways to read data from a database. For simple SQL we can use org.springframework.batch.item.database.JdbcCursorItemReader. Below snippet code shows how to define a reader using JdbcCursorItemReader.

In CarRowMapper.java will map the database row record to the car object.

3. Write data into flat file

If no ItemProcessor defined, ItemReader will pass the chunks to ItemWriter to process. We can either write the data into a database or flat file. In this article, we will write the chunks into a flat file for later used.

4. Multiresourceitemreader to merge two files

Now, assumed we have two flat file generated from step one and step two which located at “D:\spring-batch\tut\in\”. We can used org.springframework.batch.item.file.MultiResourceItemReader to read in the file one by one sequentially.

We specify the resources list at the resources property and delegate the reading job to thirdReader which used org.springframework.batch.item.file.FlatFileItemReader to read the text file. In this reader, no need to indicate the mapper as the data already being mapped and processed in step one and two.

Lastly, used the thirdWriter to write all the data into single file.