Scaling Batch Processing with Spring Batch

At Nucleus Software we had a nightly payment-reconciliation job that used to run for more than two hours. A single failure required manual re-runs and delayed downstream SLAs. We migrated the monolithic job to a Spring Batch microservice and achieved a 10× runtime improvement (down to ≈10 minutes) while also gaining fault-tolerance and restartability.

Key Bottlenecks Identified

• Single-threaded processing of 4 M+ rows
• Row-by-row DB writes
• No partitioning strategy

Solution Highlights

// Partitioner splits data by date range
@Bean
public Partitioner rangePartitioner() {
    return gridSize -> {
        Map<String, ExecutionContext> map = new HashMap<>();
        /* logic omitted for brevity */
        return map;
    };
}
    

We used a remote chunking step where the master distributes work to workers via Redis queues. Each worker processed ~50 k rows and sent aggregated results back.

Results & Takeaways

• Runtime: 120 min → 10 min
• CPU utilisation: 15 % → 80 % (horizontal scale)
• Restartability: automatic on step failure