Asynchronous task execution using Redis and Spring framework

Sonu Kumar
6 min readNov 1, 2019

In this article, we are going to see how to use Spring boot 2.x and Redis to execute asynchronous tasks, with the final code that we will have after following the steps described here.

Spring/Spring Boot

Spring is the most popular framework available for the Java platform. Spring has one of the largest communities in open source. Besides that, Spring provides extensive and up-to-date documentation that covers the inner workings of the framework and sample projects on their blog, there’re 100K+ questions on StackOverflow.

Spring 3.0 was the first version that supported the annotation-based configuration, later in 2014 Spring boot 1.0 was released that completely changed how we look at the Spring framework ecosystems, it provides out-of-the-box configurations and many more. Timeline

Redis

Redis is one of the most popular open-source NoSQL in-memory databases. Redis supports different types of data structures e.g. Set, Hash tables, Lists, and simple key-value pairs just to name a few. The latency of Redis call is sub-milliseconds, support of a replica set, etc.

Why Asynchronous task execution

A typical API call consists of five things

  1. Execute one or more database(RDBMS/NoSQL) queries
  2. One or more operations on some cache systems (In-Memory, Distributed, etc )
  3. Some computations (it could be some data crunching doing some math operations)
  4. Calling some other service(s) (internal/external)
  5. Schedule one or more tasks to be executed at a later time or immediately but in the background.

A task can be scheduled at a later time for many reasons, for example, an invoice must be generated after 7 days of order creation or order shipment, similarly, email/notification(s) need not be sent immediately it can be delayed. Sometimes we need to execute tasks asynchronous to reduce API response time, for example, delete 1K+ records at once if we delete all these records in the same API call then API response time would be increased for sure, to reduce API response time, we can run a task in the background that would delete those records.

Delayed queue

We can run scheduled tasks using cron jobs, and a cron job can be scheduled using different methods like UNIX style crontabs, and Chronos if we’re using Spring frameworks then it’s built-in Scheduled annotation ❤️. Most of these scheduling mechanisms suffer scaling problems, where we scan database(s) to find the relevant rows/records. In many situations, this leads to a full table scan which performs very poorly. Consider the case where the same database is used by a real-time application and this batch processing system, this will further degrade the application performance and it will also add spiky nature in resource(database, CPU, memory)usage. A delayed queue can be used in such cases where as soon as the timer reaches the scheduled time a job would be triggered. There’re many queuing systems/software available, but very few of them provide this feature, like SQS which provides a delay of 15 minutes, not an arbitrary delay like 7 hours or 7 days.

Rqueue

Rqueue is a broker built for the spring framework that stores data in Redis and provides a mechanism to execute a task at any arbitrary delay. Rqueue is backed by Redis since Redis has some advantages over widely used queuing systems like Kafka and SQS. In most web applications' backends, Redis is used to store either cache data or other purposes. In today's world, 8.4% of web applications are using the Redis database.

Generally, for a queue, we use either Kafka/SQS or some other systems these systems bring an additional overhead in different dimensions e.g money which can be reduced to zero using Rqueue and Redis.

Apart from the cost if we use Kafka then we need to do infrastructure setup, and maintenance i.e. more ops, as most of the applications are already using Redis so we won’t have ops overhead, in fact, the same Redis server/cluster can be used with Rqueue. Rqueue supports an arbitrary delay

Rqueue Usage

Message Delivery

Rqueue guarantees at-least-once message delivery as long data is not lost in the database. Read about it more at Introducing Rqueue

Tools we need:

1. Any IDE 2. Gradle 3. Java 4. Redis

We’re going to use Spring boot for simplicity, we’ll create a Gradle project from the Spring boot initializer at https://start.spring.io/, for dependency we would need 1. Spring Data Redis 2. Spring Web 3. Lombok and any others. The directory/folder structure would look like the one below.

We’re going to use the Rqueue library to execute any tasks with any arbitrary delay. Rqueue is a Spring-based asynchronous task executor, that can execute tasks at any delay, it’s built upon the Spring messaging library and backed by Redis.

We’ll add the Rqueue spring boot starter dependency using com.github.sonus21:rqueue-spring-boot-starter:2.7.0-RELEASE

We need to enable Redis spring boot features, for testing purposes we will enable WEB MVC as well.

Update theApplication file as

Adding tasks using Rqueue is very simple we need to just annotate a method with RqueueListener. RqueuListener annotation has multiple fields that can be set based on the use case, for example, set deadLetterQueue to push tasks to another queue otherwise the task will be discarded on failure. We can also set how many times a task should be retried using the numRetries field.

Create a Java file name MessageListener and add some methods to execute tasks.

We would need Email and Invoice classes to store email and invoice data respectively. For simplicity, classes would only have a handful number of fields.

Invoice.java

Email.java

Task submissions

A task can be submitted using the RqueueMessageEnqueuer bean. This has multiple methods to enqueue tasks depending on the use case, like retry use, retry count, and delay for delayed tasks.

We need to AutoWire RqueueMessageEnqueuer or use a constructor to inject this bean.

Create a Controller for testing purposes:

We’re going to schedule invoice generation that would be done in the next 30 seconds, for this we’ll submit a task with a 30000 (milliseconds) delay on the invoice queue. Also, we’ll try to send an email that will be done in the background. For this purpose, we’ll add two GET methods sendEmail and generateInvoice, we can use POST as well.

Add the following in the application.properties file

It’s time to fire up the spring boot application, once the application starts successfully, browse

http://localhost:8080/email?email=xample@exampl.com&subject=%22test%20email%22&content=%22testing%20email%22

In the log, we can see tasks are being executed in the background

Invoice scheduling after 30 seconds

http://localhost:8080/invoice?id=INV-1234&type=PROFORMA

In conclusion, we can schedule tasks using Rqueue without much of the boiler code. We need to consider a few things while configuring the Rqueue library and using them. One of the important is whether a task is a delayed task or not; by default, it’s assumed tasks need to be executed as soon as possible.

The complete code can be found at my Github account https://github.com/sonus21/rqueue-task-exector

Rqueue library code: https://github.com/sonus21/rqueue

If you found this post helpful please share across and give a thumbs up.

--

--