Tag: data processing

Real-time data processing is paramount in many modern applications across multiple domains, from monitoring financial transactions to analytics on IoT sensor data. Although historical solutions required the implementation of a complex distributed system involving possibly many JVMs, we are able to create a more simplistic yet powerful solution for stream processing by using Micronaut and Kafka Streams in combination. This blog demonstrates how to develop an entire real-time word counting application that will cover the important foundational areas of stream processing, and it will also use modern best practices to create a simple executable jar that can be deployed easily.

Why Micronaut + Kafka Streams?

Before we get to coding our application, it is essential to understand why this combination is one that many developers are now considering for stream processing applications.

Micronaut Benefits

Micronaut is a new framework that, instead of handling dependency injection and configuration at runtime, processes both at compile time. This offers several advantages for microservices and stream processing applications.

The benefits include:

• Fast Startup Time: As there is no reflection processing because it is done at compile-time
• Reduced Memory: Fewer runtime dependencies and smaller bytecode
• More Cloud-Friendly: Built for container-based and serverless environments
• Improved Developer Experience: Shorter development cycles due to shorter restarts

Kafka Streams: Stream Processing Without Separate Clusters

As Kafka Streams allows for stream processing code to be executed as a library in the application rather than relying on some sort of cluster, operational overhead is reduced.

Key advantages consist of:

• No Separate Infrastructure: Library running as part of your application
• Exactly-Once Processing: Data consistency guarantees built-in
• Automatic Scaling: Additional instances to increase throughput
• Fault-Tolerance: Automatic recovery of state and re-balancing
• Architecture Overview

The application follows a clean architecture pattern, with REST APIs handling communication to the outside world and Kafka Streams managing the real-time processing pipeline.

Data flows:

1. REST API that accepts text messages
2. Messages are sent to Kafka Topics
3. The messages are processed in real-time by Kafka Streams
4. The results are saved to state stores and output topics
5. Query APIs allow for access to the processed results

What We’re Building

We’re building a word counting application that might seem simple on the surface, but it demonstrates all the core patterns you’ll need for more complex stream processing scenarios. When you send a text message through our REST API, it gets broken down into individual words, counted in real-time and stored so you can query how many times any word has appeared.

The beauty of this approach is that the same patterns work whether you’re counting words, tracking user behavior or processing financial transactions. The topology stays conceptually the same; you just change the data models and processing logic.

What makes this particularly interesting is that everything runs in a single JVM. No separate stream processing cluster, no complex deployment orchestration. Just a regular Java application that happens to be really good at processing streams of data.

Setting up the Development Environment

Prerequisites

• Java 17 or later
• Docker and Docker Compose
• A text editor

There is a full Docker Compose configuration in the project repository, which has:

• Kafka broker, with auto-topic creation
• Zookeeper, for cluster coordination
• Local development with proper networking configuration

Start up the infrastructure:
git clone https://github.com/mahitha-ada/kafka-streams.git
cd kafka-streams
docker compose up -d

Creating the Micronaut Project

The repository contains a complete Micronaut project that has support for Kafka Streams. The project has important dependencies like:

• micronaut-kafka-streams: Core Kafka Streams integration
• jackson-databind: JSON serialization support
• jackson-datatype-jsr310: Support for Java 8 time types for JSON serialization
• micronaut-http-server-netty: Implementation of HTTP server

Note: You need to include the dependency on the jackson-datatype-jsr310 for you to be able to handle java.time.Instant and other Java 8 time types in your message models.

Configuration That Makes it Work

The Key Configuration Points in Application.yml

• Application ID: Your Kafka Streams application will have a unique ID. This is important because Kafka uses this to track which consumer group your application belongs to and where it left off processing.
• Exactly-Once Semantics: This guarantees that each record will be processed exactly once, even if your application crashes and restarts.
• Commit Interval: This controls how often the state store gets flushed to disk. Lower values mean more up-to-date queries but higher I/O overhead.
• Bootstrap Servers: This is how your application connects to the Kafka cluster. In production, you’d have multiple servers here for redundancy.

Constructing the Stream Processing Pipeline

The foundation of our application is the Kafka Streams topology that describes the flow of data through the processing pipeline.

The topology does the following:

1. Read Messages: Creates a stream from the text-messages topic
2. Split Into Words: Each message gets split into individual words
3. Group and Count: Groups by word and count occurrences
4. Store Results: The counts are stored in a state store called word-counts-store
5. Output Results: Finally, sends the results to the word-counts topic

The key insight here is that this processing happens continuously. As new messages arrive, word counts get updated in real-time, and you can query the current state at any time.

REST API Integration

Stream processing handles the data transformation, but REST APIs provide the interface for external systems to interact with our application.

There are two primary controllers:

• MessageController handles message ingestion — a simple POST endpoint that accepts text messages and sends them to Kafka.
• WordCountController provides query access to results — you can query individual word counts or get the top N words.

Important Implementation Note: Our working implementation uses a direct KafkaProducer<String, String> rather than Micronaut’s @KafkaClient interface. This ensures reliable JSON string serialization and avoids common issues where messages appear to be sent successfully but arrive as null values in the Kafka Streams processor.

Testing the Application

Before testing the application, you need to create the required Kafka topics:

docker exec -it kafka kafka-topics –create –topic text-messages –bootstrap-server localhost:9092 –partitions 3 –replication-factor 1

You Can Run the Application With: ./gradlew Run

The application starts up in a few seconds and is ready to process messages on localhost:8082.

Send Message:
curl -X POST http://localhost:8082/api/messages -H ‘Content-Type: application/json’ -d ‘{“content”: “hello world hello kafka streams”, “userId”: “user123”}’

Query Word Counts:
curl http://localhost:8082/api/wordcounts/hello # Returns {“word”:”hello”,”count”:2}
curl http://localhost:8082/api/wordcounts/kafka # Returns {“word”:”kafka”,”count”:1}

Common Gotchas and Remedies

• Jackson Time Serialization: When deserializing java.time.Instant, JsonMappingException occurs. Add the JSR310 module dependency and register it with your ObjectMapper.
• Producer Implementation Issues: Messages sent via REST API, but Kafka Streams shows null values. Use a direct KafkaProducer<String, String> implementation instead of the @KafkaClient interface.
• Topic Creation: Application does not start with ‘topic does not exist’ errors. Always ensure topics are created before starting the application.
• State Store not Operational: Application does not start after ungraceful shutdown. Implement proper cleanup policies and health checks.
• Memory Use: High memory consumption for large state stores. Configure appropriate cache sizes and use windowed operations for time-based data.

Next Steps

In this blog, we have covered the basics of building real-time stream processing applications using Micronaut and Kafka Streams. Here are some directions to explore:

• Add windowed operations for time-based analytics
• Implement joins over multiple streams
• Add schema registry for compliance
• Add metrics and health checks

Conclusion

You have a simple yet powerful approach to building real-time stream processing applications using Micronaut and Kafka Streams. By leveraging compile-time enhancements and embedded stream processing, you can build systems that are both performant and operationally simple.

The combination works well because it eliminates a lot of the operational complexity that comes with traditional stream processing platforms. You don’t need to manage separate clusters, worry about complex deployment orchestration or deal with the networking issues that come with distributed systems.

The patterns you’ve learned here scale up well to more complex scenarios. Whether you’re processing financial transactions, IoT sensor data or user behavior events, the fundamental approach remains the same.