IBM i has been around for many years, and many legacy applications run on the platform. By implementing Kafka, a distributed platform that enables near-real-time communication between systems in different locations, you can use an event-driven architecture alongside your IBM i application when designing a distributed communications system, rather than treating it as a legacy system.

By Joydip Kajilal

IBM i applications have processed business transactions reliably for decades, providing reliable, high-volume batch processing. Many IBM i shops also run interactive transaction processing, real-time database queries, and modern web and API workloads on the same platform. These applications have run high volumes of batch jobs on schedules such as end-of-day settlement processing, daily inventory reconciliation, file transfers, and many other jobs.

Traditional IBM i integration relies on a combination of batch jobs, scheduled database exports, and polling tightly coupled point-to-point interfaces. When the IBM i system was at the center of the application landscape, these approaches did work well. However, the technology landscape has changed considerably since then with the emergence of newer technology and tools.

The problem is not that these legacy systems are aging. Instead, the challenge is helping these applications adapt to modern technologies. Today’s applications use containerization, Kubernetes, and cloud solutions across numerous data centers. At the same time, mobile applications and APIs require data to stream in real time, so operations must run automatically.

What is Event-driven Architecture? Why does it matter?

Before we discuss event-driven architecture, let us understand how a traditional request-response architecture works. In a traditional request-response architecture, an application sends a request, often to the same or another third-party application or service, and then waits for a response. Figure 1 below shows how a traditional request-response architecture works:

Figure 1: An IBM i application with tightly coupled components

The web application shown in the preceding diagram directly depends on the order API, which in turn depends on the legacy IBM i application. If another application needs to access the order data, you usually add another path. As the number of such applications increases, these dependencies increase and become difficult to manage.

When you introduce an event-driven architecture, the interaction changes completely, as shown in Figure 2.

Figure 2: Illustrating how an event-driven approach decouples the application components

Event-driven architecture (EDA) is a software design pattern that decouples services, allowing them to communicate and operate independently through asynchronous event production and consumption. In this design pattern, applications create, publish, view, and respond to events they produce.

An event is an action that occurs within an application because of another action or as a result of a change in state. Typical examples of events include: a new order has been received; an item's available stock quantity has changed; and an order has been processed. An event is any change of state or an action performed within the application.

It refers to an occurrence or an incident with facts that indicate something has happened, rather than waiting for a request from one application to another. Essentially, an event records something that happened; for example, an order was placed, an inventory item was updated, or a payment was made.

Message Queues and Kafka

While Kafka and message queues can often solve related problems, they differ in their distinct use cases. Message queues are typically used for reliable message delivery, and they are usually deleted once the message has been successfully delivered. They are well suited for command-style workloads such as the following:

• GenerateInvoice
• ProcessPayment
• SendShipmentNotification

On the contrary, Kafka is well suited for event streams such as the following:

• InvoiceCreated
• PaymentCompleted
• ShipmentDispatched
• InventoryAdjusted

IBM i as an Event Producer

In an event-driven architecture, IBM i need not be a cloud-native application—you can continue using your existing legacy RPG and Db2 workloads while building an application that uses an event-driven approach. There are several ways your IBM i application can produce events, such as the following:

Application-Level Events

In this case, the RPG program can publish an event whenever a business operation completes execution. For example, in a typical process order workflow, you may need to validate an order before updating it in Db2. Finally, you can publish the order as part of an OrderCreated event.

A typical OrderCreated event might look like this:

{

  "eventId": "3f9a2b8e-2b41-4e9a-9a63-1a2e6f4b7c10",

  "eventType": "OrderCreated",

  "schemaVersion": "1.0",

  "source": "ORDAPP01",

  "correlationId": "REQ-88213",

  "orderId": "O-2026-0125",

  "customerId": "C-0001",

  "businessTimestamp": "2026-08-18T08:30:00Z"

}

Here's what each attribute in the preceding code snippet implies:

• eventid: This is used to enable consumers to deduplicate and trace a specific occurrence.
• schemaVersion: This is used to inform the consumers about the schema version of the event they are reading.
• source: This is an attribute that helps you identify the system or the program that produced the event
• correlationId: This attribute is used to help trace the event back to the request that triggered it.
• businessTimestamp: This records the time when the order was actually created. Note that this may not often be the same time the event was published.

Implementing the Outbox Pattern Using RPG, Db2 for i, and a Kafka Publisher

The order application shown in this example, when performing validation and update operations, will insert an OrderCreated payload into a Db2 for i outbox table as part of the same unit of work. All statements are committed or rolled back simultaneously, ensuring business updates and event recording never get out of sync.

CREATE TABLE ORDERLIB.OUTBOXEVT (

OUTBOX_ID BIGINT GENERATED ALWAYS AS IDENTITY,

EVENT_ID CHAR(36) NOT NULL,

EVENT_TYPE VARCHAR(50) NOT NULL,

AGGREGATE_ID VARCHAR(30) NOT NULL,

CORRELATION_ID  VARCHAR(50),

PAYLOAD CLOB(2M) NOT NULL,

CREATED_AT TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

PROCESSED_AT TIMESTAMP,

PRIMARY KEY (OUTBOX_ID)

);

The following code snippet shows an RPG program that updates the order and writes it to the outbox inside the same commit control cycle:

Exec SQL

UPDATE ORDERLIB.ORDERS

SET ORDER_STATUS = 'CREATED'

WHERE ORDER_ID = :OrderId;

Exec SQL

INSERT INTO ORDERLIB.OUTBOXEVT

(EVENT_ID, EVENT_TYPE, AGGREGATE_ID, PAYLOAD)

VALUES (:EventId, 'OrderCreated', :OrderId, :PayloadJson);

Exec SQL COMMIT;

If the insert operation fails, the entire unit of work rolls back, resulting in a consistent state for the order. If the insert operation fails, the entire unit of work rolls back, resulting in a consistent state for the order and the corresponding logging. Db2 for i journaling used in the application provides the audit trail needed to recover from issues if necessary.

Additionally, an independent publishing process runs outside the main application's RPG runtime, periodically pulls all successfully committed records from the outbox table, and forwards them to Kafka or IBM Event Streams.

The following pseudocode outlines the logic of a typical polling-based publishing program:

rows = db2.query("

SELECT OUTBOX_ID, EVENT_ID, PAYLOAD

FROM ORDERLIB.OUTBOXEVT

WHERE PROCESSED_AT IS NULL

ORDER BY OUTBOX_ID

FETCH FIRST 100 ROWS ONLY

")

for row in rows:

kafkaProducer.send(

topic = "orders.order-created",

key = row.EVENT_ID,

value = row.PAYLOAD

)

db2.execute("

UPDATE ORDERLIB.OUTBOXEVT

SET PROCESSED_AT = CURRENT_TIMESTAMP

WHERE OUTBOX_ID = ?

", row.OUTBOX_ID)

We must wait for Kafka to send back the acknowledgment receipt of successful message delivery before we can mark the corresponding row of data as processed. If the program responsible for sending messages crashes right after initiating message transmission, before it has time to update that row's processing status, the same event will be sent a second time.

Change Data Capture Based on Db2

You can also capture database changes as another approach—you can monitor changes to Db2 for i and then convert those changes into events. This approach uses CDC to read Db2 for i journal receivers for the involved file to identify row changes and convert them into events, without needing RPG. The downside is that CDC is suitable only for smaller data volumes.

IBM i Integration Services

This approach is about an integration service that connects IBM i and Kafka. Unlike other integration methods that use RPG, this integration service can be a Java application running on IBM i or a PASE application running under AIX on IBM i. You can build it in Java or any other programming language supported by IBM i, such as Node.js or Python, and use any Kafka client libraries.

Comparing the IBM i Integration Options

Integration options usually fall into two categories: those that require RPG modifications and those that do not.

For example, if you are comfortable adding an inline insert statement to your existing RPG code, then the outbox is the best place to start. You do this by adding a new SQL insert statement to your existing transaction; this way, you control the event's formatting and content, and it is guaranteed to be reliable.

If you don't want to make any changes to your RPG code, then you can use either a Journal-based Change Data Capture (CDC) model or have an external service that reads either the outbox or journal for all your events.

If you are already using IBM MQ on your IBM i environment, you can use it to send and receive command-style messages directly to and from each other, or configure the Kafka Connector to consume messages from the outbox to bridge the two systems.

Decoupling IBM i Applications

IBM i application components are often tightly coupled, which makes it difficult to introduce changes. Figure 3 below shows how these components are coupled in an IBM i application.

Figure 3: Tightly coupled components in an IBM i application integrated with third-party or external components

Here is where an event-driven approach helps. You can use an event-driven architecture as an intermediary to decouple these components, as shown below.

Figure 4: The IBM i components are decoupled using an event-driven approach

As shown in the preceding diagram, IBM i publishes events when a business operation is executed or completes execution. An event platform helps the event consumers decide what to do with these events, thereby enabling the components to work in a decoupled manner.

A Practical IBM i Event-Driven Architecture

Figure 5 below illustrates a typical IBM i event-driven architecture at a high level.

Figure 5: A typical IBM i Event-Driven Architecture

Figure 5 illustrates an event-driven enterprise architecture that modernizes an IBM i environment by creating a Kafka-based integration layer that allows event data to be sent to multiple users at once.

Overview of the architecture components

• Presentation Layer - a layer within which the users communicate with the system through their browser or mobile app.
• API gateway - serves as a single point of entry that authenticates requests and then routes the authenticated requests to the backend system, providing a central access point for implementing cross-cutting concerns, security, throttling, rate limiting, and more.
• Application services - a layer of the enterprise application dealing with business logic for the request to reach the backend system as microservices or a modular monolith.
• IBM i (RPG + Db2 for i) - another layer consisting of RPG programs and the Db2 for i database. The main advantage of this layer is that it stores important transactional data.
• Event publisher – this is a software component or a service, usually a change-data-capture mechanism, that makes data flow from IBM i to Kafka.
• Messaging layer - The messaging layer comprises Kafka, a fast message broker used for exchanging messages between a data producer and a data consumer in a scalable way.
• Microservices - the services that use incoming data for running processes and keeping results in their own databases.
• Analytics - this is yet again a consumer that uses the incoming data to make updates to reports and keep results in the data platform.
• Notification - this is another layer that allows triggering notifications based on the data received.

Performance Considerations

An event-driven architecture does not automatically improve an application's performance. Instead, it helps distribute work across components and makes your application scalable and fault-tolerant.

An IBM i application should not wait for the message broker to be available, as that would create a performance bottleneck. Network latency can also affect transaction performance. So, use an asynchronous publish-consume approach to keep your application responsive and scalable.

An event-driven architecture often uses the transactional outbox pattern, which helps the application perform better and scale more effectively. Let's look at how this works with an example. Consider an IBM i application that needs to update Db2 and Kafka independently.

Instead of writing to Db2 and Kafka individually, the application can write the transaction information to an outbox asynchronously. A publisher can then read the data from the outbox and send the events to Kafka, as shown in Figure 6.

You don't need to perform two separate operations to write data to Db2 and Kafka. Your application can execute the update task and the outbox records in the same Db2 for i transaction to commit—either both items are saved successfully, or both are rolled back. After this transaction commits, an independent publishing process reads the outbox records, converts them into events in the background, and sends them to Kafka. This ensures transaction consistency because writing updates in Db2 does not happen separately from writing to the outbox; instead, only the publishing process runs asynchronously.

Figure 6: An event-driven approach using the Outbox pattern

Consumers

Any application or service that reads events from an event producer and then does something with them is called a consumer. Once IBM i events reach Kafka or a message broker, the message consumers can process them independently. Let us understand this with an example.

Consider the OrderCreated event as shown in Figure 7.

Figure 7

Refer to Figure 7. As you can see, the inventory service can handle item stock. The analytics platform can provide a real-time dashboard, while the notification service can send notifications such as emails or push notifications. None of these systems needs a direct connection to the RPG program.

Delivery Guarantees and Duplicate Processing

Synchronous and asynchronous (or event-driven) calls work differently. Let us understand this with an example. In a synchronous call, when a downstream service fails, the caller is informed about the error instantaneously because the event is processed in the same transaction. In typical event-driven architecture, the producer commits the transaction before any consumer processes the event.

In other words, by the time the consumer processes an event, the producer has already completed the transaction. Hence, if processing fails, it fails later, i.e., it fails outside that transaction, with no caller waiting on it. Hence, you need to implement retries, dead-letter processing, idempotent consumers, and offset management, rather than relying on the caller to handle the error. Because of this, when developing event-driven applications, it is important to implement error-management techniques. One of these techniques is known as the “at-least-once” technique, used in Kafka and other queue systems; the idea is that the system resends messages when it doesn't receive a response from the consumer.

Operational and Security Considerations

Here are a few best practices you should follow as part of your operational and security considerations:

• Ensure all content transmitted is encrypted, and verify the identity of every connection that accesses the message relay server.
• Never allow unauthenticated connections to access the system. When assigning access permissions to different users, restrict permissions to the individual topic or queue level.
• If the downstream program that consumes messages does not need sensitive content such as payment information and personal data, desensitize these fields, either by masking them or not transmitting them with the message.
• If relevant regulations require that the data be protected by encryption, also implement field-level encryption for these fields. It is also a good practice to set up automatic alerts to receive notifications when any issues arise.
• Monitor the processing lag of the programs that consume messages on the Kafka side, i.e., when the speed of receiving messages cannot keep up with the speed of sending messages, leading to a large backlog of unprocessed messages.
• When setting the message retention period, you should base it on the furthest point in time you need to trace back and replay messages.

Migration Strategies

• Keep the IBM i application unchanged, i.e., instead of trying to change the RPG and Db2 processing inside the application, move the integration layer around the application.
• Choose a bounded event to start with that is limited enough to avoid any risks, like OrderCreated for a single order type.
• Set the reconciliation criteria before going live. Define how you will compare event data with batch results.
• Execute the event pipeline together with the existing batch job and do not let the new pipeline replace anything yet.
• Run the complete business cycle tests, and check if the event results align with the batch results.
• Make sure you can roll back and replay. Make sure you can go back to the batch process if the event pipeline goes wrong, and that you can replay events from the outbox or Kafka after something goes wrong.
• Do not retire a batch job until the event stream for this has just run in production during at least one business cycle and a rollback plan is available.

Takeaways

• Event-Driven Architecture is an approach that helps build resilient, scalable, adaptable, responsive, and extensible systems by focusing on events.
• Your business-critical data is still hosted on your trusted IBM i, and maybe it always will be. Still, event-driven architecture lets you expose this microservices data without replacing anything running in these legacy applications.
• Kafka was designed from the outset for high throughput and durable event streaming. Use Kafka if you need features like event replay or would benefit from loose coupling between systems/applications.
• IBM i application components are often tightly coupled, making it difficult to introduce changes when needed. You can use event-driven architecture to decouple these components.

References

• Db2 for i journaling (IBM Documentation): https://www.ibm.com/docs/en/i/7.5.0?topic=integrity-journaling
• Coding SQL statements in ILE RPG applications, including commitment control (IBM Documentation): https://www.ibm.com/docs/en/i/7.4.0?topic=programming-coding-sql-statements-in-ile-rpg-applications
• IBM MQ, introduction and technical overview (IBM Documentation): https://www.ibm.com/docs/en/ibm-mq/9.4.x?topic=mq-introduction
• Kafka Connect scenarios for IBM MQ, including the MQ Kafka connectors (IBM Documentation): https://www.ibm.com/docs/en/ibm-mq/9.4.x?topic=scenarios-kafka-connect
• IBM Event Streams overview (IBM Cloud Docs): https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-about
• Apache Kafka documentation, including producer semantics and design (kafka.apache.org): https://kafka.apache.org/documentation/
• Message Delivery Guarantees for Apache Kafka (Confluent Documentation): https://docs.confluent.io/kafka/design/delivery-semantics.html
• Transactional Outbox pattern, the general pattern this article adapts for Db2 for i (microservices.io): https://microservices.io/patterns/data/transactional-outbox.html

 

Joydip Kanjilal is a Principal Software Engineer in Hyderabad, India.

 

Read my blog

 

Awarded the prestigious Microsoft Most Valuable Professional (MVP) award in ASP.NET six times in a row from the year 2007 to 2012. A speaker and author of several books and articles with over 18 years of industry experience in IT and more than 14 years in Microsoft .NET and its related technologies.

 

Currently working as a Principal Software Engineer at DELL International Services at Hyderabad. Was selected as an MSDN Featured Developer of the Fortnight (MSDN) a number of times and also Community Credit Winner at www.community-credit.com several times.

 

Authored the following books:

·  Entity Framework Tutorial (Second Edition) by Packt Publishing

·  ASP.NET Web API: Build RESTful Web Applications and Services on the .NET Framework by Packt Publishing

·  Visual Studio 2010 and .NET 4 Six-in-One by Wrox Publishers

·  ASP.NET 4.0 Programming by McGraw Hill Publishing

·  Entity Framework Tutorial by Packt Publishing

·  Pro Sync Framework by APRESS

·  Sams Teach Yourself ASP.NET AJAX in 24 Hours by Sams Publishing

·  ASP.NET Data Presentation Controls Essentials by Packt Publishing

 

Also reviewed more than 10 books and authored more than 350 articles for some of the most reputable sites, such as www.msdn.microsoft.com, www.code-magazine.com, www.asptoday.com, www.devx.com, www.ddj.com, www.aspalliance.com, www.aspnetpro.com, www.sql-server-performance.com, www.sswug.com, and so on.

 

Has years of experience in designing and architecting solutions for various domains. His technical strengths include C, C++, VC++, Java, C#, Microsoft .NET, AJAX, WCF, JQuery, ASP.NET Web API, REST, SOA, Design Patterns, SQL Server, Operating Systems, and Computer Architecture. Has been exploring Cloud technologies, IoT and Machine learning these days.

 

Blog: http://www.infoworld.com/blog/microsoft-coder

Website: www.joydipkanjilal.com

Twitter: https://twitter.com/joydipkanjilal

Facebook: https://www.facebook.com/joydipkanjilal

LinkedIn: http://in.linkedin.com/in/joydipkanjilal