Discover how IBM i shops can leverage Kafka and event-driven architecture to modernize legacy RPG applications without rewriting decades of business logic. Learn when Kafka makes sense, deployment best practices, and how IBM i can participate in real-time distributed architectures.
By Joydip Kanjilal
Event-Driven Integration on IBM i: Leveraging Kafka for Modern Architectures
IBM i has been around for many years now — many legacy applications run on the platform. The good news is that you don’t have to rewrite their business logic to modernize those applications. By implementing Kafka, a distributed platform that enables near-real-time communication between systems in different locations, you can use an event-driven architecture alongside your IBM i application when designing a distributed communications system, rather than treating your IBM i application as a legacy system.
What is Event-driven Architecture? Why does it matter?
Event-Driven Architecture (EDA) is a software design approach that comprises a collection of decoupled services that communicate asynchronously by producing and responding to events in real time. EDA refers to the style in which applications publish events, detect events, and respond to events. EDA focuses on events, how they are produced and captured, how they are processed and stored, and how they allow loosely coupled systems to operate in real-time.
An event is any change, state, or action performed within the application. It refers to an occurrence or incident with facts indicating that something has happened, rather than waiting for a request from one application to another. Essentially, an event is a record of something that has happened, for example, an order has been placed, an inventory item has been updated, and a payment has been made.
Figure 1 illustrates a typical event-driven architecture for the IBM i platform.
Figure 1: Demonstrating Event-Driven Architecture in IBM i
While event-driven architectures provide many benefits, developers typically face several challenges when designing event-driven applications on the IBM i platform:
- Order of Event Processing: You should be able to ensure the order of event processing in an event-driven application.
- Reliability: You need a robust error-handling framework to ensure resilience and continuity, and to prevent data loss.
- Consistency: Ensuring consistency in an event-driven system is another challenge, because, unlike in a monolith, transaction processing in an event-driven application can typically span multiple systems.
IBM i as an Event Source
The IBM i platform can work as an event source using any of these integration patterns:
- Database Triggers & Journals
- Data Queues
- Messaging Middleware
- RESTful API Exposing
Data Queues
Data Queues are an IBM i-native method of inter-job communication. They've historically been used to send messages between jobs due to their speed and ordering (optional key). If your existing RPG programs write to data queues (indicating that a shipment has been confirmed, for example), you can build a connector from your data queues to a Kafka topic using a lightweight .NET or Java consumer running either on the IBM i server or on a gateway server.
IBM i systems have long been running critical business operations across several industries, including manufacturing, banking, distribution, and retail. It is not a good idea to replace an IBM i system at all, as decades of business logic are embedded in the legacy applications. Hence, organizations have been modernizing their IBM i systems using integration technologies. An event-driven architecture that leverages Kafka can enable an IBM i system to expose data and events in real time within a distributed computing environment without rewriting any existing business logic.
Triggers and stored procedures
If you need more control over exactly what constitutes an event, database triggers and SQL stored procedures provide hooks at the code level to determine how your event should be shaped and what it will contain.
Every time a record is inserted or updated, the trigger fires and can call an external program or write to a staging table that a Kafka producer monitors. While both options provide greater control over event shape and content, they require more coupling to the database layer.
Assume that an RPG program processes a sales order in DB2 for i. Now, a Kafka producer sends the committed sales transaction to the Kafka broker or topic as an order_placed event. Next, a downstream microservice and an analytical dashboard consume this event independently of one another, without worrying about how IBM i actually works.
Journal-Based Change Data Capture (CDC)
In a Db2 for i database, all committed changes to tables that have been set up for journaling (inserts, updates, deletes) are logged in the associated journal receiver records. A change data capture (CDC) tool can capture the committed changes directly from the journal receiver as structured Kafka messages (events) without changing any existing RPG, COBOL, or SQL application code. Each journal entry contains an image of the row's before-and-after state, a timestamp, and job information, providing downstream consumers with a rich payload of events.
Real-time Data Integration
Organizations have been adopting a real-time data integration and decoupled architecture approach to respond to business requirements at the drop of a hat — in milliseconds. In an IBM i environment, real-time integration ties together the old world of legacy processing (aka transactions) and modern analytics and AI pipelines through streaming information, which allows us to move away from batch processing, keep our systems up to date rather than running on dated information, and provide transactional integrity in real-time analytics.
Decoupled architecture makes this possible by allowing any component to run in its own process rather than being tightly coupled to other processes. So, our IBM i transactional processing function remains in place and can share information with all applications, even those that are not tightly coupled. Solutions of this sort require purpose-built infrastructure, rock-solid hardware, complete test cycles, testing for schema changes and data drift, and the ability to run in parallel with existing workloads.
Using Kafka for IBM i workloads
Kafka provides an efficient way to stream high volumes of real-time events on IBM i to multiple independent consumers, including dashboards/analytics, mobile apps, fraud detection, AI delivery pipelines, and microservices. Use Kafka to publish multiple transactions simultaneously, enabling multiple independent consumers to read them.
Kafka also supports audit trails and event sourcing, enabling you to store events (and related metadata) and replay them later. Additionally, you can use Kafka to modernize your IBM i applications to enable them to take advantage of the latest technologies by decoupling your legacy RPG or COBOL applications.
If you're using Kafka for simple workloads, it can be overkill. For example, sending periodic status updates, triggering a single application service layer, or connecting one IBM i program to one consumer can typically be accomplished using IBM MQ, data queues, or even direct REST calls at much less operational overhead.
Kafka will introduce unnecessary complexity for applications that are already tightly integrated, have limited functional scope, handle a small volume of transactional data, or where the benefits of using a distributed event-oriented architecture do not outweigh the administrative and latency overheads incurred.
When you need to read your data much less frequently, or you need scheduled reports or batch extracts, you should use a direct database connection because it is simple and avoids the overhead of maintaining a CDC pipeline.
The general rule of thumb for using Kafka is to use it when you have an IBM i application interacting with other applications as part of a larger event-driven architecture, where multiple businesses need to react to the same business event independently.
Deployment Considerations
Kafka brokers are rarely deployed on IBM i because a typical IBM i workload generally incurs significant transactional processing overhead. Consequently, long-running applications will typically use much more memory and CPU resources than a typical workload.
A recommended standard topology is to isolate your Kafka infrastructure in a Linux environment, while your IBM i environment remains the primary data source and destination.
The key deployment patterns adopted for IBM i modernization include:
- API-First Integration
- Anti-Corruption Layer (ACL)
- Event-Driven Architecture
- Strangler Pattern (or, phased refactoring)
- Hybrid Deployment
Security and Governance Considerations
If you're using IBM i in a regulated industry such as finance or healthcare, it is a good idea to leverage Kafka. That said, integrating Kafka with IBM i may raise compliance and security concerns.
Authentication and Security
You should authenticate to a Kafka broker using SASL (Simple Authentication and Security Layer) or Secure Sockets Layer (SSL), and encrypt the traffic using Transport Layer Security (TLS). It’s definitely a bad practice to store credentials in the application's source code; instead, you can store them in a secrets manager, such as HashiCorp Vault.
Auditability
You can leverage Kafka to build audit logs that capture information on security violations, such as illicit authentication attempts, access to specific topics, schema changes, etc.
Data Governance
You should set up a data catalog and a classification scheme before publishing events to Kafka. Moreover, you should identify the database tables that may contain sensitive data, such as PII, payment card details, and health records, and tag and classify them before they are ingested.
Takeaways
- Event-driven architecture (EDA) is a new approach to building scalable, adaptable applications, often by moving away from a tightly coupled, monolithic architecture using loosely coupled services.
- By focusing on events (state changes), EDA allows engineers to create systems that are resilient, responsive, and extensible. Apache Kafka is a proven EDA solution for enterprise environments.
- While IBM i continues to be a trusted platform for your business-critical data, event-driven architecture enables you to expose that data to modern distributed systems without the need to replace your legacy applications running on this platform.
- Kafka was originally designed for high-throughput and durable event streaming. Kafka should be used when you require functionality such as event replay or the benefit of loose coupling between different systems or applications.
- You can consume Kafka events from IBM i systems via gateway services that read from Kafka topics and write to either Db2 for i tables or data queues.
- If you are only going to have one consumer, using a REST API to call a server or making a direct database call will be simpler and more maintainable than using Kafka.
- Now, with Kafka, you can access the latest cloud-based systems for the IBM i without rewriting years of sound business logic.
Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.
IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn:
LATEST COMMENTS
MC Press Online