Learn how one simple character can make a program go crazy.
Several years ago, I wrote an email processing program for the System i and posted the source on my Web site. Several people have downloaded the code, compiled the program, and run it without any problems…until recently. For a few months now, people have been reporting that the program crashes. The program MAIL, which worked perfectly for several years, suddenly went haywire.
In this series of articles, I will show you how a CCSID variation can turn a program crazy, why this can suddenly become a problem, and why perhaps you've had the problem for years but didn't see it. This can happen even when your applications are Unicode-ready.
The first article will explain the basis of the problem. The second article will show you how I solved the problem. Finally, the last article will show you some tricks I learned along the way.
This article, the first of the series, explains the problem that needs to be solved.
The Coded Character Set Identifier (CCSID) is something like the alphabet. Not everybody uses the same alphabet.
So let's look at the MAIL program problem, which I mentioned in the opening paragraph. As an example, here's what one user reported. The procedure returned the following:
Invalid email address :
Sender email (
RPG status 00202 caused procedure MIMESENDER in program JPLTOOLS/MAILR to stop.
The problem appeared simultaneously with an incoming group of new users from another country. The email addresses were perfectly valid. So why did my procedure perceive them as invalid addresses? What happened with the MAIL program? It was not recompiled, so the reason had to be elsewhere.
My investigation revealed a clue: the CCSID of the job that makes MAIL crazy is different from the CCSID of the job that compiled the program MAIL. Don't ask me how I figured that out or how I discovered that this coincidence is in fact the primary reason for the haywire behavior. Sometimes, we are just lucky.
De facto, when the job that uses MAIL has the same CCSID that the compilation job has, the problem never appears.
Why on earth do I have to take care of this strange parameter of the job that compiled my program when I am calling this program?
What Is the Link?
The CCSID is an old parameter. We can retrieve the CCSID parameter from many places on the IBM i (example: SBSD, JOB, FILE, FIELD).
Moreover, even DSPPGM shows the following:
Coded character set to identify. . . . . . . . : 65535
But CHGPGM does not allow me to modify it.
The CCSID(65535)—sometimes named CCSID(*HEXA)—means that the object is a binary object. Let me explain. When you copy a program from one machine to another, you hope to obtain on the target machine exactly the same program (object binary code) as the one on the origin machine. When it is a binary copy, you get the same object, bit to bit.
CCSID is the parameter that tags an object precisely for translation purposes. The CCSID explains to the system whether, during this copy, a character translation is mandatory or forbidden. On a binary object, translation is forbidden (it is the good choice). Therefore, after a copy, you get the same object (bit to bit). Since a program is a binary object, after a copy, a program continues to work.
The special value CCSID(65535) or CCSID(*HEXA), which is used to tag *PGM objects, is here exactly for that purpose: automatic translation of bytes is forbidden, so a copied program continues to work.
Nevertheless, this generally good process has a side effect: in my program, there is a literal constant, the "at" sign (@).
There are some places in my program (a binary object) where I have stored text. Oops. Text translation is sometimes mandatory.
I resume: For a program, automatic character translation by the system is forbidden. In my program, there is a hard-coded @, and the @ is a character that varies from one CCSID to another. Remember: the initial issue is that the program goes haywire when CCSID varies.
Therefore, the reason for the unintended behavior is well outside of the program, but the problem is in the program. If the @ was not hard-coded in my program, I would never have had an issue.
The problem is precisely here:
// * one @ is mandatory
i = %scan ('@':email);
What is wrong in this code? The @ is hard-coded as a literal text constant.
What I have just displayed is copied from the source code. It comes from a file (and a source file is one kind of DB2 file), and this file has the CCSID 297 (I am French). My job and my 5250 emulator also use CCSID(297). When I compile my program, the compiler reads the source code and transforms it into a binary object (a *MODULE).
At this moment, my @ character becomes X'44 '.
dsppfm jpltools mailr
i = %scan('@':email);
84746A889474778988955
90E0C2315DD4DA54193DE
What happens in the program is therefore really a search of X'44' in the email field.
While the job that uses MAIL is in CCSID(297), the problem is hidden. As soon as the CCSID varies, the problem appears. If the job is, for example, in CCSID(037)—the preferred CCSID of Americans—the job transmits to the program a correct email address, with an American @. In CCSID 37, the code for @ is x'7C '.
Therefore, the program becomes crazy:
Invalid email address:
But why does the problem appear only now?
Since this program walks from end to end of the Internet, the problem should have immediately appeared.
If I had published the object program, the issue would have appeared immediately. But I published only the source code.
The problem is hidden because the source code comes from the Internet. The source code I just showed you comes from my IBM i, from a source file with CCSID(297). When I want to publish it, I first copy it to my PC via FTP. Automatically, FTP translates the source code (EBCDIC) that comes from my IBM i to the CCSID of my PC (ASCII). Then the source code is published on my Web site, therefore into an ASCII file, and more precisely, in CCSID(1252), which is the Windows CCSID for America and West Europe.
Then, a programmer gets the source code. Say it's Dan, my American friend. Dan copies the program from the Internet to a PC, and then he copies it with FTP to his IBM i source file. Dan's source file has CCSID(37). So FTP executes automatic conversion from the ASCII CCSID(1252) to the EBCDIC CCSID(37), the CCSID of Dan's source file.
When Dan looks at the code, he sees no issue (the problem is hidden):
dsppfm jpltools mailr
i = %scan('@':email);
84746A889477778988955
90E0C2315DDCDA54193DE
But if there is no issue, why am I writing this article? Because there is truly an issue. After Dan has compiled the program, the object code scans for x'7c' in each incoming email address. This works perfectly. Well, it works perfectly until I try to use it. Remember, I'm French. In the French CCSID, the @ is coded x'44'. When I try to use the program compiled by Dan, the object code tries to scan for x'7c' in each incoming email. This never works.
So, while people use only the default CCSID for their machine, there is no problem. The problem is for people who develop for multiple languages. This may happen suddenly when your company incorporates or is incorporated into a world group.
Let's continue: When the CCSID of the job that uses MAIL is different from the job that compiled MAIL, the program becomes crazy. The exact cause is a literal constant keyed in a line of the source code. After I identified this problem, I found on the IBM Info Center these "Recommendations and guidelines for using CCSIDs." Here's one line from that document: "Avoid using characters that are not in the invariant character set for names and literals in programs." Now, I understand what that means. Go to that page; you'll see a lot of "Be aware of…"
This first article has described the issue. We now are sure the problem is with CCSID.
In my next article, I will go deeper into the CCSID, its definition, the reason why I have a problem, and the method to implement a solution in the RPG program.
as/400, os/400, iseries, system i, i5/os, ibm i, power systems, 6.1, 7.1, V7, V6R1
More than ever, there is a demand for IT to deliver innovation. Your IBM i has been an essential part of your business operations for years. However, your organization may struggle to maintain the current system and implement new projects. The thousands of customers we've worked with and surveyed state that expectations regarding the digital footprint and vision of the company are not aligned with the current IT environment.
TRY the one package that solves all your document design and printing challenges on all your platforms. Produce bar code labels, electronic forms, ad hoc reports, and RFID tags – without programming! MarkMagic is the only document design and print solution that combines report writing, WYSIWYG label and forms design, and conditional printing in one integrated product. Make sure your data survives when catastrophe hits. Request your trial now! Request Now.
Forms of ransomware has been around for over 30 years, and with more and more organizations suffering attacks each year, it continues to endure. What has made ransomware such a durable threat and what is the best way to combat it? In order to prevent ransomware, organizations must first understand how it works.
Disaster protection is vital to every business. Yet, it often consists of patched together procedures that are prone to error. From automatic backups to data encryption to media management, Robot automates the routine (yet often complex) tasks of iSeries backup and recovery, saving you time and money and making the process safer and more reliable. Automate your backups with the Robot Backup and Recovery Solution. Key features include:
Business users want new applications now. Market and regulatory pressures require faster application updates and delivery into production. Your IBM i developers may be approaching retirement, and you see no sure way to fill their positions with experienced developers. In addition, you may be caught between maintaining your existing applications and the uncertainty of moving to something new.
IT managers hoping to find new IBM i talent are discovering that the pool of experienced RPG programmers and operators or administrators with intimate knowledge of the operating system and the applications that run on it is small. This begs the question: How will you manage the platform that supports such a big part of your business? This guide offers strategies and software suggestions to help you plan IT staffing and resources and smooth the transition after your AS/400 talent retires. Read on to learn:
LATEST COMMENTS
MC Press Online