Regardless of the enterprise's plans for using an artificial intelligence app, preparing the data for use with it is an essential step.

by John Ghrist

Preparing to adopt an AI app is usually a paradigm-shifting event for any organization. Checklists abound for setting AI business goals and priorities, managing changes to corporate culture, addressing employee attitudes, securing managerial support, choosing the right technology, developing employee skills, raising awareness of ethical considerations, securing vital records, and addressing governance issues. All are vital, but creating accurate models is the cornerstone of successful AI because reaching valid conclusions requires starting with the best possible data.

So, setting aside these and all the other considerations (about which whole books have been written), let's focus for a moment on an overview of just how the data any AI will be feasting on needs to be gathered, enriched, prepared, and cleansed to provide a future basis for enterprise insight.

Overall Basic Steps

In general, preparing data for AI training requires six basic steps, although each can have many substeps.

• Raw data needs to be collected and consolidated.
• The data must be cleansed to correct errors, eliminate duplicate records, provide missing values, and provide consistency.
• The data must be translated into a consistent AI-compatible format.
• The data must be labeled to help the AI analyze patterns within it.
• The data must be filtered to avoid such problems as statistical bias from outliers.
• Finally, the data must be validated (essentially checking that the previous steps were accurately carried out) before starting AI training.

Collecting Data

Accurate predictions can only come from data that's as accurate and complete as it can be made. This means tapping the widest assortment of good data that time and initial resources enable. Primary data sources will include sales, marketing, and other existing records on customers, past enterprise business activity, and markets the enterprise may be seeking to enter.

Attention should also be given to enriching existing data using available information and external sources. To help organize this data, it can be helpful to map the data by its groups. First-party data is drawn directly from customers, including contact information, subscription data, browsing behavior, and customer feedback. Second-party data is data purchased from another company in a similar line of business. Third-party data is obtained from sources such as data enrichment providers, governmental agencies, and general internet search tools.

Data enrichment can also broadly be subdivided into three general areas: behavioral, geographic, and demographic. These areas can be distributed among the three data groups mentioned above. Behavioral data is available, for example, by analyzing (first-party data about) how customers have browsed the enterprise's existing website(s), their past purchases, and comments they have made about the site. This helps determine what interests customers have expressed by their actions and what kinds of advertising have been most successful so far. Geographic data (from all three data groups) comes from location and contact data, which can help refine overall views of purchasing (or other relevant) information about customers generally. Demographic information (from all three data groups) can provide insights into customer characteristics such as income level, median home value, credit scores, education, family composition, and other factors. This data can enrich AI training data and, for example, tailor marketing approaches to specific customer groups or personalize marketing tactics for existing or potential customers.

Data can be further enriched by incorporating secondary resources – data research of all three types done by others that relate to the enterprise's aspirational needs. These include governmental census data, syndicated data compiled by commercial entities specializing in data relevant to enterprise goals (e.g., general economic trends, television ratings, commercial market forecasts), and other external data sources that may bear on enterprise goals. While some care must be taken to determine the timeliness and relevance of this data, as well as to screen for potential biases and faulty methodology in the data sources, such enrichment can offer benefits. Using such sources means existing data can be enhanced, the enterprise doesn't have to take the time and expense to gather such data itself, the process can streamline the sorting of potential customers into categories (market segmentation), and can help the organization find and correct errors in existing enterprise data (e.g., address or contact information changes, inaccurate e-mail addresses and titles).

Finally, the data must be assessed for completeness. Is there enough data of the kinds needed to meet the organization's goals for adopting AI? What might be missing? Where can any pertinent additional data be found?

Data Preparation

Data cleansing involves tasks such as detecting and eliminating duplicate records, providing missing values for some records, correcting keystroking or other errors, and ensuring that critical fields (e.g., dates and times, pricing figures, currency values) are expressed in a consistent format.

Integrating data for AI use requires further processing, generally referred to as data transformation, which converts "raw" data into a unified format or structure. There's currently no standardized process for doing this. Still, it's necessary to format data for large language models (LLMs) (and other AI models) to avoid inaccuracies or inconsistencies that can lead to non-representative conclusions or make it hard to eliminate data bias after data has been fully integrated. Data binning (also called discretization) is a preprocessing method that divides the data into small ranges (bins) and replaces each bin with an aggregated value, helping AI identify patterns more effectively.

Once the data is assembled, it must be formatted so that an AI algorithm can meaningfully assess it. Mapping can help divide data into its basic types (i.e., structured, unstructured, and semi-structured) to determine what preprocessing the data may need to be more useful. Structured data, such as spreadsheets or transaction records, is easiest for most AI models to work with. Unstructured data, such as email messages, online comments, or textual articles, may require natural language processing or other methods to be standardized. Semi-structured data — such as the JSON open-structure file format for application programming interfaces (APIs), spreadsheet-based comma-separated values (CSV), and eXtensible Markup Language (XML) formats used by older systems — all need to be mapped out to facilitate AI processing.

Data Labeling

Supervised-learning AI models benefit from data labeling, which assigns raw data labels that define its context for machine learning models. This process helps a data team annotate or categorize data, excluding some records from processing to prevent anomalies from influencing overall analysis results. Labeling can be done in three ways: automated with a commercial tool, human annotation, or a combination in which humans create a labeling standard and automate later labeling with a tool. What’s essential is that labeling enables an ML model to use “supervised learning,” which is possible only when data is labeled, as opposed to “unsupervised learning,” which is learning an ML model undertakes when it’s presented with unlabeled data. A decision on which path to follow is best left until a decision has been made on what AI product, or at least model, will ultimately be analyzing the integrated data.

Labeling is also essential for regression and classification functions, helping AI models learn patterns. Regression tasks benefit from the assignment of numerical values to data and for classification, enabling labels to characterize specific data categories.

Data Filtering

Data Filtering eliminates data records that are significantly outside the dataset's norm, a process accomplished through preprocessing techniques such as normalization and scaling. Normalization scales data features to a standard range based on the data's minimum and maximum values. The objective is to prevent a single feature’s values from dominating other feature values in the dataset. There are five major formulas for accomplishing this, which vary according to desired effect, but are beyond the scope of this article to explain in detail. (For example, the most common one is Min-Max or z-score scaling, which rescales data to a range between zero and one by subtracting the minimum value and dividing by the range of the data.)

There are other techniques for data filtering.

Trimming is a process that removes redundant elements from a model (e.g., records with extreme values). Capping sets upper and lower bounds on feature values, reducing the impact of outliers in the training data. Smoothing reduces data irregularities by calculating the average of a range of data-point values, assigning decreasing weight to older data, and using a weighted average of points with similar values to create a smooth curve of results. Filtering isolates data subsets from a larger one by using predefined criteria.

Any or all of these techniques can be employed to avoid outliers from unduly affecting overall results.

Hopefully, by now, an AI model has been selected, so that the final data integration steps can be customized with that model's capabilities in mind. It's also time to consider feature extraction, the process of identifying which information in the dataset will be most important for the AI to make decisions. One difficulty often encountered at this point is resolving inconsistencies arising from different data sources that categorize data differently (e.g., one source organizes expenses by activity type and another by corporate department).

Sometimes, representation learning occurs, during which the AI model takes a preliminary pass over the data to determine data relationships that guide further processing. For example, checking whether similar items map to the same bins to measure similarities or dissimilarities between data points, mapping data to a new coordinate system, or suggesting more detailed coding methods for data, each of which can facilitate machine learning.

After these (and perhaps other custom methods) data-massage techniques have been applied, the data needs to be integrated into a unified dataset because data drawn from multiple sources often varies in quality, structure, and format. This process converts the data into a format compatible with the designated AI model. It normalizes data points (e.g., by adjusting values based on criteria such as minimum and maximum values, the maximum absolute value, the mean value, or logarithmic scaling) to ensure that no single value distorts the results by dominating others.

The final step is to split the dataset into training, validation, and test subsets. The training set is the core dataset that essentially teaches the machine learning part of an AI app to isolate data relationships and patterns. The validation set helps evaluate an AI model's performance during training. The test set helps determine how well the model performs on previously unseen data after initial training. There are various ratios for how much data goes into each of these datasets, 80-10-10 or 70-15-15 being common allocations. The splitting can be done randomly, based on time sequencing within the data itself, stratified according to human-assigned priorities, or using several cross-validation methods, with the best method determined by the enterprise goals for initiating the AI system in the first place.

In today's data-driven world, high data liquidity indicates how accessible data is within an enterprise. Still, it's an essential goal to achieve to make the best use of adopting AI technology.

John Ghrist has been a journalist, programmer, and systems manager in the computer industry since 1982. He has covered the market for IBM i servers and their predecessor platforms for more than a quarter century and has attended more than 25 COMMON conferences. A former editor-in-chief with Defense Computing and a senior editor with SystemiNEWS, John has written and edited hundreds of articles and blogs for more than a dozen print and electronic publications. You can reach him at This email address is being protected from spambots. You need JavaScript enabled to view it..