Data Cleansing & Standardization

Extracting value from your data to prevent revenue leakages

5 minutes read 

The Backdrop to this Project

Effective data quality initiatives aren’t isolated. A good data quality program must be effectively integrated with an organization’s overall data governance vision

Many procedures required for excellent data quality also contribute to strong data governance. Data governance requires business rules to be applied across the firm and an enforcement mechanism.

Our foray into this experience, happened at a time when Data Quality was being touted as the most important factor of any businesses success.

Based on the merit of our solution-ing and experience, we were selected against larger foreign companies to help our Client put our theories, approach and solution to improving data quality to succeed in business.


What our Client was trying to overcome?

Our Client’s major goal in embarking in this effort was to forestall and put a halt to the income loss that was brought on by cyclic churners.

Their capacity to engage in any aggressive marketing initiatives and to diversify their business was significantly hindered as a result of revenue leakages.

Identifying Circular Churners

Circular churners are inactive customers with outstanding balances. Most customers subscribe to a service, use it to capacity, and then cancel. When another appealing offer is provided, they open a new account by presenting their details differently.

Without validating controls, the system treats the same consumer as a new account, giving them all the benefits of a new customer – AGAIN. This causes other business issues.

Fraud Detection & Payment Recovery

Due to the absence of a uniform and clean client database, it is challenging to detect fraud. In this context, fraud refers to the practice of clients subscribing to incompatible packages, such as programs in a language other than their native tongue or packages that do not correspond to their economic bracket.

Without the capacity to identify cyclic churners, it is practically impossible to convince them to settle their outstanding debt.

Achieving a Unified Perspective of Customers from Data Quality

Lack of data cleansing and verification processes and a large number of data entry points, fragment and destroy customer data over time. One customer may have several aliases and accounts in the database.

This hinders the company’s ability to execute “real” client on boarding and loyalty programs. Even writing letters to clients becomes difficult and expensive under these scenarios. This didn’t offer a unified view of customers because the concept of a customer is different from the concept of an account.

Achieve Better Data Accuracy

Without automation, data accuracy depended solely on manual data entries. Due to the nature of the customer’s business, localized data entry from where the customers are situated was essential for speedy on boarding of clients, but posed the unique problem of requiring human input and dependence to ensure data accuracy.

This resulted in incomplete, inaccurate, miscoded, and many variants of the same data. These data quality concerns significantly hindered the client’s capacity to progress their business and manage their clientele.

Accurately Project Market Statistics

Due to a lack of comprehensive data, it is impossible to make an exact projection regarding the market penetration and subscription pattern for a particular target area.

This has a knock-on effect on the clients’ strategies for expanding their businesses.

Our Take on Solving Our Customer’s Problems

We were of the firm belief that in order to put a stop to the problem of revenue loss and circular churners, the data belonging to our Client needed to be cleansed and standardized.

Only after that will they be able to obtain insights into the profiles of their clients, which will assist them in making better decisions and growing their business.

Cleansing and standardizing existing customer and address data

Before a business can begin to focus on its consumers, its operations and data challenges must be resolved. Likewise, this was the case with our Client. Before attempting to utilize their data, they were required to cleanse and standardize it. The data quality terms standardize and cleanse have diverse meanings.

Given the uniqueness of Malaysia’s addresses, especially in the country’s outskirts and smaller towns, the greatest difficulties were in address cleaning and standardization. The outcome of the cleansed addresses, must conform to a pre-defined standard and must make sense to the postman.

Transforming dirty data to clean usable data involves these 2 (two) main aspects:

  • Data standardization – Have your data follow a certain format and rules for consistency
  • Data enrichment – On top of standardization, fill in or reformat missing data.

The application of the data quality rules, were applied across all addresses and contact data fields such as:

  • Mobile Numbers
  • Landline Phone Numbers
  • Phone Extension Numbers
  • Contact Names & Salutations
  • Identification Numbers
  • Emails (with special handlings)
  • Ethnicity
  • State & Postcode Fields


Uploading Cleansed Data

Following the completion of the data cleansing and standardization procedure, the data were put through a round of user verification using a data verification user interface that had been specifically built for that purpose. This verification was based on the data changes scoring on top of user defined data filtering capabilities.

It was planned that the certified data on that system would be uploaded to the source system at a later time without interfering with or interrupting the regular activities of the users or their customers.

As a result of the activities that were carried out, our client was able to have access to a consolidated CIF (Customer Information File) data view, which was of great assistance to them in the expansion of their company.

Real-time Data Cleansing Interface

  • The initial cleansing’s cleansing and standardization rules are used to cleanse and standardize incoming data in real-time before storing it in the database. This ensured database data consistency. The company also integrated this interface on additional customer-information systems. This comprehensive process ensured the company always had clean, reliable data.
  • Implemented rule-based intelligence for fraud detection to automatically analyze “suspect” clients or accounts and retain them. This method helped the organization avoid new capital and hardware expenses for churning accounts, saving money over time.

A strategy for real-time data purification and standardization was supplemented in order to prevent the data that had previously been cleansed and standardized from becoming corrupted.

How our Customer benefitted from this initiative?

As elementary as it sounds, our customer gained tremendously from this data cleansing & standardization exercise.

These benefits laid the foundation for their growth for decades to come and into greater cost savings and data hygiene best practices.

Quick Return on Investment

Our client saved a lot of money by eliminating income leakage from circular churners. Fraud detection techniques have also boosted the amount of money that can be recovered through the payment recovery.

Within a year of deployment, our client saw a return on their investment.

Robust and battle-tested Solution

Even now, more than a decade after they were first implemented, the solutions we created are providing excellent service and cost savings to our customer.

Wealth of Knowledge and Experience

The collection of best practices and implementation experience that we acquired as a result of this project was extremely beneficial to us.

This foundation allowed us to assist our other clients in overcoming data issues and attaining business success via data quality and digital transformations.

Contact us today and explore with us how we can help your organization