Data duplication can be any record that inadvertently shares data with another record. Basically, this is data that has been entered by other systems, by users, imported, and it can relate to customers, suppliers, products, and more. This data duplication costs companies vast amounts of money. This can lead to missed opportunities, leads, etc.
The main difficulty on de-duplicating data is how is the data row duplicate to another record. There are many factors to determine this. It could be the name, address, phone number, or other fields. At Cloud Performer, we use multiple rules to follow that will determine duplicate rows. For example, we use a company name rule that helps normalize rows. It eliminates things such as LLC., LLC, Inc., Inc, Incorporated, etc. Additionally, we use algorithms such as Levenshtein distancing, Soundex, and others.
Levenshtein will match both of these as they are less than the typical 2 space limit. Another approach used is combining multiple fields together to create a uniqueness test.
1st four letters of Name, 1st 7 letters of Street, 1st 5 letters of City, 1st 2 letters of State, 1st 5 letters of zip code. This to helps in distinguishing unique and non-unique records in your source. You will notice this is two different account names. Is this a duplicate? Did they change their name? Or is this a typo. This is why we need business experts to help in the clean-up effort. This isn’t a one-person show, we will need your help in the clean-up.
This process is normally the longest to properly get correct. Just imagine if this were millions of rows of account data. If done correctly we will get a clean dataset to start new. If not, you will be spending a lot of time and money cleaning up manually the bad data that was loaded.
A picture is worth a thousand words may be cliché, but that doesn’t make it any less true. Life in 2020 looks differently than anyone could have imagined and for many of us the challenges…
There is nothing better than sharing BBQ with family and friends especially over a holiday weekend! Several of us at Cloud Performer often share our experiences and recipes. Even when we can’t get together in…