After the quality check finds null values ​​or duplicate values, if only one warning email is sent, the problem will often reoccur. Data quality requires processing paths and knowing which anomalies actually impact the business.

First clarify the problem and conditions

Set rules according to field usage, assign responsible persons and processing deadlines to exceptions; distinguish between blocking access, isolating for verification, and allowing storage but prompting.

Put rules into the actual process

For example, a missing address may not affect summary sales, but it will affect distribution analysis. Therefore, it should be evaluated according to usage scenarios instead of giving all empty fields the same severity level.

Check with specific scenarios

Check whether the problem can be located in the source batch, whether it can be rerun and repaired, and whether repeated alarms have been merged; after processing, you should also check whether the same exception occurs again in the next synchronization.

Check the processing results item by item

  • The quality problem page needs to provide actionable information: specific abnormal values, corresponding rules, impact reports and the latest occurrence time.
  • When a task fails and thousands of identical exceptions occur, they can be summarized by source and reason, allowing the person in charge to locate the root cause first and then view the details.
  • Rules cannot only increase but not decrease for a long time; verification conditions should be modified for fields that are allowed to be empty in the business, and alarms whose sources have been retired should be exited.
  • When repairing, keep the original value and correction basis, and make it clear whether the source system or the platform copy is changed, otherwise the next collection will still be overwritten.
  • For exceptions that are temporarily accepted for business, a valid period and explanation can be set, and the period will be rechecked.
  • Alarm closure and data repair need to be recorded separately to avoid taking confirmation as the completion of management.

How to review before and after going online

Rule maintenance can record the reasons for false positives and distinguish between omitted business conditions and source data errors. Rules that are continuously judged as reasonable exceptions by humans are worth redesigning, rather than having the person in charge close the same alarm repeatedly every day.