When report numbers change, troubleshooting often begins with questioning back and forth between multiple systems. Data lineage can connect sources, processing tasks and final indicators to narrow the scope of the problem.

Start from requirements

Record input-output relationships, conversion rules and task running versions, and retain verification results for important processing steps; put business specification descriptions and technical dependencies in the same traceability path.

Things that should be paid attention to in the design

For example, if a report adds conditions for excluding return orders, the lineage record should indicate which step the change occurred, rather than just showing a string of connections between tables.

For example, handle it like this

Try to trace the total number of exceptions to the corresponding batch and filter conditions, and test the impact prompts after the upstream field is renamed; in addition to the dependency graph, the person in charge of the processing also needs to be traceable.

Check the processing results item by item

  • The traceability record should cover manual corrections.
  • If the analyst changes the data in the exported file and then sends it back, the final result cannot be explained by database task relationships alone.
  • Manual adjustment can be used as a clear step, and the reason for adjustment, original value and confirmer can be recorded.
  • Lineage can also help evaluate changes: before you prepare to delete a field, see which metrics, exports, and subscriptions are still referencing it.
  • For complex calculations, keep readable rule descriptions and do not expect business personnel to judge the impact from the program name.
  • The task running log must be able to correspond to the results of a certain report, especially the data version after re-running.
  • After the investigation is completed, associate the reasons to relevant nodes so that there is a basis for investigation when similar differences occur later.

Check the processing results

If multiple input snapshots are used for one processing, the traceability information should indicate the batches respectively. Merely knowing the task running time cannot restore the data combination at that time. When troubleshooting, it is likely to rerun with a new source but not get the old results.