Not every table needs to be synchronized in real time. Putting data that is not urgently needed into the real-time link will increase the burden of operation and maintenance; conversely, too long a delay will also make inventory or customer service judgments lose reference value.

Start from the needs

Select the method according to the delay allowed by the business, the frequency of data changes and the source capability, and indicate the update time; establish checkpoints for incremental access to handle duplication, disorder and retry.

Things that should be paid attention to in the design

For example, the business daily report can be summarized at night, and the status of the orders being processed needs to be more timely. The two can coexist, but the difference in data freshness cannot be hidden on the same page.

For example, handle it like this

Simulate the source to be disconnected and restored, confirm whether the increments are completed and there are no duplications; check whether the total number is stable after the batch task is rerun, and then determine the alarm and manual intervention conditions.

Check the processing results item by item

  • When the incremental task resumes, you need to know which checkpoint to continue from.
  • Just remembering the last startup time may miss late records, so overlapping windows and deduplication keys should be designed in conjunction with source capabilities.
  • Batch import must also distinguish between files that have been received, content verification passed, and results officially available. After the user uploads the file, it does not mean that the report has been updated.
  • The task page displays the latest success time, the scope of processing and the expected next batch, making it easier to locate problems than only showing running tasks.
  • For high-frequency synchronization, a single error can be entered into the isolation area to avoid the entire batch being shut down; however, whether to continue when key fields are missing should still be implemented according to the business agreement.
  • When the business dashboard quotes data with different frequencies, explain the cut-off time of each data.

Check the processing results

When the source interface is current-limited, you can reduce concurrency and schedule replenishment instead of continuously initiating failed requests. The recovery plan clearly indicates the point in time to be supplemented, and the report page can also prompt that there is still some data waiting to be synchronized.