Finding new or changed records in a large dataset

There are a number of platforms from which data are poured on the target platform.
A large number of platforms are database, part of the platforms provides access via API.
The total number of records of the order of several million.
Also the data on the source platforms can vary — need to track changes and update data on the destination platform.

Suggest how best to implement the mechanism to fill the new data change tracking and update?
October 8th 19 at 00:30
2 answers
October 8th 19 at 00:32
The conditions are too vague, much will depend on the specific API. Here is one solution to these problems for mysql tables

maybe some ideas will be useful.
October 8th 19 at 00:34
Native replication in databases is more efficient IMHO.
If the database can send signals to the application (seen in Interbase), is generally high.

If the database is heterogeneous, then the knee will have to write the whole kitchen up replication from scratch. With their bikes and southcrete old rake.
However, I recommend to look through OLAP techniques and/or integration of the data bus. Also bikes, but the industrial scale. - Haleigh_Bailey44 commented on October 8th 19 at 00:37

Find more questions by tags AlgorithmsWeb DevelopmentPHP