ELT (Extract, Load, Transform)

A modern approach to data migration, relying on the fact that today massively parallel databases are very good at transforming data. It's far more efficient to load the data quickly into the target database and transform it once it's in compared to engineering complex workflows of joining and aggregating disparate data sources, joining and aggregating as intermediate step and pushing the result to the target. Omni Loader does transform schema and data on the fly, but it does a minimal transformation necessarry to bring everything over in a way compatible with the target database. It maps source data types to the closest available data types in the target database. It re-encodes data, if necessarry, on the fly. And it never touches the disk while it does so.

Massive parallelism

To efficiently migrate a lot of data, parallelism is essential. To efficiently ingest very large tables, data needs to be partitioned. To efficiently migrate very large single table, data from that single table needs to be sliced into smaller parts so that individual slices can be copied in parallel. Omni Loader does all that, automatically, while allowing adjustment of the defaults.

Distributed migration cluster

On cloud, access to powerful virtual machines with plenty of CPU cores is readily available, on-premises installation often runs on a more modest hardware. To enable the massive parallelism enabling the performance needed to push many terabytes to the cloud or to a powerful database server, Omni Loader controls agents running on as many machines as necessary. Each agent further runs many jobs in parallel. Both orchestrator and agents are native Windows and Linux applications and you can mix and match with no limitations.