High Performance

Omni Loader is a database migration solution with a massively parallel migration engine. It does not require any third-party systems for transforming data.

Omni Loader’s engine is a self-tuning, high-performance streaming engine. It does not require a setup and is ready from the get-go. It enables you to get on with your task in a few clicks and does not require expert-level knowledge.

How does it operate?

Omni Loader reads data, transforms it into a form optimal for the target database, and implements customizations specified by the user.

Streaming data saves you time because the databases continuously serve and ingest the data without waiting for the batch process to complete.

Omni Loader does not write or use the local disk. Instead, it pushes data into the target database without writing intermediate results onto the disk at any point. Data is streamed from the source database to the individual agent, transformed in memory, and streamed to the target server.

Our migration engine is massively parallel, subdividing work into smaller units and distributing it among multiple workers for efficiency. Each worker is independent, executing work uniquely assigned to it, with no stalls in the pipeline. Parallel workers enable stellar performance, even when copying terabytes of data from a single table. Parallelism allows the workers to execute tasks simultaneously, and because each is communicating with the orchestrator individually, each focuses on its uniquely assigned tasks, unaffected by other workers. Omni Loader automatically sets the optimal number of workers for your machine.

All the transformations necessary to correctly bring the source database into the target server are applied automatically. Source data types are mapped to the closest available data types in the target database.

What features improve performance?

Omni Loader’s key feature is its ability to optimize your machine’s full potential by employing all CPU cores, generally assigning each worker a CPU core. Sometimes you may want to use more workers than there are CPU cores, but that is only the case when the task is not CPU-intensive. More workers mean more tasks being executed simultaneously, meaning shorter process time.

Due to Omni Loader’s high performance, it can easily saturate either the CPU, network or the target database’s ability to ingest data. Saturation is always a bottleneck, but the steps to remove the bottleneck are easy.

CPU saturation is resolved by scaling the machine, either vertically (upgrading your machine) or horizontally (running a cluster). Learn more

‍If your database server has very fast network connectivity but an individual machine running Omni Loader reaches its limits in network bandwidth, running Omni Loader in Cluster Mode allows the multiplication of said network bandwidth by a factor of “number of agents”.

Deliberately slowing Omni Loader down

If data transfer Omni Loader is capable of would cause a disruption in other business processes on the network, you can choose to cap the network bandwidth Omni Loader is allowed to use. This functionality is built into Omni Loader and exposed as a network-throttling option. One can specify the desired number of megabits per second and Omni Loader will abide by it.