Distributed Migration Cluster

Omni Loader can seamlessly utilize a number of local machines working in parallel to speed up the migration process.

How to start?

In order to get started, select the orchestrator machine and run Omni Loader in Cluster Mode. Then, run Omni Loader Agent on any additional machine which you wish to be part of the cluster.

When is Cluster Mode useful?

You might find yourself in a situation where you need to migrate billions of records and finishing the process on your machine alone would take time you might not have.

Naturally, it would be optimal to vertically scale your machine for better performance (upgrade to a more powerful CPU). If this is not possible, horizontal scaling is the next best solution. Joining machines into a cluster is effectively an alternative to adding more cores to your machine.

What sets it apart?

Cluster Mode is designed to look, act and feel the same as Omni Loader’s Standalone Mode.

By default, Omni Loader assigns each worker a CPU core. When your workload is not CPU intensive, you can customize the worker number to be higher than the number of cores on your machine.

There can be an unlimited number of agents within the cluster, and you can mix and match Windows and Linux systems freely. What this means is that some agents could be running on Windows and others on Linux.

How is this important for you?

The objective behind Omni Loader’s Cluster Mode is to save you time and execute your desired tasks as effectively as possible. It is designed to make the user experience as straightforward as possible while doing all the hard work behind the scenes.

How do we deal with latency?

A common scenario people face when migrating or comparing data within clusters globally is latency. Because the machines within the cluster are connected via an encrypted global network, this gives way to latency as source and target databases are geographically apart and communication goes through multiple hops to reach the destination.

Omni Loader minimizes latency by executing processes parallelly, workers are simultaneously performing multiple tasks. While awaiting acknowledgement on certain network packets, other network packets are being pushed through the pipe, effectively cancelling out the latency.

How does Omni Loader’s Cluster Mode work?

Omni Loader communicates with the agents through a WebSocket connection which allows for ultra-fast communication. It sends commands and receives status updates nearly instantaneously. Each worker has a dedicated connection to the source and/or target database and does a full range of tasks as required.

To execute tasks in the shortest amount of time, each worker is assigned tasks by the orchestrator. All tasks that can be parallelized are parallelized and ran on multiple workers.