Connecting to Azure Synapse Analytics is easy, almost the same as connecting to the SQL Server. We will be loading the data into Synapse by using the cloud storage as an intermediate storage layer and will need to specify the storage details as well.
Server hosting the Synapse database is an Azure server with a URL in form of <account>.database.windows.net. No need to specify the port.
Omni Loader supports the following authentication schemes:
Once the server and authentication are provided, you can type your database name, or select it from the drop-down.
Synapse can load data from Azure Blob Storage container, or from Azure Data Lake Storage Gen 2 (ADLSv2) container. We recommend ADLSv2.
Data format can be CSV or Parquet. CSV is inefficient row-based textual format, while Parquet is well-designed and fast columnar binary format. We recommend Parquet as it is significantly faster to ingest and produces smaller data files.
Both formats can be used uncompressed and compressed. One should always use the compressed format even if it means longer data preparation. Parquet can compress data up to 5 times, which directly impacts the efficiency of Synapse ingestion. While compressing terabytes of data takes time, Omni Loader is designed for efficiency and will use all CPU cores of the machine it is running on to compress the data. For further acceleration, you should use Omni Loader in a cluster mode to utilize the CPUs of several machines.
CSV can be compressed using a slow Gzip, but Parquet can use very fast Snappy compression as well. We recommend using Parquet with a Snappy compression.
Data handling defines the way Omni Loader will generate the files and clean up:
We support three modes of storage authentication:
The only thing left to specify is the name of the container in the storage account. Of course, if you use Shared Access Signature of a specific container, that's the name you need to specify.
Optionally, if you would like to put the data into a subfolder of the container, you can specify a subfolder name.