Azure Data Factory Reviews and Ratings

Use Cases and Deployment Scope

We live in a world where half of the data for analytics come from SAP and half from non SAP sources. We use Azure Data Factory to load non SAP data from different source systems into Azure lake house. The project follows medallion architecture where Azure Data Factory takes data from multiple sources and stores them in the bronze layer of the medallion architecture. Since our SAP datasphere has limitations connecting to non SAP sources as good and native like Azure Data Factory, we use Azure Data Factory for these scenarios. Further modelling of data in the next layers (silver layer and gold layer) is done using Azure Data Bricks, where the final data product is created. The Azure Data Factory also helps in applying transformations which loading the data from different source systems. Datasphere often relies on ODBC/JDBC/OData connectivity, whereas Azure Data Factory provides maintenance-free connectors for our web applications, like partner portal, cloud applications like one crm, on-prem Oracle systems, and also to NoSQL dbs like MongoDB. To summarize Azure Data Factory is used in our organisation to ingest non SAP data from different sources into our Bronze layer for the Databricks to further clean and curate the data for data product creation. Without Azure data factory connecting the data from different source wouldnt been possible because SAP Datasphere has limitations when it comes to connection with non SAP source systems

Likelihood to Recommend

Best scenario is for ETL process. The flexibility and connectivity is outstanding. For our environment, SAP data connectivity with Azure Data Factory offers very limited features compared to SAP Data Sphere. Due to the limited modelling capacity of the tool, we use Databricks for data modelling and cleaning. Usage of multiple tools could have been avoided if adf has modelling capabilities.

Use Cases and Deployment Scope

Azure Data Factory is a data integration technology used to integrate data to the cloud, especially. Azure cloud. Today, any organization wants to integrate its data into a cloud data platform. This cloud platform serves data for various purposes, such as sharing it with downstream applications, developing analytics, and even building AI applications on it. Azure Data Factory integrates data from various sources, such as on-prem applications, databases, and files, into this data platform.

Likelihood to Recommend

Azure Data Factory is a great data integration tool for developing a cloud data platform, especially within the Azure ecosystem. Azure Data Factory is very good for the Data Ingestion part. It can work for simple data transformation with its Data Flow, but it will also need cluster configuration, and there is some cost. Also, it is an excellent tool for orchestrating data pipelines. But for complex data transformations, you may need to use technologies like Databricks and PySpark.

Use Cases and Deployment Scope

Using SHIR to pull records from on-premise databases and storing in ADLS storage. From ADLS storage, bringing data into databricks for analytics use. Roughly 50 different pipelines in each environment, with 3 separate environments. Code is stored and deployed from Azure Dev ops. Alerting is handled via LogicMonitor and Azure Functions.

Likelihood to Recommend

Pulling data from Databases in a scheduled and controlled pattern. We heavily use the self hosted SHIR option to pull from existing sources. In addition, we have several API's directly from Azure Data Factory using keyvault for passphrase storage. Utilizing GIT in Devops helps promote changes within our multiple environments in a timely matter.

Use Cases and Deployment Scope

Data Integration: We harness Azure Data Factory's capabilities to move data from various sources – both on-premises databases and cloud storage – into our Azure data storage solutions like Azure SQL Database, Azure Blob Storage, and Azure Data Lake Store. This ensures all our data, regardless of its origin, is consolidated in one place.

Transformations: Azure Data Factory's data flow transformations help us clean, transform, and enrich our data before loading it to the destination. This is crucial for maintaining data quality, especially when dealing with diverse datasets.

Likelihood to Recommend

Well-suited Scenarios for Azure Data Factory (ADF):

When an organization has data sources spread across on-premises databases and cloud storage solutions, I think Azure Data Factory is excellent for integrating these sources.

Azure Data Factory's integration with Azure Databricks allows it to handle large-scale data transformations effectively, leveraging the power of distributed processing.

For regular ETL or ELT processes that need to run at specific intervals (daily, weekly, etc.), I think Azure Data Factory's scheduling capabilities are very handy.

Less Appropriate Scenarios for Azure Data Factory:

Real-time Data Streaming - Azure Data Factory is primarily batch-oriented.

Simple Data Copy Tasks - For straightforward data copy tasks without the need for transformation or complex workflows, in my opinion, using Azure Data Factory might be overkill; simpler tools or scripts could suffice.

Advanced Data Science Workflows: While Azure Data Factory can handle data prep and transformation, in my experience, it's not designed for in-depth data science tasks. I think for advanced analytics, machine learning, or statistical modeling, integration with specialized tools would be necessary.

Use Cases and Deployment Scope

One of the best Data Integration tools for both ETL and ELT. I have been using ADF for the last 6+ years and it helped me in extracting several data feeds within our organization that meets our specific business needs. The tool provides many features such as Move and Transform, Data explorer, Azure Functions, Data bricks, Data Lake Analytics, Blob Storage, Linked services, Machine Learning, and Power Query.

Likelihood to Recommend

In a data pipeline, you will be able to add different kinds of activities for example connect from your on-premise SFTP and move CSV files to storage accounts. As well data factory has its own data flow if you are an ETL developer who experimented with maybe you have worked with SSIS, thus, you will start quickly with this new feature of the data factory.

Use Cases and Deployment Scope

Orchestration platform for the Databricks notebooks. Have used an ETL for loading csv files into SQL server based database.

Likelihood to Recommend

Well Suited:

<ul><li>Offers low code/no code features executes against spark pool.</li><li>Batch processing features, Tight coupling with Databricks &amp; ETL jobs. </li><li>Offers Logic apps &amp; Azure functions invoking API. </li></ul>Less Appropriate:

<ul><li>Not much inherent features of Stream analytics (Liasing Azure Stream analytics to DF might be good option).</li><li>Advanced load options viz . Upsert type operations missing. </li></ul>

Use Cases and Deployment Scope

We use Azure Data Factory to orchestrate ETL and ELT pipelines in our projects and it has a wide range of data connectors right from classic(FTP) and modern data lake storage like ADLS and AWS S3. Using the Dataflow and SSIS integration runtime server, it can perform complex transformations without the need for another tool. Executing and Monitoring ETL loads the data factory is very simple and user-friendly.

Likelihood to Recommend

Azure Data Factory can perform better with Azure services and can easily do cloud migrations from on-premises services like SQL Server. It has a limited set of functionalities to transform data using SSIS integration services and data flow.

Use Cases and Deployment Scope

Get full data integration at scale with Microsoft Azure Data Factory's management service for serverless data integration,‌ hybrid data integration is easily and agilely possible through this software. We are creating ETL and ELT workflows as well as orchestrating and monitoring pipelines without writing any code. Full management and serverless integration with default features installed on the system and various connectors reduce costs. It is used in all company departments and project management units of our customers. Since ADF adopts an intelligent intent-driven mapping methodology, it enables copy activities to be automated.

Likelihood to Recommend

If you know a bit about database management everything is pretty easy, based on my personal experience. You can build a lot of things entirely in design, even if you do not know all the syntax, [since] having ready-made templates simplifies everything to get started and build the first pipeline.

Use Cases and Deployment Scope

I've used it to perform PoC's and work with data transformation processes that interact with other applications or tools.

Likelihood to Recommend

It works better than other tools from the same range, it has a beautiful UI and it makes work easy. Its also very easy to integrate with other tools, tools, apps and ecosystems.

Use Cases and Deployment Scope

Azure Data Factory, particularly V2, offers a good option as a cloud-based ETL tool if you are leveraging the Azure cloud. We are using it as we begin to hybridize our on-prem data warehouse and applications with Azure. Up until now, we have leveraged SSIS for these purposes, but are beginning to migrate ETL and other data movement functions to the cloud, with Azure Data Factory as the primary utility.

Likelihood to Recommend

If you are just getting started and all your data is resident in the Azure cloud, then Azure Data Factory is likely to work fine without having to jump through too many hoops. However, in a hybrid environment (which is most of them these days), ADF will likely need a leg up. It works well for scheduling and basic scheduling/orchestration tasks, but the feature set is not at a level with SSIS (which has been around for 15 years so...). As ADF now supports deploying SSIS, it is also a good candidate if large amounts of your data are resident in the Azure cloud and you have an existing SSIS investment in code and licensing. We are using it in a hybrid fashion for the data warehouse and will slowly transition over to ADF as the feature set improves. We are also using it for cloud-native applications that only require supplemental data from on-prem resources.