Driving CI/CD ~ Azure Data Factory ~

Hello. This is Akiyama from Azure Cloud Solution Architect. In this article, we will introduce the CI/CD initiatives we are pursuing, focusing on Azure Data Factory (ADF). I won't go into the introduction of ADF itself this time. It's a little old, but there is an article about when I was hands-on before, so please refer to it. Publish hands-on materials to get started with Azure Data Factory

Why are you driving CI/CD?

Do you have too much time? I always want more. If I had time to sell it in bags, I would like to buy it.

In order to make effective use of such a finite and equal resource, time, the promotion of CI/CD is important.

In a project with three environments, development, verification, and production, there is a need to automatically deploy the same deliverables as the development environment for the environments described later than the development environment. The reason for this is that we want to avoid boring problems such as bugs fixed in the development environment occurring in the verification/production environment. If a person is manually deploying three environments, these trivial problems will occur quite often. People are creatures who make mistakes, so instead of spending time on unavoidable problems, they want to spend time creating value.

Azure Data Factory CI/CD Construction Example

The configuration image is as follows.

• Developers interact with ADF's development portal in the development environment to develop
• Developers do publish in the development break
• When you publish, the ARM Template is committed to Azure Repos
• When a commit triggers Azure Pipelines, use an ARM template to deploy the resource to ADF in validation/production
• Developers manage environment variances in the ARM Template Parameter and update them accordingly

Reference: https://docs.microsoft.com/ja-jp/azure/data-factory/continuous-integration-deployment

Conclusion

In this article, we introduced CI/CD for Azure Data Factory, which is used as ETL when creating a data analysis platform. At Colorkrew, we drive CI/CD and DevOps initiatives.

If you are interested in supporting the construction of a data analysis platform, please contact us below. Support for Data Analysis Infrastructure Construction |