Translated from 日本語 • View original Hello, this is Hara, who is in charge of Azure.
Today, we will introduce the pricing structure of Azure Databricks, a PaaS for data infrastructure that has recently been recommended. Azure Databricks is a highly recommended service in terms of functionality and performance. By understanding the fee structure, you can use it more actively with peace of mind.
*The prices listed in the text are as of March ~ April 2022.
For more information about Azure Databricks itself, please see the official documentation
What is Azure Databricks? | Microsoft Docs
Azure Databricks pricing overview
Azure Databricks usage fees can be broadly divided into two parts: (1) Azure resource usage fees and (2) Databricks software usage fees.
About Azure resource usage fees
When using Azure Databricks, the following resources are deployed on Azure. The "data lake" on the left side of the diagram refers to the storage account that stores the data you own, not the Azure Databricks-specific resources. The Databricks workspace box on the right is the resources that are generated when you use Azure Databricks.
A resource that is created where the "User Subscription" is visible to the user. Azure Databricks automatically creates and deletes them, but the resources themselves are charged in line with the Azure price list, such as regular virtual machines, disks, and storage.
Resources in Microsoft Managed Subscriptions are not visible to you and are not explicitly billable.
Source: Manage virtual networks - Azure Databricks | Microsoft Docs
Azure Databricks Software Usage Fees
Besides Azure resources, you will be charged for Databricks software. The unit is "Databricks Units (DBU)", and the price for running a 1 DBU node for 1 hour is as follows. Actual charges are calculated in seconds.
Try Calculating Cluster Charges
Now let's calculate the price. Azure Databricks data processing is done in clusters.
There are several types of clusters, but here we will use clusters for SQL Compute. SQL Compute clusters have a preset combination of size and number of worker nodes, as shown in the table below.
Example "Small" Cluster Pricing
A "small" cluster consists of one driver node and four worker nodes. The types and amounts of resources used for running this size for one hour are as follows:
If you use a Spot VM for an Azure virtual machine: In this example, the cost of the virtual machine is 83% off. If the workload is retried and there is no problem even if the processing is interrupted, the cost can be significantly reduced.
※spot vm
Cluster Size Cluster Cost by Workload
The cost per DBU in Azure Databricks depends on the workload in your cluster. The cluster cost per hour calculated by size workload based on SQL compute cluster size is shown in the following figure. For All purpose and Job workloads, you can flexibly set the driver specifications and the number of workers, but here we have calculated them according to the specifications of SQL workloads for comparison.
When Should a Cluster Be Launched
Essentially, the cluster must be up when accessing user data. Clusters are used not only when SELEC table data, but also when viewing table lists and looking at file systems (Databricks File System = DBFS). You don't need a cluster to view and edit notebooks, configure jobs, manage clusters, etc.
Managing Usage Fees
You can attach tags from the cluster settings. The tags you set here are automatically added to the Azure resources generated by Databricks, so you can allocate usage fees according to the use of the cluster.
Consider Pre-Purchase Plans
For DBUs, there is a mechanism where you can purchase Databricks Commit Units (DBCUs) for one year or three years in advance. It works similarly to virtual machine reservations, but unlike virtual machine reservations, which expire every hour, it is a credit that can be consumed at any time for one year if paid in advance. It can be said that the mechanism is close to the commitment of Azure EA.
If you use up the DBCUs you purchased in advance, you will be charged the DBU at the list price. The cluster will not be stopped or performance will be limited.
For DBCU plans, visit the Azure Databricks pricing page.
Azure Databricks pricing | Microsoft Azure
An example of DBCU calculation is shown in the table below. The lowest-priced DBCU plan (25,000 DBUs per year) is the equivalent of running a "Small" cluster for 9 hours per day, 20 days per month.
Example Datastore Pricing
Storage accounts are available for the datastore. I will omit the details because there are no elements specific to Azure Databricks in the estimation of the usage fee, but if you use Data lake storage gen2 for 10TB with local redundant storage in East Japan, it will be about 200~300 USD.
Storage accounts are also discounted by purchasing pre-orders. Please see the documentation for details.
How reservation discounts apply to Azure Storage services | Microsoft Docs
Conclusion
In this article, we've learned more about Azure Databricks pricing. Overall, it's a simple mechanism that charges for cluster uptime. It is an easy-to-handle service not only in terms of performance and functionality, but also in terms of price, so I hope you will try it.
Colorkrew provides support services for the implementation of data analytics platforms based on Azure Databricks. If you are interested, please contact us through the following service page.
Support for Data Analysis Infrastructure Construction |
