FinOps for Databricks

Introduction  

The rise of SAAS Data platforms has given users and admins the ability to create on-demand resources such as compute, RAM, and GPU. While this is beneficial, it can also lead to situations where costs spiral out of control. That is why it is crucial to proactively monitor costs; a practice known as "FinOps." 

FinOps, short for Financial Operations, is not just about managing cloud finances. It also promotes a cultural shift within the organization, fostering collaboration between finance, engineering, and business teams. One key aspect of FinOps is shared responsibility. Another important aspect is making data-driven decisions. By actively monitoring costs, organizations can identify areas for optimization and make informed choices. 

The goal of FinOps is to achieve a balance between cost optimization and meeting business needs. It is about managing your cloud finances effectively to catch waste early, optimize resource utilization, stay within budget, and make informed decisions. 

Databricks and FinOps Phases  

There are three phases of FinOps, namely Inform, Optimize, and Operate, which can be applied to Databricks, too. These phases allow organizations to gain visibility into costs, optimize resource usage, and align their plans with business objectives. 

• During the Inform phase of FinOps in Databricks, organizations need to gather and analyze data to gain a comprehensive understanding of fully loaded costs and benchmark performance.

  This involves identifying all the costs associated with running workloads on Databricks, including compute, storage, and data transfer costs. 

• In the Optimize phase, the focus shifts toward enabling real-time decision-making and optimizing usage. This involves implementing cost optimization strategies such as rightsizing instances, leveraging spot instances, and optimizing data storage.  

• Lastly, in the Operate phase, organizations align their plans with the business objectives and optimize rates. This involves negotiating contracts with Databricks and other cloud service providers to ensure favorable pricing terms.  

FinOps in Databricks 

To effectively manage expenses in a Databricks workspace, consider the following strategies: 

• Understand and verify DBUs: DBUs represent computational resource usage on Databricks. The number of DBUs used depends on the cluster's node count and VM instance capabilities. Different cloud providers have different DBU rates, so verify the rates for your specific cloud service. 

• Implement Cluster Policies: Use cluster policies to control DBU usage. Set parameters such as maximum node count, instance types, and auto-termination rules to optimize resource allocation and prevent unnecessary expenses. 

• Monitor Compute Costs: Keep track of compute consumption, including VM expenses. Identify underutilized or overutilized resources and adjust cluster sizes accordingly. Consider using auto-scaling to dynamically adjust cluster capacity based on workload requirements. 

• Manage Storage Costs: Monitor storage expenses within the workspace. Understand data growth patterns and optimize storage distribution. Make use of managed storage solutions provided by the cloud service. 

• Regular Evaluation and Monitoring: Continuously assess DBU, compute, and storage utilization. Look for opportunities to reduce costs and adjust resource allocation as needed. Ensure that the workspace operates within budget constraints. 

Understanding DBU  

The fundamental unit of consumption is a Databricks Unit (DBU). The number of DBUs consumed, except SQL Warehouses, is determined by the number of nodes and the computational capabilities of the underlying VM instance types that form the respective cluster. In the case of SQL Warehouses, the DBU rate is the cumulative sum of the DBU rates of the clusters comprising the endpoint. 

At an elevated level, each cloud provider will have slightly different DBU rates for similar clusters due to variations in node types across clouds. However, the Databricks website provides DBU calculators for each supported cloud provider (AWS | Azure | GCP).  

Cluster policies: Managing Costs 

An admin can use a cluster policy to control configurations for newly created clusters. These policies can be assigned to either individual users or groups. By default, all users can create clusters without restrictions, but caution should be exercised to prevent excessive costs. 

Policies limit configuration settings. This can be achieved by setting specific values, defining value ranges using regex, or allowing an open default value. These policies also restrict the number of DBUs (Database Units) consumed by a cluster, allowing control over settings such as VM instance types, maximum DBUs per hour, and workload type. 

Node count limits, auto-termination, and auto-scaling 

Clusters that are underutilized or inactive are a frequent problem with computing expenses. Databricks provides solutions to address these issues dynamically without requiring direct user intervention. By implementing auto-scaling and auto-termination functionalities through policies, users can optimize their computational resources without any hindrance to their access. 

Node count restrictions and automatic scaling policies can be enforced to ensure the activation of auto-scaling capability with a minimum number of worker nodes. 

Also, when setting up a cluster on the Databricks platform, users can configure the auto-termination time attribute. This attribute automatically shuts down the cluster after a specified period of inactivity. Inactivity is determined by the absence of any activity related to Spark jobs, Structured Streaming, or JDBC calls. 

Spot Instances and Cloud instance types 

When configuring a cluster, you can choose VM instances for the driver and worker nodes, each with its own DBUs rate. To simplify instance management, you can use the "allowlist" or "fixed" type to restrict usage to a single type. For smaller data workloads, it is recommended to choose lower memory instance types. GPU clusters are ideal for training deep learning models due to their higher DBU requirements, but it is important to balance the limitations of instance types. 

If a team needs more resources than allowed by policy restrictions, the job may take longer and incur higher costs. 

Spot instances are discounted VMs offered by cloud providers. However, they can be reclaimed with minimal notice (2 minutes for AWS and 30 seconds for Azure and GCP).