We create a serverless instance to cater for unpredictable workloads.The following figure shows the solution and workflow steps: Therefore, when a workload is moved from one Amazon Redshift data warehouse to another, the workload can continue to access data in the initial Amazon Redshift data warehouse. In addition, moving unpredictable workloads into a separate Amazon Redshift data warehouse allows each Amazon Redshift data warehouse to scale resources independently.Īmazon Redshift Data Sharing enables data access across Amazon Redshift data warehouses without having to copy or move data. The solution Safet圜ulture used is comprised of Amazon Redshift Serverless and Amazon Redshift Data Sharing, along with the existing Amazon Redshift provisioned cluster.Īmazon Redshift Serverless caters to unpredictable workloads in a cost-effective manner because compute cost is not incurred when there is no workload. Continues to allow models to be created and modified based on production data.Separates unpredictable workloads from predictable workloads to scale compute resources independently.Caters for unpredictable workloads in a cost-effective manner.With plans for further growth in dbt Cloud workloads, Safet圜ulture needed a solution that does the following: To balance the needs of unpredictable and predictable workloads, Safet圜ulture used Amazon Redshift workload management (WLM) to flexibly manage workload priorities. Whenever models are created or modified, a dbt Cloud CI job is triggered to test the models by materializing the models in Amazon Redshift. A source of unpredictable workloads is dbt Cloud, which Safet圜ulture uses to manage data transformations in the form of models. Safet圜ulture runs an Amazon Redshift provisioned cluster to support unpredictable and predictable workloads. In this post, we share the solution Safet圜ulture used to scale unpredictable dbt Cloud workloads in a cost-effective manner with Amazon Redshift. Together with price-performance, Amazon Redshift enables you to use your data to acquire new insights for your business and customers while keeping costs low. Its operations platform unlocks the power of observation at scale, giving leaders visibility and workers a voice in driving quality, efficiency, and safety improvements.Īmazon Redshift is a fully managed data warehouse service that tens of thousands of customers use to manage analytics at scale. Safet圜ulture is a global technology company that puts the power of continuous improvement into everyone’s hands. This post is co-written by Anish Moorjani, Data Engineer at Safet圜ulture.
0 Comments
Leave a Reply. |