Today it feels like everything is moving to the cloud. Even large enterprise data warehouses that many people thought would never leave the on-premise data centers are moving to the cloud to take advantage of today’s cloud technologies. In this blog post, I will guide you through the benefits and challenges of moving your data warehouse to the could.
Scalability & Ephemeral Infrastructure
The first benefit of data warehousing in the cloud in the scalability of the cloud architecture. The cloud offers you the ability to quickly and easily increase and decrease disk space, CPU, and memory resources to accommodate spikes in usage. Typical data warehouses run batch processes. These batches usually run daily but could run every few hours to every couple of minutes. For those warehouses that only run a nightly batch process, the CPU and memory sit unused as they wait for the process to start. If these servers are in your data center, you are paying for servers to sit idle and run near capacity for a few hours each night. When large processed are required, like end of quarter or end of year processing, you may max out your servers causing delayed loading times. If you use the cloud, you can easily add additional ephemeral CPU and memory to account for the extra load and complete the process in the expected timeframe. Once the processing is complete, you can release those servers back to the cloud provider, thereby paying only for the minimum CPU and memory needed until the next batch is executed. This elasticity of infrastructure is what makes the cloud so compelling to most companies.
Ghost Expenses
The challenge to scalability and ephemeral infrastructure is the cost. Although cloud providers tout their ability to save their customers money, most companies never realize those cost savings because they don’t maintain their cloud services. We have been conditioned to misuse servers (CPU and memory) and disk space because we have already paid for the server and put it into our data center. Once the server is installed and it is up and running, we often forget about it until we need it. When this happens in the cloud, the costs can be painful. Far too often, I have heard potential customers ask, “Why is my cloud bill so high?” The issue is most often because they spun up servers to address specific challenges (development, testing, temporary scaling, etc.) and have left the servers up and running, even though they are no longer in use anymore. To avoid this “bill shock,” you need to add alerts and task a person or group to be responsible for keeping the cloud costs in line with the projected budget. This watchdog group will often pay for itself in cost savings.
Transformation of the Traditional Data Warehouse
The next benefit of cloud data warehousing is the ability to expand and transform all of your data. Today’s data warehouses are more than just the traditional Inmon or Kimball models of the 1990s and early 2000s. These data warehouses now include structured, unstructured, and semi-structured data stored in databases using relational (SQL), NoSQL, multimodal databases, and/or using polyglot persistence and data lakes using object storage. There are some fantastic tools to help you leverage your traditional data warehouse with today’s big data explosion. Vendors like Snowflake, Amazon Redshift, Microsoft Azure, and Google BigQuery are modernizing today’s data warehouses into real-time reporting and decision-making engines.
Vendor Lock
Another challenge is the vendor lock. You have to be very careful when choosing a cloud provider. Some vendors make getting data into their cloud easy and sometimes even free, but getting data out of that cloud to move back on-prem or to another cloud provider is difficult and often very expensive. Not all could providers are equal. Make sure you do your homework and research the current offerings of each cloud provider against your current and future business needs. Make sure to ensure you are choosing the correct cloud provider for both your short and long-term needs.
Security
The security of today’s cloud providers easily supersedes that of your in-house IT team. You may have an amazing and gifted IT security staff, but I would bet on the cloud vendor security every time. They employ thousands of highly talented security engineers implementing security patches and researching and resolving potential issues every moment of every day for the cloud providers. Their reputation and overall success depend on the security of the services they are providing. It is highly unlikely that your in-house IT team can achieve the level of protection the cloud providers offer.
These are a few of the benefits and challenges that I feel are the most compelling as you move your data warehouse to the could. Let me know your thoughts in the comments below.