BigBear.ai
  • Home
  • Industries
    • Academia
    • Government
    • Healthcare
    • Manufacturing
  • Solutions
    • Cyber
    • Data Analytics
    • Enterprise Planning and Logistics
    • Intelligent Automation
    • Modeling Solutions
    • Professional Services
  • Company
    • About
    • Investor Relations
    • Partners
    • Team
  • Careers
    • Benefits
    • Culture
    • Explore Jobs
    • Military and Veterans
    • Applicant Login
    • Employee Login
  • Resources
    • Blog
    • Events
    • Newsroom
    • Resource Library
    • Online Store
  • Contact
Search

Home Data Warehousing The Benefits and Challenges of Cloud Data Warehouses

Blog

The Benefits and Challenges of Cloud Data Warehouses

Jim McHugh
July 1, 2020
  • Share
  • Share

Today it feels like everything is moving to the cloud. Even large enterprise data warehouses that many people thought would never leave the on-premise data centers are moving to the cloud to take advantage of today’s cloud technologies. In this blog post, I will guide you through the benefits and challenges of moving your data warehouse to the could.

Scalability & Ephemeral Infrastructure

The first benefit of data warehousing in the cloud in the scalability of the cloud architecture. The cloud offers you the ability to quickly and easily increase and decrease disk space, CPU, and memory resources to accommodate spikes in usage. Typical data warehouses run batch processes. These batches usually run daily but could run every few hours to every couple of minutes. For those warehouses that only run a nightly batch process, the CPU and memory sit unused as they wait for the process to start. If these servers are in your data center, you are paying for servers to sit idle and run near capacity for a few hours each night. When large processed are required, like end of quarter or end of year processing, you may max out your servers causing delayed loading times. If you use the cloud, you can easily add additional ephemeral CPU and memory to account for the extra load and complete the process in the expected timeframe. Once the processing is complete, you can release those servers back to the cloud provider, thereby paying only for the minimum CPU and memory needed until the next batch is executed. This elasticity of infrastructure is what makes the cloud so compelling to most companies.

Ghost Expenses

The challenge to scalability and ephemeral infrastructure is the cost. Although cloud providers tout their ability to save their customers money, most companies never realize those cost savings because they don’t maintain their cloud services. We have been conditioned to misuse servers (CPU and memory) and disk space because we have already paid for the server and put it into our data center. Once the server is installed and it is up and running, we often forget about it until we need it. When this happens in the cloud, the costs can be painful. Far too often, I have heard potential customers ask, “Why is my cloud bill so high?” The issue is most often because they spun up servers to address specific challenges (development, testing, temporary scaling, etc.) and have left the servers up and running, even though they are no longer in use anymore. To avoid this “bill shock,” you need to add alerts and task a person or group to be responsible for keeping the cloud costs in line with the projected budget. This watchdog group will often pay for itself in cost savings.

Transformation of the Traditional Data Warehouse

The next benefit of cloud data warehousing is the ability to expand and transform all of your data. Today’s data warehouses are more than just the traditional Inmon or Kimball models of the 1990s and early 2000s. These data warehouses now include structured, unstructured, and semi-structured data stored in databases using relational (SQL), NoSQL, multimodal databases, and/or using polyglot persistence and data lakes using object storage. There are some fantastic tools to help you leverage your traditional data warehouse with today’s big data explosion. Vendors like Snowflake, Amazon Redshift, Microsoft Azure, and Google BigQuery are modernizing today’s data warehouses into real-time reporting and decision-making engines.

Vendor Lock

Another challenge is the vendor lock. You have to be very careful when choosing a cloud provider. Some vendors make getting data into their cloud easy and sometimes even free, but getting data out of that cloud to move back on-prem or to another cloud provider is difficult and often very expensive. Not all could providers are equal. Make sure you do your homework and research the current offerings of each cloud provider against your current and future business needs. Make sure to ensure you are choosing the correct cloud provider for both your short and long-term needs.

Security

The security of today’s cloud providers easily supersedes that of your in-house IT team. You may have an amazing and gifted IT security staff, but I would bet on the cloud vendor security every time. They employ thousands of highly talented security engineers implementing security patches and researching and resolving potential issues every moment of every day for the cloud providers. Their reputation and overall success depend on the security of the services they are providing. It is highly unlikely that your in-house IT team can achieve the level of protection the cloud providers offer.

These are a few of the benefits and challenges that I feel are the most compelling as you move your data warehouse to the could. Let me know your thoughts in the comments below.

Posted in Data Warehousing.
BigBear.ai
  • Home
  • Industries
  • Solutions
  • Company
  • Careers
  • Blog
  • Investor Relations
  • Contact
  • Twitter
  • Facebook
  • Linkedin
  • Google My business for BigBear.ai
1-410-312-0885
[email protected]
  • Privacy Policy
  • Terms of Use
  • Accessibility
  • Site Map
© BigBear.ai 2023
We value your privacy
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Privacy Policy | Do not sell my personal information
AcceptCookie Settings
Manage Consent

Cookies Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
JSESSIONIDsessionThe JSESSIONID cookie is used by New Relic to store a session identifier so that New Relic can monitor session counts for an application.
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
CookieDurationDescription
__atuvc1 year 1 monthAddThis sets this cookie to ensure that the updated count is seen when one shares a page and returns to it, before the share count cache is updated.
__atuvs30 minutesAddThis sets this cookie to ensure that the updated count is seen when one shares a page and returns to it, before the share count cache is updated.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
_ga2 yearsThe _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_ga_NK4L4Q320Q2 yearsThis cookie is installed by Google Analytics.
_gat_gtag_UA_163894009_21 minuteSet by Google to distinguish users.
_gid1 dayInstalled by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.
at-randneverAddThis sets this cookie to track page visits, sources of traffic and share counts.
CONSENT2 yearsYouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
uvc1 year 1 monthSet by addthis.com to determine the usage of addthis.com service.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
CookieDurationDescription
f5avraaaaaaaaaaaaaaaa_session_sessionbusinesswire.com cookie
loc1 year 1 monthAddThis sets this geolocation cookie to help understand the location of users who share the information.
VISITOR_INFO1_LIVE5 months 27 daysA cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSCsessionYSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devicesneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-idneverYouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
Save & Accept