The Published Research

If you need further convincing, a cursory examination of published research provides startling insight. Some studies provide insight into why Snowflake Data Cloud is important.

International Data Corporation (IDC) ( www.idc.com/getdoc.jsp?containerId=prUS46286020 ) stated that in 2020, more than 59 ZB (59 trillion gigabytes) of data was created, captured, copied, and consumed. Most of these are copies of data (or worse, copies of copies), with the ratio of replicated data (copied and consumed) to unique data (created and captured) being approximately 9:1, trending toward less unique and more replicated data. Soon, the number of replicated copies is expected to be 10:1, and this growth forecast is set to continue through 2024 with a five-year compound annual growth rate (CAGR) of 26%.

The Economist Intelligence Unit, in its Snowflake-sponsored report, “Data Evolution in the Cloud: The Lynchpin of Competitive Advantage” ( www.snowflake.com/economist-research/data-evolution-in-the-cloud-the-lynchpin-of-competitive-advantage/ ), states that 87% of 914 executives surveyed agree that data is the most important differentiator in the business landscape today, where 50% of respondents frequently share data with third parties to drive innovation in products and services. But in contrast, 64% admit their organization struggles to integrate data from varied sources. The 50% of organizations that frequently share data must be doing something right . How are they leveraging their infrastructure and tooling to share their data? Where did they draw their inspiration from? We return to these themes later, but an immediate conclusion we can draw is that someone, or some groups, must have changed their thinking to embrace a new paradigm and found themselves in an environment where they could try something new. Likewise, the 64% who struggle to integrate data from varied sources are at least acknowledging they have a problem and hopefully looking for solutions. If your organization is in the 64%, keep reading because out of the same sample, 83% agree their organizations will routinely use AI to process data in the next five years. Without integrating more disparate data, less complete, potentially inaccurate results will arise .

The Global Perspective

Our organizations are a microcosm of the global perspective . Whether we know or acknowledge our position in the digital economy, our organizations are subject to external market forces that force change and adaptation to new working methods. We need a global perspective to underpin our organization’s success, and the Snowflake Data Cloud does just that. A single platform is delivering tooling out of the box enabling frictionless, seamless, and rapid data integration with the opportunity to monetize our digital assets.

In every organization, whether immediately obvious or not, there are data silos. Those repositories which we think may hold value but are inaccessible; data sets we are not allowed to see or use, and those data sets hidden in plain sight, the obvious and the not so obvious. Data silos are discussed in Chapter 2, where we start to challenge received wisdom in tightly segregating our data. Continuing with our theme of asking why, there are typically two reasons why we don’t get answers to questions. The first is because knowledge is power, and the second is because the answer is unknown.

The volume, velocity, and variety of data are ever-increasing. I touched upon these earlier and later discussed the same themes in Chapter 10. As a sneak preview, we also explain how to address some of the less obvious data silos, and for the avoidance of doubt, this was my favorite chapter to write, the one with the most scope to unlock the potential of our organization’s data, bear with me, as there is much to understand before we get there, oh alright, you skipped straight there.

Unavoidable in today’s increasingly complex, international, and multi-jurisdictional organizations, data governance policies, processes, and procedures oversee and increasingly control every aspect of our organization’s software development lifecycle (SDLC) . How policies and procedures impact what, where, and how we can use our data are covered in Chapter 11. It’s not all doom and gloom, there are very good reasons for data governance, and we lift the lid on some hidden reasons why we must comply.

We all know software is increasingly more complex. Anyone using a desktop PC without a deep knowledge of the underlying operating system and tools to keep both process and memory hogs at bay looks to upgrade hardware far more often than might otherwise be needed. Subtle software interactions and edge cases cause unforeseen scenarios, and we don’t always have the tools to identify and remediate issues. Chapter 13 offers some ideas and concepts to remediate these challenges; not an easy subject to resolve, but we can do something to help ourselves.

Having read this, I must be clear: organizations cannot fund constant hardware, storage , network, and associated infrastructure upgrades. Increasingly costs are associated with software patching, products to help us manage our infrastructure, and improvements to our security posture dealing with both current (what we know) threats and emerging threats (those we don’t yet know about). All of which is without considering up-skilling and increasing our staffing levels while the regulatory environment tightens and imposes greater controls and reporting obligations.

Every day, we focus on what we need to do to keep the lights on. Our operational staff faces more demand from ever-growing data volumes but with the same people and infrastructure, constantly reacting and firefighting to the same challenges daily. If we are fortunate, we have robust documentation explaining how we do our day-to-day jobs. But documentation only remains relevant if we devote sufficient skill, time, and energy to its maintenance. We all know the first thing to suffer in highly stretched teams is the documentation, offering a litmus test in and of itself.

If we are ever to break this never-ending cycle, we need to think differently, challenge the status quo, and quickly pivot to focus on why. Embracing automation, low-code tooling, reducing dependencies upon key people, and implementing appropriate controls and alerting mechanisms reduce the daily noise and allow us time to understand why.

We have the means at our fingertips to resolve some of these seemingly intractable problems. But we must start now because the longer we delay, the big issues compound, become harder to understand, and head toward an inflection point beyond which we will never recover. A friend described this as a diode (a discrete component that only conducts current in one direction) at some unknowable point on our upward trajectory of increasing complexity. Once passed, we can never go back because the complexity, inertia, complacency, and all other reasons make it far too hard, and the only way from this point onward is to start again.

Hardware and Software

The most obvious difference between cloud and on-premise (on-prem) is hardware provisioning, with software provisioning a close second. Most organizations have two or more data centers filled with hardware with high bandwidth network connections facilitating both application failover and disaster recovery in the event of server or, worst case, data center outage.

Hardware, air conditioning, physical security, staff, UPS, halon protection, miles of networking cables, switches, routers, racking, test equipment, servicing, repair, renew and replace. The list is almost endless, and each item has a fixed cost; ever depreciating, the upgrade cycle is endless and increasingly demanding.

With the cloud, organization-owned data centers are no longer required. Cloud service providers (CSPs) provision everything, available on demand, 24×7×365, infinitely scalable, fault tolerant, secure, maintained, and patched. While we pay for service provision, we do not have the headache of managing the infrastructure. Moreover, apart from storage costs, we (mostly) only pay for what we consume. Storage costs are very low. Snowflake simply passes through storage charges, and depending upon location, at the time of writing, costs vary from $23 to $24.50 per terabyte.

Figure 1-1 shows the relative costs of retaining fixed on-prem infrastructure compared to moving to cloud-based elastic provisioning.

Performance

On-Prem dedicated hardware is always limited to the physical machines, CPU, memory, and disk. Regardless of whether the servers are in use, we continue to pay for their provision, yet servers are only utilized for about 5% of the time. The same is true of network bandwidth; every organization has a limit to the amount of data carried on their internal network with utilization peaks and troughs.

With cloud-based applications, instant performance scaling and per-second costing provide performance when required, for only as long as required, with predictable costs, and then scale back. We pay for what we consume, not what we provision, and have both elastic performance and storage at our fingertips. We don’t incur network bandwidth charges for cloud-to-cloud connectivity as data transits the public Internet; however, we may pay cloud provider egress charges.

When hardware failures occur, they are invisible to us. We benefit from always-on, infinite compute, infinite storage elastically provisioned on demand. We need to pay attention to authentication and network transport layer security, all discussed later .

Staffing

The highest costs to an organization are its people. With on-premise hardware, we incur security guards, physical installation costs, specialists to perform operating system and application patching, servicing, installation costs, PATS testing, and management to run teams.

With the cloud , costs are significantly reduced, but not to zero. We need more cloud security specialists instead. However, the cost savings are significant when compared to on-prem equivalents.

The COVID-19 pandemic has shown that our organizations can operate with at least the same or higher efficiency than otherwise thought possible, with reduced office capacity needed to support staff that can work remotely and connects to cloud services natively.

Control

Each organization has exclusive control of hardware and data in its own domain with on-prem implementations. Total control is physically expressed in the infrastructure provisioned.

Cloud implementations have a different perspective. Absolute control is only as good as the security posture implemented, and in a shared environment, such as may be provisioned for managed services. Across our estate, both on-prem and cloud, the security boundaries must remain impenetrable, and this is where our cybersecurity team is most important.

We also realize benefits from seamless patching and product updates that are applied “behind the scenes” are invisible to us. Vendors release updates without downtime, or outages lead to higher uptime and availability. While we give up some administrative oversight, we benefit from integrated, joined-up automated delivery throughout all our environments.

Another benefit is the feature-rich cloud environment offering built-in tooling and virtual hosting for third-party tools and applications. Driven by the need to increase market share, cloud providers continually improve their tools and integration pathways, enabling developers to innovate, reduce time to market, and find solutions to difficult business problems.

Data Security

Data security is of paramount importance to cloud-based solutions. Snowflake has a comprehensive suite of security controls—an interesting subject, which is discussed later.

Typically, each business owner determines the business data classification of each attribute in a data set with specific protections mandated for each business data classification. For on-prem data which never leaves the internal network boundaries, security considerations do not arise to the same extent as they do for cloud, where data in transit is expected to be encrypted at the connection level (conform to TLS1.2, for example) or be encrypted at rest (Bring Your Own Key, SHA256, for example). We will discuss data security in more detail, but for now, it is enough to note the distinction between on-prem and cloud-hosted solutions.

Compliance

For on-premise solutions, most regulators are content with knowing data is held in their own jurisdiction. But for cloud-based solutions, some regulators apply additional scrutiny on a per data set basis before allowing data onto a cloud platform. Additional regulation is not identical between regulators; therefore, an individual approach must be adopted to satisfy each regulator’s appetite.

Additional governance may apply before approval is granted to move artifacts to the cloud, particularly where data is hosted in a location outside of the governing authority’s jurisdiction. This is covered later, though it is important to note data governance and controls follow the data, not the cloud, ensuring we protect the data while maintaining the highest levels of security and compliance .

Data Volumes

You have seen from the research I identified that the typical number of data copies to original data is currently around 9:1 and expected to increase to 10:1 soon. These alarming figures hide many underlying issues. Replicated data sets as pure copies (or copies of copies) result in the inability to reconcile across data sets due to different extract times and SQL predicates.

Unreconcilable data leads to management mistrust and a desire to build siloed capabilities as the central system must be wrong. You see the problem emerging, particularly where manual out-of-cycle adjustments are made, further calling data provenance and authenticity into question.

Other direct consequences arise with increased data storage requirements and higher network bandwidth utilization as we shuffle data to and fro while requiring more powerful CPUs and memory to process data sets. There are others, but you get the idea.

With alarming regularity, new data formats, types, and challenges appear. And the velocity, volume, and variety of information are increasing exponentially. With the Snowflake Data Cloud, many of these challenges disappear. I will show you the how and the what.

Business Lifecycle

Evidence for Snowflake being at the bottom of the growth stage is not hard to find. Look at Figure 1-3, which illustrates the standard product lifecycle showing the various stages every product goes through. Supporting my assertion, financial statements are at www.macrotrends.net/stocks/charts/SNOW/snowflake/financial-statements . Snowflake is a Unicorn by any standard; big institutional investors at IPO include Berkshire Hathaway and Salesforce.

The six-year pre-IPO period might be regarded as the introduction stage in the diagram with low sales, high costs, and low profits. The growth stage started post seed funding with increased sales, reduced costs, and profits. Snowflake is certainly not into the maturity phase; the sheer volume of new features and product enhancements demonstrate Snowflake is ever-growing in capability and scope.

Snowflake published the numbers shown in Figure 1-4. Draw your own comparisons. The latest figures are at https://investors.snowflake.com/overview/default.aspx .

Diffusion of Innovations

Snowflake’s innovative design separating storage from computing is a paradigm shift from old thinking, representing a clean break from the past. Snowflake architecture is explained in Chapter 3.

When presented with new ideas, we are often challenged to accept new thinking. The Diffusion of Innovations theory provides some answers by breaking down the population into five distinct segments, each with its own propensity to adopt or resist a specific innovation. Figure 1-5 presents the bell curve corresponding to adoption or resistance to innovation. I leave it to you to determine where your organization falls.

Snowflake adoption is no different. In Figure 1-5, Snowflake is somewhere to the left of the early adopter’s profile. This is where those of us looking to make a strategic leap forward find ourselves, an opportunity to embrace a rapidly maturing product, enabling the Snowflake Data Cloud and providing another opportunity to monetize our data.

On the left, the innovators, 2.5% of the population, have already adopted Snowflake, endured the challenges of dealing with an immature product, unknowingly conducted user acceptance testing, and suffered sleepless nights creating workarounds for previously broken features—the list goes on.

The next 13.5% are early adopters. If you have read this far, you are likely one of them. We take advantage of the hard work the innovators have put into Snowflake and seize the day, grasp what is before us, and move forward.

At our fingertips is Snowflake, a mature product of which the next 34%—the early majority—are just becoming aware of. It offers a window of opportunity to create the greatest commercial advantage and develop into new markets to monetize our data.

Let’s pause our discussion on Diffusion of Innovations here; few want to be in the late majority, and less still in the laggards, but this is where 50% of the population find themselves, whether they know it or not.

Better to be early than just on time, never to be late.

Future State Enabled

Our world is often changing in unpredictable ways. For example, face recognition is now commonplace but unthinkable ten years ago with a consequential rise not only in volume but the velocity of data too. By the time you read this book, Snowflake will have released support for unstructured data, a subject for later in this book, which mainstream databases struggle to deal with.

And new challenges arise. If you want to extract billing information from a photographed invoice, Snowflake has the tools to do just that. If you want to save paragraphs from a Microsoft Word document into a structured data format, Snowflake can do that too. I will show you how to do both in Chapter 9.

This represents microcosms of data sets requiring particular tools and techniques to process, and Snowflake has the answers. New features arrive all the time. As I wrote this chapter, object tagging became available; see Chapter 11 for details.

Emerging themes on future data growth (all discussed later) such as the Internet of Things (IoT) generating sensor data, imaging, semi-structured records, and more, big data and analytics showcasing how we can make sense of these huge volumes of data, machine learning decision-making affecting our everyday lives.

First-Mover Advantage

Combining the Diffusion of Innovations theory with the business lifecycle gives us the knowledge to understand we have a window of opportunity and first-mover advantage. We can identify and create new markets, be the first entrants into the market, develop brand recognition and loyalty, gain competitive advantage, and consequently, a marketer’s dream.

But time is not on our side. Your organization’s competitors have already done their research and begun executing their strategies. Some are reading this book just as you are.

Your Journey

Your journey to the cloud is just that—a journey with some lessons to learn. Some are outside of the scope of this book, but many are illustrated here.

Not only are there significant cost savings, performance benefits, and flexibility to be gained, but also wider security considerations to address. And for those who enjoy technology, Snowflake has plenty to offer with an ever-expanding feature set delivered faster than most of us can assimilate change.

Global Data Mesh

The Snowflake infrastructure underpinning the Snowflake Data Cloud is its global data mesh . Snowflake addresses the challenges of building data platforms in unique ways which readily facilitate the delivery of robust data architectures.

With some prior thought on how RBAC enables data ownership and access, adherence to published standards for data ownership, and clarity of thought on logically and physically separating data sets, delivery of the Snowflake Data Cloud is readily achievable. But only if we do things the right way from the outset, or more probable for organizations with an existing data lake, develop a plan to bring structure and governance together along with clear direction to remediate the past.

Engagement Criteria

To successfully create our contribution to the Snowflake Data Cloud, we must adopt a domain-based approach to data ownership. In practice, we must know “who owns what” and enforce data ownership as a toll-gate before allowing data onto our platform. Not as easy as one might think, our organizations are becoming more tightly regulated, and while attractive to “just throw the data in,” any audit or governance function wants to know “who owns what.” Better to set the standard at the outset rather than suffer the consequences later.

With multiple data domains in our data cloud, new opportunities exist for cross-domain reporting. Subject to correctly segregating our data using RBAC, we simply need to grant roles to users to enable data usage. Simple, right? No, not simple in practice because we may not want to grant access to all data in the domain, so we must restrict data to the minimal subset required to satisfy the business usage. And Snowflake has the tools to do this. Finer-grained RBAC, row-level security, object tagging, and the creation of custom views are some techniques we use.

As I will unpack shortly, data discovery and self-service are central to Snowflake Data Cloud capability.

Secure Direct Data Share

Secure Direct Data Share is a built-in, highly secure core Snowflake capability reliant upon cloud storage to share data with other Snowflake customers; that is, consumers both inside and outside your immediate organization. We unpack Secure Direct Data Share in Chapter 14 with hands-on examples.

With Secure Direct Data Share, the data owners decide the data sets to share, and technologists provide the implementation. At all times, full control is maintained over which external customers have read-only access. Addressing “who can see what,” Snowflake is extending its monitoring capabilities to provide metrics on consumption. Chapter 14 provides a template.

Data is maintained in real time. As we update the source object data, our customers see the same changes in real time as they occur. Data sharing occurs behind the scenes. Once we declare the objects and entitle customers to access the share, our clients consume it; it is as simple as that. No more SFTP, data dumps, process management, and other cumbersome data interchange scheduling and tooling.

Costs differ according to CSP and actions performed, noting as Snowflake product capabilities evolve, the costing model may also change. For further up-to-date information, see the documentation at https://docs.snowflake.com/en/user-guide/billing-data-transfer.html#understanding-snowflake-data-transfer-billing .

Suitable for all applications, Secure Direct Data Share explicitly controls which customers have access to shared data and are a great way to securely and safely monetize your data while retaining full control.

Many organizations suffer from incomplete, incorrect, divergent reference data where inconsistencies lead to rejected records, unmatched attributes lead to missing content, and over time, slowly lead to mistrust of the system.

If we can address this one seemingly simple (it is not simple) issue across the many hundreds or thousands of applications in our organizations by sourcing our reference data once and distributing it across all our applications seamlessly, we would soon enjoy the benefits of corrected, conformed data which can more easily be joined as many more attribute values will match.

And there are more opportunities; the story just gets better.

Data Exchange

Utilizing Snowflake’s built-in data sharing capability, Data Exchange is the organization’s internal place where data is published to consumers, that is, people and organizations both in and outside the publishing organization boundary. Data Exchange is also where organizations and individuals discover available data sets. We unpack Data Exchange with worked hands-on examples in Chapter 14.

Data sets controlled by business owners are published for consumption by others, either by approval-based subscription or freely available to all. Truly the beginnings of data democratization, data publishing and data discovery allow everyone to fully participate. They are both able and entitled to access data sets. Moreover, the silos begin to crumble, allowing data discovery, enrichments, and utilization in previously unthinkable ways.

Marking a significant step forward in an organization’s capability to seamlessly interact, Data Exchange quickly allows subscribers to access their entitled data sets. Time to market is slashed by hours, or at most days, to implement data integration between organizations—no more SFTP, authentication, firewalls, or handshaking, and previous impediments disappear.

Central to the concept of data sharing is security. With Data Exchange, the data owner retains complete control of their own data sets and can curate, publish, manage, and remove entitlement at will. Naturally, one year retained immutable full audit trail is available; this is discussed in more detail later.

Snowflake Marketplace

Organizations use Replication and Secure Direct Data Sharing capabilities to create listings on the Snowflake Marketplace, the single Internet location for seamless data interchange.

Snowflake Marketplace is available globally to all Snowflake accounts (except the VPS account, a special case) across all cloud providers.

Because Snowflake Marketplace uses Secure Data Sharing, capability, security, and real-time features remain the same. The integration pattern differs as replication, not share, is used as the integration mechanism.

However, there is a hand-shake between subscriber and provider as information may be required to subscribe, and data sets may require pre-filtering, resulting in custom data presentation.

Once complete, the source database is replicated, becoming available to import into the subscribing account. Both options require local configuration to import the database, which becomes accessible similarly to every other database in the account.