Why Data Share?

We mostly work in environments where data silos predominate, and legacy thinking pervades our conversations, stifling innovation and preventing true data democratization. Consequently, we may only have access to curated data sets via limited access paths or, worse still, partial data sets from which to derive meaningful answers. Without wishing to repeat Chapter 2, we are largely stuck with the legacy paradigm born in the 1980s of FTP/SFTP, API, ETL/ELT, and most recently, cloud storage. Yes, it is surprising to see the latest emergent technology considered a legacy. But the stark reality is that many organizations have wrapped up cloud storage in legacy constructs because these are both familiar and well understood.

I recognize the irony. Most of this book has been concerned with ingesting data via legacy techniques. But the reality on the ground (at the time of writing) is that most organizations are not yet willing to move away from their comfort zone and embrace the new data-sharing paradigm.

I hope you are among those willing to consider a trial and then adopt what is considered the future of data interchange . Snowflake has envisioned a new paradigm, Secure Direct Data Share, where service on demand is seamless and transparent, and data access is available out of the box. Secure Direct Data Share is a built-in, highly secure, core Snowflake capability reliant on cloud storage to share data with internal business colleagues, partners, and Snowflake customers.

Suitable for all applications , Secure Direct Data Share explicitly controls customers’ access to shared data. It is a great way to securely and safely monetize your data while always retaining full control.

Data sharing removes barriers, regardless of cloud, region, workload, or organizational domain, and gives instant access through a single copy of data. The number of data copies is set to rise from 9:1 to 10:1. Reusing a single copy of data should cause everyone to sit up and pay attention. Data sharing is a true game-changer.

Data sharing enables business continuity by eliminating disruptions and delivering a seamless customer experience, with no more processing chains or single points of failure from source to destination. Utilizing Snowflake’s global data mesh guarantees consistent, reliable delivery with full security embedded.

Incorporating object tagging gains the distributed benefit of our organization’s perspective on data categorization applied consistently across all uses of the subject data. Cross-domain governance is more easily managed centrally and consistently rather than distributed or fragmented across geographically disparate teams; one consistent implementation pattern and multiple aspects satisfy all use cases.

How to Data Share

Shares are a first-order object whose declaration and management are typically performed using the ACCOUNTADMIN role, the use of which should be tightly restricted. Snowflake provides instructions on devolving entitlement to lower privileged roles at https://docs.snowflake.com/en/user-guide/security-access-privileges-shares.html#enabling-non-accountadmin-roles-to-perform-data-sharing-tasks . In our view, devolving entitlement should not be done lightly as the law of unintended consequences quickly follows https://en.wikipedia.org/wiki/Unintended_consequences . I recommend that share entitlement is not devolved.

To identify shares.

Replacing <your_consumer_account> with a consumer account in the same CSP and region.

To identify your Snowflake account, select current_account();.

We should see the new table csv_test_2 in our imported share. Naturally, all other share consumers can also see the new table.

Further information on consuming from shares is at https://docs.snowflake.com/en/user-guide/data-share-consumers.html#data-consumers .

Data Share Summary

Consumers may create a single database per share. An inbound database must be created before the share can be queried, and created databases can be renamed and dropped. As stated, roles and entitlement are not currently carried forward with the share. They must be rebuilt in the recipient account. However, row-level security, column-level security, external tokenization, and data masking policies are honored.

Managing the share is trivial. Adding and removing objects from the share does not require much effort, and all share consumers also receive the same shared objects.

Most importantly, no data is copied. Metadata alone enables the secure sharing of data in underlying storage. Since no data is copied from the provider account, the Snowflake consumer account is not charged for storage.

Only the most recent micro-partitions are available in the share. The Time Travel feature is not supported for objects imported via a share, although it is supported for the original objects. Not all schema object types can be shared, and for those shareable object types, their secure counterparts must be used to preserve strict control of access to data. Further information is at https://docs.snowflake.com/en/user-guide/data-sharing-provider.html#working-with-shares .

Reader Accounts

Reader accounts are provisioned for organizations who are not yet Snowflake customers enabling data query but not upload, update, or delete of data. A key advantage of reader accounts is reduced data integration cost with direct access to live, ready-to-query data.

For Snowflake customers, the compute cost is paid for by the consuming account. For non-Snowflake consumers using a reader account, consumption is paid for by the provider.

We do not provide an in-depth study of reader accounts but instead refer to Snowflake documentation at https://docs.snowflake.com/en/user-guide/data-sharing-reader-create.html#managing-reader-accounts .

Tracking Usage

Before investigating data access, we must be mindful of latency when accessing Snowflake’s Account Usage views. For query_history, latency can be up to 45 minutes, and for access_history, up to 3 hours. Latency is not a major consideration for tracking usage as the information is not of critical importance. More information is at https://docs.snowflake.com/en/sql-reference/account-usage.html#account-usage-views .

Data Sharing Usage

Snowflake is developing a suite of monitoring views where we can track shared data usage. Current monitoring capability is quite limited and does not allow identifying the rows accessed by any specific query. The latency may be up to two days.

Please also refer to Snowflake documentation at https://docs.snowflake.com/en/sql-reference/data-sharing-usage.html .

Both classic and Snowsight browsers provide access to the Data Sharing Usage views. The classic view is shown in Figure 14-6.

A screenshot of a new worksheet with a search bar to find database objects. The data sharing usage is highlighted under the snowflake option.

I recommend periodic checks of Data Sharing Usage views. The following are sample queries.

USE ROLE      test_object_role;

USE WAREHOUSE TEST_WH;

USE DATABASE  snowflake;

USE SCHEMA    snowflake.data_sharing_usage;

We expect zero rows unless shares have been consumed.

SELECT *

FROM   snowflake.data_sharing_usage.listing_consumption_daily;

SELECT *

FROM   snowflake.data_sharing_usage.listing_events_daily;

Intriguingly, during testing, history has been retained for prior implementation of Data Exchange .

SELECT *

FROM   snowflake.data_sharing_usage.listing_telemetry_daily;

Centralized Cost Monitoring

This section explores the potential for implementing a common cost monitoring pattern using data share and local tables.

Conceptual Model

Building upon the code developed in Chapter 6, let’s propose an extension to (almost) plug-and-play capability. Figure 14-7 illustrates a conceptual model of how a centralized cost monitoring share-based implementation may operate.

A schematic of the A W S cloud region that consists of a provider account A, provider account B, and a consumer account. Provider accounts contain databases with monitor schema and share with U D T F. A consumer account contains database a, and database b with import schema and U D T F and reporting database with reporting schema.

Figure 14-7 shows two provider accounts (A & B) with a common suite of baselined reporting components deployed in a monitor schema. Each view overlays the provider Account Usage store or information schema with tabular SQL user-defined functions (UDTFs) , one per view. Each provider account shares its UDTFs, and the consumer account imports each share into a separate database, one per provider.

Taking advantage of UDTF capability to return a result set, scheduling via a task in the consumer account calls each UDTF inserting data into its own tables, one for each UDTF.

In the reporting database, result sets from each provider account reporting component are joined using views for common reporting.

Replication

Replication has been discussed throughout the book. Now, let’s work through setting up replication.

This example uses two accounts: primary and disaster recovery (DR). It replicates a database from Primary to DR. Figure 14-8 illustrates the scenario.

A schematic of the A W S E U Dublin region and E U London region. The primary account of the Dublin region is replicated to the D R account of the London region.

While we can configure replication in the Snowflake user interface, we contend deeper understanding is gained from working through command line invocation recognizing any automation we develop works through scripts and not using the interface. In other words, our replication strategy depends on more than a single user interface-driven strategy, probably with complex dependencies including failover, failback, and client redirect.

Not every object type is currently available for replication. Please refer to the documentation at https://docs.snowflake.com/en/user-guide/database-replication-intro.html#replicated-database-objects . Replication has further limitations. For example, RBAC is not replicated; therefore, the receiving account must implement RBAC locally; databases with external tables cannot be replicated; databases created from shares cannot be replicated.

This example illustrates simple database replication in support of data shares and is not a full treatise on failover and failback, which falls beyond the scope of this book. Also, note there are cost implications to replicating data across accounts, CSPs, and regions.

We need two (or more) accounts enabled for replication. Your organization may prefer each account to be enabled individually, in which case a Snowflake support ticket should be raised. Response times are usually very short. Otherwise, Snowflake support can assign accounts the ORGADMIN role, as documented at https://docs.snowflake.com/en/user-guide/organizations-manage-accounts.html#enabling-accounts-for-replication .

You should see <your_database> in the list of available databases for import. Note that is_primary is “true”.

And refreshing the left-hand pane of our browser where our new database is listed.

With our database replicated, we have two remaining tasks: create and apply a security model and then share desired objects, which I leave you to complete.

Request Data Exchange

Enabling Data Exchange can take 48 hours.

Snowsight

This section moves from the classic console to use Snowsight , the future-state Snowflake console, because there is no approved command-line approach to implementing Data Exchange features.

If using AWS PrivateLink or Azure Private Link, you may need to create a CNAME record as described in the documentation at https://other-docs.snowflake.com/en/marketplace/intro.html#introduction-to-the-snowflake-data-marketplace .

Figure 14-9 highlights the button to click.

A screenshot of the highlighted snowsight icon on the bar of a tab.

A new browser window and login dialog appear alongside our existing classic console. Log in, accept the prompt to import tabs, and take a moment to familiarize yourself with the new look and feel.

On the day of writing, Snowflake released Snowsight as generally available, prompting migration to the new user interface with a button on the left-hand side of the browser to revert to the classic console. You may need to click the Home button before progressing to investigate Data Exchange , as shown in Figure 14-10.

A screenshot of a worksheet tab, where the home button is highlighted.

Accessing Data Exchange

When ready, change your role to ACCOUNTADMIN by clicking your login name. Then click Home ➤ Data ➤ Private Sharing, as shown in Figure 14-11.

A screenshot of the admin bar with the data option highlighted. The private sharing button is highlighted under the data option.

We are now presented with all private data exchanges shared with our account, as shown in Figure 14-12. Two shares provided by Snowflake overlay the imported Snowflake databases created when our account is provisioned. The corresponding data exchange entry is also removed if a sample database has been deleted.

A screenshot of the data exchange tab where the option of manage exchanges is highlighted on the top.

I leave the investigation of data from direct shares for your investigation and focus on XYZ Corporation Data Exchange, our fictional data exchange ready for configuration.

We may have multiple data exchanges for our organization, which appear alongside our initial data exchange, as shown in Figure 14-12.

Tony Robbins said, “Complexity is the enemy of execution.” I strongly recommend a simple approach to data exchange is adopted by provisioning the minimal number of data exchanges necessary to meet business objectives while retaining meaningful data isolation boundaries.

Managing Data Exchange

In contrast to data share, accounts may be in any region or CSP. Beware of egress charges, however.

Before proceeding, a brief reminder of our intent is to configure one-to-many data sharing in a designated group from a single central Snowflake account, as shown in Figure 14-13. In the following discussion, there is a single publisher account with admin privilege to manage exchanges and several consumer accounts without admin privilege. The distinction is evident in the request made to Snowflake support to provision Data Exchange, as illustrated in Figure 4-12.

A schematic of the data exchange for data sharing with a designated group. Your account represented by a circle is connected to surrounding circles via double-headed arrows.

We can now manage our data exchange by adding and removing Snowflake accounts. Click Manage Exchanges ➤ XYZ Corporation Data Exchange, as shown in Figure 14-14.

A screenshot of the manage exchange tab where the X Y Z corporation data exchange with admin as a provider in the exchange role is highlighted.

Figure 14-15 shows two data exchange member accounts, in this example, those used to configure Secure Direct Data Share discussed previously in this chapter.

A screenshot of the X Y Z corporation data exchange tab. Under members, account number, A D 77003; region, A W S U S east north Virginia; and exchange role, consumer; is highlighted.

Adding a new member to our data exchange is simple. Click the Add Member button, populate with the Snowflake account URL, and select the role, as shown in Figure 14-16. By default, both Consumer and Provider are enabled, which may not be appropriate for your data exchange where one-way traffic may be preferred. Populate the account with a valid URL noting the user interface validates the entered URL, set the appropriate role, and then click the Add button. Accounts may be located in any CSP or region.

A screenshot of add member tab with a section for an account name, and role as consumer and provider.

Selecting an existing data exchange member enables their role to be changed, or clicking the three-dotted box allows removal from the data exchange.

If you selected the data exchange owner, you would see the Admin role checked and grayed out. It is not possible to reassign data exchange ownership. This is a Snowflake support function.

We might also wish to add a provider profile, which is useful for informing consumers of our organization’s contact information.

Add Standard Data Listing

With data exchange members added to our data exchange, let’s consider how to manage data sets in the data exchange. We have already encountered the two roles available, which are self-explanatory: providers and consumers.

From the main menu bar, click Share Data, as shown in Figure 14-17.

A screenshot of the shared with me tab. The share data option is highlighted, with a drop-down menu with options, share with other accounts, and publish to X Y Z corporation data exchange.

After which, a new dialog opens, as shown in Figure 14-18. Populate the listing title and select the desired listing type. The “See how it works” links provide useful contextual information. I recommend you investigate both links.

A screenshot of the create listing tab. A section for the title of the listing and how the will consumer access the data product. The options are standard or personalized.

In our example, we use Test Listing as our title, select Standard, then click Next. The title is presented in preview form requiring further information, as shown in Figure 14-19. Note additional tabs for Consumer Requests and Settings for your further investigation.

Unpublishing a listing reverts to preview format, which is discussed later.

A screenshot of the test listing. The required information below the listing is as follows. Basic information, details, data product, business needs, sample S Q L queries, and region availability.

I strongly recommend your business colleagues are actively involved in the creation of content to populate the listing details. Each listing is a showcase and shop front for your organization’s internal colleagues and selected external customers; therefore, well-crafted descriptions and documentation representing approved messaging are essential.

Although each entry is self-explanatory, there is a lot to consider when provisioning objects into our data exchange. We do not dwell on populating. I leave them for your further investigation.

When complete , click Publish Listing.

Manage Shared Data

In the pop-up dialog box shown in Figure 14-20, we may choose to manage our shared data by clicking Manage Shared Data .

A screenshot of the listing going live. The text reads your listing is visible to consumers in all regions. You must attach a secure share in each region in order for consumers in those regions to gain immediate access to your data. The options below are to manage shared data and set it up later.

According to the regions enabled in Figure 14-21, there are options to make our shared data available in consumer accounts. Note the requirement to log in to each account. Simply select the required region in the drop-down box where further options become available (not shown).

A screenshot of the manage shared data tab with region and configuration. Under region, A W S Europe, Ireland is chosen. Under configuration, a sign-in option for account and user is given.

When complete, click Done. The listing banner changes to show two new buttons— View on Exchange and Live, as shown in Figure 14-22.

A screenshot of the test listing tab. The highlighted view on exchange option is live.

Clicking View on Exchanges provides a single page view of listings with summary information which I leave for your further investigation. Navigating to Home ➤ Data ➤ Private Sharing, as shown in Figure 14-12, displays a second listing alongside the original (not shown).

Unpublish a Data Listing

The second option, Live, allows you to unpublish a listing noting existing consumers retain access, as shown in Figure 14-23.

A screenshot of the unpublish listing tab. The text reads test listing will no longer be visible in the X Y Z corporation data exchange, existing consumers of your data will continue to have access to it. The options of cancel and unpublish are given below.

Unpublished listings revert to preview status, as shown in Figure 14-19.

Accessing Data

Assuming we have shared our new data listing with our second snowflake account region, we can view it in our second Snowflake account. Log in and navigate to Home ➤ Data ➤ Private Sharing ➤ Test Listing. Figure 14-24 illustrates the dialog box now enabled in the consumer account.

A screenshot contains options for free unlimited search with query data and view database.

The View Database screen (not shown) displays the Database Details tab by default. There is a second tab for Schemas. I leave these for your further investigation as the content is self-explanatory. Note the context menu navigates to Home ➤ Data ➤ Databases.

In common with Secure Direct Data Share, RBAC is not imported with shares. You may need to set your browser role to ACCOUNTADMIN or configure a new role with IMPORT SHARE privilege.

Click the Get Data button. Figure 14-25 shows the opportunity to assign a role, which should be pre-created.

A screenshot of the test listing tab. The add roles drop-down menu is highlighted under the create database section.

Once RBAC entitlement considerations have been resolved, the dialog in Figure 14-26 is displayed.

A screenshot of the path to source data, to data is ready to query with a query data button.

A new browser tab opens pre-populated with the sample query and comments from the listing Usage Examples. Add warehouse to execute the sample query and execute.

Data Exchange Summary

Having walked through hands-on examples of configuring and accessing objects provisioned via Data Exchange using a Standard Listing, I leave you to develop data access paths for your user community to interact with provisioned data sets.

Provisioning steps for Personalized listings are broadly the same as for Standard Listings, except the consumer must request access to the data set, and the provider must approve the request. These steps are self-explanatory.

Setting up as a Provider

When ready, change the role to ACCOUNTADMIN by clicking your login name. Then click Home ➤ Data ➤ Provider Studio, as shown in Figure 14-27. The direct URL is https://app.snowflake.com/provider-studio .

A screenshot of the snowflake marketplace navigation option. Provider studio is highlighted under the data menu.

If our account has not been entitled to participate in the Snowflake Marketplace, the message shown in Figure 14-28 appears, contact your Snowflake account executive, or fill out the form at https://other-docs.snowflake.com/en/marketplace/becoming-a-provider.html#step-1-submit-a-request-to-join-the-snowflake-data-marketplace .

A screenshot of the permission required tab. The text reads to manage listing in snowflake data marketplace, select a role with current permissions or contact your account administrator.

Assuming your account has been approved to participate in the Snowflake Marketplace, the next step is to create a profile, as shown in Figure 14-29.

A screenshot of the snowflake data marketplace profile set up tab. The text reads publish data tp the snowflake data marketplace. A set-up profile button is placed below.

We are now required to populate our profile. Note the subset of fields shown in Figure 14-30.

A screenshot of the create profile tab. The queries to be filled below are company icon, company name, company description, and consumer contact email. The options at the bottom are save draft, cancel, and next.

Add Standard Data Listing

Snowflake Inc. requires content in Microsoft Word for its operations team to review and approve prior to creating the first listing. This is Snowflake Inc.’s standard practice.

For all listing types (see Figure 14-31), prospective consumers must register their interest by populating contact/company information.

There is an overlap with Data Exchange insofar as private listings may be made on Snowflake Marketplace, and long term, it is possible Data Exchange will be folded into Snowflake Marketplace.

Snowflake Marketplace listings attract a rebate on credits consumed by usage.

When the profile is populated, click Next. The screen shown in Figure 14-31 should appear.

A screenshot of the create listing tab. The queries below are what's the title of the listing, who can discover the listing, anyone on the marketplace or only specified consumers; how will consumers access the data product, standard, purchase with a free sample or personalized. Below are the options for cancel and next.

Once the listing has been created (not shown but very similar to creating a Data Exchange listing), there is one further consideration.

There is one further consideration. Consumption analysis may indicate data set publication may best be focused on particular regions; therefore, I recommend analyzing available data before publishing to all regions.

Let’s decide how our listing will be published, as shown in Figure 14-32. The options are automatically published upon the Snowflake operations team’s approval. We might prefer to set up multiple listings and decide to coordinate release as part of our marketing strategy.

A screenshot of the publish setting tab. The publishing options are automatic and manual. The manual option is selected. Below, there are options for cancel and save.

Accessing Snowflake Marketplace

We are now presented with the most recent data sets made available via the Snowflake Marketplace, as shown in Figure 14-33. Your content will differ.

A screenshot of the snowflake marketplace content tab, with four different company content. The I T S unemployment data by the local area and Insig A I filing D B global data have a personalized section highlighted.

Immediately we can see the distinct difference between internal Data Exchange and external Snowflake Marketplace. The two listings are standard and personalized as outlined.

Using Snowflake Marketplace

A closer reading of the Standard data set listing shown in Figure 14-33 illustrates the marketplace nature of published data. Both are sample data sets.

To gain experience with Snowflake Marketplace, click Home ➤ Marketplace or select Providers, as shown in Figure 14-34.

A screenshot of highlighted providers drop-down menu in a tab.

Then scroll down and select Snowflake Inc., where the user interface refreshes to display the content shown in Figure 14-35.

A screenshot of the snowflake inc financial data tab. The quarterly financial statements are highlighted in the data tab.

Note content may also be searched by category and business need.

Content can also be referenced directly from the corresponding URL https://app.snowflake.com/marketplace/listings/Snowflake%20Inc . You see one or more published data sets.

Selecting a data set results in a detailed screen similar to the one seen when consuming Data Exchange content, which I consider sufficiently familiar to not require further explanation. from our work

Managing Snowflake Marketplace

Assuming all obligations have been met and our request to become an approved data provider ( https://spn.snowflake.com/s/become-a-partner ) to the Snowflake Marketplace is successful, we may now participate in the Snowflake Marketplace. Our participation may impose restrictions according to the type and nature of the data we provide.

In contrast to data share, but common with data exchange, be aware of egress charges. And like Data Exchange, there is no approved command line approach to implementing Snowflake Marketplace features.

Before proceeding, a brief reminder of our intent is to configure our Snowflake account to publish data into the Snowflake Marketplace and/or to consume data from third parties sharing data in the Snowflake Marketplace, as shown in Figure 14-36.

A schematic of the snowflake data marketplace has three parts. The parts from right to left are as follows. Sectors such as business, demographic, weather, health, S A A S, and more. Snowflake data marketplace. Your account.

Accessing Data

To access a data set, we might be required to validate our email address when selecting a listing, as shown in Figure 14-37.

A screenshot of the verify email address tab. The text below reads your user account email verified. To submit a request please check your inbox to complete account verification first. Options for the close, update the email address, and resend verification email are placed below.

Otherwise, data should immediately become available for download, as shown in Figure 14-38. The database to be created (in this example, FINANCIAL_STATEMENTS_SHARE) can be overwritten; if done so, I recommend a naming convention be established to facilitate tracing back to the source.

A screenshot of the import database tab. The database financial statement share is highlighted along with the add roles drop-down menu. Below the drop-down menu, a question reads which roles in addition to S Y S admin can access this database. S Y S admin is highlighted in the question.

The Add Roles dialog allows another existing role with IMPORT SHARE entitlement in your Snowflake account to access the imported database.

Then click Get Data.

If all is well, our data is imported into a new database and made available for use; otherwise, you may see an error.

A screenshot of an error message. The message reads your selected role S Y S admin cannot get data. Choose a role that has the import share privilege to get this data or please contact your account administrator. Insufficient privilege to accept data exchange listing terms.

Change role and retry, after which we should see a dialog shown in Figure 14-39.

A screenshot of the imported database of financial statement share for which the data is ready to query. The options below are query data and done.

Clicking Query Data opens a new browser screen showing imported databases with sample queries pre-populated and ready for our use.

Snowflake Marketplace Summary

Snowflake Marketplace represents a pivotal point in Snowflake development and delivery, facilitating citizen scientists to rapidly acquire desired data sets into a single account and providing an opportunity to monetize data sets. Participation is critical for Snowflake’s success, and the ease of data set integration presents an almost frictionless opportunity for our organizations to derive immense value.

Automating integration with Snowflake Marketplace requires tooling outside of the user interface. Allowing programmatic interaction, along with improved consumption metrics, will encourage adoption. And Snowflake is signaling steps in this direction.