Freshness

Freshness, sometimes called timeliness, is a measure of how out-of-date a particular piece or collection of data is. Does the data represent the actual situation, and is it published soon enough? This is not the same thing as the age of the data: week-old data can be fresh, and minute-old data can be stale. Freshness relates to the time difference between the available data and the latest update or additional processing input. That is, data is fresh until new information comes in that renders previous information less useful.

Out-of-date information can cost companies time, money, or credibility. How much it costs is a function of the data’s rate of change, the impact of staleness on the use case, and the periodicity of the use that relies on that data. To set an expectation on data freshness, ideally you can measure when and how often users access the data, and calculate the freshness level that keeps the impact within your service requirements.

Think of a weekly report that goes out every Monday morning to the board. In such a case, we might determine that the data it is drawn from can be incomplete or stale during the week, as long as the data is fresh and reliable by Monday morning.

• Example SLO: 99% of data for the previous week (Monday through Sunday) is available for reporting by 09:00 each Monday.

With this well-understood expectation, you might design the system as a weekly batch job to process the week’s data and generate the report in time. Or you might decide that the potential business cost of a late failure in that single job would justify redundant jobs, or incremental processing through the week.

Contrast the preceding to a real-time dashboarding tool, which demands continuous freshness.

• Example SLO: 97% of data is available in the dashboard tool within 15 minutes of an event occurring.

The expectations from this SLO make it clear that the system design will require a near-real-time event-based solution, such as streaming.

Note that both of these example SLOs also include reference to data completeness, covered in more detail later in this chapter.

Data with high freshness concerns is found in:

• High-frequency trading systems that need to respond to the market within milliseconds

• Any service where a user is making critical real-time decisions based on the data (military, air traffic controllers, etc.)

• Concurrent user systems, such as chat, multiplayer gaming, collaborative documents, or video streaming with real-time comments

Architectural considerations that may be guided by a freshness SLO include determining when to sync an event pipeline to disk; keeping copies up to date via streaming, microbatch processing, snapshotting, or replication; and determining when to use caching or materialized views/projections. Alternatively, query results can be calculated at the time they are requested. The more volatile the data, the more often it’ll require refreshing.

Completeness

Completeness, aka comprehensiveness, measures the degree to which a dataset includes all data items representing an entity or event. In a data pipeline, completeness is defined as the percentage of events available to a user after they are sent into the system. This property is correlated to durability and availability, since losing objects or access will result in incomplete datasets.

The importance of completeness is familiar to many of us: incomplete information might be unusable or misleading. Let’s say you’re notifying your customers of an important change. You need each customer’s email address—without it, the user contact data is incomplete. Missing records of the users’ notification preferences may result in the wrong messaging behavior.

In another case, you could be using a MapReduce or other distributed worker mechanism to build a dataset, and decide that waiting for 95% of records to return is a better balance than 99.999% because you’re ultimately looking to query a statistically sufficient sampling of records, rather than laboring under a financial and/or regulatory requirement for every transaction in a system to be recorded.

For data that can be rebuilt from a persistent source, or has a predictable size and shape (e.g., a top-50 leaderboard), completeness may be straightforward to measure.

• Example SLO: 99.99% of data updates result in 50 data rows with all fields present.

This can be a useful measurement if this exact record count has significant impact; if leaderboard presence is a driving mechanism for user engagement, then any omissions will generate distrust. However, in many cases the expectations for consistency, accuracy, or freshness may be a more compelling representation of the user experience.

For the many provider-generated data sources that have no other source of truth, completeness can be much trickier to measure. One approach might be to audit the counts for all ingested data.

• Example SLO: 99.9% of events counted during ingestion will be successfully processed and stored within 15 minutes of ingestion.

Approaches to optimize completeness will depend on the size and scale of the data, as well as the use case. Are you processing financial data where there will be monetary implications to every discrepancy? Or is your application focused on big data for marketing impressions, where accurate aggregate counts are business-critical, but a thousand missed events out of several billion are a rounding error?

Remember that we can sacrifice this or any other property when it makes sense. When would ensuring a high level of completeness be a poor use of resources? For analytical data, completeness might not be a primary concern, as long as the data is directionally correct. Sampling can often give you enough data to make inferences, as long as you make sure that sampling bias doesn’t affect the results.

• Example SLO: Each query will process 80% of its input set.

In this case, though, the real concern may be getting the data quickly so it’s useful for analysis. If so, perhaps you could better measure the service quality with a performance or freshness SLO.

Data with high completeness concerns includes:

• Regulatory data, used for legal and regulatory reporting

• Financial records

• Customer data where you need a record for each customer, with required fields: name, contact info, product subscription tier, account balance, and so on

Consistency

Consistency (an old stalwart of the CAP theorem) can be a transaction property, a data property, or a data application property. It means that all copies of files, databases, and other state stores are synchronized and in agreement on the status of the system or a particular datum. For users, it means that all parties querying that state will receive the same results. After a “write” is acknowledged, any and all “reads” ought to result in the same response.

Does the data contain contradictions? We’ll use an example from the healthcare field: if a patient’s birthday is March 1, 1979, in one system but February 13, 1983, in another, the information is inconsistent and therefore unreliable.

We might lack consistency when we choose storage techniques that store data in multiple independent locations, but for some reason do not ensure that all locations are storing the same data before marking the logical write as complete. Often this is done for reasons of performance or better failure isolation, and consequently better availability. Many use cases tolerate eventual consistency. How much delay is acceptable?

If both scalability and consistency are critical for your data application, you can distribute a number of first-class replicas. This service will have to handle some combination of distributed locking, deadlocks, quorums, consensus, and/or two-phased commits. Achieving strong consistency is complicated and costly. It requires atomicity and isolation and, over the long term, durability.

Measure consistency by quantifying the difference between two or more representations of an object. Data can be measured against itself, another dataset, or a different database. These representations can be the source upstream signal, redundant copies of the data, the downstream storage layer, or some higher-dependability authoritative source.

• Example SLO: 99.99999% of monthly financial data processed by the service will match the company’s general ledger system (authoritative source) by the 5th of the following month (month close).

This kind of matching to measure consistency can be expensive, in both service resources and maintainer hours, since it generally requires a dedicated processing job that can iterate over all the data. The SLO can inform decisions about how rigorous this extra processing needs to be. Are incremental checks enough? Are there any filters or sampling that would make the problem more tractable, or the results more focused on the most critical data?

Following are some examples of data with high consistency concerns:

• When an authoritative source of data needs to match audits against derived or integrated systems. For example, a hedge fund balance sheet’s gain/loss totals need to reconcile with the transactions in the credit risk management system.

• When user queries cannot tolerate out-of-date values. If a user performs aggregate queries where one target dataset has been updated and another hasn’t, the results can contain errors.

Accuracy

Accuracy measures the conformity to facts. Data veracity or truthfulness is the degree to which data correctly describes the real-world entity, property, or event it represents. Accuracy implies precise and exact results acquired from the data collected. For example, in the realm of banking services, does a customer really have $21 in their bank account?

When evaluating accuracy, we can measure and consider both granularity and precision:

Granularity

The level of detail at which data is recorded (periodicity, cardinality, sampling rate, etc.). Granularity helps us reason about whether a dataset has a degree of resolution appropriate for the purpose for which we intend to use it. This is heavily used during downsampling and bucketizing of data for time series monitoring and alerting, and so on.

Precision

The amount of context with which data is characterized, adding clarity of interpretation and preventing errors arising from accidental misuse of data. Are these actually apples and apples, or apples and oranges?

Precision is a measure of how well typed and characterized our data is—not how well it conforms to type (validity), but whether it’s defined and described clearly enough to survive reuse in another context without generating erroneous conclusions or calculations. Consider a set of temperature readings where the temperature collection method varies, or where we measure temperature, then administer a medication, then measure temperature again. Without capturing context in the definition/labeling of either case, we could get errors in some uses of the data. With it, we expose additional avenues for normalization, correlation, extrapolation, or reuse.

When measuring accuracy, we typically need to talk about the ways to audit datasets for real-world accuracy. That is, it is perhaps reasonable to set an SLO that 99.99% of our records will be accurate; if we adhere to that target our system may well be fit enough for purpose, and our users may well be happy. But the interesting part is usually how we determine that real-world percentage. Data accuracy can be assessed by comparing it to the thing it represents, or to an authoritative reference dataset.

Is this a running tally as we reference records in the course of using them (for example, every time a cable technician goes to a customer site they check the records and note any inaccuracies, then we track the rate of corrections needed over a 30-day or 90-day rolling average)? Or do we pull some authoritative third-party source and benchmark to that? Or do we have a periodic real-world collection true-up, like with census data or a quartermaster taking inventory?

• Example SLO: 99.999% of records in the dataset are accurate.

• Example SLO: 99% of records are within two standard deviations of expected values.

• Example SLO: 90% of records have been verified against physical inventory within the last three months.

Accuracy is where our world of data can sometimes become the very real world surrounding us, and may require observing it independently of our normal data intake mechanisms and manipulations (whether through manual work, or RFID scanning, or national security means).

Data with high accuracy concerns includes:

• Personnel records, which must remain accurate for the company to operate smoothly

• Medical data used for diagnosis

• Bank account information, including the balance and transaction history for each customer

Validity

Also referred to as data conformance, validity concerns the question: does the data match the rules? Validity is a measure of how well data conforms to the defined system requirements, business rules, and syntax for format, type, range, and so forth. It describes the data’s shape and contents, and its adherence to schemas; it includes validation logic and nullability. A measure of validity indicates the degree to which the data follows any accepted standards.

Quantifying validity involves measuring how well data conforms to the syntax (format, type, range, nullability, cardinality) of its definition. Compare the data to metadata or documentation, such as a data catalog or written business rules. Validity applies at the data item level and the object/record level.

For example, a nonnumeric value such as U78S4g in a monetary amount field is invalid at the item level. A six-digit zip code is invalid for US addresses but valid for Canadian customers, so the rest of the record needs to be considered in determining validity. Keep in mind that a zip code of 00000 can be valid, but inaccurate.

• Example SLO: In the dataset, 99% of values updated after Jan 1, 2018, are valid.

Data with high validity concerns includes:

• Any data that users depend on to be in a certain schema or format and that, if incorrect, will be deleted or cause errors in downstream systems

• Messages flowing through a schema-managed data pipeline, where any nonconforming message goes to a dead letter queue

Is it better to land bad data or no data? If you validate on the fly, any record that doesn’t conform to a schema is rejected. If you want to be safe, send it to an error log and/or dead letter queue. Alternatively, land then validate. You can process and analyze a batch of data in a staging environment or isolated table(s). Maybe some of it is recoverable. This is a tricky trade-off because you don’t want to encourage producers to provide you with bad data. By accepting malformed inputs, you’re expanding your service’s robustness (see “Robustness”), but taking on additional maintenance loads.

Integrity

Data integrity involves maintaining data in a correct state by controlling or detecting the ways one can improperly modify it, either accidentally or maliciously. It includes data governance throughout the data life cycle: tracking data lineage, determining retention rules, and auditing data trustworthiness. Integrity is not a synonym for security. Data security encompasses the overall protection of data confidentiality, integrity, and availability, while data integrity itself refers only to the trustworthiness of data.

Data with low integrity concerns may be considered unimportant to precise operational functions or not necessary to vigorously check for errors. Information with high integrity concerns is considered critical and must be trustworthy.

In order to safeguard trust across that spectrum, integrity can deal just with local file data on our local system via a mechanism as simple as checking and/or updating the cryptographic hash of important files, or it can extend to secure boot mechanisms, TPMs,² and verification of the integrity of the runtime environment itself. File integrity checks are useful, but are also a lagging indicator. A more timely metric might be transaction integrity. Rather than measuring all transactions, we would likely focus on integrity when we process a transaction class that is particularly important, like the banking and digital rights management transactions that help make sure that we can continue to keep Fluffy in the Fancy Feast manner to which he has grown accustomed.

If, in addition to the checks of the transaction itself, we check the integrity of the attesting system before and after each transaction, and all the checks are successful, then we assume the transaction can be trusted, too.³

Combining all these, we can detect and measure the following:

• Files that don’t match integrity checks during periodic scans

• Files that don’t match integrity checks on periodic access

• Sensitive transactions that don’t pass remote attestation runtime integrity checks

There is no perfect mechanism for validating integrity—even less so in a distributed system, where getting every component to agree on something as “simple” as incrementing a counter can be a challenge. But scans, checks on access, and transaction/attestation checks can work together to give a regular overall check, an as-we-go run-rate spot check, and an approaching-real-time check that the system isn’t fooling us on the other checks.

• Example SLO: 99.9999% of data has passed audit checks for data integrity.

Data with high integrity concerns includes:

• Billing application code, which must be unaltered in order to ensure proper application function and auditability for compliance

• Critical system logs, which must be unaltered in order to ensure proper detection of intrusions and system changes by security

Is the data required to remain uncorrupted? Can this data be modified only by certain people or under certain conditions? Must the data come from specific, trusted sources? If data integrity is a primary concern, make sure you have a comprehensive data management strategy. This includes procedures related to backups, access controls, logging, and data lineage, along with robust verification, validation, replay, and reconciliation processes.

Durability

Durability measures how likely it is that a known-healthy copy of your data will be in the system when you attempt to access it. This is not a measure of whether, in that moment, you will actually be able to access it, but rather whether the data exists to be accessed at all. Recovering from a true durability failure—where no copies exist and the data cannot be re-created—is usually impossible. Because of this, in cases where durability is important, we need to invest extensively in both robustness and resilience.

Azure claims between 11- and 16-nines durability, depending on the service that is used. Amazon S3 and Google Cloud Platform offer 11 nines, or 99.999999999% annual durability. This durability level corresponds to an average expected loss of 0.000000001% of objects per year. That means that if you store 1 million immutable objects, you can expect to lose a single object once every 100,000 years.

• Example SLO: 99.999999999% annual durability

Examples of data with high durability concerns are:

• A general ledger accounting service, which is the authoritative source of a company’s transactions

• Any system storing important data that cannot be re-created, such as sentimental data

Keep in mind that not all data is equally valuable to the user. Durability for legacy data might not be as critical as it is for current data. Also, certain datasets may be more useful for decision making, even if they’re stored in the same tables as other data.

Of course, not all data is created equal, and not all of it is irreplaceable. A service’s durability SLO should reflect the inherent value of its data (whether intrinsic or sentimental) and the benefit derived from the preservation of the data.

Sometimes data can be reingested from the original source, or derived from related data. Where the reconstruction expense (in terms of processing, transit, and opportunity cost) is relatively low, a durability SLO can be more relaxed.

• Example SLO: 99% monthly durability for all hourly summary reports.

Relaxed durability can also make sense in cases where the lifetime or relevance of data is brief. When the time or expense of enabling recovery would exceed the valuable life or the lifetime value of our data, we might reasonably decide that You Only Log Once and we’re fine with the reasonably low chance something will happen to this ephemeral data between creating it and the point at which we will be done making use of it.

On the other hand, there’s risk in dismissing critical data as “transient” or “metadata.” Decryption keys, user-to-block indices, edge graphs for user notification preferences, or group editing rights and history are all distinct from user-provided data, but still critical. Consider carefully the importance of even automatically generated data to how your users interact with your services and your core data, and how much pain losing it would inflict upon them.

We’ve seen operators invest in globally distributed, highly replicated architectures for storing and reading data, only to relegate large swaths of “metadata” to cold start active-passive systems that meant days of recovery were needed to restore user access to all that carefully guarded primary data in the event the hot copies were lost.

That could be a perfectly acceptable recovery strategy—unless you make money when users are able to access their data in real time, and lose money when they can’t. It doesn’t do much good to tell them their groundbreaking O’Reilly manuscript is completely safe and they have nothing to worry about in spite of the fact that your entire West Coast data center just slid into the Pacific Ocean, if you have to turn around and tell them that you should be able to restore the “unimportant” metadata that will allow you to locate the “real” data they care about approximately two weeks after their print deadline.

When it comes to the intersection of durability and availability, the world usually doesn’t care if the dog ate your data center. Regardless of the cause of data unavailability, your users only see the system as a whole, and are unlikely to be empathetic to the fine nuances of differences between “real data” and the data critical for that data to be operationally available.

• Example SLO: 99.9999% of data will remain intact for the year.