Bartosz Mikulski

Six words can destroy your credibility and jeopardize your career: “This number does not look correct.” When you hear those words, it is already too late. People don’t trust you anymore. They are suspicious about everything you have done so far. Suddenly, you become a person who makes mistakes—the person who has probably been feeding us false data for years.

Is there anything to worry about? What if the timing was terrible, and somebody recently made a wrong decision that caused the company to lose tons of money? Suddenly, every decision becomes data driven. Even the people who ignored the data for years claim that they were making data-driven decisions. The problem is that the data is incorrect.

The data is wrong because you have made a mistake! Congratulations! You’ve been promoted to a scapegoat! Everybody in the company is going to know your name. You have just become the Excuse of the Year.

Do you want that? No? What are you doing to avoid such a situation? How often are “hope” and “luck” a vital part of the data pipeline? We hope that the data format does not change. If we get lucky, the files get uploaded before we generate the report for the CEO. Of course, if it fails, we have a person who restarts it manually, and nobody notices the problem.

Data engineering is no different from other branches of software development. We can apply functional programming principles and turn our pipelines into something that resembles a function. Sure, the input is huge, and the function runs for a few hours, but for any given input, there is only one correct output, and we can write tests to ensure that we get what we expect. Does it seem trivial and obvious? So why are there so many data pipelines without a single line of test code?

The other best practices we can copy come from site reliability engineering (SRE) teams. These include relentless automation and monitoring. Manual configuration leads to systems that are fragile, unpredictable, and cannot be fixed. You never know what is going to happen when you change something. You can’t create a test environment because you don’t even know the current state of the production system. It’s a nightmare.

Similarly, the lack of monitoring will make you wake up screaming at night. Do you know whether your pipeline processes all of the data? What if it drops 3% and nobody knows about it? We should gather instrumentation metrics about the code we run and about the data we process. You would be surprised by the number of issues you can detect by comparing the histograms of input values and the pipeline output.

Of course, all of that must be automated as a part of the pipeline, and the whole process must fail if any validation detects incorrect values. Otherwise, it becomes an expensive toy that nobody uses and nobody cares about.