Amanda Tomlinson

We’ve all seen the cool infographics, but no one needs to tell a data engineer about the immense and ever-increasing amount of data being generated every day. We’re all living it.

We’re extracting it, transforming it, landing it (now considering whether we should maybe have started landing before transforming), cleaning it (what, you’re saying we should stop cleaning it?), deciding where and for how long to store it, standing up new infrastructure to handle the sheer volume of it, filtering it, joining it, building KPIs and models from it, creating workflows for it, exposing it, cataloging it, monitoring it (easy enough when we started, but increasingly difficult a few years in). With so much to do and so much demand from our stakeholders, it’s not at all surprising that so many data teams, especially those serving internal customers, get so caught up in the technical aspects of data engineering that they forget to consider who their users are and what they actually need. Data is data, right?

Before too long, this will lead to frustration, wasted effort, and a lack of confidence in the data team. Yep, all of the stuff I’ve listed is hugely important, but it’s just as important to realize that not all of your users are the same, and to take a step back and consider what they need from the data you produce and what they need from you.

But you shouldn’t cater to their every demand. A certain level of satisfaction comes from delivering requirements and closing tickets, but simply churning out solutions creates long-term problems. A fine balance needs to be struck between delivering what your users need and maintaining a sustainable, scalable data function.

For me, this starts with a vision and strategy for the engineering team. If that seems too ambitious, at the very least set some design principles or guardrails. Next, start treating the demands received by your team as nothing more than conversation starters. These are simply insights into what your customers want from you, not fully formed requirements.

To turn them into requirements, you need to dig a bit deeper to understand the problems your customers are trying to solve with your data. Create personas to allow you to understand the various data users and use cases within your organization; look for the commonalities and apply your design principles to allow you to produce data products to serve these. Take ownership of your data and become the expert in serving it up to your users in the right way.