We live in a connected world. The data ecosystem around us is huge. And to top it off, almost all data is connected to other data.

In such a connected world, it becomes almost impossible to traverse each node and each edge stemming from that node. If we decide to do that, it will be almost impossible to arrive at any answers in the allotted time frame/SLA. Thus, another challenge while performing data processing is when to decide not to traverse a particular network path.

It is extremely difficult to make such a decision, because you do not know what lies on the particular path that you have decided not to pursue.

This kind of knowledge does not come at once, but rather is a constant endeavor by the data scientists to find which paths make the more sense for a given use case and which ones can safely be ignored without having a huge impact on the outcome.