Based on the concepts that we recently introduced about Pipe-and-Filter in the field of enterprise applications, we use this kind of architecture in several scenarios in order to process a large amount of data (or large files) that trigger several steps (or tasks) that need to be processed. This architecture is highly beneficial when we need to perform a lot of transformations in the data.
To understand how Pipe-and-Filter works, we are going to review a classic example of processing payroll records. In this example, a message is being sent through a sequence of filters, where each filter processes the message in different transactions.
When we apply a Pipe-and-Filter approach, we decompose the whole process into a series of separate tasks that can be reused. Using these tasks, we can change the format of the received message, and then we can split it to execute separate transactions. As a benefit of doing this, we improve the performance, scalability, and reusability of the process.
This architectural style makes it possible to create a recursive process. In this case, a filter can be contained by itself. Inside of the process, we can include another Pipe-and-Filter sequence, as illustrated in the following diagram:
In this case, every filter receives an input message via a pipe. The filter then processes the message and publishes the result to the next pipe. This repeatable process continues as many times as we need it to. We can add filters, accept or omit the received input, and reorder or rearrange the tasks into a new sequence, based on our business requirements. In the following section, we will detail the most common use cases for applying a Pipe-and-Filter architectural style.