Pipe-and-Filter Pattern
With the pipe-and-filter pattern, each component called a filter is responsible for a single transformation or data operation. Data is streamed from one filter to the next as quickly as possible, and data operations occur in parallel. Loosely coupled filters can be reused and combined in different ways to create new pipelines.
The pipe-and-filter pattern is prevalent in data analysis and data transformation use cases. If you’ve ever piped Unix commands together in a terminal window, then you have firsthand experience with the pipe-and-filter pattern. See the table.
Category | Component & Connector |
Elements |
Filter—A component that reads data, transforms it, then writes out the transformed data. Filters may begin processing data as soon as it is read. Filters must define expected inputs and produced outputs. Pipe—A connector, which transports data from one filter to the next, preserving data order. Pipes have a single input and output, and do not alter the data in transit. Some variants of this pattern also include source and sink elements. The former only produces data whereas the latter only receives it. |
Relations | Attachment—Connects the output of one filter with the input of another by way of a pipe. |
Rules for Use | Pipes can only connect filters with compatible inputs and outputs. Filters should be completely independent of one another and have no knowledge of upstream or downstream filters. |
Strengths | Promotes performance, reusability, and modifiability. |
Weaknesses | Pipe-and-filter systems are not interactive and cannot include a user interface without modifying the pattern. Reliability is not specifically promoted by the pattern but can be designed in by introducing filters to handle error cases. A naive implementation can harm performance because having many filters running in parallel can be computationally expensive. |
Here is an example diagram of the pipe-and-filter pattern:
The batch sequential pattern is similar to pipe-and-filter but has one major difference. The stages of a batch-sequential system operate in turn, one at a time instead of in parallel like in a pipe-and-filter system. Instead of streaming data from one stage to the next, batch sequential systems usually write all data to disk for the next stage in the sequence to read.