Google Cloud Dataflow is a fully managed service for creating data pipelines that transform, enrich, and analyze data in batch and streaming modes. Google Cloud Dataflow extracts useful information from data, reducing operating costs without the hassle of implementing, maintaining, or resizing the data infrastructure.
A pipeline is a set of data processing elements connected in series, in which the output of one element is the input of the next. The data pipeline is implemented to increase throughput, which is the number of instructions executed in a given amount of time, parallelizing the processing flows of multiple instructions.
By appropriately defining a process management flow, significant resources can be saved in extracting knowledge from the data. Thanks to a serverless approach to provisioning and managing resources, Dataflow offers virtually unlimited capacity to solve the most serious data processing problems, but you only pay for what you use.
Google Cloud Dataflow automates the provisioning and management of processing resources to reduce latency times and optimize utilization. It is no longer necessary to activate the instances manually or to reserve them. Automatic and optimized partitioning allows the pending job to be dynamically redistributed. You do not need to go for keyboard shortcuts or preprocess your input data. Cloud Dataflow supports rapid and simplified pipeline development using expressive Java and Python APIs in the Apache Beam SDK.
Cloud Dataflow jobs are billed per minute, based on the actual use of workers in batch mode or streaming of Cloud Dataflow. Jobs that use other GCP resources, such as Cloud Storage or Cloud Pub/Sub, are billed based on the price of the corresponding service.