Dataflow and Pipeline Processing

Dataflow is a managed service for running data processing pipelines. You describe transformations — read, filter, join, aggregate, write — and the service provisions workers, distributes the work, handles failures and scales as needed.

Its defining idea is that batch and streaming are the same problem. A batch job is simply a stream that happens to be bounded. The same pipeline code can process a historical file or a live stream, which means you do not maintain two implementations of the same logic.

The traditional alternative is running one system for nightly batch processing and another for real-time, with two codebases that inevitably diverge. Unifying them removes a persistent source of bugs where the batch and streaming results disagree.

It also removes cluster management. Autoscaling and work rebalancing happen automatically, which matters for streaming pipelines that must keep up with variable input rates without falling behind.

Batch processing has a natural boundary — the file ends. Streaming does not. To compute anything aggregated, you must decide when a group is complete, and in a distributed system data does not arrive in order.

• Event time — when the event actually happened, according to the source. This is almost always what you want to aggregate by.
• Processing time — when your pipeline received it. Easy to use and usually wrong, because network delays and retries shift it unpredictably.
• Window — the grouping of events by time. Fixed windows are contiguous intervals; sliding windows overlap; session windows group by activity separated by gaps of inactivity.
• Watermark — the system's estimate that all data up to a certain event time has arrived. It is a heuristic, not a guarantee.
• Late data — events arriving after the watermark has passed their window. A mobile device that was offline for an hour produces exactly this.
• Trigger — when to emit a result: at the watermark, early and speculatively, or again when late data arrives.
• Accumulation mode — whether a re-emitted result replaces the previous one or adds to it.

• Pipeline — the full graph of transformations from source to sink.
• Transform — one operation in that graph, such as a map, filter, group or join.
• Autoscaling — the service adds and removes workers based on backlog and throughput.
• Templates — parameterised, reusable pipeline definitions that can be launched without recompiling.
• Drain versus cancel — draining stops accepting new input but finishes in-flight work; cancelling stops immediately and may lose buffered data.
• Dead-letter pattern — routing records that fail processing to a separate destination rather than failing the pipeline.

• Streaming clickstream or telemetry events into a warehouse with sub-minute latency.
• Real-time aggregation for dashboards, such as revenue per minute by region.
• Enriching events in flight by joining against reference data.
• Batch transformation of historical files into a cleaned, modelled form.
• Migrating and reshaping data between storage systems.
• Detecting patterns in streams, such as anomalies or session behaviour.