Skip to main content

Cross-language pipelines in Beam

Java is famous of its paradigm – Write once, Run anywhere – which was defined in 1995 by Sun Microsystems to illustrate the cross-platform benefits of the Java language. Apache Beam follows the similar principle but in regard to cross-platform data processing engines – Write pipeline once and Run it on every data processing engine. Beam achieves that by leveraging a conception of Beam Runner – the programming framework, which is responsible to translate a pipeline, written in Beam model way, into a code that can be run on required processing engine, like Apache Spark, Apache Flink, Google Dataflow, etc. All translations actually happen in runtime – users don’t even need to recompile their code to change a runner if all dependencies were already provided in compile time. Therefore, Beam already supports a bunch of different runners that can be easily used to run a user’s pipeline on different platforms. 


However, the classical runner translates user code only from and to the same SDK – from Java to Java, from Python to Python, from Go to Go. That is why, if we want, for example, to use Spark Runner (that allows to run Beam pipeline on Spark), written in Java, we have to use only Beam Java SDK. If we want to run a Beam Python pipeline on Spark, then we had to implement either Spark Runner written in Python - which is interesting but not trivial task - or… 


That was in the past until Beam community started to work on Portability – a quite new Beam feature that is supposed to eliminate such restrictions and reuse already written code in different SDKs. That being said, users will be able to run a pipeline, written in Python or in Go, with Spark Runner written in Java. And even more! It will be possible to run cross-language pipelines – for example, Java pipeline with one or several Beam transforms written in Python or in Go! The goal of this post is to give a brief explanation how Beam Portability works and show how to run cross-language pipelines like these. As an example, we will run Java/Python pipeline on Apache Spark.


Beam Portability and Cross-Language pipeline

Let’s talk about Beam Portability in few words. If you need more details, I’d recommend watching these great talks: 


As it was said above, the “classical” Beam runner translates a user pipeline code to the code of processing engine that will be used later to run the pipeline and user has to use the same language SDK for the pipeline, runner and engine.  


The main goal of Beam Portability and Portable Runner is to eliminate this restriction and use the components based on different SDK in the same pipeline.  In that way, it will be possible to reuse the Beam transforms of different SDKs (e.g. IO connectors or Beam SQL) and create cross-language pipelines. 


Let’s take a brief look on how it works. While using Portable Runner, SDK will translate the pipeline into Protobuf representation via Runner API. Then, it will upload all required dependencies from class path (in case of Java SDK) or specified by user location(s) to Runner side (for example Spark Runner). After that, pipeline is submitted to the Job Server via Job API and, finally, Runner translates and runs the pipeline on data processing engine (Spark in our case).


Actually, Job Server is a new Beam component (it’s a daemon that should be run separately), which is responsible to receive a pipeline in portable way and run it on specified runner.  


Though, it’s still not clear how to execute a code of different SDKs with a runner that is written only in one SDK – like Spark Runner is written in Java. For this purpose, Beam uses a conception of SDK Harness – this is a separate component of Beam Portability which is responsible to execute a small part of pipeline (PTransform) with dedicated SDK. Beam supports two main ways of SDK Harness – Docker and Process based. It means that either a Docker container or a process with required Beam SDK will be launched to execute a code of transform.


And finally, for cross-language pipeline, we need to run and specify Expansion Service for every transform. By default, it will use the Expansion Service that is running along with Job Server – so we don’t need to specify for transforms with the same SDK as Job Server. Though, for other SDKs we need to run a standalone Expansion Service and specify it for every external transform. So, the final architecture of Cross-Language pipeline will look like this:



In the next part we will talk about how to run such portable pipelines with Portable Spark Runner.


Comments

Post a Comment

Popular posts from this blog

Apache Beam for developers: Creating new IO connectors, part 1

By this post, I'll start a series of blog posts about creating new IO connectors in Apache Beam . Introduction to Beam IO Before getting into Beam IO internals, let's take a quick look on what actually Beam pipeline codebase is. In general, logically all code, that required to run a user pipeline, can be split into 4 layers - Runner , SDK , Transforms & IO , User Code .  On the bottom level, there is a Runner  code, which is responsible for all translations of user pipeline to make it possible to run on preferred data processing engine, like Apache Spark, Apache Flink, Google Dataflow, etc.   On the second level, we have a SDK  code. This part of code allows to write a Beam pipeline in favourite user programming language. For the moment, Beam supports the following SDKs: Java, Python and Go. Scala is supported through 3rd party SDK called Scio .  Third level incorporates different Beam Transforms , like ParDo , GroupByKey , Combine , e...
1 comment
Read more

Developing data processing job using Apache Beam - Windowing

Talking about Streaming data processing, it's definitively worth to mention "Windowing".  An example, showed in the previous post about streaming, was quite simple and didn't incorporate any aggregation operations like GroupByKey or Combine. But what if we need to perform such data transforms on unbounded stream? The answer is not as obvious as for bounded data collections because, in this case. we don't have an exact moment on a time line when we can conclude that all data has been arrived. So, for this purpose, most of the data processing engines use a conception of Windowing that allows to split unbounded data stream into time "windows" and perform such operation inside it. Beam is not an exception and it provides rich API for doing that. Problem description Let's take another example to see how windowing works in practice. Just to recall, the previous task was the following – there is unlimited stream of geo coordinates (X and Y) of some obje...
Post a Comment
Read more

Developing data processing job using Apache Beam - Streaming pipeline

This time we are going to talk about one of the most demanded thing in modern BigData world nowadays – processing of Streaming data. The principal difference between Batching and Streaming is type of input data source. When your data set is limited (even if it’s huge in terms of size) and it is not being updated along the time of processing, then you would likely use Batching pipeline. Input source in this case can be, for instance, files, database tables, objects in object storages, etc. I want to underline one more time that, with batching, we assume that data is immutable during all the processing time and number of input records is constant. Why we should pay attention on this? Because even with files we can have unlimited data stream when files are always added or changed. In such case we have to apply streaming approach to work with data. So, if we know that our data is limited and immutable then we need to develop batching pipeline, like it was showed and explained in the fi...
Post a Comment
Read more