Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Running Apache Samza on Kubernetes - Weiqing Yang, LinkedIn Corporation

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Running Apache Samza on Kubernetes - Weiqing Yang, LinkedIn Corporation

Apache Samza is a distributed stream processing framework that allows you to process and analyze your data in real-time. It has been widely used at Linkedin and other companies on a large scale. Recently, we added Kubernetes as the new scheduler backend for Samza to run in distributed mode. In this talk, we will deep dive into the technical details about how Samza runs natively on Kubernetes by leveraging the primitives provided by Kubernetes for scheduling, storages, etc. We will also compare running Samza on Kubernetes with other existing solutions such as YARN and standalone mode. Finally, we will share some practices about running Kubernetes as a container orchestration framework for other big data processing engines.

https://sched.co/Uacc

A

Hi everyone and welcome to the first session of the day my name is Anirudh. I lead the big data user group in the kubernetes community. The first talk we have today is by waching yang from LinkedIn and Jian. He from Alibaba and they're going to be talking about running Apache Samsa on kubernetes.

A

Couple of logistical items: please hold your questions till the end. I'll come with you with a microphone and leave feedback on scared, calm at the end of the session. Thank you.

B

Hi I'm waiting for linking and I'm working on this from Alibaba and our topic today is running them their own communities.

B

First we'll talk about what Center what Apache center is. Then we will dive into having it's as cute in a center job on kubernetes, and then we will do a demo to run a center job on agile, kubernetes cluster. After demo, we will will talk more about different deployment options for center users. In the end, we'll share more practice about using abilities as a campaigner as container actors, a framework for other big data processing engines.

B

So Sansa is a distributed stream processor framework that allows you to build words they for job they'll process data at skill. In real time it was developed, a link in 27 2013 and was helping has been widely used and linking and other companies like when were slack extra.

B

Center supports a fully palatable mode for input, sources and output systems. It can process and transfer data from any source. Seminar often offers built-in integrations with APIs akafuku ADA, like m-80s kinases, as your even hub and hub, and so on. Also, it's quite easy to integrate with your own sources. There are different development options. You can run them the English standard, standalone mode or in cast mode like runs on a passion or kubernetes. We'll talk more about the job deployment later.

B

One is a typical central use case. Let's look at this picture event.

B

Event producer. It could be a service database changed or magics event. Producer writes the event into messaging system like Kafka and agile. You will have from where a standard job reads: input events to do the computation after that, send out right job results into output systems, which could be a database Brasserie service or measuring systems.

B

Then the downstream application can consume the job result to do their own analysis and so on.

B

This slides brief stems are features: Center provides flexible API in Cinco, Java and Python. It has flexible deployment model for running the applications in any host environment, enemies. Cluster managers like young and kubernetes, so the same Center job code can wrong everywhere from public cloud to continue right, environment to bare metal, Halbert massive scale center provide the first support for local state and support incremental check pointing of state instead of for snapshots.

B

This enables Em's up to scale to applications with very large state center support applications that use several terabyte upstate and runs on thousands courses. This empower multiple large company, including in cream and some for tolerance, Center, transparent, transparently, migrates tasks along with their associated state. In the event of failures, its support, host ability and incremental check, pointing to fast recover of the failures stems, are currently at least ones processing a semantics.

B

Next, we will introduce them the standard terminology, Center process, your data in the firm of strings. A string is a connect of immutable messages, usually at the same type or category data in a string can be unbounded like Akaka topic or bounded like a set up. Files on HDFS Center supports prodding systems, and can that can implement string abstracts, for example, Kafka implements the string as a topic a while. The database can May increments the string as a sequence of updates to his tables.

B

In this picture there are two input strings and one output stream and the sender job filled out the event which a lot below a string is charted into multiple partitions for skinning. How is data is processed? Each partition is an audit, replayable sequence, or order of records when mastery is written into a stream is ends up in one of partitions and them's are skills. Your applications by logically breaking it down into multiple partitions tasks. A task is a unit of Paradiso of your application.

B

Each task consumes one partitions of its input strings, the assignment of partition to tasks never change. They say if a task is on one node that fails, the task is returned. The task is really started elsewhere, but still consuming those things during partition.

B

Just like the logical just like a task is the logical unit or paradin, and a container is the physical you need. Application typically have multiple containers distributed across hosts. You can think of each container as a GVM process, which runs one or more tasks.

B

So each application also has a job coordinator, which masters which, which manages the assignment of tasks across the individual containers. The coordinator monitors the liveness of individual containers and redistributed, which redistribute the tasks amount, reminding ones the remaining one during a failure. The coordinator itself is pluggable, so this in labels. This enables them to support multiple deployment options.

B

As we know, kubernetes supports long-running jobs well and them the stream processing framework and its job are long-running jobs. So communities is a good another good option for Center user to deploy their jobs. We integrate assembler with communities for running streaming processing as a managed service. We averaged kubernetes for isolation, multi-tenancy resource management and deployment for center application. In this mode you write your Center abdication and some may choose gadget or kubernetes cluster Center then work with communities to provision resources for your application and run it across a customer shrinks.

B

It also handle the failure of individual instance and automatically restart them.

B

We will dive into how a sender job runs only qualities.

B

First, let's recap a bit of communities. The master component includes API server controller scheduler, a scheduler Watson Union, created path and select one node for them to run for concern or cooperate. History of building controllers like a stay for that user users can also develop their own controllers for a slave node component. It includes cuba net and part cumin. 'it is a agent to launch and manage it like they called apart.

B

Let's talk about the workflow of offender jabroni own peak abilities, so first the job, the user use the wrong app script to submit their job. The squeak will initialize an application runner, which is the main entry point responsible for Ronnie Center application, the application, runner parses your configs and writes them to a Kafka topic for distributing them. Then it process it proceed to submit a request to a keep API server. Now a job coordinator pot, the cumin. It will start the job coordinator part. The coordinator.

B

The job coordinator is then responsible for managing the overall application in race configs from the Kafka and computes work assignment for individual parts. It also determines the hosts. Each part should run on taking data locality into account. It proceeds choose then the work report, creation request to API server. The cabinet will watch the requests and start a work report.

B

If the applications dependency are hosted in the remote artifactory repository like HDFS. Those dependency need to be downloaded into the pot, so in step 8 the worker will ask the job coordinator for is assigned tasks.

B

Network had more about the state step 8 when the work report is started, it first queried the job, queries the job coordinator for his work assignment and can fix the end process to ask you to is assigned a tasks in this picture. The job has two inputs, which has four partitions repast respectively.

B

Each worker has its own assigned during partitions, which then can read input from nests. Look at more about. Let's look at into the note cuban. It will start container and then continuously watched the house up the container it then restored container. If the container fails, the container can be constricted to right state into local or remote store. If it's conveyed to the remote, a remote store will be mounted into the pot for the case. In the example, in the picture, the agile remote store is mounted into the pot.

B

Next, we'll do a demo how to run a job on a chaos cluster in the cluster. Besides kubernetes we also deploy kafka. The job will write the result into Kafka. What is the jobs doing is to read real-time feeds from Wikipedia extracts of metadata of the events and calculated count every 10 seconds for all added that are that were made during the window. It outputs the counts into a current topic: pod Wikipedia state topic. Let's look at a demo.

B

I'm going to run a slender job called Wikipedia application in 80s. This is to add your portal. I've created an 80s crafter they're, the resources discs to storage account and three virtual machines to use agile file as a kubernetes. For now we need to create storage account and no file share for it, so the standard storage account is for that. The other storage account is used to process the files for cloud the shell. Let's look at them. The storage account.

B

This is the file share I created now here there is no file here, we'll go back to here up to around the job. Let's open the shell to run the job.

B

First, let's look at the services wrong in the cluster in the cluster. Besides kubernetes, we also deployed caca because the job used Kafka as the messaging system. Let's check the namespace. Besides default namespace, we created similar namespace for running, send our jobs we can see here.

B

There is no sender job running in the cluster as there's no pots in stem learning space.

B

In default, namespace there is a pot called Kafka client. We all love into it to run all job.

B

Before running the job, let's check the existing kafka topics. This is the command to these two kafka topics for a Wikipedia application job. The job results will be sent to a Kafka topic called Wikipedia States, and now we can say, there's no such topic created.

B

We can submit a standard job use the wrong app script. We need to provide two parameters to the script. Why is the configuration and the other is a factory class? That's used to create a configuration file, other bugs sent achieves reach the prop property config factory, but you can also implement your own config factory as well before running the job. Let's look at what configurations we set up in the configuration file.

B

This is the configuration file since we want to learn the job on kubernetes. We set up these two configurations tooth health lamda to choose community's operator to run the jobs instead of our other cut cluster managers like Jung.

B

We also set up the application class application, name and ID job partition, number container account and the base directory of the log and job state files.

B

These two configurations are related to capture system. We also specify the image name for the job and the namespace where a job runs. This config is to enable the job to use agile files as the communities volume. Other configurations are ready to Kiba restore serializers and job metrics.

B

Let's go back to cloud shell to run the job.

B

Okay, the job has been submitted to a chaos successfully.

B

Let's check the job status.

B

Now we can see the arthropod created by the job. One is the job coordinator part. This is the job coordinator, part and the other two are worker parts. Their status is running region check, pot status in dashboard as well.

B

Let's switch to send our namespace.

B

We can see the CPU usage memory usage also, we can see the part information and PV claims until now, there's no errors or exceptions showing in the dashboard. Everything looks up, everything looks, ok, everything looks fine.

B

We can lock into the containers to check the job state and logs.

B

Let's log in to one of the worker containers.

B

Let's check if the log file in the state folder have created successfully or not. Okay, we can see the log files and the state folder.

B

Now, let's check the job results.

B

We log back to the calc aligned pot.

B

We check the Kafka topic again.

B

We can see there are some topics created by the job, the coordinator stream change rock stream and saw in the median streams job results are saved into the Kafka topic, Wikipedia States we can use.

B

We can use this command to read the messages from the result topic.

B

Now we can see there have been some results already.

B

Okay, if the container is gone, the file saving the container will also come to check the job state and marks. After that container gone, we can check them in agile file because before running the job, we can take the job to use agile file as the remote volume monkey into the pot.

B

The data in agile file will always be there, no matter whether the job or container is come a lot. Let's go back to the agile file and check the logs and state files.

B

Now we can see there are the job logs and state files.

B

This is, this is the folder for the job job coordinator pod, the other two are for a worker container.

B

There are the logs and the job states. There's all the demo.

B

Next Jim will talk about a different deployment options.

C

Nope yeah, so I'll talk about the other deployment options for Sansa, that is in the running scanner or learning on how to be yarn and also I briefly mentioned how conics kunis works for other data processing engine like spark Fink.

C

First, the stand alone mode of sensor, so in the previous part we mentioned that sensor works on communities and Sansa has also has a way to run static, ray on the host, meaning. The user can simply start the center stream processor manually on the pair host and in this mode. So all the stream processors uses the zookeeper for communication, so basically it used the suitable for membership management and tasks or Nations. So, in contrast in this mode, it does not use kubernetes or yarn for scattering, so there will be limitations life.

C

If one node fails sensor stream processor cannot automatically fill to a different fit over to a different node. So there will be some limitations in this mode so and that's not mode that similar runs on yarn. So in this mode, sims are the average yum for scheduling and resource management and deployment. So Jana is very similar.

C

Octo is similar to kubernetes. It's also a master/slave architecture that Yan has a master called a resource manager, the iron, so that is similar to the API server and schedule of the communities. It is responsible for scheduling the containers across clusters and the slave is the node manager that is similar to the row of cookies in communities, and it is most possible for launching the containers and also there's an application master that is similar to the control, the operators concept in communities.

C

So the application master is basically a piece of user, usually written code for user to interaction, interact with young and start their continuous.

C

The way it works is so the sender young client first talked to the resource manager, it submits the spot, sorry, the Scimitar application, and then the resource manager will make decisions and scheduling the center. The same time, application master to a particular node. So once the sensor application master is started, it goes back talking to the resource manager and asked him for continuous to launch the sentence or tasks so the center tasks as a real task. That's running inside the container.

C

So once the center talks is started on the know major, then it can start with its own business project like reading data from the Kafka broker,.

C

If we do a comparison between kubernetes and yarn, so communities is historically good for Romani service, as people know, so the design of kubernetes is also called the lab of sugar design. The way it works is user specify the intent in the phone like llamo foul, so it specify the intent what the state or the what is the expected state of my app and then the operator or the controller running on kubernetes.

C

It will try to drive the current state of the cluster towards the desired state, say example: if we deploy app with five traffickers and the cluster has only say three replicas, it will try to match up. There were two remaining ones and if it's more than that, it will automatically delete the extra ones. So it's always trying to match the current state towards the desired State and that's how the create the controller the operator works in encode.

C

So it always does this control loop every period all the time it checks the current state and the desired state. So this sort of creates a self-healing mechanism so, like eventually, the system will somehow go back to the original track. So in every such loop design patterns it's ideal for automated operations, so people can similarly say well. My obsession water state I wanted my application to be, and then the controller operator will go towards that state. So, in contrast, the young based on it's not like that Soyoung is mostly known for big data processing engines.

C

So it's a platform for scheduling data process. Engine starts Park, Map, Reduce and so on. So it's known cool for best jobs and Jung. Has this first-class job concept since I've built features for job priority, so we can scheduling based on job the job level and- and there are the the scheduling part of the arm is also I. Think is more more feature: extensive, attend communities.

C

It builds a lot of scheduling, advanced features there so, but in contrast of kubernetes scheduling is mostly based on power level, so it's schedule based on Paul, so there are some difference in your communities other than that we. So we all talk about some. How kubernetes works for other differences in engines like spark and Frink, so first spark spot is in here it's very similar to what we just? How spark works on communities? Is that spark? Still?

C

It also has the driver and accurate, so the driver is is pretty much similar to the same job coordinator is the essential component for coordinating the tasks across the casters, and the actual executor is a real and real worker. That's similar to the sensor Walker and the job declined versus um it's the the driver, port and the de job. For once the job report started, it will go back to the communities API server and create its own executor port. So, once all these ports started a spark rasterized phone.

C

So that's how spark works to with the communities and there's another thing community built, that's called spark operator, so the spark operator either averages. The operator concept in communities, as I mentioned so operator in community is a is a is a way for user to specify their IP application in a declarative form so say in a form of a llamo, foul and user submits the llamo powder to the cluster and the operator will watch the yellow flower definition and tries to create or delete certain ports to match the Yamaha definition.

C

So here, similarly spark root its own spark operator that the user can submit the spark definition definition through the co I and the spark operator watch that definition fallen to whatever it needs like creates. The driver, port and the job was done internally. The actual parts and also the operator monitors the port status and tries to set the operation status based on certain port status. So this whole thing is very similar to the well-known controllers on communities such as the staples, a deployment and so on. So the llamo foul of spark operator looks like this.

C

It shows the spat and the status the spatter is. The is the specification of the spark application inside it specifies things like the driver and the executors. What is the memory sizing, the cost limits, and so on for the spark and also the image? What is image for the application and also the instructor cross name of the spark gap and the stator side? It shows the status of the app and other informations right, the the web UI address and so on.

C

So that's how spark works on communities and so communities also integral integral ways of print, so the spring is another popular, very popular streaming, processing, engines and think architecture wise, it's also kind of like the master slave. So the spring has the job manager that's similar to the center job coordinator and also the task manager, that's similar to the center worker. So in cranes it actually uses the communities department primitive to launch its own job managers and task managers. So it does not it's not embedded in the finger framework up to this time.

C

So the process of this approach is that ravaged the exist, robust communities, worker primitive, and this requires minimum coal change. But the disadvantage of that is it's not as factual as the center spark approach. So in the below examples, the job, so the left side shows the llamo definition for the job manager. That's the de prompt that uses the department primitive increment is to to deploy one instance of the Fink job manager and the right side shows the department yeah no foul for user to deployed ramicus of task managers.

C

So with this way the drink ravages. The department launched instances, however, so in contrast with the previous mechanism in this mode, it has limitations, like you cannot say, I want to run a port on a particular node with data locality considered, so it's computed average, the underlying controller primitives. So it has such limitations so that the conjure a primitive core is hard holy. So you can't easily change it. So that's one thing and yeah, so that's it. It concludes our session. Thank you.