Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Introduction to Strimzi: Apache Kafka on Kubernetes - Jakub Scholz & Paolo Patierno, Red Hat

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Introduction to Strimzi: Apache Kafka on Kubernetes - Jakub Scholz & Paolo Patierno, Red Hat

Strimzi is a CNCF project focusing on running Apache Kafka on Kubernetes. Apache Kafka has emerged as a leading platform for building real-time data pipelines. It provides support for high-throughput/low-latency messaging, as well as sophisticated development options that cover all the stages of a distributed data streaming pipeline, from ingestion to processing. But running it on Kubernetes can be complex and tedious. This talk will introduce you to Strimzi - an operator which makes it easy to run Apache Kafka on Kubernetes. It addresses the whole lifecycle from creating, managing, and monitoring Kafka clusters to managing topics or users. This session will go through the main challenges of running Apache Kafka on Kubernetes, explain how they are solved by Strimzi and show a live demo.

https://sched.co/Zevl

A

So, thank you very much for joining us. We are going to talk about stream z, a project about running kafka. On kubernetes I am a paolo patierno and I am here with jakub schultz. We are both maintainers of the stream z and we are going to talk about it in the next um few minutes.

A

uh Let's start from uh kafka, so kafka is an open source project originally created by linkedin and at the beginning, kafka was defined as a publish subscribe messaging system, because at the end, kafka is a messaging system. But over the years the use cases around kafka are changed. There are more use cases about using kafka for doing some analytics on uh data in real time. So today, kafka is even defined as a data streaming platform, but at the end, kafka is a distributed fault.

A

Tolerance high available commit log, but kafka is not just about the brokers, so the components which are in charge to exchange the messages between the clients, so the consumer and the producer. But it's more a broader ecosystem with more components. On the right hand, side you can see. For example, there is kafka connect, which is a framework for uh moving data between a different system um using kafka like, for example, moving data from one database to another.

A

You can also see mirror maker mirror maker is used for mirroring data between two different kafka clusters running maybe on two different data centers in the cloud, for example, coming to the clients. So talking about consumer and producer and the apis for the clients, there are a kind of low-level apis, so consumer and producer api that you can use for developing your application for sending and receiving messages.

A

But there is even a more complex api, which is the streams api for writing. More complex application for doing some analytics in real time time on the data flowing through your kafka. So kafka is able to ingest this data in real time and you can use this streams api for doing some complex processing like filtering mapping and so on. There are also some third-party tools so other tools in the kafka ecosystem, which are not part of the kafka upstream project, but can be used for interacting with kafka.

A

But why at this point running kafka on kubernetes? So on one side we have kafka, which is a a system distributed by nature and even the applications and workloads using kafka are also distributed as scalable by nature.

A

On the other side, we have kubernetes kubernetes, provides a grid, abstraction layer for running your software everywhere, so you can run kubernetes on bare metal or you can run kubernetes on any cloud provider like um azure, aws, ibm and google, and you can move your software on any of these platforms or on bare metal or on a cloud provider. If you have kubernetes under it, so you have just using kubernetes for running your application and you can move your application everywhere.

A

So uh kubernetes enables the cloud native development, but having on one side kafka and on the other side, kubernetes people who has no village about kubernetes, why they should learn more about kafka, not using the knowledge that they already have about kubernetes for running kafka on kubernetes.

A

This is where uh strings comes. So streamsy is an open source uh project.

A

It's a cncf sandbox project since last year, more or less, and it's mostly focused on running kafka on kubernetes, as I already mentioned, so it provides some container images for kafka for zookeeper, because uh kafka needs a zookeeper and sam in order to work today, but even images for all the other components that are parts of the kafka of the stream. The ecosystem, as we will see uh in a little bit uh streams, provides some operators.

A

So it's based on the operator pattern in order to deploy handling your kafka clusters and even all the entities around the kafka. But we will see more with the the demo that jakub uh is going to show us. So stream z provides a really kubernetes native experience for us in order to handle kafka on kubernetes. Why?

A

Because um it's not only about ikafka clusters, but even about the users and the topics and even all the other kafka components in the kafka ecosystem, but why I say that stream z provides a kubernetes native experience for kafka, because um in kubernetes we have different objects right like, for example, pods stateful set deployment, secrets, configmap and so on. They are the different kubernetes native resource that the user can use today, but in kubernetes there is also a way for extending the kubernetes api.

A

So stream z provides some custom resource definitions in order to uh to make real this kubernetes negative experience for kafka. So it means that with strimzy we have a new kafka resource, so you can create your kubernetes resource of kind kafka where you can describe your cluster and all the configuration, but it's not just about kafka, as you already mentioned, even for all the other components in the stream.

A

The ecosystem like, for example, you have resources for kafka, users and kafka topic for handling users and topics in the cluster, but you also have kafka, connect and kafka connector resources for deploying a kafka connect deployment. So all the worker nodes related to kafka connect, but even for interacting with kafka connect for running the connectors, uh which are the components needed for moving the data between the system using kafka connect through kafka.

A

Then there is a kafka bridge resource for um deploying a bridge, a http bridge in order to allow http client to interact with your kafka cluster. So using the http protocol, then we have a couple of resources for mirroring, so kafka mirror maker and kafka mirror maker 2 because they provide the way to deploy the two different version of mirror maker. That today are in the kafka upstream project so for doing mirroring across kafka clusters.

A

And last but not least, we have a kafka rebalance resource which is used for doing a rebalancing on the cluster, so moving the partitions across the the the cluster through the brokers uh in order to have a more balanced cluster and not having uh heavy loaded brokers compared to some other brokers that are not uh hosting partitions. So they have no traffic from the other clients. So it's a way for rebalancing.

A

The cluster using the cruise control, which is supported by uh streams, say that uh I would like to hand over to jacob in order to show you everything I just said in order to see how all this kind of stuff works, using stream, z,.

B

Thanks paolo, uh let me share my screen and uh I'm here already connected to my uh burnities cluster and in this demo I will really show you that deploying kafka streams is fairly easy if you ever worked with the operators before you know that the first thing which we need to do is to deploy the operator so to deploy streams operator. I will uh use a script from our website, so https streams, dot, io, slash, install latest and my namespace is my project.

B

Equals here and that will install the operator for me, uh the installation really consists of all the usual files which you will find in installing applications on kubernetes.

B

It's the custom resource definitions uh for the custom resources mentioned already by paulo, it's all the different arabic files to give the operator the access rights, and then things like a service account and then last but not least, it's the deployment which actually uh contains the operator. So if I do now cube ctl get pot, I can see the operator is already running.

B

There are also other ways how you can install the operator so on our website or on the github page. You can download the yaml files directly to install it from there. We have also a helm chart. That's your preferred way, and then streams is also on the operator hub.io. So you can also install it easily uh from there.

B

So you can really choose whatever fits you best now, if you know the operators, you know that the next thing which I will do will be to create some custom resource for those who don't know that much about operators, yet the custom resource really extends the kubernetes api and allows us to add some own resources.

B

In our case, I will be using here kafka resource and when the resource is created, then the operator will see it and it will make sure that it deploys the kafka cluster accordingly and it doesn't just do the deployment and installation, but it will also make sure it is managed.

B

While it is running it keeps running and it can take care of things like certificate renewals, upgrades to new versions and so on. The custom resource really serves basically as a blueprint for the kafka cluster. This one is one of the more simple one. It really just specifies the version of kafka, which we should use number of replicas.

B

It configures the kafka listeners where the clients will be connecting some configuration, it configures the storage and then because kafka depends on the zookeeper as a kind of dependency to bootstrap the cluster and do some hard beating and leader election. Then we need to deploy zookeeper as well. So zookeeper is part of this resource.

B

Also, now what I will actually deploy is uh a bit more complex custom resource, where I specify some more things, such as how many resources should the brokers get?

B

uh I configure the encryption and authentication and authorization to have security in the cluster and then I can also enable, for example, the prometeuse metrics from here, and then I do the same for the zookeeper as well, and I deploy also something what's called kafka exporter, which helps with metrics and all I need to do to actually deploy the cluster is to apply the resource and the operator will basically immediately see it and it will start deploying the bots.

B

So what we can see here is that it's already deploying the zookeeper cluster, so we can see zookeeper ports, 0, 1 and 2, and the operator will wait until the zookeeper cluster is ready until it is bootstrapped and prepared to work, and only when that happens, it will actually move to the kafka deployment.

B

So you can see that the zookeeper pods are now getting ready, and now it will provision the kafka storage using the storage class and persistent volume claims, and we can see now the kafka pots starting as well and again, it will wait until the kafka cluster is ready and available, and then it will deploy the entity operator as paolo mentioned.

B

We have also custom resources for users and topics and that's what the entity operator is responsible for and, as you can see, the whole process is quite fast. uh It really depends a lot on how fast your internet connection is and uh how quickly the container images will be pulled, uh and so on so now we can see that kafka is there.

B

The entity operator is now deployed and when that is ready, it will deploy the kafka exporter and in this demo, I'm really showing just the deployment of the kafka cluster, but more or less in exactly the same way. You can also deploy the kafka connect or the mirror maker or, for example, the http bridge.

B

So the entity operator is now deployed now, just the kafka exporter.

B

And once the kafka exporter is ready, then basically the whole cluster will be deployed and available and we can check that by doing cube, ctl get kafka which shows us that we have this. My cluster kafka resource with free kafka notes and three zookeeper notes, and we can also do get it in yammo and we can see that these are, for example, the addresses where the kafka clients can connect.

B

This one is using load balancers from the outside, for example, from my laptop. This one is from applications running inside the cluster, and then we can also see that the state of the cluster is ready.

B

So now we have the cluster running and we should deploy some application right. So, let's start by preparing some user for that application and the topic and that's where we will use the entity operator. So for the user we have the kafka user resource.

B

In my case, the user is named, my user and inside this resource I can say that this user should be allowed to use tls authentication, and I can also because I'm using the authorization. I can also specify the authorization rules.

B

So, for example, in my case, I give it the right to send messages to kafka topic, which is called my topic, and I allow it to consume the messages from this topic as well and again, I just apply the resource that we created and because we selected uh that we want to use tls client certificates for authentication, then I can check and there should be a secret called my user, which is the same as the username which was created by the entity operator and which contains the authentication details for connecting to for authenticating to the cluster.

B

And when I check the details what's inside the secret, uh we can see that it has basically the user key and user certificate which I will use for uh authentication.

B

And if I would choose some different mechanism, for example, uh uh scramcha for using username and password, then the secret will contain uh the password for authentication instead of the certificates.

B

And so this is when you want to manage the users locally. But it's not the only option which we have. You can, for example, use for the authentication oauth using oauth tokens and manage the user centrally in your oauth server, and you can use all also for authorization or, for example, we have also an option to use open policy agent for authorization, so that was the user. Next. We need to create a topic for that. We have a resource called kafka topic and uh I'm great at creating unique names. So I call it my topic.

B

It will have 12 partitions and three replicas and I can also add the config section where I can specify all the different details for this topic and uh again I just apply resource to get the topic created, and the great thing about this is that I do not really need to remember all the commands from kafka all the tools, all the options, all the parameters and execute them somewhere.

B

But I really just have this custom resource in yaml, which I can use and then, with the topic and the user ready, we have to deploy the application right- and I have here really just some simple hello, world producer and consumer which run as a deployment. uh What's the nice thing is that, uh because the entity operator created the secret with the credentials for the user, I really just need to mount the secret as a volume or use it as environment variable, and I can use the certificates from it for the authentication.

B

So I do not need to copy them anywhere. I can't really just use the the kubernetes api for this and when I apply it.

B

We can uh check that the pots are starting and we can check uh in the lock of the consumer that it is actually receiving some uh messages.

B

uh As you can see, it's really just a sample application, so it's really just sending every second some hello world message but yeah that shows uh that it works and uh the great thing about uh having the kafka, user and kafka topic resources is that when you develop your application, you can basically take all these resources, the kubernetes deployment, the kafka user and the kafka topic.

B

You can, for example, store it in the github, together with your application and then, for example, use some github's tooling to get that automatically deployed and it can travel with the code together with your application uh through the test, environments, production and so on, and you don't really need to have some complicated installation guide book which would explain how to create the topics or the users in kafka, and I also wanted to show that while streaming makes it easy to deploy and manage the kafka cluster, it's not really just something for playing locally.

B

We really support the features which you need to run production clusters and we have, of course, users using streams in production. So, for example, what I get automatically out of the box are things such as prometheus monitoring. So I can see here in my grafana dashboard that I have free brokers. I have 186 partitions.

B

I can see all the different metrics uh right now, of course, with the hello world application. The throughput in the cluster is very small, but you get this all built in including the kubernetes, uh the grafana dashboards and some sample alerts, uh and uh things like that so uh uh yeah, that's it for this demo and uh back to paulo.

A

So, thank you very much jacob. It was a great demo showing what is the kind of kubernetes native experience for handling the kafka cluster right, so uh jakub showed us uh some of the features that stream z provides. But uh stream z has a lot of features, so you can see here showing up different uh features like scaling up and scaling down easily. You can just change the number of replicas on your kafka resource.

A

You can even use affinity and toleration in order to define the nodes where you would like to deploy your kafka brokers instead of other pods. You can even enable the encryption so having tls, as shown by jakub with clients, but even internally, between the brokers.

A

You have acl sendling for handling the rights for writing and reading from the topics for the users. As you already mentioned, integration with cruise control kafka connect, the http breach, mirroring uh secrets are used for storing the credentials and the certificates.

A

You also have um metrics from all the components so from the the the operator itself, from kafka from zookeeper kafka connect and so on, and we provide some grafana dashboard that yakub showed about showing all these metrics through prometheus, of course. uh So these are a bunch of features or part of the features that stream zero course provides today, but other than uh the built-in features.

A

We can also say that stream z really integrates pretty well with some other uh cncf projects like, of course, kubernetes uh also with the helm, because we provide helm charts for deploying z. There is, of course, the integration with prometheus, exposing the metrics and showing them in grafana fluency, for example, a support for kafka that you can deploy on kubernetes using stream zeo chords.

A

There is uh integration with jaeger and open tracing for uh for doing some tracing so tracing uh so getting some tracing information about the messages exchanged between clients uh through uh kafka topics and even integration with the opa, so the open policy agent for the authorization so that you can specify and describe using opa.

A

um What are the topics that a consumer can read? The producer can write and so on, and there is also integration with kida, which is um pretty new in cncf today, and it's a project for event, driven auto scaling application. So can you can auto scale your kafka based application using a kafka scaler provided by kida, and you can of course deploy your kafka cluster alongside kida using streams. So there is a great integration with all this kind of project in the cncf family.

A

um At this point, what's the way, a kind of call to action for engaging with the community with the upstream community in stream z? First of all, you can just go on the streets website and download the uh stream z, try it locally on soundcloud provider. So on your kubernetes installation, you maybe can find some bug, running streams and try and seems it. So it will be great submitting the bug.

A

So you can open a github issue in our github repo, explain about the bug and if you have a fix, it will be great that you could even open a pull request on the github repo so submitting your code, but you can even submit your code if you have some new ideas around stream z. So you want to propose some features to add to streams itself.

A

You can also help with the the documentation so um not developing stuff, but uh writing documentation, improving the documentation that we have today. If you find some typos.

A

If you would like to make clearer some part of the documentation that you didn't get well while going through the documentation itself- and you can even help translating the documentation in your language so for making streams easily available uh across the world in different languages, you can even join our chat room. We have a room in the cncf slack workspace.

A

You can of course um follow us on twitter. uh We have a stringy twitter account where we tweet about a new release: new blog post uh new features um in strimzy, but you can also help us spreading the word so talking about streamsy and maybe the project where you are using streams or some ideas that you have around shreemzy at conferences at meetups or even writing articles or blog posts. And if you don't have a place for blogging, you can of course blog with us.

A

So you can contact us and you can write your article on the official streamsy website in our blog so say that what's the way for reaching the upstream community, uh of course, there is the official website, so streams dot io there is a stream, the organization under github, it's an organization where all the components in the stream the ecosystem live so like, for example, the operator of course, uh the the http breach the oauth library, but even some client uh examples for for interacting with the kafka cluster uh deployed by stream z.

A

The twitter account that I mentioned, and even the streams room in the cncf select workspace. We also have a mailing list in order to interact with us just sending an email.

A

So I guess that that's all on our side. I really hope that you enjoyed this session and that you will go and try streamsy, and we will be really happy to to to know from you from having some feedback for improving this project and maybe engaging with us in the future. So thank you very.

B

Much thanks for joining our session. uh I hope this was useful for you. uh There are a lot of questions, uh so I will try to go through some of them.

B

If I don't manage to answer your question yet or if I don't get to it in the live q, a then in 30 minutes, we have mid maintainers session where you can join and we can continue the discussion there, and you can of course, also ask in the in the slack channels uh as well, and uh we are also running uh streams survey. So uh if you are interesting to help us know more about your requirements and missing features and so on, then please go to our website there.

B

You find the link and you can fill it in.

B

So let's have a look at uh some of the questions which we got. uh First of all, there's are two questions which are a bit similar. One is uh whether there are plans to replace zookeeper uh and another. One is uh uh whether there is any initiative for unifying common dependencies like zookeeper, which are used in a different project.

B

So maybe the kafka perspective is a bit different. So if you follow the apache kafka, community zookeeper is being slowly removed from the kafka project and should be not needed in the future and since strimsy focus on apache kafka, then we will basically follow that up and uh as it is removed from kafka, we will then use only kafka and not need zookeeper anymore.

B

So uh uh that way we will just upgrade and that would not deploy zookeeper anymore and I think the current time expectations are maybe around end of this year or beginning of next year. When this might happen in the apache kafka project, then another question was: uh did you try to run it uh with istio? uh So I personally didn't run streams with istio, but we have users uh using strimzy together with istio strimzy right now doesn't kind of tightly integrate into istio and run as a part of his steel.

B

So the users who are using this boat together. They always configure steel to kind of exclude strimzy uh from uh the uh from kind of the istio side, cars and not uh use them to do all the traffic and instead kind of rode, the traffic to strimzy and the kafka brokers as to an external application. So you can run it together, but it's not fully running uh strimsiest part of istio.

B

Another question was uh whether there is uh any roadmap for all the different projects and features of strimzy. If you go to our github organization, there's a github project which is used as a roadmap, where you can have a look and uh yeah. If you think we are missing something, uh then we can definitely edit uh right now in the stream z, there's a lot of work going on around the ui, which would allow to manage uh strimzy and kafka itself like the kafka topics, consumer groups.

B

So that's, for example, one of the big things uh coming in in the future, uh then another question was: does streams also include components such as asr rest, proxy kafka manager?

B

So we include some components, but not all of these. We have our own rest proxy, which is integrated into strimzy.

B

B

Have support for cruise control for balancing the cluster and that's all built in and, as I said, we are working on uh on the ui and some kind of rest api for also managing the things a bit more. So not all of these are really built in, but some of these are and the rest can be usually connected uh really as yet another kafka operator. So to be honest, uh you do not always need operator for anything. Some of these things are just easier to deploy as a regular deployment.

B

That, for example, includes uh schema registries.

B

uh Then another question was uh how would one perform performance tuning on streamsy, for example, number of disks which can affect the performance uh or, if you can specify on which types should the brokers running? So you can definitely do it because strimsy supports all kind of different uh environments, starting from the kind of basic mini cube environment which the users can run locally up to the big public clouds. These are not always enabled by default, but you can, for example, use affinity in the configuration to define on which nodes should the brokers be scheduled.

B

We also support the jbod storage, so if you want, you can specify that you want the kafka clusters to use multiple disks, and we will then just create this disk and mount mount multiple of them into single port. And that way you can kind of use more storage or get better performance from the individual brokers, and you can also.

B

Use most of the different kafka configuration options to tune the performance, so the operator itself managed some of them related to things like tls and listener configuration, but all of the different buffer sizes and number of threads and so on. You can easily configure it yourself, and that was the last question which I managed to answer.

B

uh There will be a threat on the slack uh where you can continue asking the questions and, as I said in around 25 minutes, we have me the maintainer session so feel free to join there as well, and we can continue answering the questions there.

B

Thanks for coming for our talk and q, a.