Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A] - Nenad Bogojevic

Description

Kafka Operator: Managing and Operating Kafka Clusters in Kubernetes [A] - Nenad Bogojevic, Amadeus

In this talk we will demonstrate an approach to management of kafka clusters in kubernetes deployments. We will show how we can provision kafka clusters and configure it using kubernetes concepts and an operator process. The kafka and zookeeper cluster elements will be provisioned using StatefulSet. As these applications benefit from high performance storage, we will also show how we can use node selectors or persistent volume claims to schedule instances on correct hardware. In order for clients to use it, the necessary message topics have to be configured in kafka cluster. We will show how using an operator process, based on kubernetes custom resources or ConfigMaps we can manage this configuration in descriptive manner and ensure consistent configuration across different development and operations stages as well as cluster restarts. Finally we will discuss how all this ties in with service catalog.

About Nenad Bogojevic
Nenad Bogojevic, platform solutions architect at Amadeus, has 20+ years of experience in software development. He has worked on e-commerce applications, natural language processing tools, and high-performance network middleware. In his job, Nenad is an architect who codes, a technical lead and speaker, open source contributor and an evangelist for application platforms, online and web technologies. Currently, he is driving design of new kubernetes based platform for new data driven services. In his spare time he travels, manages small home clusters or builds and programs robots with his daughters.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

My name is Ned beverage and album sharing with you how we run and manage kafka clusters in kubernetes, a tomatillos and amylose is provider of IT services for travel industry. So for those of you who came by plane and sting or hotel, you might have used be using our services without knowing we're celebrating 30 years this year, which makes us much younger than our biggest competitor, which is a texan company.

A

We have been running kubernetes since its inception, so since three years, which is basically 30 years, equivalent in kubernetes years on our own premises and in public clouds, I'm solution, architecture Madhu's, I'm helping our business units, I'll developed, new platforms or migrate and 30 years we have but the number of applications- and today's migrations are over almost always towards kubernetes.

A

So we use Kafka, we use Kafka outside kubernetes. We use Kafka next to kubernetes. We actually did it since we started using it for logs and for events collection.

A

We have been installing it using puppet, which is an experience we don't like to repeat, and recently we had this idea to start using actually Kafka for more functional stuff, and we are building this streaming platform where we have a number of events, operational events from the airlines or things like bookings or boardings which go into the platform there's a whole number of micro services which process this in the pipeline and then certainly some actions executed at the end and use Kafka is underlying messaging infrastructure. So it's supposed to be advanced session.

A

So I guess you all know Kafka to be on the same page choice. Kafka, it's a streaming platform. You have a cluster of servers and servers are called brokers. Storing streams of Records in topics.

A

Topics are split into partitions, which are spread over all brokers, which allows for horizontal scalability. You can add new brokers, a new partition and accept more traffic. You can replicate those partitions to have a higher ability if I'm broker dies. You have a backup on another and you have producers and consumers as clients of this platform can so can we run it in kubernetes?

A

Well, when you run a normal replica set or you do deployment in kubernetes, you get something like a pod, which is called with a random ish extension at the end, which is not fits well with Kafka basic in Kafka in a calc, a cluster, each broker has its own unique identity, which is both an ID, but it also has to have your own unique network address so that brokers can talk between themselves and also that they can clients can talk with brokers. They also need persistence to store this partitioned log files.

A

In addition, you don't you need another thing. Next, a Kafka cluster is a zookeeper cluster, because Kafka stores a lot of metadata inside zookeeper, which is basically exactly the same thing. You need another cluster with each other identity and persistence, and luckily we have now stateful sets in kubernetes. Actually when we started those were called pet sets they evolved and they will soon be exiting better.

A

So what is a stateful set, unlike traditional pods, if we can save something three years old, traditional is stateful set, pods have stable Poland entities, so they will called pod zero pod one was two: they. You also need to create headless service, which is a kind of sub domain within your namespace, so your pod address would be pods erode some domain.

A

If it's the name of the service dot name space, whether they provide stable storage, they provide orders, start up and start down or shut down, so they will start press 0, then 1, then 2 and shut down in reverse and since recently there is a rolling gap, it's so to run, Kafka and zookeeper. We need to state process and actually know medias. When we rank Afghan zookeeper deployments, we always run one Kafka and one associated zookeeper for those who operate Kafka. You know you can share zookeepers across many several clusters.

A

It's not what we're doing two bridges we're deploying one one-to-one and in addition, there is a discovery service. So, unlike headless service, which doesn't have any cluster IP, the discovery service has the cluster IP and it's actually allows clients to just say: okay I want to connect to Kafka, you would strap them, you don't have to tell them go to Kafka zero, we tell them go to Kafka and it will fall on one of the brokers and then learn about for cluster.

A

And of course, if you want to deploy things in kubernetes first thing that you need is containers and you need your descriptors and then you need probably things like I want to facilitate this and to replicate and to do several times deployment. So you things like charts. So there is a number of projects out there on github I think most are inspired by Owens kubernetes Kafka. Those involved are the ones which are inspired by what how we operate inside the Madhu's.

A

You have a chart and you have the operator so I'll be trying do small demo, so I hope the demo gods are on my side today. It worked on plane coming over here, so the first thing I will do. I will deploy to a producer in the consumer.

A

And, of course, we'll see because I haven't deployed Kafka cluster, they will probably fail, probably so they'll failing as there's no Caprica. Their nose knows nothing in the cluster. So let's delete those two and I would now deploy class. Who you a time ideas. We are actually running or open shift as October neatest distribution, so we're using openshift templates to deploy here, I'll be using home charts.

A

How is package manager for Benitez and it's a great way to reproduce your deployments or to have multiple deployments if you want in an environment, so I will here deploy my cluster.

A

And she takes just a few seconds and we'll see what hang will deploy all the elements necessary, so several stateful sets and services that are needed. So let's have a look. First, we have our staple sets up there, two of them Kafka and zookeeper I, requested three of each and we have our services that were so mentioning just before. We have a headless services with no IP, no cluster IP and we have a cluster IP services for the discovery. There are few other things that are deployed here by talking about those later.

A

So while it spins them up and just share a few of the practices we use actually in on ideas when deploying Kafka Kafka is pretty much disk. I/O and network performance per capita disk I/o network base, so we will actually want to land Kafka Brokers on the instances which have good performance disguise. So basically it means SSD, and we use note selectors for this, for all the brokers have known selectors that have want to deploy them on the nodes, which has have a label disk fast or something similar. But we also want to do something.

A

Is that we don't want it all our capture brokers, land on the same node with this label, because if we lose that node, then we lose all of the cluster. So we are using an T affinity, just featuring kubernetes, which allows us to tell you, take this a pod which have this specific label and spread them across some topology key, and here we are using a host name G. So we are saying basically two kubernetes. When you deploy these Kafka brokers, please make them across different machines. We are using preferred, you can also enforce it.

A

You can say it has to be on different machines.

A

The other thing which is necessary for Kafka is persistent storage.

A

When you're using stateful set, you can use a volume volume claim templates were actually specified, a kind of volume claim you want and each part would get exactly the same volume claimed type. So you have a six part. You will have six different claims and for those of you who were reading this license and of listening of me, it's written things. They are completely different because that's a common wisdom when you're running Kafka you want to keep your logs. You want to get persistent volumes, you get its provisioned its attached to your pod.

A

It's stored, your pod dies. You will get it back again, our particular case. We are building a streaming platform which has pretty strict isolates from the moment that an event comes inside. It should be all the action should be taken within a few seconds at most a few minutes, so we can keep the amount of data fairly limited on the brokers and we want this high performance, so we could use host host path, but that's not good security wise.

A

So what we actually do is we for each of the pod in a poll set with two PC okay use empty volume, it would be local disk, it will be an SSD. It will be very fast if the container crashes we get actually the same empty there and all fine if the node crash is what we live with it, but we are running Kafka and one of the selling points of cafes. This replication, you have a copy of your partitions replicated.

A

So when a pod is spun up on a different note, eventually to get up-to-date in sync with the current leaders, and it will be able to serve it, of course, to be able to do it, you have to have enough brokers, and you have enough replicas, so yeah five book, brokered cluster and two replicas well, you can afford to lose. One of the brokers have to wrap, because you can only afford to lose one brokers.

A

If you lose to you, you might be in trouble what's coming soon in a queue, Benitez is actually in alpha already is local persistent volumes, which would be volumes which are on a local machine, and the pod would then be always scheduled on this particular machine. It's something that we'll be looking into future in future.

A

Only currently behavior with the empty gear was sufficient for our you use cases you have to monitor. What's going on, you won't have there several approaches called to monitor, Kafka, consider also in github. You may use Kafka scripts. What we actually do in a me do is we use TCP socket, who actually will open the sockets on the character brokers? Because that's what's telling us it's running, it's accepting connections and we use Prometheus NJ mix.

A

We are we're repackaging ourselves containers. So basically, we are deriving from what has been done in fabricate projects which, for everything which is JVM automatically exposes primitives and J mix entry points. So you have the nice dashboards and if, when the operators needs to do something, they can go directly and connect to the pod and look into the gem X.

A

That's all diving into operators. Let's have continued with our demo power. So.

A

Deploy again our consumer and producer and.

A

See what happens there? Oh it's not working! So well. Actually, it's connected. It's no longer a big section. You should Java exception, we'll see there it's connected, but we are running in a multi-tenant environment, so we have micro services which are published there. So we have dozen of the themes worth publishing those independently and we want actually to control who connects to do which cluster. We don't want that. Okay, you go there just that Kafka, 1992 and you're connected to a cluster, and you start publishing.

A

We want to use to identify clients and we are actually before I have installed as it's a process. We create separate secrets and there are secrets published for each of the cup clusters which allow clients to connect, so there's a jazz file inside it. So let's do it. Let's deploy the secured version.

A

A

Eventually, it will hopefully.

A

Yes, it's there and it's not running it's still failing, but it's a different error, it's failing with unknown topic, because if you want to run things with Kafka, you need to create topics, and there are two ways: basically how you do it. You can do it by saying. Okay, anyone can create the topics which basically reads to Caltech thing.

A

In there you have a hundreds of topics or you may use Kafka scripting Capcom scripts to do it so either there is a person who is typing or you have ansible or whatever tween you use to create those topics which brings us to the subject of the operators.

A

So what are the operators? It's a pattern of transposing the domain knowledge of essary operations or releasing teams into executive opcode to automate behavior space on some kind of descriptors you describe what you want to have and the tooling will do it for you, anima dos. We actually love all operators. We use couple dos there's open source primitives, the views the blue ones are actually doing is that we have written and some are open sourced already workflow and some will be are like Redis, cluster and I'm twisting the doctors do. Is they provision clusters?

A

That's what primitives are, for example, does or that, but that's where our radius cluster operator does, among other things, as you see, I can actually fairly easy provision clusters using a hand, charts or open ship templates or apps with sort today at keynotes, and once the platform is up and once Kafka is running for us, it stays in place. It will be there for a fairly long amount of time. We can scale it up if there's a problem and that's usually what we need scale up, scaling down and evacuation.

A

If you have to do upgrades of the nodes and upgrades are tricky. Some of these things work because of the category applications talked about a little bit at the end. You have this question of topics. Is we're deploying dozens of micro believe doesn't even more of topics being created and for us we want to have the topics present in target environments when we deploy micro services, so we can start to mediate we're running.

A

We want to delete them if a micro service is not there, because, particularly they can be rearranged and for different customers want to have the same behavior in environments, in development, in QA, in production, but also across different production clusters, as we may have clusters running on our premises or in the public cloud, we want to be able to react on things like ok. This now is no longer having disk space, let's reduce retention time for Kafka and we'll actually want to deliver all this as a code.

A

So when developers actually finish their coding, they do the pull request. There's a container bill. We actually from their description of their project generally deployment Yama file and generate this descriptor of the topic, which then goes even kubernetes, and then there is a process in kubernetes, a cooperator which looks into it and applies it.

A

So how does it look topic as a goer? It's a config map for us it might be soon customer resource. It's a config map which basically says I want a topic like this name, which is the name of the config map, where the partition count with the replication factor. Letters, and maybe someone knew what how to configure more into details, the properties and whenever this config map is created, then in queue burn.

A

It is the operator, that's managing the cluster will create a topic for it and whenever a config map is deleted, it will delete it and it's actually exactly the same behavior as Service Catalog provision and provision behavior well intentionally map it, because we are going towards offering internally IT services as a Service Catalog. So let's do it. Let's create this.

A

Config map, so there.

B

A

Let's have a look at what's going on here, so have long, a little list here, oh and it's working. So as soon as the config map was there, topic was created and the publisher started working and the consumer started. Consuming can just show you what happened inside the operator.

A

So inside the operator there there's a lot, which basically says: ok, I've seen that you have did the config map and I created a topic for this config map.

A

So, let's find it allows us to manage all the topics, but one actually to go a step further there conscious about security. We want to control the access to the topics. We don't want that a developer hard-coded a topic name in the code that suddenly the micro service which is deployed, starts publishing to a topic which shouldn't be there, so we are actually using access control where we have.

A

Each deployment comes with a notation, saying: OK, it's an I'm consuming this topic and I'm publishing to this topic a bit, I'm not multiple, actually and about what the operator does it monitors this behavior this deployment and will actually apply actually choose. First, one: random user, assign it to this deployment, create a secret and assign the rights into Kafka to use it. So this part of the demo.

A

The plank here, yes y'all, and if we look at the operator logs, we would immediately see that operator has seen that there is a new deployment. They're called Kafka producer and has assigned a user for it and then the same thing for the consumer, lower and assign the user great. And if you look at the secrets.

A

Actually see that we can now have credentials created for our producer, and somewhere there should be like here should be credentials created for all for the consumers and those credentials contains users which, in this particular case for consumer, you can only read from one topic and producer can only publish to one topic previous use previous case, where, with the credentials it's it was, it could publish to anything.

A

Okay, a little bit ahead of at the time so which calf cop grades, it's the thing that we wanted to build since the inception inside calf cooperator during our work on this I think there were like three or four changes in Kafka that had to do upgrades and basically each of them had a little bit different scenario, which is like not really looking like something that's easily can be easily automatized.

A

So, for example, if they change into broker protocol, you might need first, when you upgrade to use the old protocol. So you roll out one upgrade. Then you change. Do the configuration change then roll the second upgrade, but maybe the the Kafka change storage format. Hopefully now they are one zero, so those things will happen. Will less you change, storage format, so you might have to first to update consumers, then go on on the servers. I had this idea like don't upgrade, but instead to recreate cluster.

A

So basically it means we have one cluster running the old version. We create a new cluster with the new version they're running next to each other.

A

We can publish actually to book, though, to both clusters, and then we switch it's kinda Bluegreen deployment. It comes with its own set of problems, which is actually wouldn't suggest that any more well more looking into automated izing that in the future, finding a way how we can do this, the first step simply and.

A

Performance as I said, Kafka performances dominated by disk I/o. That's what we have experienced and having a good desk, we're hoping for SSD but having a good disk.

A

Even if you have network storage, maybe it will work if a good network storage, then the second is by network and it's almost never by CPU or by memory fairly low. Even for throughput, like hundred K messages per second have issues, there's a couple of things which pure strange when doing tests thinks of some like sometimes Kafka brokers and Duke, a broker and the zookeeper land on the same node.

A

So it actually reduced quite a lot network throughput there, because they're talking to themselves on the same node, sometimes that actually even clients will land on the same node. So you might see that you have like twenty instances of a micro service pod and then two of them are having this super high performance and all the others are behind the same, so they landed actually same node as ask Africa brokers and they're talking between their self things to be ready in a cloud environment.

A

That's not everything behaves always the same and I think that would be it for representation. If you have any questions, feel free to ask.

A

Can you repeat order this.

B

Is your Kafka cluster zone away or how are you handling gauging.

A

Okay, so we are come, our clusters are a spread, so kubernetes clusters are spread across our are close out our infrastructure and we basically rely on the fact that they are they're not there. No we're realizing in fact that there will be split across different zones in our data center, but not we don't make them aware of the zone. They're, not right, there's nothing like haricot vert or things like that.

A

We have our, we operate our own, that data a data center and we will deploy to water clouds but prepared on datacenter yeah mm-hmm. Okay, you can say it I repeat:.

A

The question was: do we have any constraints, a number persistent volume we create per node, as I said, when we run Kafka, we are running them with empty there, so we don't create persistent volumes. We are not using much persistent volumes at the moment we use for some of the monitoring tool. We used it for the Kafka, but and but hadn't put in place any specific constraints there.

A

A

The question was called to a handle of cooperator upgrades, but it's tough cooperate is completely stateless, so it's actually upgraded out of the band. It can be updated in any moment in the time and actually all the updates it does it does by Delta it's like kubernetes. It's checks. What is Kafka cluster, what we're having kubernetes and applies the Delta.

A

Yes, yes, it's watching changes, undercover. The question: was it? That's tough culprit or listen to the changes on accompaniment.

A

To access the questions do provide a way to get to Kafka from outside class. Sir meaning clients are outside of the class know.

A

If someone has the solution for that, we were very interested in ok.

A

The question was: did we write the operator, the Java client for the kubernetes, with the Java client for the kubernetes and Java client for the new Java client for the Kafka admin client?

A

A

It's didn't repeat or.

B

So confluent suggested two wrong Kafka cluster on bear motto for performance reason. What do you think about that? Okay,.

A

Our approach is that we tend to run as much as possible things in VMs and then, let's run into kubernetes, maybe we will be running to bring to some bare metal, but currently we are running on the new VMs. We had the same exactly the same suggestion from confluence to run at on bare metal, but we went for a VM I think there was a sinner yeah.

A

It's needed for a date. The question was: why is this headless service is needed, its need for stateful set it by design. That's how staple said behaves you need to provide an applet service and then the pods will be in with the DNS name, which is pod name, state, Harless, service, tandem, I. Think.

A

Our in this platform are typical. Cluster would be five brokers, we have bigger ones, but I'm not sure I can't share formation.

C

A

Can either come or because I really.

C

The order of deployment is that a true keeper has to come up and running before you deploy. Oh yeah thanks.

A

We don't have force order of deployment, we actually I, don't think I put get port Alex. Let's do just that. You sit here yet actually seeing that the kafka zero restarted once I was in starting because it's waiting was, it said, there's no zookeeper Andrew twist it crashed and we started again. You.

C

Are asking the Kafka to try.

A

And recover until the zookeeper is up and running I mean just speaking to the mic. You.

B

Put the link up of your github operator yeah: do you have any other custom code that you used to run it or is that kind of the everything.

A

That is needed now, it's not what we run inside house and that's your question. What.

B

Do you be able to describe any of the the customizations that you had to do? We.

A

Have few customizations around security of mostly which are inside it's pretty much a similar thing also, we are less up to date with the recent version of kubernetes and openshift inside house. Compared to this one.

B

So you don't see any restrictions on running the published version and so.

A

Does actually, we are in talks with some open-source players that to make this really open source web eff doesn't get anything, we will be open sourcing it from am at your side, but and if there is to be a community, we think there should be someone who is more into this panel. Wonderful, let's talk.

C

A

Can you just repeat Oh.

C

What kind of stories do you use when we run it on public cloud.

A

So we don't run I say that we don't run this with persistence, volumed at the public cloud. We use instances we have which have asses these even on public cloud.

A

Can you describe your general strategy for scaling down.

A

No scaling down.

A

General strategy to scaling down would be a lot of manual work by moving partitions and replicas and then scaling down, and one of the things is that usually system may be, like usually most of the problems that we experienced were due to the humans, so when the humans scaled it down without taking steps, but they have to do before so it would be really great if that can be automatized questions. Oh thank you very much. I hope you have. There are nice.