Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Demystifying Data-Intensive Systems On Kubernetes - Alena Hall, Microsoft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Demystifying Data-Intensive Systems On Kubernetes - Alena Hall, Microsoft

Distributed databases, stateful stream processing workloads, caches, and machine learning frameworks often require persistence for storing data, operation progress, and more. Managing state while running systems like Cassandra, Kafka, Spark, Redis, or Tensorflow on Kubernetes is different than with VMs or physical servers. Let’s examine why we might want to run these systems on Kubernetes, and look at foundational Kubernetes concepts (e.g. Stateful Sets) that help us get those systems up and running. But up and running isn’t always equal to operating correctly. We will go over best practices for managing data-intensive systems on Kubernetes, existing challenges, as well as solutions (e.g. CRDs, custom controllers, operators) and a possible future. You will learn about operational things to take into account even if you haven't worked with data systems systems on Kubernetes before.

To learn more: https://sched.co/GrW8

A

My name is Leena Hall I work at Microsoft on Azure and today we're going to be talking about data intensive workloads. So what are first things that come to mind: databases, caches stream, processing systems, any systems that work with state and data.

A

Another type of systems that we're going to be talking about today is not necessarily stateful systems, but those that operate in a way that isn't fully covered by predefined kubernetes abstractions.

A

So it can be any of the popular distributed systems like Cassandra or Kafka. They definitely need persistence. Word can be systems like spark that operate mostly in memory, but still require persistence for certain scenarios like persisting trained machine learning model, or it can even be a system that you have built that might need to store State and manage state. It might have very specific lifecycle rules of operation or need to provide certain guarantees, like consistency, no data loss and other similar things.

A

So just a quick show hands who here feels like you, have more experience with data systems than kubernetes okay about sixty percent I. Think interesting, so majority of information on how kubernetes is used today. It reflects mostly stateless services, but in some cases, however, it might make sense to run something more complex on kubernetes and there is a big difference in technical process of how we run.

A

Data systems and more complex systems and kubernetes versus simpler, stateless services and I know that we have a very wide range of audience here. I have looked at the attendee profile and have seen engineers, Lucian architects, engineering managers, CTOs open-source project, maintainer, x' and others. That's why we will address the topic from various perspectives. First, we're going to talk about making a decision whether or not kubernetes is good for your systems about the whys, the risks, the benefits and challenges, then we're going to cover existing solutions, how to run data intensive systems and kubernetes correctly.

A

And finally, we will talk about the future. What we can expect, what we can already start preparing for decision making is always fun. So if some organization with similar technologies is using kubernetes, it doesn't always mean that it will work as well for your use case. They might have more resources or expertise. Their risk acceptance barrier could be lower than yours.

A

They also might have different goals so when making a decision on whether a kubernetes is or isn't a good system for your use case, there are quite a few things to take into account what issues and challenges you would like to solve that you have in your current environment, where you are running those data intensive systems which of those challenges do you think kubernetes will solve, but then what new challenges will it create that you would have to deal with? How much will it cost? How much benefits will it bring?

A

What changes will need to happen in your team in your project and organization, for it to be successful and what are the benefits of choosing kubernetes to run your system over going with a managed service? So it's good to have answers to those questions before making a further decision, because really only you can answer them.

A

So if you're wondering what are some of the reasons, companies are choosing kubernetes, they're, Manning samples, of course, someone to achieve better workload, portability for their systems or to benefit from convenience and flexibility of deployment operations, automation for creating new resources or to get more independence from various vendors or various providers.

A

Some running to get faster, startup times for new resources in environments for cost savings, but some companies are choosing kubernetes because they're already running kubernetes for their existing services and they want to use it for more of their solutions. A lot of projects find extensible and open api. The kubernetes has very useful for building solutions on top of it.

A

They also prefer to trust it because of open and rapidly growing ecosystem, but on the flip side, there are, of course, a quite a few challenges: some companies by being a situation where there are no available solutions that satisfy their scenario and if they decide to go with kubernetes, there would sometimes have to build their solution almost from scratch, because there are no suitable options to gain new skills to troubleshoot new environments and to spend a lot of resources on training and time to support it.

A

There are other challenges like stability and reliability of existing components integrations with various systems and many more, but the one thing that I can actually say with great certainty is that if something in kubernetes ecosystem was unstable or it didn't exist a few months ago, there is a big chance that it's fixed by now, it's really a fast-growing ecosystem.

A

So how do we determine possible risks and challenges? To do that? We need to understand what converters can and can't do what it is and isn't responsible for. How do we determine if kubernetes can satisfy our guarantees that are required by our system? We need to learn abstractions and instruments that kubernetes and its API have to guarantee or implement our system requirements.

A

So the big takeaway here is before we can make the right decision. We have to get more specific and take a look at details. So let's do that so you're all are most likely familiar with at least several built in kubernetes types like paws jobs, deployments, staple sets, services and more those and many more our core kubernetes api objects or types that we can use to build our apps when new data people are discovering kubernetes, they see stateful sets and it might seem like a good solution for running their data systems on kubernetes.

A

But if we look a little bit closer stateful set is just one of the many predefined kubernetes types. It supports certain predefined functionality like replication, stable, pod, startup ordering predictable named for pods, stable links to persistent storage volumes, but nothing else really. It is great for some use cases, but is it really enough for all of the data intensive system scenarios?

A

In other words, what if stateless says, don't really work for our scenario? Our scenario might need more specific data, distribution or life cycle rules that are not implemented on the level of stateful sets. So what? If none of the core kubernetes types work for our scenario? The reality is yes built in kubernetes. Abstractions might not be fully sufficient for running more complex distributed data systems and it comes down to application specific things that need special care objects like stateful sets are powerful, but they alone have no idea what's different between a Kafka or Cassandra cluster.

A

Even if they have special operations, I mean common operations like scaling.

A

It doesn't have any knowledge if things should be managed differently, unless we teach kubernetes how to treat our systems and take pronounces specific to a stateful application under consideration.

A

So kubernetes simply doesn't know all of the specifics of internal operation of all distributed systems that we might want to run on it. The definition of operating correctly is different for many different systems. The good news is that we can still run many of the complex systems on kerbin Ares correctly, because there is a way to do so with customers or definitions and custom controllers. So, let's make it more concrete and look at a few examples.

A

Let's take cluster scaling as a first example. Many clusters can support scaling, so, let's think of it as if I took off cluster for a second. If we were running it using stateful sets and decide to increase number of brokers in the cluster. A stateful set would add. A new replica pod attach a new persistent volume to it issued a new, predictable name. It isn't actually enough for correct operation of the kafka cluster, adding nodes to a kafka cluster correctly.

A

It requires a few things first of all, manually assigning some partitions to new Brokers because by just by itself, new pods don't automatically get assigned new topic partitions, so we need to take care of it. Another important thing is cluster configuration. New pods will not automatically be updated with all necessary configuration values.

A

Another example: our cluster restarts, which is relevant to many systems, but it's very specific to each separate system, so for Kafka restarts need to be done carefully. So what is it meaning to have a good restart? We need to make sure that there are no under replicated partitions before doing this, and we need to make sure that the cluster is in a healthy state.

A

Then we can start the restart operation and do it one broker at a time each time making sure that the the broker is caught up to the leader, so also a graceful shutdown of the each broker is very important for this operation.

A

Basically, these are just some examples. What could go wrong if we don't implement it correctly and because kubernetes is very extensible and open, we can implement new and missing functionality really easily. We can use customers, definitions and custom controllers and a custom resource is an extension of kubernetes api that is not necessarily available on every kubernetes cluster.

A

It can be added so after its added users can work with objects of the new resource just the same way for working with built-in core objects like pods, so we can think of CR DS as new kubernetes entities, new data types that we define that become a part of kubernetes api, but on their own custom resource, doesn't actually contain any logic. It just defines a new resource, but we can combine a CD with a controller which allows us to declare logic to say what is the desired state for our resource.

A

For example, we could create a new CD called a Cassandra cluster or a Kafka cluster, or a spark job or a tensorflow job, and then we can write a custom controller for our C Rd and specify the logic that defines what correct desired state means for operation over a specific stateful application. If we take a sander, for example, it could mean that the controller won't let kubernetes restart a replica pod until the replication of certain data on some partition is complete.

A

So just as each built-in object in Copernicus has a core controller behind it. Custom controllers enable us to create new logic, to manage our new cds to extend and add new functionalities. You might have heard of an operator pattern which is just a combination of a customer's or definition and a custom controller operators are interesting because they are important not only for container people, but they are also important for Kafka engineers for Kassandra engineers, for SPARC engineers, tensorflow engineers and basically for all engineers who want to benefit from running and managing their distributed systems.

A

Reliably consistently and correctly, when built in kubernetes resources are not enough.

A

So now that we know what an operator is, we can take a closer look at internal aspect of it. First of all, it's helpful to understand the notion of events in kubernetes. So an event is an action on some resource. It can be create, update or delete and those resource available to the queue and control a worker is going to block on a call to get the next item from the queue, so we can think of controllers as brains behind our resources themselves. There is another important aspect that is worth mentioning: it's called an informer.

A

An informer is like a connection between where those events originated from and between our controller. So what informer does it? Registers event, handlers for these three different types of events, add update and delete and inside event, handler functions of the informer we add resource key to the queue to let content to let controller logic handle those events appropriately.

A

So, let's take a look at an operator in action. One of the existing operators that I really like is a tender flow operator or TF operator for short, and it is part of a cube flow project.

A

Flow project aims to provide a convenient way of running tensor flow jobs on kubernetes clusters and this example actually runs on Azure carbonated service, but you can run it anywhere else if you'd like, because workload portability and I have already installed, could flow on my cluster. That means that the cluster is already aware of the new CRD and custom controller that could flow needs to run tensorflow jobs. So, let's take a look at the example.

A

The first of all, we need to generate the specification that looking used to submit new tensorflow jobs and could flow uses case on it to to do generation of that such as specifications, and here it it gave us path to the specification that we can look at if we copy the file and inside this file. We can see that this is some tensorflow specific parameters. Then we can you just use if we want to run the test job and if you notice that it has TF job kind.

A

So what we can do now is we can start I mean deploy this specification to the cluster and start the job using the apply command. So after this is done, I think it would be interesting to look at the CR D in our cluster, so we can take a look at what's here. Ds exists and we can even filter them using the prefix. So I know that the CR D is called T of jobs, and we can even look at more details of what it represents.

A

So it has a kind TF job. It has tender flow, specific parameters and properties. For example, replicas can be of different types, like chief parameter, serving eval and there's a lot more information here, but now, let's actually check the status of our tender flow job that we submitted so and we're working with it. The same way, we would work with the built-in object, but behind the scenes it uses parts because it is implemented that way, and we see that there are two pods behind the scenes, parameter, server and worker.

A

If we look in logs behind the worker pod, we will see that our total flow job is running and it has it is using ResNet 50 model. It is in training mode and it worked so once the operator is running, it watches all of the new API requests related to the TF job resources and uses the logic defined in the T of jobs, controller to manage tensorflow jobs based on life cycle rules.

A

So let's go a little bit deeper behind the scenes and find out how the TF operator actually have job operator actually works, and the code for it is online. It is really interesting to check it out. The link is below so github.com. It could flow /tf operator, and this is the CRD customers or definition example.

A

You can see that the kindest is TF jobs and on the flip side, there is also a definition of the type called TF job, which is a struct that we're going to use when controller is going to work with the job. It has two fields: Joe specification and job status. This is important because the controller is going to compare those values and make sure that the job is in the correct status that we specified in our specification inside the job specification.

A

There is a map of replica types and replica specifications, because we know that one job can have many replicas and each replicas specification defines well how many replicas of a certain type there are what template is used for replicas and if you were being a session that you could see that the pods pod template specification is used here and that's why pods are being created behind the scenes. When you're writing your own operator, you can choose your own abstraction or several.

A

If you want to use it, and also it has a restart policy for the job as well, for the replicas, and here we have additional types to define types of replicas that we're gonna work with restart policies that we're operating and job status, and here is an interesting party.

A

This is an informer and it basically watches the events, and it is like a connection between the source where they come from in the controller- and this is the most important part of the operator, which is the controller itself, and it has a function, called reconcile, T of jobs that checks and updates replicas for each given replica specification. So in case something goes wrong with the job or the replica. It will Riku the job or restart the replica and make sure that the status of our job is correct.

A

So knowing exactly what operators are, how they work, it's useful to know how to build one if you would like to- and you can do it manually, using code generators for the client and former Lister and writing your own controller. But you can also use things like operator, framework bike or OS. That can actually significantly simplify the process so check that out, if you're interested and then next a reasonable question comes to mind. Now that there are operators, CR, DS and custom controllers. Where can we find all the operators we need so well?

A

There are a lot of existing operators. There are open source on github, for example, a Cassandra operator from insta cluster or tensorflow operator. They are all open source and in the list some operators are actually closed source or in preview by various companies, for example, check out confluent operator for apache kafka and some other operators simply don't exist. Yet so remember that the CRD concept is relatively new for other than kubernetes people, and there are a lot of people who work in data industry that are not familiar with go and don't know everything about kubernetes.

A

So it takes a bit of time to learn and start creating widget operators. So if you'd like to check out more of the operators, there's a list of on github as well, so you'll have to type that link, github.com, slash operator framework, slash, awesome operators and another important note to add.

A

There is no consistency as for how you can use existing operators, for example, some operators, some operator creators choose to use helm as a main mechanism for end-users to work with operator some use case on it's like you have seen, but not not to refer tensorflow some people build custom CL eyes, so there is no one single way everybody uses to work with operators and by the way, even though we're looking at operators from the lens of data and consist incorrectness.

A

The same applies to other aspects of distributed systems like managing security, customer lifecycle, workflows and other important things. So before we talk about the future, let's, let's summarize some takeaways from this section, so especially for solution, architects and engineers, knowing how to build kubernetes objects, how Doulton kubernetes objects work and how operators work.

A

It will definitely help make better decisions for when kubernetes is and isn't the right tool and will help make sure that you're running your data systems and communities correctly versus just running so, and this brings us to the next point- you might know everything about CR DS and custom controllers. But if you don't know what correct operation means for the system that you're running, then there won't be much success, because this is one of the reasons. Current data systems related operators are being developed through collaboration between kubernetes experts and particular domain distributed system experts.

A

So now, let's talk about the future majority of operators for data are in extremely early stage right now, but we are registered, seeing big progress and there are new operators coming out one after another and in the future there will be more operators that will continue to improve, be more stable, provide better functionality and, as more people start writing operators, we will see further progress on making them easier to write and kubernetes is really one of the most extensive all systems for running other systems. On top of it, it's also becoming really easy to add.

A

New storage options and the quality of performance storage options is getting better. Local volumes is one of the examples we will see. Our industry become even more focused to work on better workload. Portability, for example, cloud native application bundle or C nav is one of the examples of this work and we will observe the continuing shift from low-level abstractions to more focus on being effective.

A

Writing our applications, so things like meta particle and draft, for example, bring our attention back to code and Helmand Brigade help automate many routine operations, so kubernetes already runs on any cloud on-premises. It allows us to add missing functionality and with its powerful foundation layer.

A

It is already on its way to becoming a platform where we run or coordinate most of our systems, because starting with open service broker as a gateway for managing cloud services and resources from kubernetes continuing with operators as a beginning of something that we might call managed service, but not tied to any cloud and now looking at things like cross plane for multi cloud control plane all this means that we'll have more freedom to decide where and how to run our applications without sacrificing benefits of using alternative systems when appropriate.

A

So, based on what you have heard and what you have tried, I would like to learn about your experience with data intensive systems and kubernetes with the purpose, to better understand your goals, requirements and challenges and make it easier for others in the industry. So you can use the link here bit. Dot ly slash data k8s to share your thoughts with me. If you would like I would appreciate that and some resources, if you're curious, to look at my blog if you're interested as well.

A

There are my previous presentations and that's me again: I'm Lina Hall I work at Microsoft. If you want to reach me, I am on twitter at lean android and if you want to talk to me, I'll be around here outside of the ballroom after the talk and also you can come down to Azure, both in the expo hall. There are a lot of my colleagues who can talk about interesting tools and interesting problems and I'd love to chat with you and I'm, also on the program committee for cough. The summit Kafka summits next year.

A

So if you would like to submit I would really encourage you to do that.

A

Ml for all, is a conference that I organized about machine learning with a few other people and if you're into your machine learning, you can check it out. It should be in Portland next year and the last thing just remember that you can do powerful things with kubernetes and keep that in mind for your next project. Thank you.