Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: eBay Geo-Distributed Database on Kubernetes [A] - Chengyuan Li & Xinglang Wang, eBay

Description

eBay Geo-Distributed Database on Kubernetes [A] - Chengyuan Li & Xinglang Wang, eBay

Database as a Service is one of the most interesting and challenging domains on the cloud industry. In eBay, we implemented a cloud-native geo-distributed document service based on the kubernetes. eBay extended the kubernetes to support local disk volume on bare metal machine, which enables the high performance DB can be deployed on the kubernetes as a Pod. On top of the kubernetes platform, we develop a control layer to orchestrate the databased pods and enable it can be distributed on multiple cluster, and expand the WISB model to use a workflow to auto manage the database cluster.

About Chengyuan Li
Chengyuan Li is a member in eBay Kubernetes team, his focus area is host-runtime and storage in Kubernetes. Before joining Kubernetes project, he worked in computer and network area for eBay cloud.

About Xinglang Wang
Xinglang Wang is a architect in eBay Data platform, he is working on ebay next generation geo-distribute database, and his main focus is the distribution and control layer of the database. Before he is the architect of ebay real-time behaviour data pipeline, focus on real-time stream processing.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello, hello, everyone could afternoon this session, we were going to talk about the eBay Co distributed database on culinary's. We come from eBay first, let me introduce myself. My name is the single I worked in the eBay data platform team, actually I'm, a kubernetes user and another community developer, and this Autopia included two parts in the history. Today's agenda so first ever introduces a eBay geo disability database, where I will share the experience of how we deploy such a database on the kubernetes. The second part of my colleague of Chania, please from ebay community.

A

We will do a deeper dive as a storage management in kubernetes at the ebay. We did a few extensions of the storage apart on top of the kubernetes, and this is the core causes to support us and running the database on top of the kubernetes.

A

So first, let me start from the lesion of the EP IOT trillion to the base. As you know, ebay, our vision is, we want to people to shopper any item from ebay from any place anytime, while any device. So we have the requirement that the one to the data close to the user. So that's why we develop as a Tod database and the vision of the these databases.

A

We want to provide a resilient color, the native and a cost-effective data platform to supply eBay close global business, so we want to eat, are always available and also is, can be highly scalable and to support as a business guru and also attention to be high performance. Since we deployed our cloud, the native environment, so we also want our operation agility we want to everything, can be fully automated, otherwise it cannot survive if we don't have this automation, our cotton native environment. So this is our vision of our CEO to achieve a database in eBay.

A

So now, let's look at how it looks like so. Actually our distribution tott database idea is quite a similar like other, but this really the base, so why we want to dispute it in the base is just that the dealer said is too large what kind of fitting you one node, so distributor actually have two meanings: one. We want to split it. The logic data set to a small chunk of data set in this case. We call it a shot, actually, it's a shot or ruin G of a Kieran Xia and physically.

A

We call this replica set. This is a duplicate in our database, not the same as the because there, the counselor in the Canaries, it's quite a similar, so each shraddha actually contain a small range, is just a small data set of the global data set and we was the second of the differences we want to. This data have a few copies and these copies will be distributed. Our multiple machines, different rooms. So that's the the logical view of a dispute.

A

Database Knology I have a graph to show how physically deployed in a real environment, so the database were deployed. Other multiple zones, unimportant layer record is the storage layer, the storage layer, just many a storage node. You should notice our mental machines. It's a physical Hardware in each storage node. We spawn a few database engine process on this storage node and in the Kaurava same color actually means the data copy is the same under the number I putting in here.

A

It just means a shadow ID of the of the data so logically Arshad, for example, the number to shut it will be in the first and second under the force node. Actually, this storage layer is quite a large. The scale at Oak, Valley pH will be assaulted, so these database engines, so how our application to access this database data to the reader right. Since we have thousands of the storage engines on the storage layer, we cannot a letter the application to connect to the other director.

A

So we introduce our service layer, this service layer to rot in the application requests to the storage layer database engines, so this service layer activated sister Denise, but how it can't do the routing. Actually we have in the service layer. You have a publishing map to know the layout layout of the storage layer. But how do you need to know this layout? We in the left side, we call our situation layer. We have a coordinator, this coordinated surface layer and a storage layer.

A

The coordinator active layer monitor all of the actually the main control all of the storage engines on the storage layer in a man, Jesus state and main since eternity and Intel has his own when the datastore distorted they stayed. Meanwhile, the coordinator also push this change to the service layer, so that will make the service layer can knowing our top notch layout the physical layout of a storage layer. The answer in a kind wrote in the request based on this topology map.

A

Meanwhile, we also have the modular system because sometimes for the distorted ending it may have. Some slowdown has some other problems, so this modular system also a monitor actively monitored its database engines in a storage layer, and it just send some signal in case. Something is a failure to the coordinator and the coordinating take some action. So now what I have talking about it says gesture for TE, aute, III database and how it relates to the kubernetes. So actually we deploy our whole things include the storage layer and also our insufficient layer on the kubernetes.

A

So on the orchestration layer and a service layer, actually it's a template and also as Diddley's. So it's not a big program when we deploy on kubernetes. The cool part is how we mention these storage things on the kubernetes. We know cognates is great, but is it easy to deploy such a complex little cloud? Our communities so answer is, is not easy. So now I'm going to talk about when we implemented such a database, we found our challenges in channeling when we deploy our kubernetes.

A

So the first one is that we are talk about a distributed database. We have a geo distribution, so typically we would deploy these Saints on multiple clusters. Typically kubernetes suggests our local availability, zoom or local data center. It's just one cluster. So first we press each problems. We have to manage multiple cogniser clusters. We need to distribute our database. Engines are multiple clusters, so once we deploy on the multiple classes, we know kubernetes raises their own internal network inside it.

A

You can have a good connectivity, but if you curse, multiple clusters, like I, have a database that has a three copies on two data. Centers. You know won't want to do some real time reputation, so that will have. A second issue is how we enable the network connectivity between the database engines about different classes, so their second problem of the network aside, since we are talking about what I want to do a high performance, a cost-effective data platform so but kubernetes by default.

A

If we want to persist in the data we use the persister volume, a possible volume typically is shared, remotest origin, but if we want to Intel high performance, cost-effective, listen, we cannot has fulfill our requirement because the remoter stipulates that means to slow it and maybe low IOPS and the share. That means that sometimes people can sum the process running our same storage. It will impact these hazard and the next one is. We purchase the data and we want to view data.

A

We want to control the data placement because we are running our database and we don't want to seem copy running our same machines so well, we are risk. So actually we want to know that data location. We want to control the data placement. In this analysis, we want to do some of my optimization. For example, when I wrote in the the request from our service layer, though, to our storage layer, we want to choose some closer data. Node then can minimize the latency, so the database is more than like a pet.

A

It's a very complex pet. We need a more control of all things next. Why is that? In the database we mention the data. It are not just likely or deployment. You have a police car. You create another new product to edit with the deployment we are. We not only need a creator port. We also need a move- the data from somewhere to restore the data, so the changes we need. Some data lifecycle management either include letter dated back harbour date, a movement that between few database engines.

A

So this is the main challenges we faced when we try to deploy such a workload other communities.

A

So now, let's look at how we reserved this issue so ie base. Actually we have a little bit other words that would render community so I just saw the seeing light, the container network interface. So this maybe can also support the community. We were dealing in eBay, so you eBay, we assign it supporter as a global IP, so that will makers can different that cognates cluster, the product of the network activity. So we have a controller who my schedule pod.

A

It will assign a globe ID, and this color by P is global, wrote a book between the different clusters.

A

So so this will reserve the network connectivity issues and the next we found if we want to use a very high performance, rather high, IOPS and also very dedicated volumes. We found that only the local disk or the disk adjuster may be on same rep, can be mounted like icicle see these local scenes.

A

We can achieve this high our peers and are also dedicated without any impact for other neighbors. So this by collaboration with our ebay kubernetes to the teams, we utilize the local theater as a positional volume. That means we can request the low position volume and, but actually it had returned a local district to this port and the meanwhile, our when we registers is persistent volume as local disk. We put our physical location info like the rec and a host inside as the labels for the local position. Volumes then can allow us in the application layer.

A

We know the location of this disk and to know which a record is and which hosts Livi's mean why we enhance the actually is also done by our kubernetes team. The enhance to the just part of any affinity features from the water level to the volume level, because when we schedule some things, we were maybe first I need a system volume. Then I schedule pod. So we have some special implementation can also support a volume level, any affinity that means I for force in shadow data.

A

I, don't want to this shard the two positional volumes not eating as simple or similar, so this will be cos, can get to the high FPS with mega or database high performance. Also, we can support the entry affinity. This is location in sense. A next one is we found over to the database database always need as a backup. If we don't have a bad habit, you know somebody's broken.

A

Even we have a multiple copy, but we don't have like the time machine capability to know the history of the data so that it back up is so they're quite important. For us. We have a solution to to utilize the local disk. This just like a device on the device level, would do the snapshot and the keeper sending the snap Delta to to sing credit to a network of volume and use this. You can listen to for our backup. They said.

A

Look at ISKCON, castilla, walk with the higher-ups, and then you just keep it doing the regular deltas and the remote volume, because if we just use the remote shared storage it you can maybe do the snapshot and the remotest share story. But since we now we use a look at this, so we have a addition. These features to enable this backup scenes, and now this is just a infrastructure level. We reserved the networking issue and also reserved the discrete issues, but we found it after we utilize this feature. We are not only.

A

We cannot use like a deployment or silver set, because this just means that kubernetes can control your total lifecycle. Now I cannot utilize because we have rare special things like whiny the controls and look at the disk, and the code must be together. You cannot utilize these features, so we so then we choose a solution.

A

We could just use the pod, we user communities to just imagine these containers and the other a plain we created our own, which is adjacent layer to maintain the wispy state, so that is our institution layer I mentioned in previous slides. Just we maintain our whiskey state so, but how we can maintain the whiskey's data, so first we have our own configure store to store our desired state. We also actively monitor all of the physical resources in the kubernetes. Then we compare the difference to to take some action built like a loop.

A

Let's we watch is a cop needs change and they take some action to fix to make the match our desired state, but this is different, like cognitivist always have even more like a first multiple data center because of multiple calmness clusters and the mechanism is being our own. Why? Our own data store? Also we utilize us, but since we call the workflow to maintain the wispy, because, like I said before, we have of you like a database data move plate operation are also cognizant oppression.

A

These are together, sometimes I I, do a create a new pod. I need to move some data, so not just like a real, simple operation. It has a few complex steps, so we utilize the workflow. We can listen to implement these complex things and the methane is a Swiss B state. Now here is a example for how I do the self remediation? For this example, I have a replica.

A

This is maybe the host Daystar kubernetes deleted this replica. This is in red and we are watching we're watching this event who appear to be deleted at this event, will be sent to our coordinator, and our coordinator will actually will have to level called coordinators. Each do have its own local one. This will avoid that one didn't die yet others, but the we have a logically global one. Actually, it also contributed our multiple data centers.

A

So this local, before the event to the caliber one and the global one, will make us a position and if your triggers are work triggers of whiskey, workflow, the whiskey of workflow, we found okay, I have one replication missing: I just send us a mess, even center of code to the local community to create a new replica editor, this sharp. So this is quite similar like a hug networks, but we just expanded around multiple communities of clusters.

A

So here's all my talk actually I just to share our base to the geo database and how we the changed, were facing when we move on communities and how we serve these challenges and I know the core part of our solution actually is based on eBay kubernetes. 30 management solution and I also think about this pattern is quite a typical. Maybe it can be extended to as a higher appears G ot printers that were workload. So that's all my heart ever forwarded to my colleague agenda for next part.

B

Okay, thank Seema. My name is changa, I'm from ebay commnity. My walking tour me is the own OS management and water management on the ebay kubernetes cluster. Next I'll give introduction of storage management on eBay kubernetes cluster. It's a support to geo distributed database. It can also support or at the eBay application as well- and here is the agenda of my topics here. First I will introduce our storage classes. Next I will introduce our local volume solution network Blackwell morning, routine backup solution.

B

Lastly, I'll give a simple summary of our future work: okay, storage classes in our cube next cluster, which you find to start class first day, is the performance it can provide 50k, LPS and also maximum 500 megabyte per second throughput. The pic of typical user case is an Oracle database or some other eBay in-house application. The storage back-end is local SSD on kubernetes, it's our local SSD volume ii. Starch class is a standard. It can provide the 300, PS or maximum 300 megabytes per second throughput, and the user keys like backup, service or EBC ICD.

B

That Cassie can show gonna introduce the yesterday's session. The storage back-end is cf HDD based trust, income. Dentist, it's ours, seen the volume.

B

Okay then go to the local volume. We started the our local worm work last year at that time, there's a quite many network volumes, Roshan income net, yes, but no local volume. We compare the network valium. We found that we still have quite many strong reason that we must increment a local volume first days. The performers actually I think is a quite common sense that local SSD performance must be better than network morning. I think, especially for the I/o latency. We ever get attached to that.

B

If we write a 4 KB block on local SSD, it's take about 200 nanoseconds. Why? If we write it on the cinder as Casa volume, it's take about one thousand, eight hundred nine nanoseconds, so so performers, local SSD, is much better. Second is, of course, we all know that commercial, hypotenuse, metro, network storage is very costly, so the reason is the availability. I think we most of us will think that one local SSD reliability is not good and compared with network volume.

B

Its availability is not good as well, but mmm you know, but actually I think nowadays, most distributed database actually management and the data ability by their own. They usually have several copies on different nodes on different disk axle. That means the data availability is guaranteed by the application level. So single SSDs reliability is not a big issue now, also at that time, kinetise only provide the empty dia and the host pass as the wait for pod to access local disk, but this to Varner also has quite obvious limitation.

B

So, first they don't support TB and PVC. Second, they cannot guarantee they are your size or IOPS. For instance, if a host paths share the same disk with the root OFS, the pod maybe can use all the disk space on the root disk root root as FS.

B

So as at this reason we start to implement our own local volume measuring we'd. Do these three changes in kubernetes code to implement our local volume? The first party is the local bottom plug in. We implemented this program to support hard disk and hard disk partition. This local volume plugging is similar, as all are the kubernetes volume providing.

B

We implemented their major local major volume in interfaces like a mount or a mount or provisional and deleter, and so we readjusted this worm plugging in the cubelet and the cube controller, so that it works as similar as there are other volume plugging in the kubernetes. Second thing we implement pv and PVC, so user can just create one PVC for the local volume and using this PVC in their poor spec, the same as the all other PVC for the TV crew issuing country with sabonis approaches that static of people creation.

B

So during the node provision we were generator can take a fire in this configure fire we will indicate which partition or which disc can be used as the local volume. So when cube litter starts, it will read this contradiction file and it will create the PV instance in the EC CD and also mounted ideas from the local on the host node. Just like this is to disk.

B

So we can mount them first on the host and when the products start, this to disk can be mounted in the Pradas namespace, so that this part can access the disk directory.

B

The last part is changes in the scheduler. The reason is that local volume, PV and PVC bonding is special. It's not seen as other Network, one on TV and PVC bounding. The reason is that when, whenever local volunteer and PVC abounded, it means that the node is also selected. This will break the later pod scheduling policies, so so the. In this reason we postpone the PV and PVC bonding in the scheduler instead of in the PV controller.

B

With your two changes first day is that we add, a local volume predict function so in this function we would check her every node it's if it can fulfill the requirements of Pradas local volume requests, and, secondly, is that the way in the kubernetes scheduler one function way to the final, a PV and PVC bounding, just behind the port 1 into the node. So all this reach, and just incoming edges for us to implement a local volume here is an example that how can we use in the local volume PVC in our production?

B

So we justify our local bottom PVC I. Think at the storage cross as local SSD, it's similar as the other PVC and in the parts back we just putted a PV name, a PVC name there. So that's it.

B

Here several parts that we ever considered during our design and the implementation of local volume. First things is still wear true to the TV Bundy I've mentioned before. If we need to do the PV and PVC bounding in the pot scheduler or in the PV controller, as mentioned before that, if we do it in the PV controller there, and we will predict a lot of pods in scheduling policy like our customer geo distributed base, they require the anti affinity feature and also they define the pod memory and the CPU resources. This kind of scheduling requests.

B

So we must fulfill these requirements, so this make us postpone the PV and PVC to the scheduler. Another thing is that some features some PD features will be. Implementation will be more complex if we put this PV and PVC in the PV controller and just like if one power requires two local volume PVC.

B

So if we do it in the TV controller, now it's possible that one PVC bun no day another PVC bun no to be there. How can this part of be scheduled? So as this kind of reason, so we decide to put this PV bonding in the scheduler, and the second point is that how do we recycle the local one? So when the PVC is dedicated, we also must delete the data on the TV.

B

This actually wait. We at the very beginning, we just start using the recycle pathway. So whenever we delete the PVC, we start a local, startup, odd and schedule this part to this node. So using this part to remove all the data on the specified PV at the very beginning, we are worried about its scalability, but we tested to 2 liters 3000 PVC at one time, and we found that this solution can still work well, so we keep it in our production still. And the third point is about disco, fader, monetary and the remediation.

B

We know that disc actually is the most unreliable device in in the machine compared with the memory asipi own network interface. So it's failure rate is high. So currently our solutions that we start a team instead pot and called the reader can yes, some real controllers, CTL, like maxi eye or smart city, L these tools to detect a disco failure and whenever we detect the face disco vidya and we send send out alerts for the remediation part country, we we can still manual.

B

We can only still to the manual remediation, and this may be, will be improved later.

B

Ok, then we will go to the network volume. I. Think networking room still plays a very important role in our communities cluster, one of the most important feature for the net. One is that it attached to a note so, but whenever the porter is the delineator or crushed, so if he also want to start a new pod or another note, the distant one can still be reattach to another node and the podcast still using this same network volume. This is a one of the most important feature of network in kubernetes.

B

Second thing is that I think when, when users are using the network volume, the the suppose, the auto data availability has been guaranteed by the back end of net one, oh right like safe, so they don't need a care too much about their application. How to guarantee the data availability and we'll make use a much thing much easier. So this is a reason that we we still need to keep the network volume in our production environment.

B

We using the single volume as our neck. That will warn a solution at we're beginning we're using the standard open stacks in the volume. But after one year's use experience we found that it has some problem. First is that it has too many depend depends on third party components. Second, is that it cannot support a pair mental. So as this reason we switch to using stand-alones in the provisional from from kubernetes community okay, so this figure shows two different resolutions: the left side. We can see that this is the standard, cinder price, Nova solution.

B

This is a kubernetes node, it's a VM. We can see how many components depends why it's one to do the volume attached attached you can see he has a queue: novel computer, rabbitmq, novel and cinder, so in furnishing so every two of these components. If they have some data drift, then will cause our bottom attach detach failure, its occurs quite offering our promotion.

B

That's the reason we switch to stand-alones in the provisioner, so by Odin cinders stand up along standalone, cinder provisioner. We can still that user to use in the sink the PVC interface, but when user create a standard, PVC interface. Actually, the standalone parishioner can help you create a PV or ask a C TV in the commonest side, so it has much less third-party dependency and it can improve network volumes attach detach reliability.

B

Okay, then we go to the backup solution. Currently, our perhaps ruching is based on scene, the cinder volume, our storage team safety has already implemented a snapshot controller so like here they have running a snapshot controller for each single cluster. So so why use a crater, a port and they create a PVC? And if this PVC is net volume and then this net one is a cinder cinder volume, so the snapshot. So if they want to do the snapshot, we can send a request to the snapshot controller.

B

This measure controller can trigger the cinder to do the snapshot and later when a client, if want to recover the data from the snapshot, they can send us requests to the snapshot. Controller to and the snare shops controller can make warf snapshot via volume in the volume and return the volume ID to the controller. The controller then returns the user ID to the snapshot right.

B

So when user gets this volume, your ID, it can create another pod by specify these volumes, your ID in the PVC and create another cinder volume PB, and this new pod can use this recovery date. This is our backup solution in kubernetes.

B

Okay, lastly, I will give a simple introduction of our future work first days that we need to align our code with our stream of local volume, especially tourist cargo up stream, has merged departure, aware bottom schedule of Pia, so I think most of our required feature of local volume has been upstream so later.

B

We need to do some work to align our implementation with up streams to implementation and the second part that we needed to do some disk and partition modeling during our not provision so so that the purpose is that we can support multiple hardware SKU in our production during the node provision, so that we can, during the provision we can using this module to specify which disk partition can be used in the local volume. It will be more generic a solution. The third point is that we will consider the volume for the local disk.

B

We want to take advantage of the powerful PI system ZFS and some of the feature of DFS maybe will help for us to enhance that for local volume, something like snapshot this kind of feature. Ok, so this is all for. For my introduction.

B