Cloud Native Computing Foundation Kubernetes Community Days Bengaluru 2021, 10 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Zero downtime deployment on Kubernetes

Description

Kubernetes Community Days Bengaluru'21

Kubernetes allows a lot. After discovering its features, it’s easy to think it can magically transform your application deployment process into a painless no-event. For Hello World applications, that is the case. Unfortunately, not many of us do deploy such applications day-to-day because we need to handle state. Though it would be much easier to have stateless apps, and despite our best efforts in this direction, the state is found in (at least) two places: sessions and databases. You need to think about keeping the state while stopping and starting application nodes. In this demo, I’ll show how to update a Spring Boot app deployed on a Kubernetes cluster with a non-trivial database schema change with the help of Hazelcast, while keeping the service up during the entire update process.

Slides: https://www.slideshare.net/nfrankel/zerodowntime-deployment-on-kubernetes-with-hazelcast

A

Hi folks, thanks to be here for this talk about zero downtime deployment on kubernetes with hazel costs. I am nicola frankel. I've been a developer for a long time or an architect or a team league. Whatever have you, while on technical roles and since a couple of years, I'm a developer and project I'm by far a kubernetes expert, but we live in interesting times. There are a lot of things to learn and kubernetes is among them. For me, I work for a company called hazelcast.

A

Hazelcast has two products. The first one is an in-memory data grid and you can think about an in-memory data grid as distributed data structures. So you can short your data. You can replicate your data across nodes in the cluster, the other one is hazel, cost chat and it's an in-memory stream processing platform, and I won't talk about it further because I will use them in my demo.

A

So the first question is: why shall we do zero downtime? Well, the first reason is because business wants it. uh If you are an e-commerce shop- and I know this is a pretty popular example um or overused.

A

um Well, you want to sell, you want to sell all the time, so downtime means first, you don't sell, and during this time your customers might buy to another shop. They will probably wait not that much time they will probably decide hey. This shop is closed. I will go to another shop, so it has a direct impact on the business and the second uh reason is a side effect is because, um like the big companies out there, the google, the facebook, the amazon apple, like the big ones, they are always up.

A

Their applications are always up so that users, who don't know about it that starts to expect that application are always up. So if google's applications are always up, why yours are not. I mean they don't understand the difference, people who work in I.t. We have been working in it for for a couple of years and yeah. We know how to deploy with downtime. We go to the proxy.

A

We set a static page, uh we're saying: hey, please come back, but the big ones that they deploy all the time they they don't do like that, and so people get used to that. And uh if you cannot compete with the big ones, then probably you will have a problem of image. Saying hey! Why are you doing that?

A

And if we are talking about zero downtime, I'm sure you are pretty familiar with the blue green deployment model. So the idea between the blue green deployment model is you've, got your production environment and, let's call it blue and meanwhile you have a similar environment.

A

Let's call it green, but for the moment the users they are directed to this blue environment. So at the moment this environment runs the version one of the application and at some points, business or in an automated fashion, decide hey, let's deploy something on the green environment, the v2 and when it's ready the users that can be directed to it and the next step, of course, is we let's deploy the free on the blue environment and again, let's switch between the blue and green all the time. So the prerequisite is we.

A

We must have those like two similar environments. There is an alternative to that uh here. We have like one single database or like one, that's a bruise per each environment, and I will come back to it later, because it's it's pretty significant.

A

It has a huge impact and it's a real problem when we do blue green environment because everything is related, every issue is related to states, and it's not only about blue green. It's about when you want to do zero and down time you need to have like to handle states and set is found in two places like in the application themselves in on the like the application nodes, probably user sessions.

A

um Even if we don't want to use user sessions in general, we use them because it's not feasible every time to go to the database or to the adapt server or to any data store. So when the user logs in, for example, you're, probably happy to store his first name and last name in the session and then display it in every screen and of course there is state in the database and kubernetes came and proposed something very interesting, the rolling of that principle. So it's like different from blue green.

A

Let's we have a single environment, we have different parts and we will like upgrade each node, sorry, each pod in turn, and you can configure how long it can take. You wait until it's ready whatever, but this is the best principle and again still it doesn't solve the step related issues. It doesn't solve sessions problem, it doesn't solve the database problem on the session uh problem. I believe it's a solved problem like we have session replication.

A

We have been doing session replication since ages, just because, if we store states on a specific pods and the part goes down for whatever reason, even if we don't want to deploy something well, we need to keep the session data, so we have been doing uh session replication since ages. I I won't go in detail over that.

A

Just in the demo, it just works and you can check the documentation, but the real issue is is on the database side and um we have actually two options that I mentioned. We can keep the same database and that's the way I used before and I I became like aware of some problems, and so let's go over this option and then we will see how we can have two different database and, like that's a much better choice.

A

So first option keep the same database. It seems like it's. The best id, however, there is one requirement is that the application needs to cope with two versions of the schema. Why is that? Well here we have three pods and we have one single database port and let's do a rolling upgrade with kubernetes. So if we just do a rolling upgrade first, we start with the first part, and then we go to the database and, of course, the old 1.0 parts.

A

They cannot cope with the 2.0 version, because there is no way they could have like inferred, the future and brags. So the idea is: oh, it doesn't work like that. The idea is that the two dot version pods needs to work with a 1.0 schema and it's completely possible. So let's do that when we do the two dot version, when we develop the 2.0 version, we say: okay, if we add this schema version, we do this. Otherwise we do that.

A

It's okay, it's a bit more work, but it works. So we just like do the rolling upgrades on each part in turn, when they're all ready, then we can migrate. The database version and everybody is happy unless we need to roll back because, yes, we are all very good engineers, but sometimes for whatever reason we need to roll back, and in that case we cannot roll back because as soon as we try to roll back well, it's the same problem as before. Again we have this 1.0 version that cannot handle the 2.0 schema.

A

How can we do that?

A

Well, let's have a very concrete example: again: I've been working in e-commerce, I'm using that e-commerce example a lot- and we have this like stupid, schema customer has multiple card lines and every card line has one product.

A

This I started as an architect. I started with the schema. My application become very popular, and now the business comes and the business as requirements as always and now what they want is they want to keep a track when the card was created and then after some time has passed, there will be a job that checks every time and say: oh, it has like 24 hours. The person has kept their cards, so they actually like didn't order, but we still have the cards and we can send them a reminder and tell them hey.

A

We have this discount for you. You forgot perhaps uh to check out your card. Hey here are 20 of discounts. Please buy. I mean this is a pretty normal for business, and so we need to redesign our application and for that we introduce now a new court table with a dedicated last modified attribute that will keep the timestamp when the card was last updated.

A

Okay. uh How can we handle that, uh like from the application point of view? Well, as I mentioned, it's a breaking change, so there is no way that your application can cope with this breaking change, so we need to handle the forwards and backwards uh migration, so the forward migration and the backward world bank.

A

So the idea is that we would need to split the braking change into a series of of changes that are compatible side by side. So we can always do a rollback if, if necessary, so how can we handle it for our e-commerce use case? The first is the first step would be a let's create the real chords table and the app still uses the old data model, and now, every time we do a change on the chord line.

A

When we add a chord line, for example, we will first check if, like the court's line, exists for this customer, if it doesn't, we will create this card and we will attach it to the customer and and this way, if, for whatever reason, we need to roll back well, that's not an issue. The the the database schema will have this card table that the app doesn't choose anyway and when I say trigger it, can be like a real trigger, because your uh database uh software can handle trigger it can be through the code.

A

It can be through anything. I I'm not prescriptive in that. The idea is just hey like when a chord line is inserted. We check if the first cast line chord line for this customer. If it is, then we create a new chord in the chords table.

A

A

The second step is now the cards table becomes the source of truths, so the application uses the card table and again there is a trigger that updates the caught line with the customer id every time it serves it now. If we need to roll back, the card line table will still be used, but it has the customer id. So it's it's fine.

A

The third step is, we will migrate, the new touch data and that's where the problem lies and the fourth will be. We will be doing cleanup. So we will remove the additional.

A

The additional customer id column in in the cartline table, so why is the migration of untouched data a problem?

A

Well, if you've got a couple of lines that that's fine, because when we are doing this like migration, we infer that people are still using our shop, but perhaps there are people who had like like chords, we were not touched and those needs to be migrated to the to the new schema as well.

A

The real issue is, it can be a lot of data. It can be a really really big lot of data, so it's going to take some time and during this time that means that uh we we need somehow to to have like the application still working. Meanwhile.

A

So it's this third step is where the real issue lies.

A

And it's not only that, of course it will require you to like you compose this change into steps.

A

Might not be that easy every time so that you can and you you can roll back and and after that, so you have this breaking change. You will like split it into non-breaking changes compatible side by side, but you can still only roll back one single baby step. You cannot roll back the whole breaking change, so you can just only roll back one step.

A

It will be needs, it will need to be planned across the whole organization. It impacts developers, it impacts uh like database administrator, it impacts the ops and probably it will impact the manager who needs to plan. So it's a lot of organizational uh light efforts and, as I mentioned, you will need to migrate data anyway. If it's a couple of lines, that's fine. If it's a lot of lines, they're, not so fine, might take too much time, and this decomposition of step is fine. It can be planned if you have like a small organization.

A

That only does it once, but imagine that you need to do those changes and they need to be deployed in parallel. It might not be possible at all, so the the second option is hey. We will need to to to migrate data anyway, so at least at least we we need to embrace it. So, instead of having a single shared database, we have the blue database and the green database.

A

So how do we handle the migration and that the real a game changer here is? We won't use jobs? We won't like start a bad job and end it. We will do streaming, meaning that we will get the changes as soon as they come and as soon as they come, we will replicate this change into the new schema and you might have heard changed the capture before and that's how we capture the change from the database site the problem of the database site. It's it's not an even source, so change data capture.

A

We will change that sql database to a different source that we can stream and the benefits is that the developers they are not impacted at all. It's completely parallel to how they work, and it's not actually like related to blue green deployment. It can work with any deployment option you can choose like for canary release, so change data capture, as I mentioned, is the way that we can like make a normal sql database into an even source. So every change is tracked. Every insert every delete and every update.

A

So here we have uh like our starting architecture and the starting architecture is actually. We have three parts of the same deployment, so pod v1, pod, v1, pod v1 and they are using the blue database.

A

The next step is, we will set up the green database. So it's just the setup, nothing happens, it's just hey. Let's create the schema and the constraints and whatever now the third step is, we will create a data streaming job that will read through data capture from the blue database and we will forward changes to the green database.

A

So every time there is a change in the blue database, it's replicated in the green database and because of like the data streaming world, we can do some transformation, because just replicating would be very, very, very easy, but here we are doing transform. So we adapt the changes from the old schema to the new schema.

A

Of course now we can now start to do the rolling update and at some point one of our v1 code will become a v2 port that is configured to use the green database and it has all the data it limits.

A

After some time, all the nodes will have migrated to this light green database so that the floor job doesn't need to do anything because well, there are no more changes and the difference with the previous, um like approach with one single database and you need to migrate into edge data, is that before job it migrates data that has been changed in the past.

A

So it's able to read stuff that happened before it was started, and so that is actually the definition of untouched data. But the rights are still there.

A

Here I was too lazy to do it, but you can think about you need to design a backward job that go the exact opposite in case. You need to roll back and if you don't need to, then of course afterwards you are happy to do some cleanups. You remove the job and you might keep the blue database for bits more just to be sure and then, after what you you do, the cleanup. So everything is fine and how can we implement it um again? As I mentioned session application, it's a sort of problem.

A

You can use hazard cost for that in memory data grid. It's perfect! It's a perfect use case in the demo. I will be using spring sessions because hazel cost is a spring session implementation for the database migration.

A

I will be using hazel cast jets for change that I capture with the help of the easing and, as I mentioned hazelcast mdg, allows you to store your data in memory so that session replication becomes an issue.

A

On the other side, we have hazel cause jets and hazel cause jets. Is a stream processing engine. It can read from different like different sources and write to different targets and meanwhile it's distributed and you can do transformation enrichment. Whatever you want.

A

So here, in that case, uh this is my design. I have like this jet cluster and I have the blue databases. I will create my job and it will use. Is the hazel question jar which uses internal division.jar, which will in turn use division mysql.jar, because the besiem is actually an abstraction layer that you can use to read from different uh databases postgres here, I'm using sql, but so that in the end I could use like nearly the same, the same codes to do it with postgres.

A

So I could even add, like a mysql blue database and the postgres screen database. That's not an issue, uh I believe I've talked a lot and now it's time for some demo.

A

So here I have everything in intellij and at the end of the talk I will give you a slide with all references that you can check, and this is in a github repository, so you can use it out of the box.

A

I've created uh an application and this application is a spring boot application. But I don't want to go into too much detail. It's just a spring boot application with a pump and how does it work? Well, it works like the following.

A

I have deployed already uh everything like the blue version on kubernetes, so I'm I'm using a kind. I have been beaten in the past by by mini cube, so I hope this demon works today. The first thing is, I will need to log in so I don't want to implement spring security. I just want to keep it simple. So how do you log in into this application and very important?

A

You cannot do anything. You cannot check your cards. You cannot browse the catalog unless you have logged in. So how do I log in I log in by like choosing a user here, so here I have user joe and now I am like authentified logged in as joe and I can browse the catalog and I can put stuff in my cart. So here you can see in the right hand corner I have like displayed the node name and the version of the application just to like show you how you can like how it works.

A

So let's say that now I log in as jack and jack has nothing in his cart. So I will add, like one t-shirt- and perhaps I will add like william- I will add like two hoodies for william and I will add a real. I will add three stickers. This is just to show you how we will migrate the data, so I need some data to start with.

A

Let's get back to joe and add like a couple of of um marks here. This is our initial state. So if we go and check on on kubernetes, I have aliased the cube ctl to k, because it's much more comfortable and I can check the parts here and normally I should have have four ports. It takes a bit of time. I have the blue database, so I have a deployment of one pod, one replica for the database and I have a deployment of three pods for the application.

A

The first step is actually to deploy, as I mentioned, the green, that that is the green environment, so I will apply infrastructure cube green. So this is the just the um database, the new environment so get paused w. Normally it should be already up. It's very, very quick.

A

So yes, it's already running and we can check the logs okay logs green dash f, so my sequel is starting and we should wait a bit until it tells me that everything is fine. Shutting down my sequel d is good, it's ready it. It starts very, very quickly. So now we have a new mysql instance. We are very happy.

A

The next uh step is to actually deploy the four jobs. So, let's deploy the forward job. Okay, apply, f, infrastructure, cube forward.

A

Okay, get parts w, so it's already started and k logs forwards.

A

F, that's very interesting! Actually you can see that it already has started to replicate data to the green database, like from data that was there before it started.

A

So now on the green database, you already have the exact exact same states as on the blue one, even though we just started the four job and if I just like go back to the shop- and I add one more mark.

A

We can see that instantly the four job cuts this change and updated the quantity on the green environment. So now, every time we do a change, it's captured, transformed and forwarded to the green environment.

A

So the next step is obviously to do the rolling upgrades, which is the least problematic stuff, so k, apply, f, infrastructure, cube and application.

A

A

And here k get parts w we can, we can start to well, we can continue shopping and the idea is that at some point magically I will be on a v 2.0 pod. So here it's a rest call, so I will need to refresh uh it won't, be um it won't be visible in into the into the right upper right corner might take a bit of time.

A

So sometimes my codes well, not sometimes my front-end skills are pretty pretty bad, um so it might like squeal a bit and I will need to refresh the page, but it's only because I don't know how to write really correct javascript, um so it it's still doing its job.

A

And it's taking some time, of course. ah No, I thought it was so meanwhile, I'm keeping um an eye on the terminal and normally at some points. This like pods, should get killed.

A

And replaced by a new part.

A

Let's see if it has been killed so uh hey, I am on that already on the 2.0 version and it was like you saw nothing and, of course I can continue now adding new stuff.

A

So how does it work? Well, there is nothing magical. Actually, um the application is not that interesting. The forward job is a bit more interesting.

A

We can see the source and we can see that it's a plain normal java application, like it, has no dependency on spring at all. It's completely like agnostic to the tech stack of the application. It's independent is just like a plain main, like application.

A

So here you start it with main, and here we can see a view of the pipeline and I won't delve into the detail of what the pipeline is. Basically, the pipeline is the code that you send to the jet cluster to get executed as a job, and here we can see that it's read from the blue environment and it maps the record, because here we are using records and we we don't need to have this class itself. We just transform it to a map and we map it further.

A

Afterwards, to a call parameters, because when we write the green environment, what happened is um what we are doing is. Actually we are calling this like function called insert card line and what this function does is just a call to a stored procedure.

A

So, whatever, like technology, you are using uh you, you can do like gdbc, you can do gpa, you can do hibernate, you can do whatever you want. In all cases, um it will work like it's just like plain simple java codes.

A

uh Yeah, you can see it here insert card line and it's a callable statement.

A

Well, I believe uh I'm at the end of the time that is allotted so um takeaways uh for you. I believe that zero downtime, if within everybody, is rich, which was not the case before I believe before I told people hey. Perhaps it's it looks like really really hard, but with this approach, anybody can do it on one side for uh likings data, that is on the application.

A

Pods session replication solved problem, just configuration and works, and on the database side you have change data capture and data streaming and it works like a dream.

A

Thanks for your attention- and you can read my blog- you can follow me on twitter. As I mentioned, you can check the git repository. Everything is on git, so you can do that at home and um if I got you interested in hazelcost, you are welcome to join our slack thanks again and have a good.

A