Cloud Native Computing Foundation ServiceMeshCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Replatforming a $4B Retailer Onto Kubernetes and Linkerd- Fredrik Klingenberg, Aurum AS

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Replatforming a $4B Retailer Onto Kubernetes and Linkerd- Fredrik Klingenberg, Aurum AS

To increase the pace of feature iteration and reduce costs, Elkjøp, the largest electronics retailer in the Nordics, migrated their application from Azure App Service to Kubernetes and Linkerd. In this talk, Fredrik describes how they built the new platform while rapidly onboarding themselves and their developers onto both Kubernetes and service mesh expertise, and how they addressed the concerns of their developer managers, who were happy with the status quo and uneasy about the proposed change.

A

Hi and welcome to this presentation called re-platforming, a four billion dollar retailer onto kubernetes and linkedin.

A

My name is freddie klingbot. I work as a developer in norway in a small consultant company, where I try to help modernize and move organizations and teams onto kubernetes. For example, this presentation will be about the largest electronic retailer in the nordics called egypt- nordic.

A

I was hired at the time by the enterprise cloud solution, architect, henry hognos and together we built and moved a significant part of edgeship nordic's applications over to kubernetes.

A

In this presentation, I'm going to tell you how we, as a small team, build the new platform while rapidly onboarding ourselves and the developers and also how using a service mesh was in retrospect a really good decision.

A

I also have some lessons learned. You can find me at the usual places like slack linkedin twitter, and I try to vlog a little bit as well.

A

So what are the goals for this presentation? What should you be left with?

A

So by listening to our story, you will hopefully get some tools that you can use on your own journey when building a kubernetes-based platform, you will get some ideas and arguments to buy. Investing in search mesh could be a good idea and learn how to onboarding on board an organization with respect to the developers.

A

We will also give you some tips and tricks on how to handle developed managers who are more interested in keeping things as they are.

A

So just to set the stage here, uh ship issue is a large company. In european sense, it consists of around 400 physical retail stores. It's a large online present and has over 25 market share.

A

It sells electronic goods and can be compared to the best buy of the best buy in these states.

A

So before the kubernetes transformation project, ailship had a micro service platform based on azure app services, it was stable, it was popular and when elsia started with a large modernization project, it was handed more load and also more services to run.

A

Obviously that increases increase the cost and operation overhead of the platform and having a microservice per azure app service.

A

um The management of those starting to get out of control, the azure app service hosting cost in 2020 was around 450.000.

A

Us dollars so, even though the azure app service platform was good, we wanted to get even better so that we someday, maybe yeah, becomes the best.

A

So not to give away the whole ending here, but after moving to kubernetes moving to the kubernetes based platform, we were able to get better scaling of the applications.

A

Because we can skip bring up those applications faster and we can use live utilize technology such as the kubernetes event, driven, auto scaling project called get on.

A

We got better performance as we could, for example, deploy applications that communicate with each other closer to each other.

A

It was uh a better developer experience where they now had full control over the dependencies. For example, the operations also got easier with prometheus and other visualization tools.

A

We also got more robust against disasters and had a better disaster recovery plan, but maybe the most impressive metric was the um hosting cost, so we were actually able to cut 75 of the hosting cost uh that is without taking into consideration the the savings that we get due to developer productivity.

A

So going back to the before the kubernetes project, like uh henry huddles, had the insights to see where this was heading and decided to get a kubernetes project approved by the lg board.

A

I was lucky enough to be hired as a consultant and together we started building the platform.

A

First off we thought that we could look at the cncf homepage for any tips and was pleased to see a link to a landscape page. So we thought great that this this this. uh This should be pretty easy. Then.

A

And then we see this, so it's probably not that easy. So we we quickly realized that we needed some guiding principles on what technology to choose.

A

So this is not an exhausted list, but maybe the most important ones. So, first off we wanted to embrace the aspect, aspect-oriented programming model and move as much as possible of the cross-cutting concerns over to the platform.

A

One such example could be mutual tls, so instead of having checklists and qas on the developers so that they would do certificate rotation and signing correctly, for example, we could just just let them get it for free by deploying the applications onto the platform.

A

We wanted to create a pit of success, and by that I mean it, should be really easy to do things correctly and but really difficult to to work against that pit to to do them wrong.

A

So this also ties into the aspect oriented programming model and that you should get as much as possible for free.

A

The biggest cost in software projects is often maintenance and not development.

A

It is good that the technology has a low learning curve, but if that means a horrible day to operations, then it's really not worth it.

A

So we're not interested in saving a week of development time in return for a year of extra maintenance time in terms of cost.

A

We wanted to introduce technology to solve actual problem without introducing an even bigger problem. So, for example, if getting neutral to less means a really big maintenance job of a service mesh, then it's probably not worth doing it.

A

We wanted to minimize the friction for initial adopters. It should be fun and easy to use our platform. uh It should be something that the developers wants to do, instead of something that they have to do.

A

And we also decided to try to be try to use a very clear language in terms of all the um or with terms that all the teams can agree about.

A

So typical example that you use when onboarding teams are words like applications system services, so we wanted to be very clear about the context and the meaning of those words. In that context, we also wanted to have everything as code or as much as possible, at least so. We had load testing as code. We had that system configuration as code alarms as code obviously have infrastructures code and, of course, all of this code should be version control.

A

We built up a sre team from linux admins along the way, as we were, building the platforms they were hand-picked and all, and we gave them a lot of responsibility on the supervision uh in the beginning and they really shined. It was a. It was a good decision, so it's much easier to do maintenance and operations now, as they have been a part of actually building the platform.

A

But during the implementation of the platform, with now an increased operation team in terms of size, we saw that we needed some better version control.

A

And this is where we introduced flux, so we used flux to manage the entire platform. We did not get so far as to use it on the individual applications running on the platform, but that is hopefully something that egypt can do in the not so distant future.

A

By moving the configuration into git and have the configuration in cml files, the initial infrastructure's code became very small.

A

We would have a bash script to do the initial bootstrapping of the clusters uh in our hosting platform and it will start flux and that flux will pull in all of those configuration and make kubernetes move towards the um the end state.

A

So it's not the exhaustive list either, but these are some of the tools that we chose for our platform, but obviously the tools that you choose for your platform could be totally different.

A

I basically just added them for reference for for the ones who are curious.

A

For the onboarding, we quickly found some champions amongst the developers yeah developers that were keen on using kubernetes and our platform and, as a consequence, had some high threshold for initial bugs and inconsistencies or missing documentation.

A

Those champions would be a good proxy towards the other developers that were a bit more skeptical. They would give us valuable feedback on what was missing compared to how they were working today, for example.

A

Also, we wanted to treat the documentation as a first class citizen, and by that I mean that it should be, of course, from the version control. You should have qa on your documentation.

A

It should be a collaborative product amongst the developers as well, so we would give a link to the documentation to the appointed champion, get feedback on what was missing.

A

Or confusing and corrected together with the developer, then the developer moves from being a student to a teacher helping another developer to do the same, and after a while of doing this, all the developers had a stake in the migration project and then written some documentation themselves.

A

We also wanted to create as good templates and processes as possible. You only get one shot on the first impression, so we want to make it count.

A

Ultimately, the platform should be something that the developers want to use, instead of being something that they are being forced to use. So we really strive towards that goal.

A

Something we often heard from the developer managers was: why bother with this? uh This has not been a problem before well, even though that's true, it's no guarantee that it will stay that way.

A

So to tackle this, we did extensive load testing to show that this actually could become a problem, and also we explained that we are on a mission here to do things even better and that we're not interested in changing things just for the sake of it to try to keep to keep them at ease.

A

We also explain um how we value their input and concerns, and we try to reflect those concerns and questions in the documentation and the presentations that we have had internally.

A

We also took onboarding as a great opportunity to focus more on reliability and alarms for teams and their applications. We created a baseline of alarms as code with prometheus alert manager and had it as part of the platform. Obviously,.

A

That way, we would automatically have a set of alarms from the applications as they were being onboarded. It actually resulted us in finding a lot of bugs that previously was a bit hidden behind azure app services.

A

We defined a clear escalation path, a policy where we used the native integration between prometheus alert manager and atlassian option.

A

We as a platform team, would act as a gatekeeper and push teams to create alarms that were more domain. Specific.

A

One of the critical design decisions that we need to get right in the beginning was whether or not to introduce a service mesh.

A

Do we really need to bring in even more technology like isn't kubernetes complicated enough? So henry gave me a week to experiment and see if the service mesh could solve two important problems.

A

We wanted to have all the traffic encrypted with mutual tls, and we also wanted to get insights into the traffic and application in an aspect oriented way without having a lot of configuration.

A

So we looked at the usual suspects like istio console mesh link d and the one that was at the time most aligned with our principles was legree is focusing on exactly the two problems that we wanted to solve. It was backed by cncf, and it was also good that they embraced the service mesh interface specification.

A

We quickly saw that they had really good documentation on day two operations and the community turned out to be second to none. It was really good and quick help whenever we needed it.

A

So during load testing of the sales portal that all sales clerks in all stores will be using, we had the rather unpleasant surprise.

A

It turned out that the application wasn't performing at all running on our platform and that this application was an external one being developed uh that we were supposed to be running on our platform.

A

We would not see the requests coming from kubernetes outside. We only see the request coming in, and this was the first case where link d and its insights really saved us. So by carefully looking at the metrics, we were able to identify bug in that sales portal.

A

It was basically not reusing sockets for outbound requests, and we could see it in a number of tcp connections that were established.

A

So, where is elsia nordic today they're, basically riding into the sunset with zero, bugs and record sales? Each and every day may no, maybe not record sets every day, but the last and first for the platform. Black friday was a great, really critical test.

A

So we were really curious and pleased when we, after black friday saw that it was performing with the civil bugs and making actually a record sale for airship nordic.