Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How Linkerd Helped bink Partner up with Barclays - Mark Swarbrick, bink

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How Linkerd Helped bink Partner up with Barclays - Mark Swarbrick, bink

In this session, Mark Swarbrick, Bink's Head of Infrastructure, will share how Linkerd gave the fintech startup the confidence to partner up with Barclays and serve millions of customers in the UK. Around 2016, the Bink infrastructure team started containerizing their apps. Not long after they migrated to the cloud and moved their apps onto Kubernetes. Still in the early days, they experienced issues with the cloud's unstable networking infrastructure, leading to multiple issues including huge amounts of random TCP disconnects, UDP connections just going missing, and other faults. After a little research, they gave Linkerd a try and, lo and behold, as soon as they started experimenting with it, network faults caused by the instabilities dropped significantly. The timing could not have been better. The team was just starting conversations with Barclays and Linkerd's metrics allowed them to monitor their SLOs and agree to the bank's ambitious latency and success rated based SLAs.

A

I work for a company called, and this presentation is going to be how linkid helped us partner up with barclays and deliver the service we required on a cloud native platform.

A

So I'll start with, who am I, I head up infrastructure at I've actually been working at for three years now um when we started we're very much in our infancy. Around our devops side of things, we've successfully migrated from a on-prem to a cloud native stack, and this presentation is going to be a little bit around some of the problems we've found and how linkery has helped support those previous to working at.

A

I worked for oracle and mysql for seven years as an implementation, engineer and pre-sales consultant working with various aspects of mysql on various different platforms. Before that I worked for level three, who are tier one isp working for their cdn function, so I've done a few different things, which basically means I know a little about a lot. um So what I'm going to do is just cover. What does really um so? I've always thought you know good concept bit of a silly name.

A

Essentially, what what does is linking loyalty and membership cards um together, um loyalty and payment cards together? So what this effectively means is, I'm sure you know any of you in the audience are listening to. This now will have loyalty cards, whether it's for coffee shops or supermarkets, or anything like that. Actually remembering to carry these with you and present them can be a problem.

A

You know leaving home without your payment card or the apathy around actually having a membership card to present is you know something that's uh a bit of a pain, and I know personally, I haven't always got the loyalty points that I should have or taken advantage of loyalty schemes because of the just the apathy of having to carry the cards around or having to remember to to present at a time of payment, so how payment linking works? What does is the customer will link their payment card to that profile?

A

This will either be done inside the app or through um one of the banking channels that we've got for a mobile banking apps, for example. So what we've done is you've added your um your loyalty cards and you've linked them to your payment cards, your your visa cards, for example. So what would then happen is all I need to do is shop as I would normally, whether it be contactless or you know, chip and pin. What, then, does we process that transaction data and we link it and match it to a customer profile?

A

We then send this information to the merchant. The merchant then receives and processes all this transaction data linked to their profile and then loyalty points can be issued automatically.

A

So essentially, what this means is that you can just pay with your payment card, as you would normally and don't need to present any of your loyalty cards at time of sale in order to get loyalty points, so the uses of payment linking could be for a merchant to upgrade their existing loyalty schemes. Very, very simply, we can plug in to transform ongoing schemes with payment linking very frictionless and very easy.

A

We're also aware that not all merchants, especially the smaller ones, will have their own loyalty scheme at this stage. So what this allows us to do is to create a new loyalty scheme and provide that loyalty scheme to them, and then you know utilize the benefits of payment linking so has partnered with barclays um in 2018 and are projected to reach a 25 million customer reach in the uk by q3 2022.

A

um We have just signed a deal with lloyd's bank as well, so we have quite a large sort of market share now. So I think you know, with the lloyd's banking group in q3 will boost our overall customer reach for over 25 million people and provide extra functionality inside the barclays and lloyd's banking apps.

A

In order for you to link your loyalty cards to your payment cards and shop without having to present your loyalty card itself, so it was quite a simple concept, one that I really thought was interesting um and, like anything, it requires a technology stack in order to deliver that. um So how did build out the infrastructure? And how do we do it to support all this cool loyalty card payment, stuff? So I'll start with a brief history lesson.

A

um You know in terms of devops, I'm old enough to remember where you had to provision you know in in server rooms, there was bespoke hardware, one of my first jobs actually was working for ibm working with ibm mainframes.

A

So essentially, if you wanted the latest computer tech, you had to physically buy the hardware install it in your own office, server room. It was very bespoke um the time to build the time to scale was massive and I'm sure, mostly in the audience, are too young to remember any of that. In the 90s and 2000s you know there was isps and colo co-location facilities um where you could house your own hardware in a data center. When I joined binc, this was still pretty much.

A

What we were doing, we had bought hardware, renting hardware, load, balancers, etc, and this hardware is, is what we provisioned our services on very quickly. This became you know a bottleneck in terms of what we were doing. You know, even in a data center, bad things can happen. um I think this is an example of a few years ago. What happened in one data center, so you have to provision for for failover for resilience and all this kind of stuff to actually do it yourself.

A

Now we weren't at the point of you know wanting to do this. You know we were a a fintech startup. We didn't particularly want to have to re-architect application stacks from the ground up, which is obviously what you used to have to do um very soon, the complexity. You know how long it would take to grow. You know we were talking weeks if you wanted an additional server to be added. You know things like that. We started look at the cloud and honestly to be totally transparent.

A

We chose azure as a cloud provider because at the time they were prepared to chuck lots of free credits. Our way, allow us time and support to to get on a you know, a cloud stack and to migrate our applications over with with zero cost.

A

There's lots of arguments you know which cloud providers best and I think, there's definite arguments in terms of um amazon versus azure versus uh gcloud, you know, and but for us it's definitely been a learning journey. um I think azure definitely had some catching up to do in terms of their kubernetes implementation.

A

We actually ended up rolling our own, um and this is where we implemented linkedin. We are currently looking at aks and migrating onto it. Now the actual sort of kubernetes is a service we feel is caught up with what we want to do, but long and short of it we've ended up with azure.

A

So what we found very soon is the cloud isn't perfect, and here at we rapidly discovered this again one of the benefits of deploying on bare metal. You know you have, whilst you have to deploy everything you do have access to everything. If there's slow network issues, you generally own the the network hardware, you know if there's issues with the os you're, the one that installed it. So you do have a good view of everything. That's going on.

A

One of the issues we found quite rapidly quite quickly were transient network failures. Applications would show gateway timeouts, we were seeing dns timeouts. Essentially our application stack hadn't actually been built with any kind of resilience in mind. um The devops team were under a fair bit of pressure to how we were going to address this um at the time we were rapidly growing.

A

You know it was new features all the time, so the the thought of having to actually go back to excuse me go back to our application, stack and add um retries, you know, add resilience on a on a service mesh and actually code that in um weren't, really, you know plausible. We even got to one point where we considered actually moving to another cloud provider or even moving back to um bare metal colo. Just so we felt we had more control over. What's going on, this isn't a particular criticism of azure.

A

You know: we've seen this across various different cloud platforms going forward, but in essence we weren't able to get the the uptimes and the reliability. We had applications hanging, for example, things like that. um As we're working with banks, it became very soon very quickly that we had the requirement to um talk tls um across everything, even internally, so every um every micro service, every application had to be encrypted. Point to point.

A

So, in order to do this, we we looked at configuring to unicorn, but that was looking like it's going to be quite difficult to do and having all of our applications being able to talk. That was quite difficult.

A

So in essence, we were really struggling to deliver the service reliably and, as I said, questions were asked over whether we needed a platform redesign or the suitability of the cloud hosting platform so either way we had problems that we needed to address.

A

So one of the things that we tried to do is look at various different products that are out there um and linkadee was one of them um and our senior devops engineer who looked into linkedin as well as some other alternatives as well, had a look through and very quickly. You know, I remember him coming to me almost within a week and saying how quickly he was able to implement this and how the problems literally went away.

A

So by introducing a service mesh, we could implement the application retry logic, and we saw immediate positive effects around this. We found it really easy to install with the automatic proxy injection. You know it mitigated any complex setup. We were able to implement mtls, which was implemented through linkedin, which immediately because linkid was the service mesh. Any application that was talking to any other application automatically was tls encrypted, which again was fantastic for what we needed.

A

One of the other things as we were maturing and I came in, we really didn't- have any slo, metrics or decent dashboards or alerting, and it was quite obvious to us that we needed to do something and because linkade sits as a service mesh, it gave us the perfect place to be able to develop our own internal slos, have some dashboarding and because of the prometheus integration, we were able to set up really good, alerting through issues that linkadee would raise.

A

um As fairly recently, we've actually invested in a subscription to buoyant cloud which gives us a really nice that dashboarding feature and the ability to go further. Looking at the metrics, looking what's going on and being able to share dashboards and have good insights into everything, that's going on um so for us, it was, you know, really fantastic and really was a quick win to what was going on. um Everyone loves live demo, so what I'm going to do is instead of just talking about um what linkedin has done.

A

I want to give you a a live run-through example of how implementing linkade really is quite quick and easy, and how, if you have an application, which you know is suffering failures or transient network failures or anything else how quickly it is to do. I always thought I'd, never do a live demo, but we'll have a go so um this will be up next, it's a live demo part.

A

What I want to illustrate very simply on this demo is how easy it is to uh deploy linkedin as an application layer to be able to mitigate any connection issues that we're seeing. So to give you a bit of context, what we were seeing across the cloud was transient network errors. We got the odd occasional timeouts connections being dropped, stuff you'd, actually like normally expect, although we to be fair, we didn't expect it to be, quite so bad initially, as stated before, we could have implemented something on the application side.

A

We could have done something you know on the application stack to implement software, retries or retry logic. The issue we've got is it's more for microservices architecture, so we've got various different applications across various different squads, so I'd have to implement it in different ways, and there was also time constraints around this as well. So what I'm going to illustrate on this demo? Is it's a really simple demo um that one of my um devops engineers put together effectively? All it is is an application.

A

That's actually designed to fail one in ten times, so it makes a request and one of those ten times it will get a 500, the other 9 and 10 times it will get a 200.. Now this is to simulate um effectively the application dropping 1 in 10 connections.

A

What I'm then going to do is apply the chaos server profile and all that does is deploy into kube the the linkedin service mesh um and then what I'm going to do is tail tell the logs, and you can see how instantly it's linkid will handle the failed retries and give us an effective 100 success rate. So what I'm going to do is one that I prepared earlier is, if you look get pods, there's no resources available. So what we'll simply do is apply the chaos, client and we'll do the same for the chaos server.

A

So this is a very simple client server application, where the client's making a request and the server is returning the request if we do control logs um for the client. Hopefully that should be up. It's not quite up yet give that a minute. In the meantime, we'll do a get pods and you'll see that they're all starting or running and all available in there so yep there we go, so you can see here um that number 200 is going up.

A

The number returned, but you'll also see that the number of 500s is going up as well with a total number of connections. So if we do a watch on linkedin vis roots you'll see as this coming, this comes up. We've got an effective success of 89 and an actual success of 89.

A

So, as you can see, pretty much one in 10 transactions are failing, um and this is pretty much from what we're seeing in production. um So you can see one in 10, failing so by simply applying the um the uh the chaos server profile.

A

What that then gives us is, is linkedid managing those connections, so if we enable that you'll see straight away instantly, the number of 500s is now pegged at 976 and then you'll start to see that the um actual success, if I reload this you'll see, the effect of success is now actually going up to 91.

A

93 and you'll see it'll start growing, as the percentage of requests starts. Passing so you'll see that it's so easy and effective.

A

You know up to 95 now it's so easy and effective to apply linkade um to give you that sort of um that level of resilience and service retries, as I stated before, there's uh another additional benefit to us that it also provides tls encapsulation for every point-to-point, um every point-to-point connection, as well so really really straightforward for us to be able to reach the level of compliance as we need without having to worry about. You know application level security between all of the microservices as well.

A

So what this is now showing you is that one in ten requests are still effectively failing, but because of linkadee doing the service retries we're now at 100 success. Now I appreciate this is you know kind of a crude, simple demo, but hopefully you'll illustrate just why linkadi was a no-brainer for us and how simple and effective it was um as a service mesh on top of what we were doing with zero disruption to our application stack, and you know zero disruptions. What we're doing?

A

um On top of that, with the sort of newer buoyant cloud, it gives us really good dashboard stats as well. So I'm sure linkadi are well represented um here at kubecon.

A

So what I'd recommend you do is go and have a look at um linkadeboos have a look at buoyant cloud because for us having metrics in the service mesh is one of the best ways of effectively seeing how applications are running and one of the things which is probably out of scope for this demo is to talk about how we can monitor slos and service level metrics using linkedin and the service mesh.

A

Hopefully this is just quick and dirty and and showing you what we can do um if you've got any questions, hopefully not too complicated ones, then uh please ask otherwise yeah. I know this, you know sounds like a complete sales pitch for linkedin, but honestly, where we've used technology and open source technology which has really helped us. I always feel that we should try and give something back and eulogize.

A

You know um sort of talk about the technology um and, and for me this is just just was a no-brainer for what we did and really affected to the business. um Thank you very much cheers.