Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 26 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: From Brownfield to Greenfield: Istio Service Mesh Journey at Fred... Shriram Rajagopalan & Lixun Qi

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

From Brownfield to Greenfield: Istio Service Mesh Journey at Freddie Mac - Shriram Rajagopalan, Tetrate & Lixun Qi, Freddie Mac

Freddie Mac is one of the two mortgage loan corporations in the United States managing trillions of dollars of assets across the country. Our infrastructure is spread across different Kubernetes providers, hardware load balancers, and large swaths of virtual machines. In this talk, we describe our service mesh adoption journey in a highly regulated financial compliance environment. We will discuss both greenfield and brownfield environments, to gain full visibility and traffic management capabilities using Istio/Envoy. We will highlight the changes to our GitOps development workflow, changes to our age old organizational practices, and how the service mesh journey forced us to foster deeper co-operation between traditionally siloed security, platform and application development teams as we tried to weave a mesh over the old and new.

https://sched.co/UaYb

A

Hello, everyone, let's get started when people's people coming. Hopefully uh you know you don't have a difficult time to land. You'll find this place with all the new changes you know this morning.

A

Okay, uh my name is Li Zheng Qi I'm, a senior tech lead at a Freddie Mac. This is my first 10 de a first time here. Kevon you talk as a first time. You know speaker, there are always things you're not prepared, for. Let me think I'm going to talk about serving match. Let me try to explain what happened this morning through. You know II still, okay. So when I come to the that Convention Center, my cell car tells me go to room 28. Okay, so I walk in up there.

A

There's one gentleman holding a piece of paper, saying: okay, Eric hold 500, okay, I would say: hey what what happened? Then I. Look at my cell car again, the control panel. You know pilot. Tell me: okay, your room you're in the end point has changed: howdy, okay, so I come down and then the hottie is in the blocked circuit. Breaker triggered they say: okay come back at 11:30, okay, so I did a retry back 11:30. Okay, no, you know 200, okay, so that's a quick story.

A

So today, what I'm going to talk about is our Cermak serviceman experience from brownfield to greenfield in Freddie Mac? Let me first introduce a free round firm at a trade page which is our server image and partner and have been working very closely with us this year and helped us navigate through the server manager knee experiment. You know in the pre here in a very unprincipled approach: haha okay,.

A

So, instead of a shorthand about who knows I know you coq burn ad can I have a shorthand who actually knows Freddie Mac Wow a lot of people hop on your titrate.

A

Okay, so for those folks are not familiar, words are in the Freddie Mac. In short, we are in mortgage business yeah. We have a two trillion in assets and 15 billion dollars as revenue last year.

A

Okay, so our mission is make home impossible. It's very for those. You know live in this beautiful city of Santiago and Bay Area.

A

So some quick disclaimer, what we're not going to talk about is a mortgage rate. Okay, if we want to lower your mortgage bill. Okay, talk to me after the session. Please, we are not going to talk about subprime mortgage crisis, but hopefully service match come to avoid the next one.

A

Okay year, 2019, you know is, will come down will go down into a Freddie, citee history as most pivotal in the revolution a year, so we have been focusing on two major company-wide IT initiatives, application and modernization. So we continue drive toward macro service architecture while evaluating refactoring monolithic application to micro services.

A

We continue enhancer to evolve to change, to support cloud native applications. We also realign our organization structures to further accelerate our development cycles. The second big undertaking is cloud transformation, so this in the picture from devil groups, really speaks well, our cloud transformation strategy.

A

We have been a major way and we're sharp.

A

So we are using the lift and shape to approach to migrate our premiums to we MC in AWS. So that's the highway at the very bottom. That's what we call our brown field approach for the green field approach. We deploy container workload into manage kubernetes on public cloud, so we foresee a long period of time. Brown field and green field going to exist coexist and we have to manage in this hybrid mode while moving to the new world, either in the middle way or the the firing of hats on the on the right.

A

So for those of you who have first-hand experience of cloud migration, you might agree with me. None of the pass is easy.

A

So, even though we are not at forefront of this cloud formation, but we consider ourselves lucky about the timing. The technology like service mass SEO have been much more quickly right, so hopefully, with those technology, our journey will be less bumpy.

A

First, give you some background information of our IT environment, like any big enterprise. We have large number of occasions to support our business need. Most application runs some VMs on a variety of software stacks.

A

You know you can see. We are a pretty big new Java, a new shop database, no SQL. Some big data service was an late way because of.

B

A

For information, we're adding a lot of new clothes services as well.

A

So he we invest heavily in our C. There were two change, so the return happy. If re, you know significant in term of faster delivery time and improved software and quality, but to move into the further level we have lot of challenge integrating with our infrastructure. I think one fundamental problem we have is the tight coupling between the application and the underlay infrastructure when we deploy the application. There's just so many new touch point in the different part of our infrastructure.

A

Some of them require manual new ticket and approval right. So it's very difficult to automate or spend a lot of time and effort to automate is virtually now. We are adding a lot of you know a component with the hybrid new cloud.

A

So the server may have many different definitions with deep, depending who you talk to when we start looking at the server smash around is Hamas. Here we really want put in bigger context.

A

We ask ourself the question: where do we see our infrastructure in coming years in our region? We think our info Church should have the following characteristics. The word first one is: we won't have a common. You know fabric that we can plug in our application easily.

A

The network vendor and the company have been talking about application aware and the African centric network for years. In my opinion, service mesh is the best coverage option so far to bridge the dead. In the last mile, the sidecar become the new team, our tation, between application and network. You sing about your pod. You actually connect your network to the local host address one 27.0 to one the same thing for your new apps running in the VM. You connect the same way. I think this is work, our philosophy.

A

This means you really the underlay new network becoming transparent, give you some example: our environment. We one leaf and spying with VSN evpn in our own prime nu network. We have nsx, you know we am environment. In AWS we have EPC ttwo direct connect right. You move to the next layer of your community. Again, you have a different network models. You know you have overlay in the network where you also have the Clos native in a network if you using manage cuban ad in public cloud.

A

So all these complex complexities should be shielded from the development team. Our second thought is: we don't want this in your actual layer to limit it to network and native long. We want. You know this layer to support more as a service model, so the application can plug in easily so I'm going to show. You know some changes because of SEO to our application design patterns in later slides.

A

Certainly, freddiemac put a lot of emphasis on automation and, in fact, where's the code so easy, oh and kubernetes really lowered automation barrier.

A

When you look at a new Cuban, ID I think you can start seeing the trending it not only used as an Orchestrator to measure and schedule worker mode. There are more, you know, people using that to prove another part of your infrastructure. They say you can. You can use it to proven your DNS registry. You know service, they cope, you guys know search or you can pair in the storage through Cuban edited. A eto is another great example right where the Cuban ITC are these.

A

It yields preventing your you know, sidecar, the Envoy, sidecar and other component of nuit still and I. Think the declarative approach are easy. Also big deal for us. When you look at you know our you know diagram the conversation of our application, manifest we're beginning start with such as Aleksei application configurations, not that percentage gets smaller and smaller. We are adding more of the declarative. You know configurations into the main universe in the file.

A

Okay, so I talked about you know what are in the reasons why we we consider service match and then why we kind of leaned toward the east or because of all the the major point that I covered so now, I'm going to talk about all the new changes is actually putting into our application new design patterns. We are moving more and more new common features used to be the implement in the software new stack onto a new platform.

A

So here is a list of some features we offer in today through an East you, they say you pick a new management, for example. Well, if you still have like TOS 1.0 in your browser, because some old applications where you understand you know the benefit you terminate here- is your your platform.

A

The chaos in the engineer- that's a good example our play. They say people have the impression once you know code in the cloud, you have the redundancy I kind of agree once you set out there, but the migration process itself. Definitely very chaotic one example. You know we have is that we keep in you know right into the issue is a when we start moving our environment to a column, batches they're, always like latency, always and said in effect right.

A

Sometimes a one or two millisecond latency delay going to throw our application performance way off the chart, but something like SEO. You can easily inject a latency that can help you a big time so for you, you can plan your migration much easier.

A

So now sidecar become a really major component for every indesign, just like the not the phone for our new life, but that become at the magic bar for us. Every time we miss something now become say: hey, let's put a Secor, so we love new Sekhar, so much so I proposed to rename our Sree team as a sad card, reliability, engineering, team.

A

Okay, so easy, when certain match also enhance our CI CV to change some quick wins. It's like a lot of people. You have been talking about. They say your canary in the deployment right so that actually speed up your release and management cycles, I think, what's a major change for us, it's really easy to enable a lot of features through confusion as a code which make our next logical step as the key tops right. So that's definitely the direction we were moving into for our growing field applications.

A

All the CIC pipeline are triggered by a pull request. So this really, you know, reduce the number of you know your hand and people that can touch our production system. We also have a more immutable in your kettles right in old world. Everything is a pain for our company. We all we have many internal and external in audit, and then we have to have in show evidence for security controls and complaint culture, but not because we have the single source of truth of confirmation in in get it much easier for us to pass.

A

The audit and also the key, becomes a new place where the different team taken we're together on new security on the under operation. They call so take a department right so in.

B

A

Opinion mesh not only match the kind of workload, but they also met other people and the internal no forces it also mesh in in in a different ways, we can steal a teacup of the department and operations.

A

Okay, so in the next few slides I'm going to talk about some concrete, you know examples where we use energy steel to address challenges. We have moving into a macro service world, so in real estate and mortgage business, everything about location, location, location for people, working IT department in the company, everything about security, secured and security.

A

We have lot of challenges: running micro services in public new cloud, the traditional perimeter base for what and working a well because it not a easy to plug into the macro service. In your data path, IP and porpoise new policies doesn't work well with the dynamic and distributed cloud cloud applications I think a newer topic we talked about is really the insider in Australia, I. Think with the the spiffy and the secret naming from the ischial didn't clearly help us reduce the directional in that area.

A

So we consider SEO as a critical component of our next-generation network design. We are moving toward a zero trust based security model.

A

Okay, we want move toward at any pace, DNS or new policies and with other like us, in your security in transit with some mutual TOS and then be able to rotate the key. Much faster and, more importantly, is everything as a cold right. That makes your set off much either so.

A

I'm going to give a a simplified, you know, example how we implement a security for our community.

A

So here we label our containers: okay, I'm, sorry, people taking more picture when I showed it in the side. Okay, so we actually label our containers with no zones right, proper zones. These help us create create the Cuban net Network policy easier. So we can mimic the traditional like a zone based application based model.

A

This example show a new requirement for from our evil second team, what we start with a so we want to create the micro segmentation on the model that we can create the application boundary in layer, 3, layer, 4, layer, 7, but our security measures. Can you further segment the external services?

A

What I mean by this is this example, is that you have we have two applications: they all have their own database, but you want, you know, application one. You know only be talking about. You know, database, one right and then for David to you should only allow application in the tool, but now, just through the you are like you know, like you know, username or credentials where in other ways so really want to have like a lower layer.

A

But the challenge is: when you, you know get into the the cloud in world management database. They all tend to have a dynamic address right. So we have new challenge. Not only the source is dynamic, your destination also dynamic.

A

If we use so traditional IP base, you have to open the bigger subnet, then you cannot defeat the purpose right and they have become a you happen and talk to you know database, one and two and then the same thing for you did African.

A

So what we did here is a we actually use the e Co new features. I don't have time to go into details, but there's a lot of good information on the eastern website. What we use were standard, you see, features like a service, an entry, no scope and are back the the easy one. Our bag features egress in UK way, and also we look at some of the mixer features which might be retired in next release and also some like OPA policies in in this to support this model.

A

Okay, so the next in the setup requirement, is really about availability in today's world, our customers, both internal and external. They really want not a 24 by 7 service with translating in a really tight SLA, and then you RTO and RPO numbers.

A

So this requires not only have a high availability in our infrastructure, but also for our infrastructure to provide building solutions for us application resiliency.

A

In our current environment, we have a some single point of failure: kind of component, like our hardware, a load balancer the concerns. Now as much as about they say the possibility to lose the each a pair but I think it's more on. Someone can go in there and then make some your own. Your changes or people don't follow the proper procedure for the upgrade right. So we still, but now we can accomplish is we can have a more decentralized load balance balancer right so that that's a big thing and then we also move more layer.

A

7 brought in policies closer to you know to on to the third side, car on to in great gateway and close to the application, and the afternoon team now has more control over the policies. So we talked about, they say to minimum the downtime for the upgrade the canary deployed model for from East you and then also I, was in another session, the gnorm session, the other day. I think we are not alone, they say in there. A lot of people have the same issue. You have a new expire, you know you're, certainly expired.

A

Then you have to pin down service chatswood and update you, sir right. So we sealed ICS. It is become a big issue.

A

Okay, so in the first slide, is talking about in our nuclear energy new design, we have worker node across different zones, but this makes easier for our development team based on their RTO RPO and the budget. They can decide whether they want to apply in a single in no AC or multi.

A

The challenge was deploying the motto AZ is by default, the community in load-balancing. They don't. You know honor locality right. They say you have application in this in those new slides right. If you have a service one, when they're trying to talk to her in a service tool, if it's a default new Cuban ad, then potentially you can send to server to in any AZ but with SEO it offered origin based.

A

So if you have an application that has really tight latency budget, then you can configure issue in a way. Okay, you only prefer the the services in your same new AZ.

A

Okay, another use case for days and also.

B

A

So we have a good number of applications. We only see the spike like an end of month or at the end of fiscal year. In the past we tend to over-provision them with the maximum. You know expectation and something you can do with secure. Bernetta yourself is, you can use h-tpa and you know scale on your recipient memory, but most often times CPU and memory it hard to relate to your application in performance. So what an easy okay Bo through sidecar is offer you a lot of telemetry informations right.

A

So now you can simply just say you know: I want to scale based on the number of requests or you can scale based on your response time or your status code right. So this make your scaling much easier.

A

Okay, so I kind of go over. You know some of the features we use utilize new you co to support our Greenfield application, now I'm going to a new chain to a new free, rom you're going to talk about some of our. You know, effort, POC, effort to bridge greenfield and brownfield.

B

Thanks so when it came to brownfield, given that these guys had, like you know, a lot of virtual machines and a lot of code and applications in the you know traditional world, we wanted to do a mesh first and then a containerized next approach, because the goal was that we need to help them migrate from Longfield to greenfield, and it's not the case that you need the mesh only in the Greenfield.

B

So what we did is we really tried to focus on getting all the benefits of the mesh which includes, like you know, simplify traffic flows, uniform security, which include mutual dealers, authentication authorization and all the stuff and telemetry in brownfield and Greenfield uniformly, so that they can take advantage of the new stack and across the board, and this in turn will then enable their development teams to gradually migrate their applications from brownfield to greenfield in their own sweet time without having to you know just to a fork and left always and specifically, when it came to traffic flows, there were a whole bunch of like you know, hairy situations like given the fact that the network was not symmetric.

B

We had to still enable like vm to kubernetes kubernetes, to VMS and a classical scenario where you want to have disaster recovery from BMS. Like you know, from kubernetes, and if you have kubernetes version fails, you want to go back to your VM version of the application, and so on so and also like east-west traffic inside the VM based system is as well, which traditionally goes headpins by a hardware load balancer. So we tried to like and optimize that as well and in our approach. So specifically, some of the challenges that we faced was like.

B

As I said, the network connectivity is typically asymmetric, odds can typically talk vm's, but vm's cannot directly talk to pods unless you happen to have a magically flat network that you find in several cloud providers. On-Prem is typically the case that you know your pods can directly talk to me. I mean VMs can directly talk to the pods on their overlay, Network I'm, a security point of view. This is where things get a lot more interesting, because traditionally, all the enterprise applications are geared towards application: specific certificates.

B

You go to your vanaf I, like you know, who or is managing when I fly inside your organization. Get your certificates for your app go, stick them into the VM and your WebLogic or you know, whatever websphere or whatnot basically uses these certs and serves these certs to f5 and that's how your entire infrastructure data flows and but we in Sto the model for security is one based on workload, certificates and not application specific certificates, so workloads have certificates and then, like application, specific certificates typically tend to move to the gateways.

B

But the migration process is where the you know the catch is because it's not like overnight. We will just change every every VM to have a sidecar, a sphere migrating like thousands of VMs or one after the other, some have sidecar, some don't have sidecars those who have site cars will have work, load, certificates and those that don't have site cars will be using application specific certificates. The mix and match of this creates a lot more of these challenges that you need to keep track of when you actually migrate.

B

Your workloads, like you know, from VMs or when you actually modernize a VM infrastructure to add sidecars to all of them, and of course, this also includes integrating with your in-house, like CA, so stores, typically when fi or what are the HSM stores that you happen to have and the other one being that most of these middleware systems. Typically don't like the all-inclusive traffic capture approach that we employ in kubernetes.

B

If we cannot magically simply say like all inbound traffic and outbound traffic inside a VM should go through the sidecar, the cause, the fact that, like in these WebLogic servers and so on, they happen to have other Middle West that would take care of session persistence and so on, and so they they have their own stuff.

B

That's running on the VM and any IP tables based traffic capture tends to like in a jeopardize that so we had to work around those scenarios as well in order to make sure that nothing in the application is affected in the slightest bit. There's like almost zero modification, with the exception of making these applications to plain text instead of HTTP.

B

And that's that's the extent of the modification that we had to do and still working on the fact and yet take advantage of the entire sidecar approach and then finally, the the fun part is that there is actually no such thing as a service registry I mean well, there is, but it's it's scattered across the place. You have whole bunch of information sitting in ServiceNow information on your V Center information on your load, balancers in form of whips and LB pools, and so on.

B

So without service registry in place, it's hard to establish a service mesh. So this is part of the challenges involves scraping through all of these data sources and aggregating, and reconciling them to make sure that this is okay. This is your service registry of like n thousand applications with so many thousand, like you know, VMs as backends, and from that that is where you can actually establish a proper client-side load, balancing solution with the service mesh and yeah.

B

So we achieved this through, like I, can give a small example of an application that we like work through at Freddy. This is a. It was one of the the first POC application that we did, which is something that has like a some portion of the code on kubernetes. Then there is a large section of the stuff on VMs, and then there is this: there's an ESB there's a whole bunch of different database appliances from so many different vendors and in the absence of sto or any of the other surface mesh approaches.

B

The the model in place was that we would use one by HTTP with basic auth as a way of like you know, securing the entire communication- and there was a hardware load balancer in between, like the VMS and kubernetes, in order to like do the load, balancing and act as a pseudo service discovery mechanism and with the essa the way we transform. This is like we got rid of the load balancers we actually added sidecars and with all the service discovery integrated.

B

This enabled us to do direct, east-west traffic, from the pods to the VMS and from the VMS to the kubernetes pods via the Gateway and, in addition to providing a uniform telemetry across the stack, I mean and uniform layer of security, because we can now impose access, control policies on VMs and kubernetes alike, so I'll just work close with another day for us, and you know, and the routing also becomes much more simplified, because it's much more direct and straightforward so and the this was achieved through a combination of various approaches like this.

B

To go back to the point that relation mentioned, the software-defined infrastructure that we actually set up and established was a full platform which had a composition of history, of control, planes that managed kubernetes clusters and VMs, and then a management plane like that. We designed that would actually in turn make the entire magic come true in terms of, like you know, establishing connectivity from VMs to kubernetes, setting up the entire security posture, and you know the telemetry and so on and aggregating them and presenting them in a decent way.

B

So it was a combination of like in multiple layers of approaches in order to achieve the software-defined infrastructure gold that listener had mentioned in the past.

B

So we will now want to talk about some of the lessons that we actually learned along the way and.

A

Okay, so something you know, we learned along the journey. I. Think really, it's importantly, so I really want. You know, follow the incremental adoption right. You don't want to say, as we said right, try to replace. You know your own. Your own technology I think that that's that the good thing about SEO in the old days, I wouldn't see in the old days right even today, like most of time you bring in a new technology. It's how it means you have to swear out new someone right.

A

How do we tolling a lot of things, but we certainly steel. It really worked well with our existing investment like different technologies infrastructures, and then we really required minimum new changes to your NLE infrastructure and also your application code race. Often time we just say hey, you know what application we approach, our development team, we say, give me your application and then we don't make any changes. Then, with you boy on a steel, then the next day we show them. Okay, yeah! This is all the benefit you can get right.

A

So this is really speed up our adoption rate.

A

The other some other new technical challenge, though, is we still haven't, figured out the way how we integrate our manager. Our pika is no sir I know like you know you can do the city door, but and then they say with the inter me CA, but there's some pushback from our InfoSec team, because we have to worry in the straightening requirement right. If you won't have you know either a roux CA or your intermediate CA.

A

You know you have to have a hard work in the base right, so that had been a issue for us and then you fall anywhere in the audience. You have, you know a good solution. You ran your production, please definite what we change the ideas.

A

Okay, I think affirmation about you, know: I, hey you don't have to in a way servers managed until you all your application. You know it's containerized right, I, think that's a some. You know confusion from our new brownfield new teams, because you know refactor or break down money, leaks application. It's a big. You know value proposition. They say you know they have to try to fight the saving, just don't just retire. So really with all the POC. You know we have. We really show them. Okay, even without making any changes.

A

Your VM application tends to benefit, even though now to the full intent. But there are lot of things you can get auto out in order box.

A

So there are a lot of discussion. You know here already during the the presentation is really about the user experience right now for the develop team, but also for the team that actually deployed the server itself, because it really touch a lot of layers in our United stack. We actually form a what we call a tiger team, so we have a you know. Member from in our team, we have definitely from a development team architecture team secured team right. So we look at we work on this one. Turning together, I think that's very important.

A

We also have a SEO coach. What we internally call you CEO new coach that actually helped develop team on board. The applications provide a lot of documentation like in manifest. No examples do they make you know. Is it much easier for the development team and there's a lot of discussion already from the community from the vendor? Try to simplify the the API, the interface you know through, like a pregnancy, RDS, I think Daphne went to see there will be like new common. You know standard suit family.

A

They were super sincere right, so we can't only settle down on one set of new single model. Rather than read the word. Anyways people come up with it's very similar. If you in the rotten through aging world, they have the open, config right that kind of bridge everything new together on our site. We also new, try our own abstraction, most practice room I'm chart. So the idea is, we all have our abstraction taking out value files and then we're going to spit out whether is standard issue and resourcing types word. The vendor, specific or solution.

A

Specificity are these right so that the approach we are we are taking well at on top of the abstraction I.

A

Think I asked some other early adopters for in term of new support I. We are still trying to figure out what's up accident support model inside inside our organization, so we still working on that.

A

Okay, so the other thing I think about I think we really appreciate the support from the community. I mean there are a lot of your own effort from community, and then there are a lot of in vendor. You can go with slice it for us. You know text. We have been goodie the vendor in helping us a lot.

A

You know along the journey, I think that that's really important you're helping us, you know not only understand the ECU technology itself, but also help you shaping up and they've, got our application design your patterns, putting the personal new practice and also as a way we can check. No back our feedback to the community to kind of drive the Rohmer.

A

Okay, okay, so thanks very much.