Red Hat OpenShift OpenShift Commons Gathering: Los Angeles 2021, 3 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps in Action William Foss,Anil Sharma, Nitin Kalra (Broadcom) OpenShift Commons Gathering 2021

Description

OpenShift Commons Gathering 2021
OpenShift Case Study: GitOps in Action at Broadcom
Guest Speakers: William Foss (Broadcom) | Anil Sharma (Broadcom) | Nitin Kalra (Broadcom)
https://commons.openshift.org/index.html#join

A

B

Welcome back everyone, we're gonna talk about get ops in action now. Thank you.

A

Hi everyone, I'm anil, I'm billy.

C

Hey nathan, caldwell.

A

Hey so we are from broadcom, and uh you know we are here for case study on gitops in action and broadcom, so you know start so uh for intro, like uh you know what is broadcom software, you know, what does uh you know sas platform engineering do and uh you know github's, uh how do we use github for our advantage? So this is our basically agenda for today right and we'll go into details there.

A

So for the intro right, um you know, broadcom is a large. You know uh enterprise company, which was primarily into the software. You know into the semiconductor market right, so anything that you do on. You know online that wherever you connect today, the more that then chances are every you know the stuff or the you know. Your data is translating somewhere it's touching broadcom technology right, so we have um all the kind of semiconductor.

A

You know um you know tips into whether it says you know cell phones or data centers or connectivity, anything which is out there. But then you know uh the company uh is about. uh You know, as of last financial year, was about 24 billion in revenue and we invest about 5 billion in our r d, and then we transitioned from basically a semiconductor primarily semiconductor company to a software company as well.

A

So we diversified in there with the acquisition of ca technologies and semantic, and that's where broadcom software came in, so you can see our legacy like how we came in and then how the broadcom software came in with the you know, c and uh symantec, so in in the software in our broadcom software right we are, you know, as of last.

A

Initially again, you know we have five billion dollar plus in revenue and uh we invest almost like 15 percent of our revenue into r d, which is you know again in you, know hundreds of millions of dollars which may be even larger than some of the company's total revenue itself right and uh we have you know thousands of patents and uh you know almost uh 80 of our workforces into the r d.

A

So in terms of our usage, almost uh 10 over 10 global companies uses. So we have, you know sas software that is used by a lot of companies, so we have infrastructure, specific sas, softwares or security base softwares with the different portfolios. So these are like you know, you see their aeops devops and uh you know networking you know security and identity software. These are like software product portfolios which are sold as a sas service.

A

So you know these products again. You know, portfolios have multiple products in there and these are all revenue based. uh You know subscription revenue based uh sas model, so within uh broadcom software uh we represent the sas platform engineering. So what we do is basically all these sas software that we sell right for subscription uh online right. These are basically all run in a standard platform, so we came in from you, know, ca technologies and then, with semantics acquisition. You know, uh platform engineering got expanded, so we uh had these softwares.

A

You know which were running in you know in the at some point, in b, metal data centers, and then we moved on to vm, based hosting and every as technology progressed. Right then, on to we started our shift to containers. um You know years back right, so you can reference our previous. uh You know presentations in the open shift commons in 2017 and 19, so we have been heavy uh focused on containerization and uh you know uh so you know what we provide within broadcom is right, because there are so many software.

A

You know products and software portfolios for sas delivery right. So we create a common platform so that all the teams, all the products which are delivered not delivering these services online. They utilize the standards. You know based platform, and these you know for that. We have, um you, know our uh standard cd pipelines, for you know, spinning up infrastructure or whether it's network or other resources.

A

You know containerized and non-containers workloads and multiple application components that we create ourselves or we use. You know where you know cloud provides the native strength. We utilize those, but we still maintain. You know security and standards, because these environments for us right, you know, can vary based on the you know, compliance and security requirements right something maybe for fed ramps, something maybe, for you know, payment security based. You know highly sensitive stuff right so and uh in sas world right for customer data and customer privacy is supreme.

A

So there are multiple, you know, audits and you know things will go on. So we try to address those as a standard platform and then you know um you know, teams utilize that so you know this is our basically a technology stack. You know you might have seen uh you know all these icons, as people have seen right there like large icons out there, so we you know, set up our own services plus we use some of you know.

A

We can't build everything on our own, so we have to use you know some of the components and try to, but still we maintain the standards and how we gonna roll that out as a so that multiple products can use it rather than everyone trying to reinvent the wheel right. So we, if there is one team which needs something there is more than more chance that hey there are other product teams which will also need it right.

A

So we try to create a standard and that prevents you know reinvention of the wheels multiple people trying to do the same thing, and then we have to clobber the solution right. So, instead of that, we look at like okay. What is the something needs? It we collaborate and maybe they can be our first beta customer internally right and then, because all these products are again multi-tenant from their perspective right.

A

So what we treat is internally within broadcom platform in any team, we treat our product team as our customer, so they are for us, it's a tenant right in our platform, but that tenant can be again a multi-tenant uh platform which is out there. So that's how uh you know we we work out in that. So.

B

Yeah, so this is going to um kind of, simplify or repeat what uh what I know was describing from a product team all the products that we support within broadcom. This is what they get to see. They just need to focus on their continuous delivery pipeline. They they describe how their application is going to be deployed, and it it goes into a sas platform. It has all those different pieces. This is just very simplified for what we're doing so yeah, but we're doing this for all these different product teams within broadcom, 30, different product teams.

B

Some of them have different sub teams, lots of different developers on these teams a whole bunch of different cd pipelines checked with one of our team like over 7 000, regular jobs running on average per day. So it's just a lot of stuff going on and we need to do that as efficiently as possible.

B

You know make this thing: go very fish efficient and so some of the ways that we do, that is using common platform foundation, make sure that the environments are as similar as possible across a whole bunch of different clusters and environments using reusable modules so that we can pick and choose because, yes, the truth is not all environments are going to be same. Production is probably going to be bigger than your dev or verify environment, enabling as much self-service automation as we can. But we have to do that with guardrails.

B

We can't give developers access to to do everything in production and then collaborating with the different product teams. Things like using inner source so that, if we're building a particular cluster and a team needs needs to scale a node pool, they can submit a pull request and actually get that change in or maybe they want a new dedicated node pool for a component. So so here's kind of a diagram of some of the different components.

B

We have a common vpc networking layer that connects various components um depending on the environment may have an openshift cluster may have database virtual machines. May have a gk cluster and there could be even like product specific workloads, things like a bucket or service accounts, or some um you know not. Everything is containerized as much as we want that containerized. Sometimes different product teams need to have a vm of their own.

B

So but through the the continuous delivery pipeline that we provide teams, they can describe their application via helm and get it deployed either to an openshift cluster or gk cluster.

B

But one of the things that we found is sometimes they need to deploy infrastructure.

B

So we gave them the pos, the capability of writing terraform, putting that into the pipeline, and so they can actually deploy their infrastructure within that that product specific project and area um to to give them whatever either cloud native resources or vms or or stuff that they need from their perspective. So that was from the product teams. um What about our team, like all of this infrastructure, needs to be created?

B

We don't go through and like manually, create it. We wouldn't be able to scale that way. So each of these components is built using a set of automation. That's packaged as an automation, so that you know if we need to build an openshift cluster. We've got automation that can build that. We can specify how many you know nodes or you know stuff we need if we need to build database machines or a gk cluster like each one. You know. Sometimes we can have one tool. That'll do multiple things.

B

Sometimes we have to have specific tools specifically for something like a core networking and those tools are, are built and run in the same pipeline method. So we we have a infrastructures code configuration repos. So in our pipelines we we define okay. This is what we want, that particular sas platform component to look like and um and it will go through a docker base or a docker packaged container and will actually build that sas piece and then that tool was actually built using.

B

You know same you know the same ci type pipeline that you would see so it'll package. You know either ansible or terraform or various tools that we're using as part of building the infrastructure, and then that then, is used within the pipeline to to build that.

B

Now. One of the things you may notice is we're actually we've given product teams, the ability to deploy to clusters and deploy their own infrastructure, but we actually managed database um virtual machines, and that was one area that was that was missing in this pipeline teams would have to to create a ticket and request a database and get it was a very manual process. So I'm going to hand it off to nittan that's going to describe how we we solve that.

C

Right, uh so, if you look at the kubernetes adoption phases uh throughout the journey of, you know how exactly uh uh from a manifest point of view, how exactly it kind of evolved, so we basically started with stateless applications, uh mainly backed up by replica sets deployments. This was like you know. More than more than half a decade back, uh packaging mechanisms were like yaml json, manifest helm. Of course, you know coming into the picture, then comes in the stateful applications. This is where you know the database applications or any any of these.

C

Let's say your redis cluster or mongodb clusters. Anything around uh those were backed up by persistent volume claims with you know, uh csr drivers coming in uh with the new formation of the whole community's part new packaging mechanism. Also kind of you know over adopted something like customized, for example, then comes in autopilot applications.

C

Operators so uh with operators brings in you know a lot of fresh features such as, like auto backups, app sensitive scaling, seamless upgrades.

C

uh uh If you look at the packaging mechanism for the operators again uh going back to the same holy ammos but alongside you know, with olm bundles, which which was a new thing altogether so within broadcom, uh these are some of the sas provided operators that we that we actually support and have, and if you look at the whole breadth of it, we base we, we have starting from the database to pubs up to uh security, something like a wall.

C

uh You know a service mesh is to all of these us like uh within the platform that goes in into any of our clusters, whatever we have uh for the engineering teams to basically uh consume, and you know, deliver it or deploy what uh billy was stating. You know we're the exact same pipeline. So the approach over here is: how do we? How do we basically take in uh the same operator method and use the same ktops approach to deploy? You know, even if it is an operator or anything of that sort.

C

So, uh for specifically for operators, uh we we finalized and chose olm as like to uh to do the whole life cycle management of the operators, uh which involves not just the uh initialization. But you know installation upgrades, seamless, upgrades anything of that sort.

C

um We we introduced our own uh uh own catalog, olm catalog, that is, um and we we also wrote our own olm bundler, which you can think of it as like, an operator hub.io, but internal tube.com within the sas itself, and this catalog specifically has those specific operators, those certified specific operators which are scanned and been delivered.

C

You know, through the same gitoff's approach, uh to do any of the deployment or upgrades effectively giving operand as like a service or think of it as like a cloud within a cloud wherein uh teams can now choose any given service and you know, do the do the deployment and architecture around that particular part.

C

So, as we were scaling uh here comes in, uh there was one of the slide earlier right uh thousand ten ten thousand plus nodes, and you know hundreds of clusters. uh We quickly realized we we need to scale further. We need to do h a on on the uh on the cluster itself.

C

You know kubernetes multi-cluster or cluster h a so as it gains adoption.

C

uh There were certain of these. These particular queries that started coming in. How do I gain geo redundancy? How do how do I observe these fleet of clusters right from from a single uh uh from a single plane? uh How do I connect native community service across the clusters and that too, from an internal native two communities not like you know, going outside, uh not via the ingress or anything of that sort? How do we do that?

C

uh How do we architect something like an edge computing, um a very classic example, or a use case that we recently saw was uh deploy a mongodb cluster and us region, while have the edge nodes in europe region, which is just read-only, and you know. Yes, there were latency issues which we kind of you know overcame, but how do we do all of that particular thing? Additionally to that, from a github's point of view? How exactly do we deliver the same solution across multiple clusters?

C

How do we, uh you know, adhere to multiple security policies and how do we make sure all our clusters are adhering to that and the uh the same solution is kind of you know uh been adopted by a specific cluster. There is no drifting anywhere within the cluster. How do we? How do we do that? So as a community? Of course, you know we need a solution for, for that particular one.

C

Some of these solutions, what we identified have implemented and certain work in process that is going on particularly submariner mcs, which helps us uh joining multiple clusters to uh have internal services communicate with each other, something like open cluster management, which is still in progress, uh giving an uh you know, observability point of view uh pushing in common manifest. There are cluster level operators uh which teams do not interact, but we we basically, as like uh you know, uh within a team or or maintaining a sas platform, need to deliver.

C

How do we do that so open customer management helps us around that particular area concept of her hub cluster, with multiple managed clusters coming in joining in the hub cluster and uh you feed in something on the hub cluster. All of it trickles down towards you, know all of the uh the managed clusters.

C

So just a pictorial diagram over here for the open cluster management uh on the left-hand side is the same concept that I explained and on the right-hand side, you can see uh same security policies or any of the cluster manifests being feeded in into the same, get off pipeline.

C

What what any of the teams are using, irrespective of whether it is for a communities manifest deployed via helm, customized, we don't care or infrastructure pipeline, it's exactly the same pipeline, which is basically meant for uh for deploying it across uh uh or using the hub cluster and deploying it across the multiple clusters and speaking of security policies. uh I'll I'll hand it over to another to speak further on all right.

A

Hey so, as you know, we described right so our environment right, basically, as a team, we focus on. um You know the side of things, so that means hundreds of clusters scale that you saw right thousand, you know ten thousand plus nodes and the environments are basically, you know, uh varied in you know, uh because these are used with the sets of applications.

A

Customers are enterprise customers, so there is a strict or stringent requirements on security compliance, and you know regular audits and all that right, so, whether that's a you know, stock audits or pci or like fed ramp or, like you know, you're trying to go into you know like say alpha. 5 kind of you know so, depending on the environment type right there are, you know, multiple environments, that we spin up, so anything that we think of these. These are like building blocks, so we think of okay.

A

We want to spend one environment, second, environment or whatever right, and how do you, you know, put all these standards across all these environments or cluster one environment make this consist of 10 15 clusters altogether right and the scale is humongous right. So how do we uh do that? So uh you know we. You know the devsecops approach there right, because we think of this. You know whatever platform that we create. We think of platform end-to-end right right from you know, infrastructure layer that we spin up. So it's not like.

A

Okay, we are just thinking of pipeline only for application deployment, right, infrastructure applications, you know, and on top of that security compliance whatever we can put that as a standard, so that the t it's easier for teams to adopt that right. So all the you know. Fourth, you know thinking organization would always plan for that, because you know you can think start small, but you know you'll realize. Oh, it's basically gone beyond a scale, then it's very difficult to handle at that time. Right!

A

That's where we try to stay ahead of the curve and try to adopt if there is one team needing, as I mentioned right, you know we think there's some other team which is gonna, need it right. So let's use that first team as a beta customer for us right.

A

So uh you know with the security same approach right, so it was not like. Okay that you build it. You run it right. It's like hey. If you're going to build it, you are responsible for security too right, so we think of shift left model all the way right. You know, even in the infrastructure that billy mentioned right, you know whatever we provide.

A

We shift left in that scenario right we will collaborate, will have a you know, inner source. Will you know you? Can you know, do a pull request. You can look into what is being done right, but it's a shift alert here. It's a shared responsibility model for the product teams right they'll be able to adopt right, like you know, operators or multi-cluster services that we provide. You can use those, but at the same time there are standards right.

A

So it's not like there's not enough freedom, there's freedom, but there is guardrails as well right so from you know, security evolution perspective right because, as I said right always our sas products used to run in data centers. Then we moved on to the you know, vm, based hosting and then on, to the hybrid cloud, public, private cloud and then on to being cloud agnostic. So um you know even security has, you know, evolved same way right. So instead of you know someone some experts doing the scanning and coming out with the reports.

A

Okay, now teams could got into the scanning, but they still it was like: okay, application teams or security teams. So now you know we have been supporting containers in real time. You know security and compliance for all our environments right up to the developer level right. They can see what is going on right. Then. The security champions can also take. You know: uh action on top of that and security teams basically have moved into more of oversight role rather than actually you know, trying to you know, implement everything right.

A

So we have built our you know, uh devsecops. You know for the our entire sas platform ecosystem by having a multi-layered approach right here where we're gonna, because we, as I mentioned rate, we look end to end. Then we say: okay here is, you know where ca process is here is where cd processes right and then eventually this environment is getting deployed in production right.

A

So, as you know, like hey how operations teams would look at from the compliance perspective, some customer is going to look for here is my compliance and security needs how you are being meeting those right. So we try to address that right right in there, so we integrate security as well as compliance. Basically, both are deployment, build and deployment things right. Like say you know with that mindset in you know, we keep that as our focus right that hey.

A

This is how things are gonna get deployed, and then we try to automate right so because, with hundreds of clusters, if you don't do that, then you know you have a lot of problem right. Then we offer teams because whatever tool we adopt right, we try to make again shift left scenario included in that that hey teams will be able to access that and they'll be able to manage, and uh you know so, uh you know to make life easier for them. So basically result for us is basically we have.

A

You know, secure environments which are with the runtime defense in depth, and uh you know we have granular vulnerability and security uh compliance for policies enforced. So basically the what that does is basically our dev and ops teams. You know they can focus on more uh strategic work than rather than you know, repetitive work, so we look at a security at a holistic layer. You know whether that's a you know, network policies or whether that say you know you know or security policies or whether that's a web policies, tls policies or ssl policies.

A

We push that out to the environments. Right and teams are, you know, can utilize those tests, those configurations and then roll it out to production. So if it is okay, tls 1.3 is going to be standard, then that policies are available in all the clusters they can just adopt. To that, and you know the policy is already pushed out so basically uh with that the basics still apply right, because we create hardened images and that's where the all the stuff starts right.

A

We have okay, you're gonna, adopt the standard images that we are putting in out there, and then we use a continuous um posture improvement right with all the kind of scans, and then here are some of the best practices right that we wanted to share with the teams. You know we want to automate and enforce with automation so that there is less friction, people understand what is being done and they are able to live. You know make you know, use of it rather than you know one-sided.

A

You know it's like okay collaborative, and then um you know, along with that right. You know it's very easy to forget the dynamic runtime scans, where you look at the policies and security, because the containers will come and go right. You may not realize what has happened, so you need to be stay on top of that right, like hey those policies prevent your. You know, or you know, uh secure your environment during the night. Also right like in that way. Something came up. Was it compliant or not? Right?

A

Take action on that one if it was something which is not indebted and then, more importantly, right like there be always some new technology tools, apis which will deprecate and all that right, like you know, it's everywhere right, whether that's a cloud apis or some product apis or even um you know, even within kubernetes right. You have to stay on top of that.

A

So that's our basically how you know some of the best practices we wanted to share. So with that. uh Thank you, and uh you know, will welcome any questions.

A

Great. Thank you so much.

B