Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Cloud Native Journey at Adobe - Carlos Sanchez, Adobe

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Cloud Native Journey at Adobe - Carlos Sanchez, Adobe

We will share our journey migrating one of Adobe's enterprise products to Kubernetes, running multiple clusters across regions, as well as the processes and technologies that made this possible, including Envoy, Helm or Prometheus. Adopting DevOps has improved the communication between teams, which are now organized by functional areas, no more Dev, QA, Ops teams. Continuous Delivery allows us to highly automate software delivery with multiple deploys a day, and using GitOps we have improved visibility of the services running in the different environments. Using Prometheus and Grafana allows us to provide visibility into the state of the system to everyone in the team, as the whole engineering organization participates on on-call rotation to foster a shared responsibility across services.

https://sched.co/ekAj

A

Hello and welcome to this talk about the cloud native journey at adobe, I'm going to talk to you about how we use kubernetes and other cncf projects.

A

What went well, what things didn't work so hopefully you'll learn some lessons that will help you on your own teams and join me in this in this journey.

A

So I'm a cloud engineer at the top experience manager cloud service. This is a product that I'll give more details, part of the adobe portfolio and and have a background on open source. I wrote started the yankees kubernetes plugging and I contributed at jenkins sex, apache maven for many years, eclipse, foundation, etc.

A

So my background is a lot on the on the open source community, so adobe experience manager. What is what it is? uh So we can understand better. What are the challenges we have to go through?

A

So this applies to my team inside this uh adobe experience manager cloud service that we we started announcing uh at the beginning of this year, but there's other teams that uh in am uh experience, manager, there's many teams in adobe a lot of them using kubernetes too.

A

So this just so you know, applies to to my team adobe experience manager is a content management system, digital asset management, enrollment forms, and it's useful by many 4100 companies, companies you, you already know and are already customers of, of the adobe experience manager, and now we are providing this cloud service. It's an existing distributed job application.

A

You have author instances where authors can create content, publish instances where visitors can see that content that was created, and this can all scale horizontally. So this already existed before the move to kubernetes. This has been already around for years.

A

The stack is using java with osgi and a lot of open source components from the apache software foundation. There's a huge market for extension developers that write modules that run on am in process and I'll go through. Why this is important to to understand later on.

A

The challenge for us was to run adobe experience manager on kubernetes, so getting into it.

A

We are running on asia, we have more than 10 clusters and we keep having clusters, multiple regions, us europe, australia and more coming, and we have at adobe a dedicated team managing clusters for multiple products. So it's not that we don't create the clusters ourselves, there's another team that takes care of that.

A

Iem environments uh so multi every customer can have multiple am environments that they can self-serve, so they can create environments on their own and for each customer we have at least three namespaces, like development stage production environments.

A

We also have sandboxes that are evaluation-like environments, and all of this is managed by the customer themselves through cloud manager, which is a separate services, the service that has their own its own web ui and api.

A

So customers can have multiple environments that match to multiple kubernetes namespaces on the names on the environments. For us, the namespaces provide the scope like network isolation, quotas permissions, so we use that that kubernetes give us already and for each environment. We use a lot of init containers, soft containers to apply division of concerns, so we have sidecar containers that do a storage initialization or we have httpd front in the java application or exporting metrics.

A

We also use fluent bit to send logs and another example- is have a java thread: dump collection, where we gather the thread, temps and store them.

A

So we have several side cars.

A

Custom developed like the threat dumb, collector, the storage initialization that are very particular to our use case, open source like flow and beat, and some that we extended from from open source like httpd, and we have to scale these to hundreds of customers, thousands of sandboxes so for us that that's the the it's a big challenge to to make sure that this, whatever we built uh services we built around it, they will scale some of the issues we face with the scaling.

A

uh You can run on azure apa rate limits, uh especially on upgrades, so we have to limit each cluster to a few hundred nodes. uh So you could solve this also having bigger nodes. So you get high bigger cluster sizes with the same number of nodes and we use the kubernetes vertical. Hang horizontal put out the scaler extensively.

A

About the vertical puddle to scaler, we use it to scale down up and down on memory and cpu, but this jvm footprint is is hard to reduce right. If you are aware of how java applications work memory, you basically reserve the memory on a startup set the hip size and then it's up to the jvm to manage that memory.

A

One thing you have to be careful with the vertical parallel scaler is that changes to the request need pot, restarts to become effective and make sure that you don't set the vpa to do it automatically, because otherwise you would get some random butt restarts and, depending on your application, that may be a problem.

A

We also use the horizontal pad of the scaler and we started to scale up on requests per minute, and one thing that you have to be aware is that you cannot use the same metrics for hpa and vpa.

A

That's on the documentation of kubernetes.

A

One service we built to manage the scale of the clusters. We implemented, something that we call hibernation for environment environments that are used by engineering and sandboxes that are sunderly used.

A

So we uh scale down these environments, and this allows us to overbook clusters and and save a lot of money by doing that and putting being able to put more resources into into one cluster how this works is very simple: we have a kubernetes job that checks prometheus metrics.

A

If there's an activity in the last number of hours, we scales a bunch of deployments down to zero.

A

The customer, the user is showing a message to the hibernate by clicking a button, and that's that's all there is to it. I mean this. This allows us to scale the clusters and get more uh packing in in the clusters.

A

Ideally, we, it would just the hibernate automatically write on a new request like uh functional service k native this sort of of things, but you have to account that jvms uh in in this product it takes around five minutes to start depending on how much content you have how many uh things you have on on the store, but um unless you use new jvm micro frameworks, uh the jvm, if you are taking a existing application, it's probably going to take a good amount of time to start.

A

Networking we we use the networking capabilities from kubernetes, which is it's a bit complex and we have to account for multi-tenancy we're running multiple customers on the same cluster. So we limit things. Services cannot connect to other services in other namespaces and we block everything by default and we open specific cases as needed. So we start for a fully blocked and then open open as needed.

A

Everything is virtual, um kubernetes we're networking. This allows flexibility. Body introduces complexity, of course, but this is managed by kubernetes and we have some issues, but overall it works. It can work pretty well.

A

We use psyllium for networking and uses ebpf instead of ip tables, it's more efficient and performant, and allows us to have custom network policies at level 7. So things like policies on path, http, headers, http methods. So it's a it's a higher level api that allows us to to define more fine-grained constraints.

A

We use the network policy, object to block or allow traffic. We block access to all the other namespaces, and we just allow some outgoing https and common ports.

A

Customers may also want to allow specific things like igris ips uh say we have a you, have a and development environment only be accessed by your ip range of your private network. So we can.

A

We can do that too.

A

For ingress we're using a contour fork that has more features and also has more features at the standard. Kubernetes ingress object, so we use block list, allow list path based routing stuff like that, and it uses envoy behind the scenes. So we use we heavily use envoy.

A

Envoy, it's something that is like a kind of love-hate relationship where you can do things. It can break in many different ways and you can pray badly. So it's! uh If you have envoy misconfigure, then uh you can cause a cluster wide outage and like if your configuration, you update your configuration and it's wrong and then you restart envoy. It will clear all the envoy routes.

A

So you get a cluster wide outage and we have issues uh when the rate of changes is too high when it it gets uh locked and things just uh start crawling by crawling down, but it's very, very powerful and we had to do some work to fix issues and to use it correctly and things that we do, for instance, now is uh validation of all the conflicts both at build time and run time.

A

So this is to prevent causing any any problems. As I said, it's very powerful, but if you do something wrong, it's being used everywhere and it can cause a cluster-wide outage.

A

For logging, we use fluent bit sidecars that sends locks to centralized store. We also use grafana lucky for for log aggregation and for monitoring and alerting typical, prometheus grafana, and then we aggregate all the clusters data and we have alerts coming from alert manager. So that's very typical stack.

A

We have a feature that is the customer login, so customers can also need access to some of these logs and we use fluent bit to send the logs to either log stash and recently loki and the customer can see them on this cloud manager ui and get those laws and see what whatever it's in the logs on broad from the application or their custom code block stage.

A

For instance, we are moving away from it because it's a jvm heavy and it uses a lot of memory and we're thinking that lucky lucky is a better option for for this multi-tenant services.

A

Resiliency and self-healing some things that you have to have be aware of so you have to have readiness, liveness probes, make sure that your services are and are marked and available if there's any problem and that they are restarted automatically.

A

If, if in this case something goes wrong, you have to have bot disruption budgets to ensure you have a minimum number of replicas and rollouts and then cluster operates so make sure that you always have one pod running of a specific service. If you want or fifty percent of the paths, so they are not automatically killed by kubernetes and paulden the affinity to distributed services across nodes and availability zones, because if we have multiple availability sounds you gotta make sure that not all the pods will end in the same availability zone.

A

So you want to use this spot on dfinity to spread out the load across hcs.

A

So we're building a very multi-tenant service, so we try to always everything we write limit. The blast radius. As I said before, customers are namespace isolated.

A

We make we enforce that all the deployments have cpu and memory requests and limits. So this is key.

A

We are also running customer code, so this is an interesting topic because um basically customer can write something that can take down or cause trouble to their instances. We run checks before that code goes into into the the production clusters, but there's always that that risk. So these spots, uh we are started testing carter containers where the pod runs in a virtual machine transparently.

A

So it's a very nice project. It's it's transparent. You could use in your cube, ctl, apis and command line and apis, and we are also.

A

Developers from this other team that manage the the kubernetes cluster for us are contributing improvements upstream, so we are helping improve cutter.

A

Containers external services we use, we not don't, run everything on kubernetes. We do have uh like external model db and that's that's running outside the cluster. We use a surplus storage, these two things for persistence and we don't use much uh kubernetes persistent volumes because it's a bit risky. Obviously it's better. If you don't need to use them, there's just that some people need to use them because of their existing applications and so on, but especially in asia.

A

We found that we hit the azure api right rate, limiting when having hundreds of persistent volumes. It's been a few months since, since that happened, but we stopped using them and using other options more like externally, storage for for data.

A

For data processing, we have a service based on kafka, and this allows us to do syncing, syncing of data between author and publish and this works worldwide when we have publishers in multiple regions, because customers that create content may have publishers and publishing information publishing their websites and their data and their assets into multiple regions and this uh kafka service kafka based service allows us to to sync all these regions.

A

For cdn, we use fastly in front of your kubernetes load balancer and it also fronts the binary content that we store in azure blobs.

A

One new feature that we have to build, for instance, was an igor's dedicated ip, so customers want to to have a dedicated ip for either firewall configuration and the igris ip, either firewall configuration or to be embedded in throttle or blocked by other tenants like if you have multiple tenants coming out of your cluster there's a range of ips that is used and it's shared across all tenants.

A

So we can provide customers with a dedicated ip and they it's exclusive for for these customers.

A

So we built this uh setting a skill set of proxies outside of the kubernetes cluster and with the round load balancer, and this is dedicated for each customer and we use the network policies to allow only the customer to access the proxy, so proxy can go out of to the internet, all the the pods sorry, the bots can of the customer can go out to the internet through this proxy and this will set a dedicated ip. That is only used for that customer.

A

For continuous delivery, how are we managing this?

A

We move from uh yearly on the non-cloud version of the product to a daily release on kubernetes, so this has its own challenges.

A

We use jenkins mostly for ci cd, and we use some also cues to trigger some jobs. There's there's a yankees plugin that can do that. For you, we use extensively githubs. All the configuration is storing it and reconcile on each commit and we have a model of pull versus pulling. So we can scale right if you have to push the configuration from jenkins to all the clusters and all the namespaces and so on. This is not going to scale and we have services in the cluster that are pulling this data and doing the reconciliation.

A

For kubernetes deployments, we use a combination of helm for the adobe experience manager, application. We also use plain kubernetes files for some services and we also use customize for some new microservices that we built. So we have a mix of of all of them.

A

We use extensively helm and things that we learn is like. Don't mix application and infrastructure configuration in the same package because then it makes it hard, and if you need to make a change because some infrastructure change, then you will have to redeploy all these helm releases. We have thousands of them and helm also needs to push to all namespaces and it's easier pulling. So we are moving towards the helm operator that will improve this situation.

A

Things we use during ci and ct to make sure things, don't break our cube valve to validate the kubernetes schemas and we have some custom crds and we created schemas for those two. So every pull request every change goes through queue, file, validation.

A

It also goes through conf test using uh open policy agents open policing rules.

A

So we validate every object in kubernetes with conf test rules, so developers can still create pull, requests and and see the the output of the of these rules, and they still have autonomy. They can see what they should do. What are the best practices by by having these rules?

A

So are things like security recommendations, leveling leveling standards, image sources like you want to make sure that all your docker images are coming from a specific registry and not from from other ones, and this it's very helpful for for um letting developers manage their own services and their own features. So all around the on the self service part.

A

So that's it. I hope you um enjoyed and get some examples on how these technologies are being used in, like a real life use case, uh existing application because a lot of times we talk about kubernetes and it's all like greenfield development, every new applications with the latest technologies.

A

So this I hope it shows how you can bring an existing application. An existing.

A

Service that has not initially been thought of on the kubernetes world. How you can bring something like that into kubernetes kubernetes makes it really easy to do this, lift and shift into into kubernetes of something that is already running, and I hope you got some ideas and that it helps you thank you for being here and we are hiring if you know any of these technologies that I mentioned today.

A

Let me know you can ping me on twitter and also you can ask me any questions now. Thank you.