Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architectural Caching Patterns for Kubernetes - Rafał Leszko, Hazelcast

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Architectural Caching Patterns for Kubernetes - Rafał Leszko, Hazelcast

Kubernetes brings new ideas of how to organize the caching layer for your applications. You can still use the old-but-good client-server topology, but now there is much more than that. This session will start with the known distributed caching topologies: embedded, client-server, and cloud. Then, I'll present Kubernetes-only caching strategies, including: - Sidecar Caching - Reverse Proxy Caching with Nginx - Reverse Proxy Sidecar Caching with Hazelcast - Envoy-level caching with Service Mesh In this session you'll see: - A walk-through of all caching topologies you can use in Kubernetes - Pros and Cons of each solution - The future of caching in container-based environments

https://sched.co/ZerY

A

Hello, everyone, I'm rafael and I'm speaking to you from poland from krakow, and I will tell you about caching, and especially about caching in the context of microservices deployed on kubernetes, but first a few words about myself.

A

I'm a cloud software engineer at hazelcast before I work at google and cern I'm also an author of the book, continuous delivery with docker and jenkins. From time to time. I do training and speaking, but my main job is I'm a software engineer and, as I said, I live in krakow in poland.

A

Also a few words about hazelcast, so hazelcast is a distributed company. We produce open source software and we have three products. First, one is the most known, hazelcast in-memory data grid. This is what people usually use for. Caching, our second product is hazelcutjet, which is a library for stream processing and the last product is is quite new, is hazelcast put into the as a service in the cloud.

A

Our agenda for today is pretty straightforward, so there will be a very short introduction about caching and the microservices, and then we will walk through all possible caching patterns. Then you can that you can use in your microservices deployed on kubernetes.

A

When I'll be talking, I would like you to think about two things.

A

First thing is in your software, which of this pattern, do you use because this list is complete, so you must use one of this pattern and the second thing I would like you to think about is: would it make sense to change to any other pattern? Would it make sense for my system for my architecture?

A

Okay, let's start so. This is a short introduction, so in general, that's how that's a diagram for micro service world.

A

So you have. We have a lot of micro services, they have different versions, they can be written in a different programming languages and they use each other.

A

And now the main question when it comes to caching is where, where is the right place, to put your caching layer?

A

Is it inside of each microservice or maybe as a separate unit in our separate deployment, or maybe we can put cash in front of the microservices and that's what this talk will be about starting from the first caching pattern is the simplest thing you can imagine it's embedded cache so embedded cache is when you put your cache as a library inside your application.

A

So the flow is as follows: request goes to our kubernetes service, which forwards the request to one of the kubernetes port and application receives the request. Then the application check checks in the cache which is embedded have I already executed such a request. If yes return it from cash, if, if no do some business logic, put it into cache and return the response, so the idea is very simple: it. The cache goes together with the application.

A

It's that simple, that we could think even about implementing this on our own.

A

If you use java that that is probably how how your code could look like, so we create a concurrent hashmap, then we process the request check if the request is in the cache, if yes get the value from the cash, if no do some processing put into cash and return the response, so you can implement it. Your yourself, however, please don't please don't do it, don't do it, because a simple collection in java or any programming language is not a cache.

A

It's not a cache, because it has no eviction policy, no max size limit, no statistic: no billion cash, loaders, no expiration time, no notification mechanism, so it's it's lacks a lot of features that you will need later.

A

So you are better way better off by using some caching library. So what is a good caching library for java?

A

A very simple if you just need anything as a library is like guava cache, is a very, very, very good caching solution, where you can define all these parameters up front in the in the while building the cache.

A

Another good solution is eh cache.

A

We can even take this idea of caching. One layer higher and put the cache into application layer. So if you happen to use spring, that is how how your caching could look like. So you just add a cachable keyword. Now, every call to the method, getbook name by isbn, will first check in a books cache if there is such value. If yes return it from the cache. Only if there is no cached value, we will execute the method find book in slow source, but be careful because spring for some reason uses concurrent hashmap by default.

A

So you are better off first thing: you do in a spring application change your cache manager, for example, to guava or any other caching library.

A

So this is our diagram for for embedded cache. So this is pretty simple, but there's one there's one big problem about this. Caching, imagine that the first request goes to kubernetes service and it's forwarded to the pot on the top.

A

We do some. We do some business logic, we put value into the cache and so on. Now there is a second request, which is the same. It goes to kubernetes service, but this time it's forwarded to the application at the bottom, so the application needs to do this business logic once again, because these caches are completely separate. They don't know about each other.

A

That is why one of the improvement to the embedded cache is embedded distributed cache. So it's still the same idea. If you think about the patterns, it's still the same pattern, however, this time we'll use a different caching library.

A

So this time we will use for caching, we will use hazelcast, which is the distributed cache, and this time it it looks exactly uh the same flow is exactly the same, but all caches all embedded caches in all applications.

A

They form one consistent caching cluster.

A

So this is how how it would look like in your spring application. If we stick to this spring example, if you would like to use for oreo, cachable would like to use hazelcast as a cache.

A

All you need to change is to add hazelcast as a cache manager and then, as a for the caching layer, hazelcast is used.

A

Now you may think like okay, I start my applications. Like my, I start two instances of my applications. They both have like cache and now how do they form the cluster? How do I configure this? Because you see on this on this slide? There is no configuration, but they will still from the cluster and we thought how to make it very convenient to the users and we came up with the idea of the discovery plugins.

A

So now for every like deployment environment, we have a discovery plugin which uses api of the given environment, to discover other members and to form a cluster.

A

It's not only to discover members, it's also to provide like high availability to distribute across distribute the partitions across the zones and so on.

A

So you can choose for the um from the plugins. There is like now very trending, like kubernetes plugin, which is the most relevant to this uh presentation.

A

uh So if you are interested in details how how to use it, we publish blog posts, but in general there are two modes you can use. One is that plugin uses kubernetes api and queries the kubernetes mastered for to detect other members, and the second mode is to use dns resolution.

A

So we read records from the dns to also to to find other members, and it's all done dynamically automatically. So that is the good the good part, so that is our diagram for embedded distributed cache. It works really good. Actually, kubernetes is like very good for that, so short pros and cons of embedded cache.

A

So from the good sides. Configuration is very simple because the cache goes together with applications, so you don't really need to do anything. The latency of the data is is very low because it, the cache, goes together with your application again, and there is no ops, team or ops effort needed because it goes together with your application.

A

However, from the downsides, the cache management is not flexible. If you would like to scale up your cache, I would like now not to have two nodes, but I would like to have five nodes of cache. You can do it only together with your application, because it's embedded, it's also limited to the your language of choice. If you, if you write in java, then it will be limited to the jvm-based solutions and the last.

A

The last downside is that data is collocated with your application, which may be a bit maybe a bad thing or not such a big deal. That would depend on your probably your enterprise.

A

From our experience like big enterprises, they don't want to have their data to get rid of the application, even if it does a cache. No, they just want it separate.

A

Okay, we covered embedded cache now the next one is client server, topology, so client server is a little bit like a database, so we set up a caching server and application connects to the caching server flow is like this request: go to kubernetes service. It goes to the kubernetes port to the application, and then application uses cache client to connect to the cache server. We just deployed completely separate completely separate from the application.

A

And if you think for a moment like what is the difference between this diagram and this diagram, like from embedded cache, there are two main differences.

A

First, one is obvious: we have this guy on the this guy on the on the diagram and this guy is this guy needs separate, like management, but it also gives us flexibility. We can scale it up down as we want. We can do whatever we want with the cache server.

A

That will also mean you will usually need some ops team to manage this or just set or just part of the developer's time like it can be a devops team. It can be like a that, doesn't necessarily need to be a separate theme, but definitely some time to manage this. To maintain this server now, there's a second thing on this diagram that is very different from the embedded cache, and this is this part.

A

So now we use now we use cache client to connect to cache server, and the thing is that cache client can be written in totally different language than car server, because there is a well-defined protocol and you can use use it. Com like in a different different language- and that is a very common strategy that you set up the cache cluster or a lot of cache clusters, and then there are a lot of applications written in different programming languages and they use this cache.

A

That is a very common approach and a micro service world. It's so common that most caching solutions like the most popular redis or a little old school bus, still popular memcache. They are they're designed to work from from client server mode, and they it's actually the only way they can work like red is, I guess, written in c, so you can cannot really embed it in a in a natural way into your application.

A

Now how to deploy like cache server and cache client? If we stick to the hazelcast example, and we would like to deploy it on kubernetes so obviously for kubernetes, we provide a hand chart or an operator where you can just start the cache server and scale it up down, upgrade whatever you want now for the client part.

A

If we stick to the java example in spring boot and spread spring that this will, this is how it would look like the cache manager, so we defined like client config and then we say, use this kubernetes plugin. This is what I described before to discover the cache server. So it's dynamically discovered, discovered and, like you don't you you see, there is no static configuration here. So that is the whole beauty it will. The client will use the discovery strategy to find the server.

A

Now all good like we have server and as a separate, separate thing on our diagram, we can do, can manage this. It's it's! It's all good.

A

Now, if you think about like usually in a big enterprise, this cache server will be managed by a separate team, and if it's already managed by a separate team, we can move one level one step further and put this part into the cloud.

A

So cloud is still like a client server, but it's very specific, it's so specific that it can be considered as a separate caching pattern and it's specific because the cloud part is outside of our organization.

A

So we have this cloud thing which we, which we ask. Some service provider like in the case of hazelcast hazelcast cloud, create a caching cluster for us and give us the way to connect into this.

A

But it's so all these parts we mentioned before, like management or you don't you don't need it because you just pay cloud provider to provide you a cash to provide a cash for you now. The thing is, like you know like um we are in the in the caching domain, is like it's very strict in terms of latency.

A

It's even more strict than a databases.

A

So so you know like when you someone creates a caching cluster for you. You must be sure that you, the latency, is low and now how we do it like.

A

So the first thing we don't own the infrastructure we, our hazelcast cloud, is based on based on aws, gcp and azure. You can choose now. The thing is, you should choose the same cloud provider on which your kubernetes cluster is run so that they are, you know they use the same cloud provider.

A

So then you should select the same region, so they are in the same region in the same cloud provider, and then we even go one step one step further and provide you a way to do a vpc ring so that they are really in the same virtual network, your kubernetes cluster and your your caching cluster, and then there is no even one router hope between your application and caching cluster, and that is very important.

A

No matter, if you set up your cache server or use a cloud solution, you you must think about latency, because here latency is what matters.

A

For hazelcast cloud, if we stick to this, if we stick to this example of spring, that that is the configuration, would you which you gonna use? So you click for to clear, create a cluster and we provide you the discovery, token cluster name and password. You put it into your configuration and that's it.

A

So there is no static ip configuration and that way with discovery, tokens that is mo like most cloud solution, works right now. I think mongodb or anything like that. That's the way like for for discovery, you get discovery token and that that's all.

A

Pros and cons of client server and cache and cloud cache from the good sides. Data is separate from the applications. It's usually a must for big enterprises. From my experience, we have separate management, so we scale up backup. It's all separate from your application. It's programming, language agnostic, because we use client, cache client, clicker server, with well-defined protocol from the downsides.

A

We need some ops effort needed or pay for the cloud solution. Latency can be higher. I mean you should con. You must think more about it that that that's the thing and yeah, if you, if you set up your client server, client you're by by yourself, think about region, the pc and and so on.

A

Okay, we we covered, like the the more classic like approaches to the caching, so now something a little a little more modern which fits very well into kubernetes world, so cash as a sidecar.

A

This is the diagram for cash as a citer. So the flaws look as follows: request goes to kubernetes service. It goes to the kubernetes port. It goes to the application, but now the cache is not a separate thing on this diagram. It's also not embedded into application, but it's in the same port.

A

It's a cache server run in the same port as application container. That's it so the application can access the cache always on the local host, because it's the same port but technically it uses cache client to connect to the cache server.

A

So it's kind of a mixture between embedded mode and client server mode.

A

It's a mixture because, like it's similar to embedded because we it's around the same physical machine because it's the same kubernetes, but it can share the same resource pool scale up and down together and there is no discovery needed because we use localhost, but it's also similar to client server, because after all, we use cache client to connect to the cache server. So it can be any programming language and there is some kind of isolation between application and the cache which may be good enough or maybe not good enough. It's on the container level.

A

How it looks like in the in the code so again sticking to the spring example cache manager. We specify the static configuration for localhost, but you know it's not static after all, because we know that the whole system is dynamic. We just know that cache server is run on the localhost in the same kubernetes spot.

A

How the kubernetes configuration looks like it, we just run two containers in the same in the same pot, so we specify deployment, we specify the pot template and two two separate containers. The first one is our business logic. So this is your application that will this will differ from use k to use case and the second one hazel cost is always the same as the cash server pros and cons of using sidecar configuration is again very simple, because you you you've seen. That was all the configuration you had to do.

A

It's programming language, agnostic, latency, is low. There is some isolation of data and applications from the downsides again. The management is not super flexible because scaling up down. It goes together with your application and, after all, the data is collocated in the same application port which again maybe a big thing, may not be a big thing, dependent use k on your use case.

A

uh It may be a big deal for you or not.

A

The last caching pattern for for today is reverse proxy caching, and this will be something totally different because in all these patterns so far in every pattern, application was aware of the cache. So it knew that such thing as a cache exists.

A

This time we will be different, we'll put cache in front of the application. So this is our diagram so request before going to the kubernetes service. It goes to the cache, and if this request is found in the cache, we don't even um we don't even go to the kubernetes service or to the application just returning value from the cache.

A

Only if the value, if the request is not found in the cache, we go to the kubernetes service to the application and do some business logic.

A

As of now, if you would like to use reverse proxy caching, nginx will be your way to go anyway. Nginx is used everywhere, like in kubernetes, so you can. uh You can tweak it with. Caching like uh like this simple configuration, will add. Http caching to your nginx um and nginx is good. You know because, like the thing about nginx is it's very mature solution, so you can trust trust it. You know you can implement it right now into your production system and you are safe.

A

However, nginx also has has some drawbacks, like uh it's only for http, but that's probably fine, but it's not distributed. It's not heavy highly available. uh That is, that is a problem data stored to the disk. It's it's like. Actually it's stored in memory, but it can be offloaded to the disk. So it's a little different like with hazelcast. You were sure that your data is in memory, so it's accessed fast in case of nginx. It can be offloaded to the disk, so you have to be careful, but it's nginx is a very good tool.

A

So you can, you can start using this, but we can even go one step further and connect. This idea combine this idea of reverse proxy caching with a sidecar and create reverse proxy sidecar cache, and this will be the last variant for for this talk, so reverse proxy sidecar caching. This works like on this diagram, so request goes to kubernetes service.

A

Then it goes to this ferrari to the kubernetes spot. But this time it's not the application container. That receives the request, but it's some. I call the reverse proxy cache container, which is an interceptor for for the requests. So we inject a sidecar which is interceptor of our of all our calls, and only if it does not find cached value, then it forwards the request to the application container and, needless to say, they should form one um one class caching distributed caching cluster because we still want to have our cache distributed.

A

And this like this whole idea, like that, like we put cash in front of the application and not like the this whole idea, that application is not aware of caching, it has some good sides and bad sides, and we need to mention both starting from the good side.

A

So the good side is that you remember this diagram from from from the beginning of the presentation, so we have a lot of micro services, they use each other, they have versions, they are different in program different. They are written in different programming languages, and this is a small diagram. This is generally a mess quickly. Your system becomes a mess.

A

Now you can look at such a diagram in your system measure this, and you can look at this and say I would. I think I need to introduce caching for service to version one and service one, and you can do it in a declarative manner. You don't even touch the code of the service.

A

All you change is the configuration of your system and say I would like to inject caching here and there and that's it. That is the whole beauty of the reverse proxy caching, so how we would inject it in like kubernetes configuration, so this time like this could be a this is actually. This is actually a configuration of what we did. We did a reverse proxy sidecar prototype with hazel cast, and that is the kubernetes, but it is a prototype. It's not something you can use on the production right now.

A

So this this was our configuration. So at the bottom we have the our application. That's clear! Then we have this caching interceptor, the caching proxy. This is the whole thing it does.

A

It intercept the network if there is a cache value, return the cache value, if no forward into the to the application, but we also needed, like init container an additional internet con networking container like this container, what it did it changed the ip tables so that the any request from outside would go to the caching proxy and any request from inside would go to the application container, because otherwise it would be application that would receive uh all the requests.

A

Now, if you think about this diagram, it you may say okay, I already seen I've already seen somewhere a diagonal direct diagram like this, and you could be right, it's very similar to the idea of istio, so the whole ideas of istio is like here. We have and some some network interceptor cycler container, and it does some additional logic and and actually that's it like.

A

As of now, there is no metro solution for reverse proxy side car caching, I the this was the issue like um opened in re in envoy, proxy and uh very recently this year. This issue was actually closed, so the the caching was implemented in istio and I think like when it when you. Finally, it gets major a little more mature easter is used everywhere, uh so it's getting more and more popular.

A

So I really believe that this with the with the adoption of istio and this feature implemented- reverse proxy sidecar. Caching, this will be the way to go. This will be the way how we do. Caching, in a lot of you use cases, so as of now reverse proxy sidecar caching is is, is at the face of not mature thing, but I think it will become very popular, but I also mentioned like before that this idea of putting cash in front of our application also has some bad sites.

A

So if the application is not aware of the caching, there is one thing that becomes way more difficult and if you look on the internet anywhere on the internet, what is the biggest problem with caching in general, everyone will tell you the same. That is the cash invalidation, meaning when to decide that my cash is stale, that I should update my cash and when your application is aware of the caching, you can use some application business logic to say, okay. I would like to invalidate this part of cache.

A

However, when your cache is in front of your application, then you are limited to the standard http things like timeouts. It tags basically timeouts to force to invalidate the cache, and this may be good enough or not good enough for you, depending on your use case, but this is like something you have to think about pros and cons of reverse proxy and reverse proxy sidecar caching, so the biggest pros is and the reason why you will use it is it's configuration based, so you there is no need to change applications to introduce it.

A

It's also programming agnostic is everything. Agnostic application does not even know about it and it's very consistent with the microservices world kubernetes easter eggs. That's where the world is going that that's. Why? I really trust in this. In this thing, from the downsides we mentioned in validation, there is no major solution yet and it's protocol based, which may be a problem for you.

A

So with this last pattern, we we are almost done with the presentation. So, as a short summary, uh I don't want to repeat anything. I said before befo because it would be boring for you. So, as a summary, what I propose as a summary is a very short oversimplified decision tree, which will which can help you decide which caching solution is for me.

A

So first thing you can ask is: does my application need to be aware of caching?

A

If no am I an early adopter if no use reverse proxy with nginx, if yes use reverse proxy sidecar with this eq implementation or use a prototype for hazelcast?

A

If you are brave enough now, if your application needs to be aware of the caching, then do I have a lot of data or some security restrictions?

A

If no do I need to be language agnostic, if no use embedded and by the distributed cache? If yes use sidecar now you have a lot of data or security restrictions, then the last question for you is, is my deployment already in the cloud.

A

Do I run on aws and kubernetes on aws or or do I deploy kubernetes on my own servers? If you don't run in the cloud use client server if you run in the cloud, use cache as a service.

A

Now, this diagram is obviously a little oversimplified, but it gives you at least idea where to look for for the right caching solution and, as the last last slide, I would like to recommend some resources, so the first one is blog post on how to configure hazelcast as a sidecar. Second, one is this prototype I mentioned mentioned.

A

We did for hazelcast reverse proxy sidecar. Caching, the third one is not related to the stock, but it's very good blog post about caching in general, and the last link is a very good video talk from nginx on reverse proxy site. Reverse trophy caching, from nginx, and with this slide I would like to thank you a lot for listening. It was a pleasure to speak here.