Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Istio - The Weather Company's Journey - Nick Nellis & Fabio Oliveira, IBM (Any Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Istio - The Weather Company's Journey - Nick Nellis & Fabio Oliveira, IBM (Any Skill Level)

The Weather Company currently runs and maintains a "home grown" caching, routing, and authorization framework that is capable of handling 24 Billion requests per day. It serves more than than 40 different data apis to customers worldwide as well as users of weather.com. This talk describes our experiences in incrementally adopting Istio to provide fleet wide visibility and operational control. We describe some of the benefits gained, lessons learned during this transition.

"About Fabio
Fabio Oliveira is a Research Staff Member of the Cloud DevOps Department at the IBM T.J. Watson Research Center. He earned a PhD degree in Computer Science from Rutgers University in 2010. His research interests include systems management, distributed systems, and operating systems.

About Nick
Nick Nellis is a Devops Engineer with The Weather Company an IBM Business who helps to support and maintain services under api.weather.com. As a member of The Weather Company he encounters a lot of unique problems due to the volume at which they ingest and serve weather data. Currently api.weather.com serves ~24 Billion requests per day and his team is in the process of upgrading to service mesh architecture with Istio and Kubernetes. His previous speaking experience has been mainly from numerous internal presentations related to adopting modern technologies in incremental ways."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So my name is Nick Nellis I'm, a DevOps engineer at the weather company work on the core gateway team and we're gonna talk about our use of sto and how it's making our lives easier.

A

So to start things out, I want to give you a little background history on our company, originally The Weather Channel after some acquisitions and bringing some other companies, we renamed the parent company to the weather company and then in 2016. We were acquired by IBM, so I am now an IBM ER.

A

So today we want to talk about API weather calm. This is the main way that we build or we sell and distribute weather data. I mean we built a platform where, on this known as the Sun platform we are currently home to over 40 micro services has served weather data. We do billions of hits per day and we operate worldwide, mainly in four regions.

A

And so, if you look at the weather, calm, you can see. I have highlighted here a bunch of api's that are actually being called through API weather calm. So you can. You can see that the weather calm has many microservices behind the scenes, and different microservices are called beat on different triggers. So if you had a winter weather advisory, the alerts API would trigger that. So you could. This I think has five or six listed here, but there are many more depending on the pages that you go to.

A

We also have integrations with Android and iOS natively, and you would know if you're calling API at wthr.com. If you see the weather channel icon on your weather, app or some say weather calm, so we also serve traffic to these as well. So the Sun platform really sits between those platforms that I showed and the weather. Api is a that serve the weather data and we serve a number of functions for both clients, like you hitting the water, as well as the developers who are making these api's and distributing their weather data.

A

So we perform about seven main functions: authentication authorization, some API, key management. We have a sales pipeline, so is as a development team you enroll in our platform, you have a sales pipeline that they can take your weather data and go sell it and distribute it. We do cage caching, routing and billing, and so these are the main functions that my team serves.

A

So the existing Sun platform was homegrown, it's cloud native. We have a router main gateway and a cache generator and the cache generator is pretty important to us. It helps us align requests that look like some other similar requests. So if you make a request to get the weather in Copenhagen, most likely somebody's already made that request. So we build these cache templates, known as edge site includes that will also serve you. The same cache data that somebody else might have already asked, for we have three also homegrown off and billing applications.

A

We run everything a docker and we've recently migrated into kubernetes. We use cloud watch for a jar. Http monitoring and that'll be important as we get through this teo slides and something to talk about is our deployments.

A

We do them every two weeks and that seems very slow in this fast-paced world and there's a lot of reasons for that. Our homegrown solution. Currently, if we want to add a new API or new routes to our system, we need to do a new, build and a new deployment, and so our builds and deployments are really dependent on our upstream servers or our upstream teams. So if team wants to add new endpoints, we align them all together every two weeks and we become dependent on those changes in that stream.

A

So at times it's kind of painful to manage and our deployments take hours currently. So these are some points that we really want to address.

A

So I want a lot about the routing and because this is a big core functionality that we offer we currently route, 240 load balanced back-end services, so every person who enrolls with us it's not service discovery, they give us an endpoint and then we call their app through API down whether I come the api's of the 40, their managed by many different teams. So we don't actually Mott. We don't really manage those api's.

A

We just managed that the traffic gets to them and of the other platform services that we offer it's all HTTP path based routing, so a team will give us a path that they're their weather data is under and we'll make sure that weather date or the requests make it to them.

A

But before we send a request to them, we actually intercept those and generate the cash that cash template and with that cash template we can all. We can also determine if that data is already been cached and we can serve that before. The request goes all the way to the backend and then some stats.

A

We do four hundred thousand requests per second globally about sixty percent of that is cat is cached, and so we can take that down quite a bit initially, but of those those sixty percent that are cached about 90 percent of the data that we serve is in that cache response. So it reduces our load significantly by generating these cache templates and that leaves about 20,000 to 50,000 requests per second per region of the four regions that we serve. So it's a lot more manageable and that's really important.

A

So when we looked at SEO, we knocked it down to probably about 50,000 requests per. Second, is our requirement per gate or per region or gateway? So this is a very, very simplified diagram of the routing platform, the edge cache it comes into our gateway, which is intercepted and then sent to our cash generator. Our cash generator then actually responds on behalf of the weather API.

A

With this cash template, the cash template goes back into our edge cache and then, depending if it's cached or not, they may it education other-- to the actual weather API to go, get the data, but if it's cached it'll just serve the cache data, some things. The note is that we log our access logs to an object, store and then our HTTP metrics are pumped to cloud watch.

A

Oh another big important thing: you can see that our gateway is running in kubernetes, but all we were pretty early adopter of kubernetes on our team, but 95% of the weather api's do not exist in kubernetes within our mash or anything they are in VP sees and load balanced. So it's a there's, a big challenge for us when we were looking for a tooling is how do we route it outside of our mesh?

A

So some of the areas improvements that we really wanted. They were looking for a tooling to help us with was deployments. We wanted to get rid of those completely for routing changes and make it so teams could self-service routing, they could add routes at will, based on some validation and that we really wouldn't have to interject as much so it allow a lot faster work streams and then teams could switch routing really quickly, especially in production if they added new versions- and we didn't really offer a lot of tooling in the routing world.

A

So we want to add circuit, breaking and retries and blue/green to those develop those developers so they're using our platform. They could try. Canaries and blue-green testing- and one thing is our visibility- is not great. I, know and I'll show you an example of that next, so the Mott, the HTV monitoring. Definitely we need to improve. We wanted to alert teams when their ap eyes are misbehaving right now, it's very manual process, and then we also wanted changes to be very transparent.

A

So we wanted to make our platform very open and very transparent to whoever's using it. So it's a lot easier to use.

A

So this is a moment that we had a couple weeks ago and when we have Auto moments, they're usually big.

A

So we had an issue and we started repeat at 500 thousand five hundred errors per second or per minute, and so this was bad and we probably got alerted late because, as you can see earlier than the earlier in the hours, we got some small spikes of five hundreds and a lot of times. Those are just a back end that had some weird requests that they couldn't serve and mind you. We have 40 different backends that flow through our system, and so this doesn't really tell you who's causing the issue.

A

It could have been us and it also could have been one of the back-end api's. So we were slow to respond. Initially, we don't have the HD feed metrics that we want or that we need, so we actually crack open access logs in our environment, we generate 1.4 terabytes of access logs a day and so cracking open access logs. You can get like a two-minute window, it's reasonable to download, so we were slow to respond.

A

We ended up finding out that it was a back-end that was causing these 500 errors, and so we had to contact them and we don't currently offer many ways for them to for us to mitigate their help mitigate their issues. We pretty much just route traffic to them, so they know that they couldn't serve this request that they were getting and we couldn't really do anything about it, so they ended up having to patch their system and then do a deployment on their end.

A

We couldn't circuit break, we couldn't cut off traffic without doing a deployment, so this was a this kind of eye-opening moment that we needed to address, and if you, if you look at the timeline it looked, it took about eight hours to resolve so you'd. Imagine how many 500 errors we threw there's a lot, and that brings us to is do so. We took we did a lot of searching for different, tooling and a lot of those issues that I addressed that I mentioned.

A

We probably could have patched or brought into our system, but there was a great talk on the keynote last week that documentation is hard to upkeep. Every time you bring on new developers. They have to relearn this code. That was, it has been written before. So we were really looking for tooling to come through and save us, and we eventually landed on Sto.

A

But why do we lie? Do we land on sto and the big thing is that as a single platform for many tools, it integrates with a lot of things and it takes care of a lot of things. For us, it was significantly less code for us to write and document and there's really good documentation out there about ISTE already said. If we brought someone on board, we could just point them to the sto site and have them follow through follow they're.

A

A big thing that we really like and being a part of IBM is that the enterprise support and testing of Sto, so you'd they're, backed by IBM, Google lyft, and you know, they're doing a lot of good work behind the scenes. There was a talk earlier about performance, and so there's teams of people already performance testing. It regression testing it and that's something that we don't have to do now. We can do a testing on top of that, so we're really happy that their use cases seem to be even bigger than ours that they're targeting.

A

So we know that we can fall under the sto umbrella and it should be able to support our load. It's also cloud agnostic and cloud native. The cloud agnostic, one is very important. One is in the newer version, o 8 I believe they're working on multi cluster support. So you can run as few in multiple clusters in multiple clouds and that with a seamless experience, so that's also very important for us. So we started looking our platform and we started seeing a lot of sailboats everywhere and we're like wow.

A

This CEO can just replace everything and we can go home at night and sleep, but in fact it doesn't really replace everything, but it does integrate with a lot of tooling and specifically routing and monitoring is something we probably could replace with this year or have replace with this year. But it integrates with billing systems and often authentication authorization. So it was really appealing to us down the road.

A

We could try and integrate it more into our system, but initially we targeted routing and monitoring so I'm going to talk about a little bit about the routing, it's pretty neat and it's, but for the O 8 version which hopefully be cut in the next couple of weeks or week or two, and it has a lot of functionality that we helped work on and test so that we could do run our workload.

A

So I'm gonna walk through a pretty simple example, and this is for our current mother. So every time you ask for current weather, this is kind of the flow that would go through through sto and I. Put up the SEO components here that we we use and they're all in the O 8 version. So in the earlier versions, you probably won't see the same names, a lot of things that renamed so traffic will come in through our edge cache into our match.

A

It'll hit a new concept, which is the SEO gateway and that gateway will decide where the where that represses gets routed and we intercept it and we send it to our cash generator and then that'll go back to the edge cache and if it needs to retrieve the data, it'll actually go back through the Gateway and then to the current weather API on the right side. Here it shows kind of what the cache generator is actually doing.

A

So, if somebody calls our current weather endpoints, which is V 3 WX observations, current the cached generator actually changes that that URL to n adds slash origin in front of it. So if it does make a second request, we know by path based routing that that request actually needs to go to the back end, and this is an ESI template. A lot of edge servers support this as well.

A

So this is probably the most important piece of this CO that allows us to do this. If we go back one slide, you can see the current weather API does not exist in our mesh. It's low balance, that's sitting in a V PC about 95 percent of our API s, or virtual just run on virtual machines and not in kubernetes and not in our mesh and so service entries are the single most important piece, probably to us and what they do.

A

Is they allow you to take external services and bring them into your mesh and treat them very similarly as if they were native and it's very powerful for people who don't aren't all in kubernetes are not in their service mesh. So what we do is, we just add an endpoint, so current weather that internal weather, calm, and so now, if you knows about it, it knows how to route. It knows how to treat it similarly to something in the mash.

A

We just define a port, and so currently we have 40 service entries, they're dynamically update able. So if a team decides they want to change their their back and end points, we can just update that in real time and you can't use them TLS, we haven't, haven't used it yet, but it is available.

A

So the Gateway is a very new piece. It kind of replaces the ingress controller in the newer version, and this is our main point of entry, so we're defining that everything from ap I doubt whether comm comes in on port 80, and so what we can do with these gateway is we can assign specific out route rules to this gateway, so it knows how to how to manage those routes and forward them on to our service entries.

A

You can have n number of gateways, so it's a lot more extensible and if we wanted to have a different control plane for different types of requests, we could have different clusters of gateways and they just run Envoy under the scenes, and you can also do it in the egress. So you can. The gateways are supported in and out.

A

So the first routing rule that we put in here is the intercept we gotta, we gotta, intercept it and send it to our cash generator. So we take that v3w X observations current and we steal it and we send it to the cash generator. This is the new route rule if you're familiar with SEO- and these are also dynamically updated and if you note that there's a gateway selector. So if you had multiple gateways, you can choose specific ones or you can just say mesh as a whole, but we assign this to the ingress.

A

So then this, if a request is coming and needs to go to the origin and get fetched that new data, this is a role that we use. So we look for the origin request, so it makes it very easy for us to identify if this needs to go to a back-end and then on the way out. We rewrite it back to its original URL, so the back-end team actually doesn't actually know that we did this.

A

We tell them about it, but they don't actually need to care what we're doing internally, and we can do a lot of things with these new rule, specifically a no 8 release. This new already supports a lot of retries and individual timeouts, but outlier detection and some other pretty neat circuit braking. You can do with this new route rule or these new virtual services so monitoring.

A

This is a probably the thing that catches the most people why they want to use SEO first and monitoring is really baked out of the box in this view, and that's really appealing really appealing to us, because it was a lot more powerful than what we currently have.

A

This is a little example that kind of shows you the isolation of what you can monitor. So each of these different lines here represent a different service and possibly a different response code. So we can do a lot of targeted, alerting and monitoring of our service backends, just just natively with SEO, and it was it's really powerful and it was something that we didn't even have to really build with Griffon and all these metrics. We can actually now tune dashboards for specific backends.

A

So if your team doesn't have great HCP monitoring, we can now offer you dashboards where you can actually come and see the the traffic that's coming to your your back-end and it be a specifically tuned dashboard for you and we don't actually have to change much. We put the variables at the top- and this is the that the default is to your dashboard, so they just select their destination or their app as a destination, and they can see all the traffic.

A

So this is the once you get all of these metrics. You can do some pretty cool things we, so we take these metrics that were sent to Prometheus and we actually pumped them into Netflix is visceral. This is a really cool visualization and it actually has some really big benefits for us.

A

So what this is showing is this is our mesh from the Internet to the mesh it. This is a demo that I set up where there are some backends that are not healthy, we're showing about 27% error rate and it's a very small load about a hundred requests for a second we can zoom in and we can see all of our mesh, and this is from Gateway to those external services as being treated natively in SEO.

A

We can then filter down to the endpoints that are not healthy, and in this case they just don't exist to prove my demo, so we filter down with these endpoints that don't exist, and you can quickly identify where the errors are happening by the red dots. The red dots mean 500 errors, so we can click on one of these endpoints and see when those started- and you can see here this one never really existed.

A

So we we know that it was all a hundred percent error rate, but we just we just found this issue within five fifteen seconds of popping open the dashboard and we it previously. You know it took us hours of scraping through logs, so this this is very important to us being able identify issues very quickly when we couldn't in the past, I also coined the term vist tio, a kind of a cool term. A cool dashboard that we can hopefully offer in the future with is tio.

A

2 look gives you a little more than the current service graph, offering.

A

So, where are we today? We've been using this tio for about eight months, API to weather, calm and our large load is currently not yet in production, and the reason for that is the oh wait: release that is being actively worked on. It has a lot of the features that we need to be in production. We've done a lot of extensive testing with earlier releases and that they do seem to already serve the needs that we we have from a load standpoint and we're actually pretty happy with the status of a sto.

A

But we just need some of the functionality to run this in production. We do run some smaller API, so it with this um production and that's been really beneficial and like working with SEO and a production like environment and kind of understanding how it works and some of the quirks, maybe that might need to be ironed out. You treat things a little bit different if it's in production, so we run on the O seven one version in production for these small api's and we're actively working with the SEO community on the o.

A

Eight support so some advice for us who use this do and if you're looking at you seeis do it starts small ingress is a very easy way to get started with this Co you could get the metrics. The sidecars are often difficult as developers a lot of time. You want to just throw that sidecar in there and see your metrics come through, but there's a lot of things you have to think about by adding that sidecar, and so you have to play with it and get used to it and understand what it's.

A

What issue is doing for you? If you're running a current version of SEO, your upgrades have to be tightly managed up until the late release. A lot of functionality was in flux, and this o 8 release is really gonna kind of nail down what we want to offer for the 1.0 release. We should be coming a couple months, so stability is coming and if you're definitely not late to the SEO party, but it'd be a really good time to start checking it out after this o 8 releases is released.

A

So, as a recap, this fee allows us to resolve issues a lot faster and I. Just showed you an example of that. We now have the ability to offer that when api's are having issues, we can offer some sort of mitigation for them. So if they know that they're gonna have problems, we can circuit break, we can cut them off until they're healthy again. It allows us to be way more transparent in the routing.

A

Currently, teams have to come to us and we have to help investigate how traffic is routing to their system, but isseo with the telemetry and the monitoring allows it to be very, very transparent about what's going on to a certain back-end, it reduces a lot of custom code, and this is a really nice feature when you've gone from a homegrown system and we also get the dynamically update routes and that's allows teams to really try things out in production and take a release, kind of beta api's and try them, and we allow that to be a lot quicker and so a lot of advantages for our argues.

A

So thank you. I.

A

Don't know if we have any time for questions if you have any questions um come up to the mics I was told.

A

Your question go for it.

A

Yeah, so the question was: how do we do it in place for placement of our current routing system, and the big thing is making sure that we have feature parity? We want to make sure that when we drop this into one of our regions and we're gonna slowly migrate traffic over to it that we have that feature. Parity and we've been doing a lot of non profit to make sure that, from the backend perspective and the client perspective, you will notice a difference and the o8 release really allows us to offer that feature parity.

A

So our current routing mechanism, our gateway, was actually written, pretty well a routing spec. So it made it easier for us to cut over when we're ready.

A

Probably within the same cluster yeah.

A

So do we need to distribute it tracing with this? You know and not yet, like I said you need to start small. We definitely are going to that as a big thing we want to target, but because our mesh is very very flat, we don't need it right away. The monitoring that is the artery gives us is pretty sufficient for our case, but we definitely are targeting it.

A

All right well, thank you guys.