Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: HomeAway's Multi-Cloud Traffic Director - Mark Tyrrell & Andrew Hodges, HomeAway

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

HomeAway's Multi-Cloud Traffic Director - Mark Tyrrell & Andrew Hodges, HomeAway

Case Study about how HomeAway built a global Multi Cloud Traffic Management solution using Content Distribution Networks and a hybrid of public and private clouds. Over the last two years we have grown a team dedicated to extending our capabilities related to routing traffic to thousands of applications, microservices and functions produced by our developers. HomeAway's stack leverages CNCF projects including Linkerd, Open Tracing, Prometheus as well as Kafka streams to support plug-and-play adoption of future observability components. The talk will be supported by live demos and example code.

To learn more: https://sched.co/GrSV

A

Hello, everyone I am Andrew Hodges I, am the engineering manager for the edge team at home, away with me on stage. I have mark Tyrell, he's a principal architect at home away and today we're going to give you a look at what it takes to run home away. It's a vacation rental marketplace, operating on multiple clouds around the world and to begin we're gonna do some background on what we do. What who we are HomeAway is changing the face of travel.

A

We have over two million places to stay in 190 countries in 2015 we joined the Expedia group. Family Expedia has over 200 booking sites in 75 countries, so that should give you a good sense of global traffic that we are Tasso managing. You may also recognize some of these brands on the left. So when it comes to directing traffic, it has become increasingly difficult to break up locations into smaller and smaller components.

A

Long time ago, traffic cops were enough to direct the flow of vehicular traffic, but as with horse-drawn carriages model, lists are being replaced by more sophisticated systems, utilizing containers and functions.

A

Some solutions don't work out as well as we hope they would such as this roundabout, which seems to cause more problems than it solves.

A

So when thinking through our approach to managing our global footprint, one quote came to mind from the CTO of fastly Tyler McMullen. He says there is no edge of the Internet.

A

While that may be the case to get a request to the destination, it has to go through a series of proxies, while the number of proxies to get you from point A to point B may grow or shrink. It is important to recognize that some of these are observable while others are not here's an example to further illustrate traversing proxies.

A

So let's say we have a request that comes out of Australia that wants to reach our data center in Austin Texas and must first reach in by a bus first go through a proxy in Hong Kong and then through a proxy in Canada.

A

Increasingly, our distributed systems are being deployed across multiple regions across multiple cloud providers. Here's an example of the data center footprint for one of the major public cloud providers. Each cloud provider offers up to 54 regions with that number growing year-over-year.

A

So that's why we're here today we're here to show you how HomeAway is able to deploy across any cloud footprint as we build out our mature composable platform, our applications must align on a few key features, the first one being discoverability. Our applications need to utilize service registries and registry aware proxies.

A

Additionally, our applications need to be observable. This is the data and metrics that drive decisions, not just for our engineers, but also for our product managers and finally, our applications are actually our customer facing applications need to be routable, so this is critical to the success of our goals. Around resiliency, high availability, scalability so no one hand over to mark to give you a look at the architecture powering our traffic director.

B

So this is AM architecture view of traffic management, as we see it home way today and I. Think one of the things I wanted to call out is: you know over time. We we made investments in different areas. So when we started out, we were primarily thinking about regional data, centers and the proxies and that layer, but it became apparent over the last couple of years as we ran a service mesh scale that we also got a lot of value from the global layer from the CDN.

B

The content distribution network so well, that provides, is logging, metrics and tracing very close to our customers, and you can see this in this again somewhat of a scared view of our stream processing pipeline that deals with the logs that come out of our CDN and you can get a lot of value from it. You can use logging tools and metrics tools and tracing tools, and we run all of those again globally distributed and they provide different kinds of insight.

B

So when you think about how you manage routing, there's an important principle around having a control plane- and we stumbled upon this really, and so what you see here is a pretty rudimentary example that I'm going to demo in a minute we're routing a path based route and we're using a control plane, which is basically an event-driven stream streaming architecture that propagates these changes globally, and we use agents that then configure regional, low balances and global low balances.

B

So, at the global level we work with vendors who provide CDNs and that's a bit trickier because you have to deal with different API so having an abstraction layer. There is really helpful because it means we can change vendors relatively easily, but you still get all the leverage and benefit of having their large number of points and presents globally. And if we look at a few failover scenarios, these are things we again kind of stumbled upon as we started to leverage our CDN more.

B

So this is a classic example where you have an app that's unhealthy and region a and it's sending back a 503. So this works quite well in a get scenario where you can just retry in a different region. However, for something like a post obviously doesn't work.

B

So one of the things we've tried is: can you propagate some kind of relatively up-to-date current view if the health checks globally and then use that to preemptively route, to healthy regions in an active, active or active passive scenario, and that works reasonably well, but we're kind of reaching the limits of how a traditional programmatic CDN would work, and so there's an opportunity here to think about.

B

Well how how do you increase that kind of interaction efficiently, especially if it's a global layer, because if you're running, multiple regional datacenters, that's quite a lot of information that has to be transmitted back to that global layer. So, moving on to what we're going to show you today, one of the things we've we've been very positive about within our team is embracing a multi cloud multi-region reality that we have to support and I'm just going to show you a very simplified view of that, some of which you can play with on github afterwards.

B

So what we're going to show you here is just a scenario where we're routing requests to a trivial nodejs micro service, that's running on three public clouds and just for my own amusement. They all have different journeys to deploy that code and I'm not going to tell you which one was the easiest, but there's an obvious winner.

B

So this URL, if you want to pull it up on your device, is where you can follow, along with the first demo I'm going to run, and please forgive my front-end coding skills. So let's go to that URL and see if the Wi-Fi is cooperating with us and sadly there's no cloud here. We've got a sad cloud. So what are we going to do about it?

B

Well, what I'm going to do is I'm running a control plane on my laptop right here and I'm gonna post a trivial JSON payload via a REST API, and that is going to add a path, mapping for slash cloud and the host that you've already pulled up. Possibly so, let's run that so I get nervous and looks like the local change took. So now we can go in so what's happening in the background here is this is being pushed to a CDN, that's globally, distributed that supports service workers.

B

So there was another opportunity for us to try out kind of functional programming at the CDN layer which definitely unlocks a bit more flexibility around how you do load, balancing and multi-region failover, and we also use it for Abe testing experiments and also canary deployments. So, if I hit refresh now yep, we've got a user interface that I built and I'm gonna play a video of what happens next, because I don't really find it easy to talk and type on this one.

B

So I'm just going to play this for you, you can interact with this on your laptop, but the idea here is we're interacting with this nodejs service and it's basically just echoing back text from different regions and I deliberately picked regions that are all over the world. Well, that was probably a mistake. So what you can see here is just a simple, a trivial example of how this multi cloud routing works. So let's go to a different demo.

B

So now what I'm going to do is- and this one getting ahead of myself, so you can play with this. It's on github comm, slash home away, / hollow cloud, and this is just a sandbox that shows you some simple things like how we use link addy, 1.0 service mesh to do convention based routing and also just how you can have a very lightweight control plane to have programmatic routing.

B

So, let's, let's try this one, so I'm just going to dock a compose, build and then I'll do a docker compose up, and so this is just orchestrating some different components, some of which we use at home, away like console and Link addy, and now that this is running. If I go over here and I break out the presentation for a moment we can follow. You can follow this. If you go to this github page. Let's read me here looks like it's loading, so one example I can show you is.

B

This is an example of path based routing now, I, don't know if you can see this so bear with me. I've got this little app here. Okay, Wow now I feel good grown up. So if you look here, what we've got is live as a tag in console hello cloud app is the name of a servicing console.

B

So this is a simple example of convention based routing and then hello, /, hello is a path that that nodejs out in response to so this is just an example of zero config writing, which is probably not practical, especially for a company like house where SEO is really important. We don't want to just open up every kind of path based routing scenario when a developer pushes a code, but it is really useful at the regional layer- and this is basically powered in this case by by what you see here in console.

B

So all of these local docker processors have registered themselves with console and that makes them routable and linker D, so in linker, D I'll just give you an example of that. So in linker d1 there was a tool and I'm going to have to go back to my links to get the right example of that. So.

B

So what I'm going to show you here is a it's a pretty useful, debug tool and what it lets you do is say hey. Where is this service running in my local topology? And then it's going to just give you? Hopefully this green section. That shows you all the different hosts that are running this service and then the other thing that you can do with link addy, which is useful, and this is going back to the previous flight of observability.

B

Is you can use a tools like Prometheus to then graph, your metrics, so that's extremely useful for debugging and offering kind of insights into how these applications are running. So with that I think we can go back to I, have am architecture view of the pieces in the demo, so we don't really have a real global layer.

B

The example of the global load, balancing is just a node script, just to show you how simple it is to interact with a control plane and then we're using link ID as a front proxy and in our little compute layer, which is just docker compose with console as our service registry.

B

So I wanted to give a shout out to some of our open source projects and we just open source stream registry, which is service discovery for streaming initially with support for Kafka, but we're going to support other streaming platforms and we also have a distributed. Trace, open, trace solution called haystack and we use sticks as a JVM based front proxy in production in multiple in many parts of expedia group today and we're hiring yeah.

A

We're hiring and I think that's it.