Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: etcd Watchers Not Working? Improving Error Handling in Your Applications - Guillermo Nunez, IBM

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

etcd Watchers Not Working? Improving Error Handling in Your Applications - Guillermo Nunez, IBM

When building services with etcd, one of the best features of etcd is watchers. It allows developers to create applications such that when a dataset is changed, the watcher for that dataset on another system receives the notification and performs user-specified action. A simple use case is a user has a program using etcd watchers to monitor the stock market and sell (user-defined action) if their stock of interest falls below price $X. There are error failures that are not automatically covered with an etcd cluster such as network connectivity issues, DC outage, and cloud infrastructure maintenance. The talk consists of the following: - what watchers are - how to build your services using heartbeat key for monitoring etcd watchers - perform locking on watchers to prevent race conditions - caching data locally for your service to keep running until the etcd cluster is back up.

https://sched.co/ZetF

A

Hello, everyone thank you for joining today, we're going to be talking about ncd watchers, not working how we can improve the error handling in your applications.

A

So today's agenda we'll be talking about what the ncd watchers are. You know how they're used how we can perform locking on key directories when using watchers how you can monitor your lcd watchers to make sure that they're running and how to cache the data that you're using from ncd locally on your applications.

A

So edcd is a key value storage system, so you can kind of think about it as a python dictionary. So for your key, you could have like a restaurant and then for your value. You can have you know like a pizza place or you can have a uh a phone number a x address, z for for the watchers. These pay attention to these specific keys and or directories, and it triggers an action so say we have a for the example I just gave for restaurants.

A

We we changed the value to instead of it saying pizza hut. We change it to domino's. So you'll see that once that change happens to that cd database and you have a watcher running, you'll get a notification that the value is changed and then you can trigger an action, as you know, as needed. So for this example here we have um stocks and I'm using the dictionary. So it just kind of creates like a hierarchy here. So we have the slash stock, slash foo.

A

So the kind of like the convention that I'm using here is the stocks would just represent what I'm meaning at a high level and then for the foo it'll, be the individual stocks that we're that we're interested in so for this example, I'm using foo um as the key and then for the value, I'm designating it as the the stock price. So you we have four different times here so time: zero, we're adding a key value here to the database um and we haven't set up the watcher yet at time.

A

One you can see that we're setting up the watchers to sell. So that's the action that I'm defining at 120. at time. 2.

A

We we see that the foo stock has increased to 130 and that's reflected in the database. Now the the watcher gets this notification, but because we're not you know trying to do any action. If it, if it increases, um then we don't have to worry about it. There's no action needed at t3.

A

Once we see that um the the price has changed to 120, then the watcher will get a notification and then we'll trigger an action, and in this case we can just say for simplicity that we're going to trigger you know to sell the stock and whatever portfolio you're, using whether it's robinhood or or fidelity.

A

So with watchers and with changing data, we'll have to make sure that we take care of the race conditions to make sure that our data is not corrupted and we can do that by using the lock api.

A

So here I want you guys to consider the situation since we've been talking about stocks, so the you know going by step by step here, so the foo will change from 120 to 119 and we have our watcher defined to where we sell if the price gets lower than 120 and we buy more if the foo is greater than 115..

A

So for for step. Two here we see the watch program would trigger. You know the cell, since it's lower than uh 120 and then a new value for the stock has gone to to 110 by a user to the to the database, and you can see here that there's a race condition happening between um between these two so and to prevent that we want to make sure that once the watch, the watcher gets the notification that the foo has gone below 120.

A

We lock that specific key, and once we finish our action, then we you know, release it and then the user can change the value to 110, and then we can trigger and watch for the second uh action which would be to buy more right, buy more once the foo is at 1 15..

A

So this will prevent the the race condition from occurring and make sure your data is not corrupted for monitoring watchers, um so we had to make sure that our watchers were up and that they received the notifications that nowhere in between the connection was dropped.

A

So a way that can happen right is that there's like network connectivity issues. You know you'll see that um if you send new new values to the database, the watcher will not pick them up. So that's why we have to monitor them and make sure that we just reestablish new connections for the watchers as needed. So here my way to monitor that is um so. I described the hierarchy here with the slash stock, slash foo.

A

So what we use in this case would be slash, stock, slash health, and then we can define a interval on how often we want to ping to make sure that these watchers are up and running. Usually, if you have you know, one watcher running, that's a good signal that the other watchers are running as well. um So you just want to make sure you're setting up that your program to send the to change the value of the health in an interval and then set up your watcher to acknowledge that you know we are having.

A

We do have established connection with that cd database and the watchers are up and running.

A

So for more complex uh software, um I'll use cloud here as an example, so it could be the case that there's a key that's changing constantly um from different places and I'll give an example as cloud right. So in cloud there's different, there's different instances.

A

um There's you know: subnets load, balancers storage, so all of these could be changing one key and the way that we can kind of you know try to minimize the downtime is to have a copy of your xcd value, stored locally and and once you have that established, then, once the connection comes back up or your cert or your program, your applications re-establish that connection with the watchers that you have defined, then you can just push the code that you've cached locally and push it up.

A

So your database is is up to date and it minimizes downtime, um and it makes sure that you know that you can come back up faster.

A

So with that being said, thank you guys.