Cloud Native Computing Foundation ServiceMeshCon EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Observability with Consul Connect - Bram Vogelaar, Hot Potatoes

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Observability with Consul Connect - Bram Vogelaar, Hot Potatoes

Things like Infrastructure as Code, Service Discovery and Config Management can and have helped us to quickly build and rebuild infrastructure but we haven't nearly spend enough time to train our self to review, monitor and respond to outages. Does our platform degrade in a graceful way or what does a high cpu load really mean? What can we learn from level 1 outages to be able to run our platforms more reliably. We all love infrastructure as code, we automate everything ™. However making sure all of our infrastructure assets are monitored effectively can be slow and resource intensive multi stage process. During this talk we will investigate how we can setup and observe a service mesh platform using HashiCorp's Consul Connect by recording its metrics. logs and traces using Prometheus, Loki, Tempo and Grafana.

A

Good day, my name is ben voclar and today we'll be discussing uh observability with console connect, um but first before we start moving into directly into to service meshes, we need to build up slowly to how we used to do stuff to how we want to do stuff. Now, um back in the days when you wanted more compute power, you would have to get management by and you would need to raise money. You would have to get a new purchase order.

A

You would have to order the thing. The thing would eventually show up to your your loading bay. You would have to install it, you would have to um install the os and then you would have new os. um Eventually that would mean like, as in this picture, the damn things were big, um so it would require building new rooms or new buildings um that wasn't that so easy. So it took years to get new compute in the 90s.

A

When the first internet bubble happened, we were at a point where we would, instead of building new computers, we tried to solve our problems in such way that we could spread it across multiple servers.

A

When I was still in the lab, that's how we used to do stuff and the list of servers was fairly small. The list of users was fairly small, so we could just say you work on server one. You work on server two or this happens to be there and there when we started to connect stuff to the internet and our users weren't internally anymore, um things like load balancers were introduced.

A

This way we could design a vanity url like google.com to our load balancers, and then we could tell our users that's where it goes, and then the load balancers works out where uh how to a how to spread the load or where stuff is um that was fine. When purchasing was still slow like in the 90s, you'd still have to raise um purchase order. You would still have to wait for stuff to show up on the loading dock and you would have to install it in your your own data center.

A

So um adding new new nodes and adding them to a static list in the load. Balancer was fine because that's how slow things were.

A

It was actually a manageable job, but when we started moving to the cloud um adding new nodes, removing new nodes, adding new services became much much quicker and that's when service discovery showed showed up and that's um basically, what service discovery is it's a method of announcing yourself into a cluster or a pool and saying I am service x or b in this case, and I am ready for action and then the load balancer you announce yourself to the notepads and no bounces starts assigning you work and that's where uh console comes in.

A

That's how it originally was envisioned as a service discovery tool that could basically route stuff to your uh to your your networks or your clusters. How do we set up a service in console? It's actually very simple: uh it's a file on disk, um it's written in a dsl called hcl. The hashicorp's uh configuration language, and um here it shows. As such, we have a service name counting and it runs on a local port 9003 and has one health check um basically and it pings the slash health endpoint every one.

A

Second, and if it's healthy, it gets announced as such into the cluster, and it is accepting work if it's no longer healthy console will take it out of the cluster and until such time, that it's fixed again console is uh you can not only ask console direct questions. Console can also act as a dns server, so it actually lives in your old school platform or environment.

A

Just ever like, like everybody else by all your services, you can dig it on a local port 8600 and there you need to ask it is a dns question in a specific way. If I want service counting, then it's counting.service.console and you can even incorporate this into your existing dns services and, like I show here for my bind server that is console zone that forwards all the questions that for services and then dot console to my local console server and then.

A

The cloud was fast: kubernetes containers was even faster instead of minutes. We now start can start and stop services in seconds. We can run services across multiple clouds and that brought a whole new layer of questions and problems to solve, and that's where service meshes came in.

A

This is where we start to think about how do services, collaborate or interact, and how can I distinguish between different services or how can I block access to these different services and that's where um console grew into console connect and that's connect is where the lovely people at hashicorp incorporated or grew the service mess level of um of console. That's what we're gonna discuss in the next couple of slides. So how do I extend my service definition for a service to become console, connected or service measure? Where and that's fairly simple?

A

We need to add just a little connect stanza with some mtv um other stanzas. That's it now. It's called console connect, it has a tls certificate and it's ready for action. So how do I connect to this? It? Doesn't we have a accounting service which is our backhand now we'll add a dashboard on top of it runs on port 9002.

A

The only thing we need to add is a similar connect stanza, but now we declare uh our upstream. So we're basically saying we want to um use another service. The service in this case is called counting and bind it on a local local.5000. Why this is cool, we're gonna discuss in the next couple of slides.

A

uh First, we need to fire up the connections, so we need to tell console um that the connect side of things need to be started and that's fairly simple: it's a command line, command called console, connect proxy and then you basically say for service accounting and for service dashboard. Let's, let's give me a sidecar and basically make it connect aware wire.

A

This is very cool. It brings us a couple of additional features that we want. um Remember I mentioned we bind it to a local port 5000. So when we use service discovery, we no longer need the vanity url of a service, but we still need to know which port is bound to and if people bind it.

A

If it's a known service that binds to its known port, then I can infer it and I can still similarly use it um if you start using things like kubernetes hashicorps, nomad as a scheduler stuff might not run on its predefined port, it might be on a random number, um so we need to either make um rules about which ports to use or we need to have a ledger of what the port to use and that's, basically, what console does for us and instead of using trying to work out a way to ask console for that port console connects actually says.

A

No, I know I'll work out the port but bind it to a local port so for your local application, it will look just like it is connecting to a local, hosting and console connect. Does the magic below the other great feature about console connect is acls or in console, speak intentions, so I I can either I can. I can block services from connect from um using my servers. I can allow stuff to to happen so we never get into a situation where our development database connects tries to connect to our.

A

uh No sorry, our development platform starts to connect to our production database or vice versa. um That should definitely never happen. You should run a console cluster for your development platform. You should run a console servers for your production platform because, having forbid, if that ever happens, due to copy and paste error, you can use all the stuff I've discussed now in previous slides. You can actually try yourself. The lovely people at hashicorp have a very good tutorial to follow.

A

That brings us to observing stuff, so I've used the words console abstracts the magic away for you. um I've only I've also used a I've carried a pager for the last 10 years and abstracting magic is scares me because I will never want to end up in a situation where restart and pray is my only way out of a problem. So luckily um the lovely people at hashicorp have ways to work to work out what is going on and that's actually what we need to do in general anyway.

A

So just three ways to work out what is going on? We either look at the metrics that the system emits so how many purchases are happening. How what's uh how many uh users do? I have what's what's the hit rate per second? um Also, can I look at the logs? The platform is emitting. How many errors are? Are there other information messages, and also since we're using well, hopefully or possibly using microservice? You also need to look at traces and tracing is basically working out where, in your platform, your application is spending its time.

A

Is it in the database? Is it um rehashing acls? Is it waiting for the accounting servers that is waiting for its own database?

A

um So the three things that we need to look at is metric traces and logs, and the rest of this con presentation is going to dig into how we're going to use the grafana stack or the to look at into where the magic is happening in console connect. So console connect is built on the shoulder of giant, so it's not an a novel implementation, but it's actually reusing a cncf tool called envoy, which is great because there's more people looking at it. It's not a not invented here.

A

Product, but it's actually a nice wrapper around around envoy. So when we I talk about metrics, I'm talking about prometheus, collecting your exposed, metrics from cncf tool it and it scrapes metrics, as I said so, instead of the classic old-school tools that push in the metrics, it actually pulls metrics. That brings in the need for service discovery, which we have and is very cool and nicely integrated into prometheus. Prometheus also has its own query language that allows us to query the data, but also ask very, very specific questions.

A

So how do I expose my prometheus metrics um in console connect? Not actually I'm not exposing console connect, console metrics, we, although you also should, but um this way we actually exposing the built-in android metrics, which is very cool, and we, the only thing we need to do is add a three-line blob to our already existing console connect.

A

Configuration basically saying I want to inject a little bit of configuration, and I want to make sure that I expose my envoy prometheus endpoint at a certain port in on my local system, um that the way I showed it in the previous slide is actually a bit cumbersome, because now we need to actually make a promise.

A

Well, I need to ask all our developers in every job you you make, make sure that this line is available. The other way around is um injecting it into the global console configuration and by basically saying enable the central service config and make sure that for every type, every kind of pro of a proxy we um we inject this. This bind address and, of course, as I said, please make sure that you're exposing the console metrics itself and that's the last four lines.

A

Basically, I have a telemetry blog and make sure you configure a few exposure as premises data how to collect this in prometheus. There's two ways: uh one is a hardcoded list, which is the top part, um which is a little bit of yaml and you say: have a straight config: that's the the way prometheus names, those things have a job and scrape all the static targets.

A

Now did the cool meta way of doing it, a cooler matter way of doing it is actually using consoles service discovery, so have prometheus query console for all services that have a certain name in this case, console connect envoy and then make sure that you just scrape those that way. We actually announce services that need to be scraped by prometheus and then the circle is round. So how do we use uh prometheus data?

A

There's two ways actually console has a ui and since version 1.7 you can actually have a nice integration with prometheus and in the general console configuration you need to add a ui config blob, and you need to point that towards your your prometheus server and that's it. Then. If you look at the ui, you now have little exploratory usage view. If you drill down into your services. In this case, I have a product, a product api that needs to talk to my database, and I can actually see data going on.

A

That's all it shows, but if you want to drill deeper, you actually can click on open the metrics that dashboard and then what happens. It goes to your local grafana instance, and grafina is a dashboard that does metrics and does metrics involved in a general way. It started out as a project that could visualize graphite data, um but over the years it's gained a plug-in system and it gained data back-end plug-ins for basically any metrics back-end, but also logging and tracing at the moment.

A

So it's very cool. How do we connect gravana to our prometheus? It's a little yammer file with a standard blob, uh offtide prometheus pointed to uh dns where prometheus lives. In my case, that's the console service. So console also does dns for this, and I do it in a wildly insecure way by not having any authentication.

A

If you run into production, please add it, but for this presentation itself, then in grafana I can use a bunch of cool dashboards. You can also download them off of the grafana website, and in this case I actually have a visualization of my console cluster. I have three nodes. There's a leader and I have a fairly healthy amount of careers going on next up is logs. um I'm introducing a tool called loki which came out of grafana labs uh fairly recently and in the byline is prometheus for logs.

A

So in the olden days or not that long ago, actually um locking aggregation systems like the ax stack, like splunk, were platforms in and of itself, so it took skills time resources to run that just for your actual platform to push stuff into so loki is a much simpler way, but also much more powerful way.

A

In my opinion of doing stuff, it's um it's heavily influenced on on prometheus um and you can run it basically off of a single go binary as a monolithic instance, or you can spread it out as microservices all using the same binary. um How do I push my stuff from from my console connect into loki? I there's a couple of ways doing this. The first one is, if you're running everything in docker anyway on kubernetes um install the docker plug-in um they kephana provide.

A

There are some issues at the moment, so it's a blocking call. So if your look is down, you'll have issues manipulating your containers.

A

If you're running still bare metal or you're a bit scared about the blocking query thing is, you can actually make console connect, write its logs into a file and then scrape it using the loki's command line, tool called prom tail and the weird ish command line on online number. One is um so we have the the first part of the command.

A

Is the command we're used to and then dash dash, which means it tells console everything after this is not your configuration, but it's invoice, so you can parse stuff directly to envoy, and this is how you this is how we now ask android to parse everything to a config file which then gets scraped and pushed into loki um again. How do I set up the data source similar like before yammer file after and then type loki, put push it towards the url and then bob's your own code?

A

What does that look like if you want to? Initially, you want to do exploratory research?

A

What is my application actually pushing? What are his neighbors pushing? Therefore, grafana has a very cool new explore tool, and then we can query you can push push in any lock, ul queries that you can come up with and then eventually you can turn that into dashboarding last. Lastly, it's traces traces can be stored in a new tool called grafana temple, which has a similar impetus as as loki the generation. One tools like zipkin, like jaeger, actually are quite cumbersome to run.

A

You require databases, you require storage and tempo is a much leaner way of running things, but not stopping you.

A

Running your own platforms, it's a single go binary. It speaks uh zipkin, jaeger, open, telemetry um dialects, so you can use your your tools you're used to, but in a much leaner way, and then it stores it directly into cloud. Cloud-Based storage. Instead of a database you have to run um making envoy emit traces, is a bit more work than than logs or metrics. So there's a two-way, two-way blob ones of json.

A

Firstly, you need to tell envoy that it needs to emit zipkin, or in this case they can type traces and then next of all, that's the more elaborate blob that hopefully hashicorp will eventually abstract away for us is there's a large blob that basically has a three line: bits of information where's the address. What port is my uh my zipkin or in my case my tempo living and please push the traces to that again. We can simply add it to um to kufana. My little yaml file pointed towards the tempo query um front.

A

End tempo at the moment needs a second component to visualize traces in grafana there in the process of removing this requirement, so that you simply only have temple, um but for the moment it's a mini, slightly manipulated, jager front end that needs to run so, hopefully, when 0.7 comes out, which hasn't happened yet at the time of this recording, you will be able to run this as a single single binary.

A

um If we have tempo traces, we can set up in such a way that we can correlate between low key to sources and tempo sources. Therefore, we need to extend the loki data source slightly by saying there's now the right field, or if you find a derived field that smells like trace id, then couple it to um then it probably is a trace id and make sure that you couple it to to the tempo data source. So what does that look like if you on the left?

A

I will we're looking at logs and if it finds its trace id then there's a new button. You can click and then, on the right hand, side a trace panel shows up. So therefore, we can very easily and simply correlate between our logs and our traces, and hopefully you can work out what happens.

A

So that's the end of my presentation. Thank you for listening. If you want to talk to me about it, ping me on email, uh you wanna, stop me on twitter. Please do, and if you wanna look at these slides a bit more slowly, then you can find them on slideshare.