Istio IstioCon 2021, 10 Mar 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Extending Engarde to Bridge the Gap Between Istio Access Logs and Envoy's Documentation

Description

#IstioCon2021

Presented at IstioCon 2021 by Gregory Hanson.

Istio allows users to enable Envoy access logs. These logs provide extensive information and are one of the first steps in diagnosing networking problems in a service mesh. Engarde is a tool which parses Istio access logs into easily readable JSON objects. With Engarde, you get the log field names, but to the average user there are still some knowledge gaps that require a hop over to Envoy’s website to understand what is shown in the logs. This extension combines Engarde with Envoy’s documentation all in one place.

A

Hello, everyone, my name, is greg hanson. I am a software developer at ibm and I have been working on the istio project since its start. First time, I'm giving a lightning talk, so here we go so what is istio just kidding? It is the end of the week and the conference has the same name, so I think I can pretty safely skip over this slide.

A

One of my areas of expertise in istio is understanding how various resources generate the appropriate configurations which are supplied to the envoy process running in every istio proxy sidecar in the form of lds, rds, cds, etc.

A

This means that, as a part of my role at work, most of the questions that get directed my way are along the lines of I created this virtual service destination rule or service entry, and my requests are failing.

A

What's the first thing, I ask for access logs, that's right access logs as a testament to how long I have been working with istio, but long before the days of istio ctl, which provided a means to inspect envoy's configuration for listeners routes or clusters.

A

There were only access, logs and config dumps if you can step your way through an envoy config dump from one of istio's sidecars props. To you find me on slack, let's be friends now I could ask to see their virtual service destinational or service entry first.

A

After all, those types of errors can be briefly summarized into the following categories: connection: errors with your service entry; usually a port or protocol mismatch; 404s, probably something wrong with the hostname or virtual service, 500s, probably a bad destination rule.

A

But if their applications are using an unknown hostname, a different port- or maybe there is another service entry with protocol tcp port definition, capturing more traffic than intended.

A

Looking at the rules is not going to tell the whole story in cases of failed requests or sidecut or sidecar traffic within the mesh or between sidecar and the egress gateway, as it attempts to communicate with an external service access logs can help give a better idea of where, in the request, path and error occurs.

A

Is that 404 coming from the outbound proxy at the source or the inbound proxy at the destination? Did it even reach the destination for that 503 failure? Does the cluster name match with the expected service name and the subset you have defined in your destination role?

A

Also, is the service entry, virtual service or destination rule even impacting the request traffic, or did the request fall through to the allow any pass-through cluster?

A

These logs provide an extensive amount of information that can be used to troubleshoot issues. There could be how sdo is capturing traffic at the source or the destination, how traffic is routed through the user, created traffic management resources or how the application itself is attempting to communicate with resources inside and outside the mesh logs also encourage novice istio users, start learning to interpret and diagnose errors themselves.

A

No more hand-holding from an access log entry users know which listener route and cluster and envoy was used during the request, and you can use istioctl to inspect the json for those objects to see if there are any misconfigured, hostnames, timeouts or tls contexts, and it even says which routing resource was responsible for generating the configuration.

A

However, most people don't know what those axis log fields mean.

A

It used to be that if I wanted someone to be able to figure this out by themselves, I would have to point users to the line in istio source code, which defines the custom log format and then provide a link to the envoy proxy.io documentation to explain what those fields mean. The process is a little convoluted at this point, and only really ambitious debuggers would bother jumping through those hoops.

A

Then, at kubecon north america, in 2019 I heard about onguard. If you have already heard about it. Great just bear with me: take an axis log entry, pipe it into the on guard tool for processing, add in jq, and you get a pretty printed json representation of the log complete with labeled field names.

A

It even included a special flag just for parsing istio's custom blog format. So I thought, aha, smarter debuggers means less pouring through access logs for me, but it wasn't quite there yet, after all, when someone asks me hey, why is there a urx response code in all these failures? What does that mean the result?

A

I still have the manual step of pointing to the envoy proxy dot io access log page. It's gotten to the point where, if I type an android proxy in my browser, it knows to send me directly to the access log page instead of the main documentation page.

A

So what can I do to make? It include everything I want in one place after some html and css educational refreshers, I wrote or hacked together. So please be gentle aui, which incorporates onguard and odd voice documentation all into one place paste in the access log entry using istio or envoy's default log format hit the button and out comes the process. Json log output from onguard; nothing new at this point, just a ui for a cli tool.

A

So what I did was I wrote a scraper for envoys documentation, pulling everything regarding access, log command operators and their corresponding descriptions from the ui. If you click on one of the json field, names or values, you get the description of the fields pulled directly from the documentation.

A

Everything in one place, hooray I'll hold for applause.

A

Anyways, I developed this mostly to make my personal workday easier, but I hope someone else will find it just as helpful as I do special thanks to the creator of onguard. You are awesome and clearly my inspiration also thanks to envoy for having docs, that a beginner like myself was able to get a scraper working pretty easily the code, firmware ui and instructions for how to use can be found. As a repository on github and just in case, I did not stress it enough earlier.

A

I am a beginner in html and css, so I welcome any suggestions or improvements as pull requests also always curious if there are any other feature, suggestions that could round out the tool and, of course feel free to reach out me on istio slack as gi hansen thanks, everyone and happy service.

A