Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 7 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Beyond OpenTracing - Allison Richardet, Asteris, LLC (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Beyond OpenTracing - Allison Richardet, Asteris, LLC (Intermediate Skill Level)

A cluster is a busy place, and lots can happen. Microservices, containers, polygot persistence, schedulers, and so on have enabled organizations to rapidly deploy a variety of applications. We can get up and running quickly, but then progress halts if we can’t understand our system. In order to keep moving forward, we need to enforce structure in the right places, while not constraining the benefits of microservices. Just as event sourcing can help us structure communication among services, an evolvable grammar can structure our measurements to enable powerful insights and analytics. This grammar can help us increase cardinality in our logging and allow more tools to provide insight into what is going on. In this talk, we will walk through how to use logging, OpenTracing, and grammar to improve event collection for later analysis.

"About Allison
Allison Richardet is an engineer for Asteris in Saint Louis, MO. She began her career doing Ada and C++ for flight systems and decided she needed more RAM, so she moved to the cloud. She currently develops in Go, Bash, and YAML and helps companies move from on-prem to Kubernetes."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everyone I'm Alison, and this is beyond open tracing hope everyone's having a great week at coop con, so I work for a company called Astra's in st. Louis Missouri in the United States we're a consulting company and we worked with various firms, we're currently working with a very large financial company and helping them move to the cloud specifically on kubernetes.

A

So let's get a level set for the audience who has experience with kubernetes? Okay, there are fewer hands that went up than I thought, so maybe some of you are asleep or in the wrong place.

A

So what about distributed? Tracing, okay and then structured logging? Okay, pretty good. So hopefully, some of what we talked about today will be relevant to you.

A

Currently we're moving from monolithic architectures to micro services, and we want our business logic to be constrained to a context, and we need to make sure that micro services are communicating with one another and we want to do things like handle infrastructure elsewhere. So a container Orchestrator like kubernetes.

A

Since we're using micro services, not everything is co-located. We need to do things like create a load balancer. We need to maybe use a sidecar for logging set up, metrics collection and visualization, and some of these tasks for observability can be daunting tasks for some engineers.

A

In the past you know people have logged into a server and gripped for logs, but now you're using something like stack, driver or Splunk for logging. So if we can bridge the gap between application developers and infrastructure engineers and get everyone talking together using the same language, I think we can make some improvements in the observability picture.

A

So an issue is knowledge of system. State is hard. You have multiple services, creating events and logging services need to be aware of overall system state and what's occurring.

A

So if we can use a common language like I mentioned, we can have application developers and infrastructure engineers working together, and if we can do this, we're also enriching our data and turning the data into usable information.

A

This may look familiar to some of you, so logging into a server and prepping for logs. You know maybe you're looking for an error or you're looking for some sort of transaction ID, but you know this could seem straightforward and easy. However, you could be looking through thousands and thousands of messages trying to find what you're looking for and it could be. You know you're sort of looking for a needle in a haystack. So, let's see if we can improve upon this, so that's logging.

A

If we use structured logging, we can impose structure and limit verbosity, so we're cutting down on the unique values. If we can do this and some people might say- oh well, you know there are so many observability tools so that you know that gets kind of out of control. How can I bring it down and you know really make things work together? So if we can agree on a structure that we want to use and use a grammar for our structured logging, how can we make it easier on operations?

A

And how can we give them the visibility that they need, possibly without using big data tools, but to to make it easier for them to use big data tools?

A

And what, if someone doesn't want to have to use blogger, stackdriver, well use and agreed-upon grammar and structuring our logs in this way, should make it easier for someone who maybe wants to do that as well?.

A

So I, don't like seeing log statements like this store some data. What are you storing where? Where are you storing it? It's not really giving me any information, so an a failed transaction. What failed? How did it fail and how do you query for open file or open file failed? What what sort of data are you going to get back when you're looking for this? So let's see what if we can do better.

A

So if we can impose a format to our logging and use structure and use fields instead of raw strings, we can tie context. We can tie a transaction ID or some error to an occurrence and we can reduce cardinality. So limiting unique values is really important and we can use just a few fields and limit just the to just the information we need to see. We have fewer columns in a database then so queries are improved.

A

We have faster lookups and we can improve analytics and get to the information that we need faster.

A

Are there any go developers here? Ok, a few so log ris is a common, structured logging library and go and you have the ability to set fields. So this is your structure. So I want to look at comparing raw text to structured logging.

A

This first example is just a raw string. The second one is an example using Loris and you use with fields.

A

Implementation can vary from developer to developer in structured logging, so you would have to spend a little bit more time. Thinking about your structure and structuring can also vary from developer to developers. So a team needs to agree upon semantics for this.

A

But I mean I think this is worth it. It looks so much nicer than raw text rings, and this is an example from logger it's from their github page. These fields will pay dividends down the road.

A

Okay, so moving on to the event groomer, if we can model events with a grammar and use common components from language, this can help us all bridge the gap with operations and developers, so the examples I'm using I'm using English, it's common in the English language, but as well as others.

A

So we can really use use our data and turn it into information that we can then use much easier, so have faster queries and improve our analysis.

A

So these are the grammar components subject and verb are required and then the different objects, direct object, indirect object and prepositional object are optional, but they do give flexibility based on your needs. A.

A

Subject is a noun, it's a person, place or thing, and it performs the action so going over each of these components, I'll be using these same two examples: Parker rolls the ball to Kennedy on the ground and check-out service updates, inventory, service, t-shirt, balance and database. So in both of these examples, Parker and service checkout service are the subjects.

A

The verb conveys action, so Parker rolls and check-out service updates those. Those two are the verbs so roles and updates.

A

The direct object is the receiver of the action, so Parker rolls the ball ball is the object and check-out service updates. Inventory service inventory service is the object in this example, so they're receiving the action.

A

An indirect object is also a receiver of the action. However, it occurs in addition to the direct object, so Parker rolls the ball to Kennedy. Kennedy is the indirect object and a checkout service updates inventory serve service, t-shirt balance, t-shirt balance is the indirect object.

A

The prepositional object is the object of the preposition it's introduced by the preposition, so a preposition is something like at for or on so Parker rolls. The ball to Kennedy on the ground, grounded is introduced by on and check-out service updates, inventory, service, t-shirt balance and database. So database is introduced by the preposition in and these objects are available. If you need more flexibility and need more information for your grammar and in your logging, but you don't always need them, but they're there. You know you have the flexibility to use them.

A

So grammar log is a proof of concept. Logging, library, I created. It extends llores, so I'm using their structure in their fields and applying a grammar. So at the top you see I'm creating a new logger and I. The first argument is commander: it's my originating service. So this is the subject, and the second argument is the destination for my logs. So your service is always going to be the same.

A

You don't need to set the subject with every log you're creating so we set it once here and also set the destination once and then I'm using debug level, and this is a simplistic example, so commander writes event Kafka.

A

This is another way to call the logger using a very attic function, so you pass in only the fields that you need so same example and it's commander so commander writes event Kafka, and you can do this. Instead of calling each of the separate functions like a verb and object.

A

Okay, so distributed tracing I'm using open tracing and Jaeger, which is a visualization for open tracing written and go.

A

So it's distributed tracing, we can get a better overall picture of a system and it gives us more context when it's used with logging. We can trace calls as they propagate through the system and see the services that were invoked. We see the time spent in each call and we can pinpoint delays. This also helps us visualize, sequential versus parallel operations.

A

This is from the commander service that we touched on and I'll be going into more details later, but I'm using cross process tracing with extract injected carrier so I'm using the built-in text, map carrier and I'm, injecting the span context into the carrier and it basically carries thee. So the carrier carries the tray state from one process to another, from service to service and lower down on the screen.

A

You'll see I'm using the event grammar, so I'm setting the verb, object and indirect objects and then I'm also setting the tags in open tracing so I'm using the same grammar I'm using the same verb object under act object for the tags so we're linking together the logging and open tracing.

A

So this shows open tracing tags and they have required parameters a key and a value. A key is a string and then a value can be a string. Boolean or numeric type and standard tags do exist, such as database error or such as database type or error. And then you have the ability to create custom tags, which is what I've done with the event grammar.

A

So you can see that the tags for for these fans are verb is credit. The object is account and then the indirect object is 32.

A

So, going beyond open tracing we're able to get context through open tracing, we're able to get a better bigger picture and overall insight into what's occurring and using the open tracing tags. We're taking advantage of the tags and using the event grammar here as well, and we're using the same language with service logging and events and also with open tracing. And we have the ability to search on these tags.

A

So using an agreed-upon semantics for application developers and operations, we're able to facilitate speaking the same language and we're structuring our data to be analytic friendly using just a few fields. Our queries are faster and we're limiting the unique values and reducing cardinality.

A

So, with micro services, we we need bount bounded context for our services and the way I'm enabling services to communicate is event sourcing, so a service will publish state change events and then any other service. That's interested in these events can subscribe to that topic, so there they receive all updates and another really good thing we get from. This is an audit log and overall application state and we do get the ability to replay these events so.

A

For my proof-of-concept application, I'm using CQRS, which is command, query, responsibility, segregation, I'm, breaking up the services by their business logic and I have an index proxy. The commander sends write, requests to Kafka. The processor takes the requests off requests off of Kafka and processes, them writes them to a database and then the query service is able to quickly query due to the processor having pre-processed the requests.

A

So a setup like this is optimized for write, heavy and read heavy workloads, so we have a queue the commands can just keep coming in and when the processor can process them, it takes off the takes them off of the queue, but this can introduce latency. So we'll look into this.

B

B

A

In the left pane here, I'm starting up the I'm starting up everything with a clean slate, so I'm cleaning out Jager cleaning out cough gun, zookeeper and while this gets started so I started off. You know: I wanted to have everything running in kubernetes.

A

But with a nine month old and a almost three year old things got a little hectic, so just use your imagination, I'm running this locally.

B

Okay, I cannot see.

A

I cannot see what I'm typing here in the terminal. I don't know. What's going on, I.

B

Don't think so it shouldn't be.

A

So I'm going to start up the applications so I'm starting up the query, proxy and Commander.

B

And I will start up a processor.

B

B

So again, whoops.

A

Here's architecture we have commands coming in from commander onto Kafka, the processor is taking the request off of Kafka and query is querying them.

A

So I'm going to go ahead and put a request on Kafka from the commander. The action will be to credit the account for this amount, so.

A

Here's the response we get back, an ID, it was put on the commands topic and the status is created.

B

Okay, so you can see here.

A

The commander is the subject: verb is credit and object is account. An indirect object is the amount that we use and the processor is using the same language. So.

A

Processor credits account 100.

A

And I'm going to create a bunch of traffic, so I'm gonna put 300 requests on Kafka and let's look back over here.

B

A

Can see them all coming in also I'm using someone else's computer today, my computer broke this morning so.

B

All right, so, let's take a look at Jaeger.

A

So, look at the commander service. We can search for the credit verb, equals credit tag.

B

Sorry ah thank.

A

You that wouldn't have worked well.

A

Okay and let's look at 300 traces.

A

So the beginning, things are moving pretty quickly and you can see we really really slowing down so over 30 seconds. For some of these requests to go through.

A

Okay, so I have the nginx proxy and then we're going through the commander.

A

And again, I'm using cross process tracing, so we're stitching together these spans to create a trace from service to service.

A

So the proxies going through the commander and we're using those verbs or we're using the tags for the grammar so verb is credit object is account an indirect object, same thing with the processor it's reading from Kafka and then it's persisting. The request.

B

So, let's take a look at.

A

One of these that are taking a long time over 35 seconds and it's taking so long that you can, you almost barely see the span. It's tiny look at all this latency between the commander and the processor, but it's working right. It's eventually consistent, but you know things are slowing down. So, let's see if we can make some improvements.

A

I'm going to spin up five processors.

A

You can see them all starting up.

A

And let's do the same thing generate 300 requests.

A

You can see all the processors are handling these processes or reading from Kafka handling the processing.

B

Ok, so let's go back to Jaeger.

A

Muster still search on that tag and let's look at 600 traces.

A

So this is the original with one processor, and then this is with five-way on the right.

A

So considerable improvements.

A

About 156 milliseconds, so we were able to see that using distributed tracing. We were able to see that we had a lot of latency in the system and you know I did only use I'm only crediting the account, but let's say we had you know debits to accounts or other operations that were occurring, and maybe it was only happening with one verb.

A

That's something that you can pinpoint. You can look at that in your logs. You can look at that in open tracing and really Zone in on what the issues are and where things are happening. So.

A

Same tags here.

A

So using this defined event, grammar we have an agreed-upon semantics for operations, engineers and also application developers to use and using cross process tracing using cross process. Tracing allows us to inject the trace context into a into a text map carrier and doing this. We're able to get a better overall picture, we're bridging the gap between application developers and operations and getting everyone speaking the same language.

A

So application developers are able to gain insights on what they do in their code and how it affects observability and how it fits into this bigger observability picture and we're tying open tracing to events and logging.

A

So using structured logging with an event, grammar I think is much better than raw strings where we're getting better more useful information, we're able to filter it, we're enriching this data and we're reducing cardinality, so we're really cutting down on the unique values that we have and we can filter on things using using this grammar and using tags we're reducing cardinality.

A

So if you're fewer unique values- and we can improve analysis, so we have faster queries, better storage and we also have the ability to set up things like machine learning and really enhance enhance that.

A

A

Anyone have any questions.

A

Okay, so the question is: do you feel like the fields you've chosen or the structure that you've chosen can be too limiting? We and do you maybe need to just add a verb rain.

A

Yes, yes, so definitely you change the verb based on whatever action, you're you're performing. So if you're writing to a database, your verb would be rights.

A

If you're, you know, if you're issuing a command or yeah I mean it. Basically, it just depends on whatever the operation is that you're doing.

A

Right so so you definitely always use a subject and a verb. So that's the that's the main, the meat of what's occurring and then the objects are there as additional fields. If you need more context or if you need more details, so the object is the receiver. So if you're, if you have a service that writes to a database database, would be the the object and then, if you need more information, then you use the other objects for that and yeah. It just changes based on whatever your, whatever you're doing. Oh.

B

A