Red Hat OpenShift Commons Briefings 2016 | OpenShift Commons, 24 Oct 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenShift Commons Briefing #52: Monitoring Microservices at Scale on OpenShift with Dynatrace

Description

Martin Etmajer, Technology Lead at the Dynatrace Innovation Lab, talks about how to monitor microservices on OpenShift.

Today’s growing demand for modern, cloud-native application architectures like microservices is changing the rules of the game. These applications run on dozens to thousands of small, interconnected services, each serving a single purpose, which are meant to be deployed and scaled often and independently. Such highly dynamic, distributed systems have always come with a premium; with greater complexity comes an increased likelihood of failures.

In this briefing, we will explain how to efficiently tackle the complexities of microservice architectures in Continuous Delivery and perform root-cause analysis once in production in seconds, instead of weeks, with Dynatrace on OpenShift.

A

A

Well, hello, everybody and welcome again to another OpenShift Commons briefing. Today, we've got Martin admire from dynaTrace and he's going to talk about monitoring micro services at scale on OpenShift. We're really happy to have him with us. We've had a few technical bugs, but other than that, we're going to get moving and I'll. Let Martin talk introduce himself a little bit more and the format of this is that you can ask questions in the chat while we're going through this and then there's Q&A afterwards and there's a few people on from dynaTrace.

A

We may be able to answer questions in chat, but if there's really good questions, we may pause him and make him take a breath again or just save them and reiterate them in the Q&A, and we will run over your allotted time a little bit today because we have a late start, but we'll be happy to do so so go for it. Martin introduce yourself and show us what you can do. Good management, I.

B

Very much as I am for having me and welcome everybody to this presentation, so my name is Martin at Maia and I am a technology leader at the dynaTrace Innovation Lab. If you happen to have any questions regarding our deep integration with open ships or on our reddit partnership in general, please do not hesitate to reach out directly to me so Rudy, Steiner, trades dynaTrace is the leading digital performance management solution by making business development an operation succeed in optimizing customer experience, modernizing their operations and accelerating innovation advantage rates.

B

We are proud partners with Reddit and it is our mission to create the best performance management solution around openshift, OpenStack and ansible, and in this presentation, I want to demonstrate that we are serious about this.

B

So let me start with a quick introduction to why micro services, when we talk about micro services, we talk about dozens to hundreds, if not thousands, of small, interconnected components each serve single purpose. The reason why micro services are so appealing that they have the potential to solve a variety of problems. Enterprises are seeing these day micro services increase velocity by enabling independent development and deployment of code and as rich sharps, the senior director of product at red, head mentioned at you at a summit in 2016 today, speed equals revenue.

B

Micro services also support growth by allowing services to be scaled independently. So, depending on traffic scenario, we can add a few instances here or remove a few instances. There also Microsoft has foster innovation. They enable experimentation and sale fad to create the right product, and they do so by enforcing agile product, centered and self enabled teams.

B

Now these features make micro services look good on paper, but what do they look like in production?

B

So this screenshot has been taken from one of South America's largest e-commerce production environments serving more than 3,000 services. A difficulty on the Left distributed over more than 10,000 containers in seven data centers. So building a system composed of Microsoft is effectively means building a fault-tolerant, highly dynamic, distributed system, and that is really a complexion to do.

B

Let's take a look at an important detail here, as you can see from those red colored nodes which are faulty services. You can see on the left that a total number of 30 30 services have failed in the entire service landscape. Most importantly- and this is what dynaTrace understands like no other tool- it says none of these faulty services have any impact on the customers and mine applications right.

B

So if you have a distributed fault, tolerant, highly dynamic system, you do not want to be alerted for each single CPU spike or any problem that you see unless it affect your users.

B

So now I would like to present how dynaTrace can help you manage the complexities when designing and running micro services on openshift, by discussing three important learnings that we've made along the way. So learning number one is with micro. Services are complex, in fact, micro services. They can themselves be fairly simple on the inside, however, the overall complexity does not vanish.

B

Instead, complexity shifts into the surrounding environment and can intercommunication between services. We also say that micro services have a lower inner vs., a higher outer complexity in the connected parts. So let's talk about environmental complexity first, so what does the micro service platform composed up?

B

We need things like service discovery, configuration management, routing, load, balancing execution environments like Dugger messaging channels, but also things like deployment, automation and monitoring are considered key capabilities, other micro services platform and, if you're, using open shift the complexity of building and running such an architecture are nicely abstracted by convenient, tooling, but still you're going to build ship and run your services on a significantly complex, distributed. Software stack and essentially you use a pass because there's no real sense in crafting your own platform and then the clustered environment.

B

You also don't really care about how your containers are scheduled, but that doesn't mean that you should fold slide line.

B

So drilling down here to see some more detail shows the dynaTrace computes a topology model of the entire system, which reflects the real-time relation and dynamics of your application right down to the data center that they running, including any services processes, our docker containers, which we use interchangeably in industry and hosts.

B

This is important for two reasons. So number one is that customers have told us that they had no idea what their system actually looked like until we showed them and reason. Number two: is that having the possibility to introspect a system at its various layers in detail and in real-time helps people in business that an Operations make better decision, and this is what dynaTrace is about, and I want to add that all of this information that you see here is also discovered for you. You don't have to do anything.

B

It's kind of serial configuration out of the box.

B

So let me talk about into service complexity. Now, one design principle of micro services is to do one thing and do one thing well, which aid separation of concerns and improves composability now to remain composable micro services. Demand robust ap is now imagine the decide scenario where your team's own separate micro services. How can you be sure that the resulting services interact as designed, of course, with dynaTrace?

B

We cannot validate that your services behave as intended by their designers, but we make you see how your services are composed and how often, and in which ways they interact in real-time. So what you see here on this screen is: what do we call this service flow and see that we have multiple services connected to each other's right? It's not a simple system.

B

We have a back-end servers, we have booked and services journey services and we see how often and in which ways those services interact and how those service calls contribute to the overall response time.

B

Now here is another way to look at the same service. Now we see on the left side and the main part of the screen. We see the de median response time. We see the failure rate of that service. We also see how many services invoke the service, so the Fannin. They also see how many services are getting invoked by these services, in this case 11 services. We also see that this service is not a single process, is actually running on four different Tomcat instances.

B

We also see that the service communicates with two queues asynchronously to communicate with other services, and, if you click on any of these buttons here in the UI you'll get further information on what these interactions look like on behalf of this very service.

B

What we also see on the right side is that we have- and this is something that would affect composability and so how well your services can be composed that we have a certain rest endpoint that has one hundred percent failure rate. So certainly there is a bad implementation happening here right.

B

We also see which are the most time consuming requests time consuming liquids are not only slow requests but are also called a large number of times right, so they contribute considerably to the overall response time of this service, and then we also see the generally slowest requests on that service. So those are your candidates for improvement right and, if you take another look on the left side, what you see here on this time series charge is, as I selected here, the response time. Okay, so we also see that the response time had had a problem.

B

We had a response, time degradation. We had various events over over that time frame right and we can also drill down into the problem and take a look at the problem. Details and I want to do this. A little later on during this presentation now, I talked a little bit about the info service complexity. What about the interest service complexities, things that go on inside the services that we have to talk about? So as an example here, a new spec article mentioned that synchronous. Rest communication effectively turns micro services back into monolith.

B

So when you read about performance in micro services, many companies many enterprises, they they actually understood that there is no way around asynchronous communication right, because there's so much communication going on over the network, which is considerably slower and communicating in memory as we used to in monolithic applications.

B

Synchronous communication is just not the way to go and enforcing asynchronous communication and queues and decentralized data management, bring another level of complexity, eventual consistency, another things like that which make your systems really hard to handle and oversee.

B

Another thing that I'd like to mention. It's also complexity that happens inside your micro services is that your fretting model and the way that you interact with your service interact with other services dramatically affect the efficiency and the resource consumption of your services. So what can you do in dynaTrace?

B

So here is what we call extreme the response, time analysis of a service, and here you see at a glance how or how the response time is distributed in terms of interaction with other services and queues at database usage and but also service execution, which means the code executed for this particular service at hand right and there's a lot of things that you can drill down into here. But what I'd like to show you is that on the very bottom, you see what we call the pure path.

B

So here you can see the actual method invocations which take place inside the service. Let me repeat again: you don't all this is some automatically for you. You don't have to do this by hand right. This is why we spell dynaTrace- and this is the level of convenience- that we wanted to create for large-scale distributed micro service environments.

B

So the learning number two is that micro services don't sail independently in a highly distributed micro service environment. The old saying the more moving parts, the higher the likelihood for failure, certainly holds true, and since failures can happen any time you have to become fault tolerant. That is, you have to ensure that your application can tolerate a potentially high number of simultaneous failures without compromising customer experience, and just because micro services can be deployed independently. That is, if defined correctly, it doesn't mean that they fail independently.

B

So more often than not, they fail in a cascade. And what is he here is a feature in dynaTrace that allows you to analyze the failure or the problem, as we call it of your services.

B

And if you look on the right, this is actually an animation that you can start which begins right before the problem occurred and then shows you how the problem evolved and how it eventually disappeared, disappeared over time, because most problems that we deal with in production, RF strengthens nature thing, CPU, spiked right and the important thing is that you can replay and see.

B

How did this particular problem affect my service and then how did it populate and bubble up to the user and visit affect mine user at all or didn't in the defect measure at all right?

B

So this is a very helpful feature and gives a lot of insight into what's actually happening inside a problem and I would like to add that we correlate a singular instance in incidents that happen in your system and we understand the dynamics of those systems, and we see we can automatically deduce whether they belong for the same problem or not. So instead of alerting you during the night is saying that the CPU spike or that Tomcat crashed.

B

We give you the replay functionality so that you can look at it as a single problem, and you don't have to look at all these uncorrelated incidents that you usually find in log messaging, based solutions.

B

So, let's assume now that your application has failed. The questions that you would like to have answered immediately are what's affected. What's the impact and what's the root cause and the diamond traits will have more than 10 years of experience in building performance, monitoring solutions and that's why we give you this information at a glance. So this is the analysis of a particular problem and we see at the first sight watts effect. We had three applications which have been affected in a given time frame. We see the time frame here.

B

We also see immediately, but the problem affected or affect real users. If we look at the table below, we see that the problem has already recovered and that it affected three applications, 15 services and two infrastructure components right. So it's actually good to know that the problem has already recovered or that the application and services and infrastructure components have recovered. But let's take a look at what what was the impact, so we already know that free applications have been impacted and in total, around 1500 user actions have been impacted per minute.

B

So this has been a user visible problem right. We also see if you look further down that one of the applications had to slow down to 2.5 minutes as a medium value. We also see that all browsers, all user actions has been affected on all operating systems across all few occasions. Now the nice thing about this, you is if the problem had only existed in an area around Boston or in an area around Vancouver.

B

We would have told you so here right and we also let you know, what's the root cause and in this case we've analyzed more than 80 million dependencies for 2nt. For this problem and analyze, the do root cause has been a cpu situation on these two nodes, with the given name right.

B

So I want to pick one of the take a step back here and go back to the number of the 90 million dependencies which we analyzed here so 19 million dependencies. Where does is large? Number come from? So when you install dynaTrace for openshift, we automatically detect how your process is communicated. We automatically detect how the services communicate.

B

We automatically baseline the metrics like response time and and many other things like those are dependencies which we analyze so that we in the end can come up with a clearer image of what has been affected in your application and and whether it's the fact that your users are not.

B

Okay, so learning number two learning number three: is the network isn't reliable? Let's quickly talk about the role of the network, so when you migrate from a monolithic to a micro, service-oriented architecture, you trade, fast in-memory, communication too much slower into service communication across networks.

B

Now, let's take a look at the service flow again, if you look at those two services, the journey service and you check destination service. So if you want to validate whether what we've done here actually makes sense, so you can see the batur new service for 99% of its invocations with a 99% de tourny service is called invokes to check destination service. So that's about one time for a request right.

B

So if it's communicating with services contributes considerably to your response time right, you might reconsider whether a high level of distribution of your micro service that actually makes sense or if you might want to read, migrate those services into a single service right.

B

So what if the network infrastructure itself becomes a limiting factor in your application and those things indeed happen. So what you see here is from our demo from Alabama application, but I recently had a discussion with CTO of a company in Europe whose goal is no less than becoming the next generation eBay in Europe, and what their problem has been, that they had a high number of TCP packet retransmissions and they didn't know where it came from and as a rule of thumb. If you have a highly networked application, that's communicating widely across the network.

B

Then a TCP retransmission of six percent already renders your application almost unusable. Now, in this scenario, also the scenario that the CTO told me about the head of the TCP retransmission percentage of around 10% and they found out that it was due to a defect pin on one of their network cables. So how could you potentially figure out this problem? If you only look at your log messages, you possibly can- and this is what you automatically detect for you- and this is why baselining and automatic based lining is so crucial.

B

Ok, so let me talk about how to dynaTrace: how can you, how can you get dynaTrace into OpenShift?

B

So let me introduce you to Dino tricks. One agent can choose one agent is our single agent technology that allows you to monitor your applications, your services, your processes, your docker containers, your hosts and the underlying infrastructure, without any configuration out of the box, and this is how it works.

B

So, basically there are, there are two major options and the option number one that I'm referring to here is the one that we that we would say is the preferred option, because it gives you the entire picture.

B

This is what we called what we call Dino face of one engine for a full sec monitoring, so what you would do is you would roll out of one agent on the host machine that make up your openshift kubernetes clusters and once you've done, that you will get insights into everything that runs on top of your machines, including the OpenShift and kubernetes infrastructure, and the only real containerized processes running on top of the platform.

B

So we've also created an instable role, which is available on the ansible galaxy for the automated deployment of of binaries one agent for select monitoring on your cluster nodes. So this is very convenient.

B

Another option that you could choose- and this also includes the full state monitoring approach- is to run the dynaTrace one agent inside a docker container, and we actually in this scenario one agent does not run in a docker container, but we use a docker container as a vehicle to roll out one agent on the nodes that that make up your cluster through privileged access right. So this is what it actually looks like and that we're proud at since a few days we're our dining system.

B

One agent container has been ratted container certified, which means for the container is secured and entrusted, and it's ready to use on open ships and on atomic based hosts.

B

So this has been the preferred approach and you can use it on any opposite platform where you have direct access to the host level. But what happens if you use open shift in a managed scenario like open ships, dedicated or open shifts online developer preview right? What can you do so? What we did is we provide dynaTrace one agent for pass monitoring. So if you don't have access to the node level, it's not a blocker.

B

What you can do is you, can use penetrates one inch in footpaths and conveniently included into each of your docker containers that you would like to have monitored by Dino gems and the way that you can do this is by using either the openshift command-line interface. You can also use the the nifty sty or a source to image tool, that's also provided by Red Hat, but you can also fall back to traditional docker files.

B

If that's what you like to do right and if you're interested in doing so, there are lots of tutorials which we provide on the openshift blog. So what this gives you is that you can use dynaTrace with openshift on any open source platform and for any openshift offering whether that's open just origin or open shifts container platform, as it's called now opens. If dedicated or OpenShift online.

B

And if we're interested in learning, how you can roll out one a check either for full site monitoring or for your data containers, please take a look at the blog articles to dynaTrace team has provided on blogs at ourchurch.com.

B

You can also refer to our landing page. We can simply find us by a search, Explorer, open ships, monitoring on Google and then directly follow to our landing page and once you're. There please check out the dynaTrace free trial and please provide feedback.

A

Martin is if there's.

B

A number on it.

A

Okay: oh, there are a number of questions that are coming in probably more on the technical side of how all this works, gonna pause and um the first one you've kind of answered it a little bit from a technical perspective. How does this all work? Is there an agent that runs in each container and how does it trace a call through many containers and does it add header information to all the packets number of questions in one and then another.

B

One that's quite another tip allows question, is how does it cuddle up house for.

A

Others does it add header information to all the packets.

B

Okay, all right, so how does it work to inject benefits? One agent into container? So basically, we've divided the insulation of one agent for pass in into three simple steps right. So the first step is: go download one agent for pass from from from dynaTrace, and this will install them choose one agent inside your container. So this can be a container that you create using source image. It can also be a container that you create using the OC command-line interface.

B

You can also use this approach to inject damages one agent for pass into existing application that already run on open ships by leveraging OSI patch or OSI edits, right and- and the approach is basically that you install the agent into each of your containers so once the installation has been done before Java. This is very simple at the simple comment: single command and it takes approximately 20 megabytes of space in your container, and then we provide additional convenience on top of that.

B

On top of that, because the one agent for path has to be injected into your particular application process in terms of Java Java, this would be H a VM. But to make your life convenient, is you just have to source a configuration file that already provides the necessary Java options so that when the child virtual machine is started, automatically picks up the agent and that's step number two step number three, for you would be to start your java application and that's it so in order.

B

So what you will get when using the pass approach is, if you will get all information you can't, you cannot obviously get any information from the hosts that you know are the foundation of your openshift cluster, but you can get all the information of your processes of your services and of your applications, including real user management right so, and the other technical question regarding header is that when you use Spanish as one agent and a web server for real user management, for example, then it would add, add header information so that we can keep track of your entire transaction from the web browser or your mobile clients across all the backend tiers and back-end services that are running on on openshift.

B

Ok, I hope this answer. The question is not, please feel free to reach out to me directly. I will in a minute show the images again is.

A

One last question here about what what overhead can be expected if we deploy the agent inside the workload containers.

B

Yeah, so from depends on what you mean with overhead, but if you, if you talk about, let's start with hard disk space, so this really depends on the technology. So, let's say for Java the the agent cost you no more than twenty megabytes of space inside your containers and I think that's a pretty fair amount of space.

B

So, if you think in terms of performance and we've taken a lot of caution, because we know that we integrate deeply into your application, so, for example, Java, depending on how deep for your instrument your application, but we take around one to two percent of the performance of your application and we believe that, for the amount of information that we give for you out of the box, it is also pretty pretty fair amount.

A

All right, I think that answered his question, he's saying Thanks. Why don't you put up that last slide that you had with your email address? If there are more questions, laughter.

B

Yes, I want to do that. I just want to quickly present a quick outlook. What what's coming up next and there's a lot of things coming down the pipe, and we want to integrate more deeply with the openshift platform. But what I would like to tell you about is that we would like to have dynaTrace inside continuous delivery, as we call it also work on openshift.

B

So if you look at the the ecosystem, around open system open shares in the work that has been done by the developer experience team of open shares, it's what what they have is. They have set up a continuous delivery project that you can quickly spin up by just using an open ship template right or kubernetes kubernetes template.

B

So what dynaTrace has been doing for for many years in the past is that we have helped customers understand whether their applications are good enough to be pushed into production or not so by analyzing, their automated tests, meaning integration test and acceptance tests, whether they use whether they'd have stretched KPIs or whether they test the user interface browser. You browser UI, web UI. It's that we can understand and whether this would have any severe performance implications or not right. So we call this an architectural validation before you go into production.

B

So what do we say is I want to keep it short now, because we're already over time is don't just optimized for speed. We all talk about speed right, but if it's also about quality, so instead release fast and with certainty, and we want to provide you with the use case where you can use dynaTrace not only in production but also in pre-production, to understand whether your containers that you have an open shape are six four tuples in production or not, okay, so in detail, this means that we can help.

B

You identify bad code before its gets checked in. We collect performance metrics from automated tests, and we feedback those metrics into champions and are able to also stop bad builds. If you want us to do so, but that's an optional thing right. This is what we were able to do.

B

So if there is only one thing that you take away from some discipline shift college breeding I would like you to be it the triple a as we call it. This means the dynaTrace provides is covering out of the box. It comes with Auto base, lining of all the important metrics that we determine from your processes from your containers from your services in your applications. It also comes with automatic problem and analysis. That means, if you don't have to manually, correlate thousands and millions of incidents.

B

We understand how these incidents relate to each other's and serve them to you conveniently as single problems that you can replay and understand how they evolved and how they actually vanished over time if they've seen transient okay. So please, if you have any further questions, try out the free trial, give us feedback and let us know how we can further improve and help you succeed in your journey. Thank you very much.

A

Martin, there is one more question that just came in john was sort of a follow up on that header one you had earlier, and how do you ensure the header information persists through various micro-services? If it gives an example, the example is if a rest call goes to a serve to service a two seconds later. A restaurant goes to service, be how you know they're related.

B

Yeah much more if I can give a medical question, because that goes deeply into how we how we implement the agent, but if John, if you send me an email, I will send you an appropriate answer and can also bring you in contact with a development team member who knows really how this has been done.

A

So please do reach out and if you can't get directly to him, just post it the question on the mailing list for openshift Commons, which most of you should be on if you're not sign up soon, it's on Commons, F, openshift, org join and adjective the main mailing list again thanks, Martin thanks for suffering through the technical soundcheck, and we're really pleased to have him here. We'll definitely do this again in probably in in 2017, because I I do I. Would love to see this running on.

A

I have a couple of large-scale folks that are using that could use this and I'd love to see that some of the working through that and and just sweet see it live in an action, though I do realize that it is a in the interest in time. It's probably I'm very hard to demo that live, but it could be would be an interesting thing to see. um So once again, thank you and folks.

A

If you have questions for Martin or about dynaTrace, just reach out directly to them and we'll be doing this again next week and hopefully you'll join us all for the OpenShift Commons gathering in Seattle and Martin art I think there's a couple folks from dynaTrace who are coming to that that have registered so normal the dirt. Alright, you can, in person in Seattle, on November 7th at the openshift gathering, but take care. Everybody and they'll talk to you all soon.