keptn Developer, 1 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keptn and mySuperMon integration - User and Developer meeting

Description

mySuperMon is a Smoke Detector For Application To Database Traffic. Its mission is to provide automated tools that help eliminate software project delays and reclamations caused by database performance issues. In this session Mikko Larkela from Devaten will present the Keptn integration service for mySuperMon and demonstrate how to use these tools together. This service is responsible for collecting database statistics performed while test sequence executed using mySuperMon endpoints. Mikko will also talk about the developer experiences with Keptn.

References:
- https://github.com/keptn-sandbox/mySuperMon-service
- https://mysupermon.com/

About the speaker’s company. Devaten provides a simple, affordable, easy-to-use and automated monitoring solution to optimize database performance. It helps software developers, DBAs and test engineers in eliminating project delays caused due to database issues being detected too late in the SDLC.

A

B

So now we should.

B

A

Okay, uh yes, so we are live! Okay, it's nice yep! So thanks a lot to me for joining today we have a short question about uh captain my superman integration. So the idea is to just have informal discussion, see how this integration works, how to benefit from performance testing, competition.

C

A

Also, what are the developer experiences for captain integrations, so I invited a few developers to the stream, so hopefully they will join and you can just proceed.

D

D

We can also tell a bit the background for for the button as a company as a very small company from finland for focusing on database consultants work. We have only two employees and they're working since 2013 and.

D

We have actually built start building my supermoon for last 2019 early and with with our partner company from india to your next, with very nice dev team, stairs and also rahul here today, as our tech lead, and he is working with the captains at integration.

D

So he is today telling how actually we have done it. The that nice captain integration thing.

D

And I can also tell few birds.

E

D

How actually we have planned this, or should we talk about commonly first, a bit uh the need for this and how actually vr came up to this thing means, while doing the consultant work in different different corporations.

D

Seen too many times that actual database is the bottleneck there of new deliveries and new things after new deployments so way too often we added monitoring and checking how and what is the bottleneck, and then we are writing report to say that these queries. This is hitting like crazy. This data query- and this is really heavy wherever it is now, if you fix this, your application is on track again.

D

So this what we have done for a while. So after this, our main idea is to make all this automate. So we can catch all these database issues in early by integrate my superman in different testing pipelines mika just.

A

To make sure, uh are you sharing, slides now or not yeah.

D

Because I can say yeah, so it's fine.

A

I followed the discussion but yeah, maybe yeah we're showing slides right now and uh yeah. I'm not sharing the screen.

D

Yes, yes, I will share one thing: I have no, not many slides, I'm going to put this on about our architecture and then later on, rahul will actually tell how and some slides and something so, but we can talk about bit uh about architecture of us. So in green area is our dashboard that can be anywhere. It can be an abs, it can be a local machine, on-prem or it can be cloud-based.

D

So here we are connected to activemc with gm server to athens, so there can be multiple agents, and this agent could be a docker image or it could be x, file and different. Sas users can put this agent in a docker image or some other place, and then we connect t to our sauce and our sauce is actually.

D

You can call it with rest, endpoints start start, recording, sorry and then.

D

uh Sorry so this topic goes start and stop recording which bill after that it will be analyze the data and get response for the user, and I will present also one small thing about our application, and then we will let rahul go so I will go right right away. Our application, which is results, are available after stop recording, and here we are presenting something nice which is query id.

D

It means that we have generate the in query id to its query, so we can identify their own with this query, id all the sql statement and all the metrics for those queries, and we can also include if we instrumenting and add a comment to the sql statement by comment.

D

I mean okay, there's no comment here, so we can actually add from where this actual query has been called from, which java class fits method, name so by this developer, can easily go and check how this query has been changed during time, but with captain we have done some nice things with this query id so there we can actually track individual queries by time.

D

But for that I will let you rahu now go and tell more about how we can actually do this and how beneficial is to integrate my superman to see database statics during the captain test.

C

Yes, thank you, so I will share my screen.

A

Here we see it.

D

Anyone has any questions anything we can just talk and have a discussion during the presentation and.

C

Yes, so we first start with what is my supermarket. The name of the my superman stand for my software use case performance monitor as the leader in the use case, westero database application monitoring, unlike most database application testing tools, rather than the benchmarking overall database performance during the set period of time. Our software as a service sas, looks at a specific use case and our you know. Innovation is in the process of being patented for its unique approach to perform the improvement early during the software development.

C

So we are focusing on detect, recombinate and optimize. So my superman mission is to provide automated tools that help eliminate a software project, delay and reclamation caused by database performance issue.

C

So we go to this architecture design. We could already explain it so on server. We have that much superman dashboard. My command dashboard has a mysql database and on client side we have agent, it may be docker image and setup file. So for communication between the dashboard and agent, we are using ftmq.

C

So uh first we are getting the database connection, information from user and save it to the agent database and agent using that connection database connection information connect with this client database and capture some statistics and send to the dashboard.

C

So when users start, the recording request go to the my superman agent, my support agent clean up some setup, then user execute some query using the generator or work to the application or directly executing the query on database. After that, we hit the stop. Recording.

C

Then agent get the statistic: data saved to the s2 database, then same data sent to the dashboard and on dashboard we are showing the statistic: data in the form of graph table, so we go to the next. So first step is a sign up. You need to kill all the information for sign up when you sign sign up. You get one mail that contain the subscription key when you log in you redirect to the download page.

C

So here is the download page. So here we are providing a link for download the agent. You can download the setup file or docker image by clicking on that tab and you can choose the which database you needed. We are supporting a mysql postcase, sql, mssql, mariadb, oracle and mongodb.

C

So next step is you need to verify the subscription key first, when you sign up you get one subscription key, so here you need to enter the subscription key and verify it. After that we have agent.yml file.

C

We need to create and apply the agent. file in the same question where captain is running here is the agent.yml file, and you can see here is the image my superman mongodb agent latest. You can find this url on that download page here is the command using this command. You can apply, deploy the my support agent on captain cluster next step is uh after deploying the agent check the log of agent using the below command that cook, ctrl, minus and captain logs minus f deployment.

C

This is the movement and deploy and when there you can find the locks- and you can see the agent unique id at the bottom, so you need to copy it and go to the my superman. Add agent page here is the form, so you need to add that uh all the required information agent name is in description.

C

You need to choose the database type that you want, then driver type, and then here is uh you need to put the agent unique id and save the agent after that, you need to add the application: here's, the application name, application, url application code and um application database types you need to enter on. Next, you can find out the database url formality stages like dave staging and production.

C

So here is the command that coop ctl get parts minus or white minus n.

C

Here is the url for that dev.

C

So next is add data source, so uh here is the form, so you need to enter the driver type that agent that you are saved application name is occur here. You need to enter the username and password and host url that in last in last screen we are find out this host url, then port number, then database, name and email.

C

You need to save it, for you need to save three database for day of stage and production.

C

uh Only difference is that that host url is different for different dev stage and production.

C

Then reply to my superman service. First, you need to create the secrets so using this command cubectl create secret generic. My superman, so here you can add your username and password. Here is the link for that. uh My superman service repository.

C

So next step is uh error. Captain endpoints captain, if I took on my superman, end point to the service dot. Why I'm gonna find my superman here is the code for that and you can deploy yours. uh My performance service using this command click ctrl, apply, minus f service, dot, yml minus end caption.

C

Okay, next step is verify that my superman service running is user, authenticated using this command qctl minus n captain log minus f's, so you can see the logs and you can see the message. Your my supernova authentication, successful.

C

So next step is to enable the prometheus monitoring add a scrap job to prometheus config map. So here is the command for that. The ctl edit config mac from ts, server, minus and monitoring. So here is the scrap job. For that here is the my superman service name.

C

It is kept in the name space and is the port 8080 running on next step is create the my super one app identified dot text file, so superman application identified file in the project specific captain's report you can find out the application identifier using this step. You need to log in the microphone app. Then click on the application management. Then there is a list of application. You need to.

E

Choose your application name.

C

And then you redirect to the application detail page on the left side. You can see the application identifier, you just need to copy it and save into the one file and save it and using this command you can add into the your captain clusters.

C

So captain add resource project. Stop starts.

A

C

D

C

Executing the performance and functional tests use, we are using a local service or we can use any other service, so my performer can collect the database statistics and here is the command for triggering the delivery for cars, so delivery project, soft soft service cards, minus image, captain example card stacks to 12.3 next step is.

E

Here the screenshot.

C

For captain cli, you can see the multi-stage dell staging and production here.

C

Here is the data we are sending to the prometheus, so we are sending a stop recording data report data we are sending.

E

The most executed query.

C

Was executed, query and table analysis data.

C

This is the stock recording later on for media's graph screenshots. This is the most executed query on chromatius.

C

B

Query on prometheus graphs.

C

And this is the table analysis, data on prometheus graph.

C

On mysupranow, we collected the data as on dashboard. We are showing all the use cases, use case, history and graph, sql, queries and query details and in report we are have execution wise and use case. Wise graph overview table analysis, data and most executed and was executed. Query.

C

So here is the screenshot for first here we are showing the all the use cases in table format and next step. Here we are showing the.

E

C

Page, we are showing the actual sql query, statistics data and also we have that table analysis data there on sql page and next on detail page we are showing the table. Row counts explanatory, jason and table analysis response here on report data we are showing uh all the most executed. Words executed table wise execution, wise graph there. So you can see the on report data.

C

Thank you now, okay, so you're, the captain. Okay,.

C

Stop recording data on this on staging.

C

Like this information.

D

Hello, can you go back to the captain side and so again that one it's back? Do you saw that axle? I don't know that captain baits, where you see the results and if you scroll down a little bit, I can explain more.

D

So the new value old value is what we do with analytics with our after stop. Recording old value means that if a user has defined a baseline, there can be user defined baseline to each use cases. So the old value is always the baseline value and new value is the existing value, and here is some differences, so we can follow the overall example. Query execution counts how many queries are executed totally in that run, so this is like overall page and during the run there we are pushing the tattoo prometheus.

D

So we can follow the query. Executions during the captain run in prometals and also after stop recording. We are boosting our reporting data to promote those as well.

D

So if you can go to the promoters page- and we can see example, one query one column.

C

And this one column here we can see on which query id that this column has, and here is the sql statement that column has- and here is the data you can see, and here is a query app reference. So uh here you can see on which application, which package, which class and which methods this come from.

D

Okay, can I say a few comments on this so by following its individual queries. Here we are presenting the query id and that query id can be found in our dashboard and our report to see to us as well statement, explains and all the details for this query, but in prometheus we can see over time how queries are executed during different deployments.

D

If, in some deployment, there is some more iterated queries, you can actually follow in prometheus graph. How individual queries are executed more or less and same is also the heaviest queries. If some query has been executed, if there is example, drop the index for some reason or the data has grown a lot in the tables, you will see that those heaviest queries are actually reading more and more data, as we are putting also the rows, lead values and that kind of metrics to promote us.

C

So here you can see the was executed. Query on our report, page.

C

And here most executed query.

C

And here the table analysis data, this table name item cards. We have this information. What is the size of the table? Execution count, storage, size.

D

Yes again, let me say a few words about this table and leak standard. This is uh based on our experience. There is usually a lot of heat static data. I mean they are iterating, some product information or some data, which is taking the most of the queries that has been totally executed. On the server side example. Very often there is 30 percent of 40.

D

Of all queries that has been executed is some kind of static data tables only and by removing those static data queries that they should not hit at the come to the database level at all by adding some query gas um application level case or some other mechanism by preventing those to come database, it will actually make obligation run much faster.

D

So by this table and stata, we can actually see how many queries has been executed on these tables and what is the size of the tables so, based on this information, we can recommend actual to development teams that hey see. This is a really small table, but you are hitting it all the time like crazy. So you should add these table handling processes in your application or some other mechanism, and then you can actually see the difference how um faster your application is running, so this is made.

D

For that reason, and this table analysis data is also available in prometheus. We are actually setting this table name and all metrics here in also in the from other side. So you can also follow the different table table wise metrics here.

C

Just like this.

C

So this is a table analysis data onto mickey's graph. So here you can see the use case. Identifier, table name and job here here is the crop for that exhibition counts.

C

We also showing here most executed was executed. Query.

C

So here's the query id here is a skill statement and you use case identifier.

C

Here's the graph.

C

Almost executed then go to the first.

C

So here is the was exhibited. Query id you can see it's called statements, and here is the graph for that. First query:.

C

These are the column execution count you can see here.

C

On both query executed.

D

Yes- and that is a link to this report- is there in captain after stop recording.

C

D

Is a link for this report.

C

D

C

So here is the report link so using this url. You can open your reports like this. Here we are showing.

C

My superman reports, so here is the.

C

Execution wise graph here here is a use case, column wise graph. So you can click on the front column to see the deep front graph.

C

And here is the overview data.

C

By clicking on detail page, you can see the explanation here.

C

Table row counts data like this and table analysis response like this json format,.

D

Yes- and we are booting this reporter, as talked before, we are boosting it to the prometal side, so there you can see the graph for different different deployments.

B

A

So any question.

A

So, for me, yeah integration site thanks a lot for the demo and a lot of details.

A

I would be actually interested to know more about the developer experience so basically to implement that you need a special service and captain right to provide all the required apis, reporting and other systems. You presented correct.

C

Yes, it was a great all. Things are documented, so we not getting trouble for that setup. Initially, uh we have some less knowledge about that kubernetes. So and all these things, so you know we require some times, but after following your documentation, we it it's.

C

You provide us all those things, so you just need to follow the steps.

A

And the integration service is uh based on the golang template right, um yeah, just opening the code right now.

C

Yes, okay, it's a second.

C

Okay, here is the service.yml file, and here we are providing the uh docker image. We are creating a docker image for that on our my supplement, docker hub. So here is the docker link for that mice command service. So you can put into the your service.yml file and.

C

C

C

Here is the matrix uh here we are sending the.

C

Start with the start, recording first.

E

A

So big question uh from to broadcast um yeah so use a python uh integration. Template right. Have you.

C

A

Moving to golang, or what's the reasons for using python these days,.

C

So you can see the uh here is the services that we are deploying on that captain cluster here is the bridge. It has service approval, helm service, here's the data source, cpr controller configuration service here is our homework agent, my supplemental agents, so here is our my supplement service.

C

We have prometheus service here and for executing the query. We are using the locus service.

C

And here is the secret that we are enter your username and password.

C

You can see also that.

C

Here is the agent.yml file, so here we are entering that. uh Our docker agent, mongodb agents.

C

Docker url here so he here is the agent.yml file that we are deploying so also, I will show you the.

A

C

A

Basically, a standalone so you're not deploying it within the database port, but you deployed as a standalone port.

C

Yes, and after deploying I'm forwarding the.

C

Here here is the s2 database that agent use this.

C

C

Here is the details who are not forwarding this here's the captain database. You can see.

C

Here is the locus service, dot, yml file.

C

So we uh here, I added the local service on my docker hub, so I'm using.

C

C

In test data, you can see here.

C

Using this url, we are hitting some query on our that's cars tv by using the local service local service use. This.

C

This health dot py file also there.

C

And this basic dot here.

C

And here is the location, big yml file.

E

C

Here is the onboarding cards. So here is the cpr.yml file. You can see here.

A

Yeah thanks a lot for showing all these details. Just in case, do you have a public demo for this integration service or how users could try it out.

C

Yes, uh we are writing docker image for my supplement service, and we also have note down this step- how we, how user can use the my supplement service on our blog.

D

And we can share this ppt what you present on. I.

C

Don't know yet.

D

This is the blog or some github where we can share these whole presentations. What you present! Yes, we.

C

Already said that github depository and here is the steps that I show you.

C

What are the steps.

D

E

C

Follow for that captain integration if they are using the my super monsters for database.

A

Really nice to try out and uh which databases you currently support, so um I believe uh h2 just for local testing, um any other production databases. You support these days.

D

Yes, yes, yes, yes, almost. What are they listening to the basis escort oracle, mysql.

D

But now as well, we have a just only for just now, but we are planning to have more more support for different data sources.

A

And uh yep just asking a question uh from uh giovanni about uh your developer experiences, so you created this integration service uh for captain. So basically, uh what were the key obstacles for you, lessons learned and how smooth was the development process for you.

D

I didn't know who you can take, but for I guess we have uh uh different guys starting this, and then they leave our company. So it will be the change people in between so rahu, then the so it will be take time consuming to learn this, but I think now it's all clear. I think so.

D

But let me comment more.

C

So all things are documented, and so we don't have any trouble for that. So, following the document we get all the information, so we already get all the template. Python templates, all the things are there so.

C

We don't have any trouble for that.

D

And also the community on slack channel uh just.

C

E

D

And we get very nice.

E

Answers and support also.

D

There, while we struggling so it's also.

E

E

Any particular features.

A

Young missile captain so, for example,.

C

The integration from your standpoint, yes, we are.

C

Means for that query comment or we require some changes in word. Cards cast tv example, captains examples that you are providing us, so we need to add some query very.

E

A

Basically, it's just uh damage changes to demonstrate various workloads and basically performance testing for this example. Yes, it would be a good addition.

A

My car is one of the examples we use widely. I want to sock shop and other things so yeah. If we can improve it, it would be awesome.

E

B

A

So thanks a lot uh for the presentation and for walking us through this integrations and yeah it's nice to see my superman integration. Hopefully uh have you already got a user's introduction? I mean for captain integration.

D

For captain I don't know, I think so, not I think we are just finishing it and now I think um yeah it would be nice if someone tried also to see if there's something missing parts, I guess have we implemented correctly and they say place all places in part right, right, implemented. So.

A

Yeah, I think uh anything that would be useful and uh database performance is one of the critical points right now, because not all databases are actually scalable. Mongodb is quite good with regards to that. But still databases are often bottlenecks.

A

Use databases exposing open, telemetry, more and more. I think that yeah there is a lot of potential for future integrations.

A

So currently, you use basically open telemetry to prometheus bridge to expose all the metrics when we just get the slice, et cetera through standard flows.

D

Which is okay, okay,.

A

By the way, uh have you ever looked into the division project.

D

Not much not much yet.

A

Yeah I was just wondering whether it could help to expose additional metrics but yeah. It was open telemetry, I'm not 100 sure about that, because division actually started before so now. Okay, expose.

D

Almost everything like that yeah we could check to check and see if there's something we could use on. Yes, okay,.

A

So, thanks again for the presentation and your closing words.

D

Thank you. I I think the captain integration and this will fit uh will fit for our purpose as well as very nicely, because our really purpose is to include database monitoring in different pipelines and with captain.

D

It really fits well on this kept an integration that side of our need as well, and our actually purpose of this our tool is really fitting. Well there that's my thinking and that's why I also interesting very much in the kept inside integrations as well.

A

That's great so thanks a lot yeah.

D

A

D

Nice to have a discussion chat and this opportunity for us to present.

A

This uh yeah well, thank you too. So, once you have everything posted, we will be happy to share it and, if you want to you, have a venue for writing. Captain tutorials and, of course, deeper integrations would be nice if there are users who are going to try my super one captain integration out, I believe all sites will appreciate feedback.

A

I was our drawing projects and, of course, uh we would appreciate any inputs so that we can uh making them better yeah. Yes, yes, that would be nice. So, okay, thanks thanks and yeah have a nice day. uh The video should be live just in any in minutes, because we just broadcast so I'll. Stop it right now uh the video will be live almost immediately.