Ortelius Open Source Architecture, 1 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ortelius Architecture Meeting September 1, 2022

Description

Keptn and blockchain are topics of this Architecture meeting.

A

B

To September 1st architecture meeting for ortilius um last time we met, we had some. We went through and broke out the kind of like the tasks of the different pieces um of what's happening in our flow here um and I got a a Discord message from Joseph on some pieces that he found with uh around the captain job executor, um and he had some notes.

B

But I tried to open the notes and uh the Link's broken for some reason may have expired, um but let's go ahead and take a look at some of the pieces have kept in. So if anybody is is like a captain, Guru uh please jump in here.

B

um So from my understanding is uh we would be able to use a job executor service, um so inside of Captain there's going to be, they call them Services, it's kind of like their version of plugins um and it allow us to run uh certain things in this case.

B

That is um out there currently like um Captain's out there listening for the cloud events to come through and when a cloud event does come through, um it will trigger I believe the job executor Service uh to go ahead and do the the actual work uh as part of that process.

B

And it looks like it's just some extra pieces that get installed into the the control plane. Let's look at the shipyard file.

B

Oh so the shipyard's pretty simple, um we just have our stages and what our task is going to be. For example, let's see what else we got.

A

B

Just going to run the task under Alpine and do an echo, um so this task here is where we would go ahead and um do like a Docker build.

B

um Let's see what some of the steps that we have uh doing, a Docker build a Docker push some artilla stuff where we're going to run our CLI and then get information back where we'll eventually get into the quality gate and I believe the quality gate is totally separate uh than the job executor.

B

So that's a different like feature set of Captain around that process.

B

Any questions so far now the one thing uh Sasha you're going to look at with oatkarsh was the um how Captain in in GitHub uh interface together. Did you have a chance to take a look at that.

C

I think I I, definitely because she's been doing some stuff in there um and he started putting some stuff together. He gave me something to look just to check out just to get familiar with all.

A

Right but see if we can dig.

C

That up uh yeah I'll, see if I can try and find it.

C

But in the meantime I don't want to want you guys to like just sit there whilst I'm yep.

B

While you're you're poking around.

A

B

A

A

B

Oh, if we go back to the um our diagram.

B

Minimize that um some of the things with like the notification piece um with the let's see uh Argo uh at this level, Argo deployment finished- um that is part of the Argo uh listener, and let me see I think I have Brad's.

A

Okay Cloud event.

B

C

Everything in Captain is a service that listens for what listens for cloud.

B

C

Events from these particular different places.

B

C

If I'm thinking of the diagram so I'll go talk to that particular service designed for Argo, is that right right.

B

So there is uh um so from so on the Argo side. It has this this concept of a Notifier.

B

um So on the Argo CD side, um you could put a hook in so when Argo goes through its different states of doing the deployment, and it finally gets the the cluster synced to um the git repo. Once all that's been completed- and you know it does all of its um blue, green and Canary stuff and all those all those pieces it will um it will.

B

You could hook in a Notifier to publish an event, and this is one of the things that Brad was able to figure out, was being able to take um that information and send it over to Captain.

B

So this is giving us that that information um as a cloud basically a cloud event going back over to from Argo over to uh Captain. That's.

C

All Jason did he had to do. Okay,.

B

Yeah Cloud events, uh Cloud events are all um all all Json. So if you look at all anything like that, yeah I think it's Cloud events.

A

Dot IO, if I remember correctly,.

B

So this is the uh this is put together, mainly by uh I, believe is Google, um but it it basically is a a specification on how we should be able to um move things around uh pretty easily and I believe it is mostly a Json format. Let's see if they have an example here in the spec.

A

Attributes and.

B

A lot of times, um yeah yeah, the cloud event is serialized into uh Json. Sometimes they will be um I. Think you can implement this with protocol Buffs as well.

B

um Yeah protocol buff is like a binary binary data stream. um If I recall, I I've never really played with them.

C

No I don't know it at all.

B

But this is the the um the the cloud event and you'll see here where you have a kind of like uh you have your well-known fields that you kind of lay out and then, um if you, in order to allow people to kind of extend the event they give, you like you'll, see a good data field or something like that that you get to throw any additional information into that that payload, um that that's one implementation, other one is you just um you know, add additional Fields into uh beyond the basically the required ones um as part of your uh event.

B

So it's more of like a well-known event.

B

um So in this state, in this case, uh you'll see this a a lot as well, where they have a type uh to describe what is in the data so like in our case, um we are, our type may be like a um a component and then their data would be the component definition uh at that level.

B

um So that would be example of how you can you can. Basically you end up using the cloud event as as kind of like a header um to pass things around, and then you have your payload yeah interesting.

B

So that's what um wrong button! So that's what Brad is uh kind of doing here is this bottom part is the the cloud event and a.

C

Building bold web server listening for AP with apis right.

A

Especially what it is right.

C

B

Exactly you know, so you have some. You have some some well-known pieces and then uh then you have your data. Payload I did to bring it into my editor, so I can highlight the payload, but that gives us that um the way to to be able to really move this data around now um I was talking to Fidelity Financial the other day and they do a lot of work with Cloud events and they actually uh persist every single event that happens in uh in the world for their um their CI CD pipeline uh pieces.

B

So what they end up doing is they end up storing every single event uh into a database and that allows them to do metrics data uh later on down the road. So, for example, you want to know when a how long it took for an issue to be picked up uh in GitHub, uh worked on and then a PR created, and then how long it took the for the pr to be reviewed and merged.

B

All those are different events that could end up being stored in there in in the database, and then you can do the correlation queries uh to build up the the metrics around that.

B

So when we, when we do our event, data we'll be able to do that that similar thing uh from our side, knowing when we look at our picture, um you know how, when, when somebody committed, how long did it take for it to get all the way rolled out and all the quality uh Gates um satisfied you know, or if the quality gate failed or whatnot, we'll be able to look at that information because we'll have we'll be able to um if we wanted to.

B

We could persist all that data into our our database as well.

B

Any luck in finding that Sasha still.

C

Looking sorry, no.

B

Worries so um I think it was if I remember correctly.

A

A

I think it was a good.

C

B

No, it says, I think it's this.

B

If I remember correctly, I thought it was an action that you had to run.

A

That's good push preview release.

A

B

Some reason: uh let's.

A

See if it's in this folder post change, log.

A

B

For my remember, the the integration um was based on a GitHub action.

B

um So what would happen is because a captain is event driven.

B

um You would grab the the GitHub action would would grab the data um formatted into the the Json string and then post it over to uh Captain uh for it to go off and take off. um Basically, the integration was doing a um a formatting and posting a cloud event over to Captain.

C

ah Okay, yes, ah you just joined, he.

B

Had a meeting he.

C

Was able to join now.

B

uh Perfect timing, of course, we're just going over the um the GitHub integration um and from I. Remember that happens uh through a GitHub action. Is that correct.

D

Yes, yes, I have something to demo as well. If you have sometimes yeah.

B

Go ahead if you're ready, let me stop sharing.

A

Oh that'd be cool to see well he's doing that Steve is the ortelius uh site down right now. Yes,.

B

um We are moving uh stuff over to from one Azure um account to another, and the Linux Foundation is um didn't, get the DNS records correct for the move and we're working with uh the Linux Foundation it to get it sorted out. Yeah.

A

B

Yeah it's it's kind of a mess because it's they did part of it, but not all of it. So if you go to like docstat ortiz.io that will work uh and I think www.orteliusio will work, but not ortiz.io.

B

It's this weird limbo that we got into somehow or we're trying to figure out.

D

Dms is a wonderful thing.

A

Okay, thank you.

D

um Okay, so this is the sample record that I'm, using for this channel thanks to Arvin I'm using arvin's project for this Captain, then Captain. uh Let me show you the the GitHub action script that I have written. It's pretty much simple. uh We have couple of environment variables configured for configured as a secret for the Repository.

D

uh What we are doing here is logging into the docker Hub, then building the docker image for the application and then fetching the docker digest and then then creating the payload for captain and then again calling the chapter and API and uh like using this payroll. So as part of this, when this action is finished, what we would see we will see here, this sequence will get here, which is a delivery sequence, basically and as part of Shipyard, another I've just kept it very simple. So we have two environments that is seven production.

D

uh Once the dev deployment is finished and validated, then only the production it takes off and deploys the changes.

D

Quickly show you the application and then maybe we can make some changes to a camera and see how long it takes to and.

A

B

It in this example, how is the um the deployment happening.

D

Deployment is uh okay, okay, so for that uh there are like some prerequisite. That is, you have to create a project. You have onboard Shipyard VMO file right right. The second prerequisite is, you also have to onboard your services. So since this activity was like one time, activity I have not configured that in bash file. Take the transaction file, but we can also include that, uh if, if that is the case um yeah, but what I was thinking, whatever processes are there that, like needs to be done, one time we can.

D

We should like isolate them and try to you know, do it somewhere else, rather than any GitHub action, because GitHub action will get to there every time you have an increment right right.

B

Yeah that makes sense to split out split out setup from everyday use.

D

Yeah, based on the inputs and feedback from whatever.

D

Okay, so this is the application. What we do is.

A

D

So this energy.

A

Okay, so this is.

D

This is basically this component, the middle component, so here here that you will find the best place. This is here in the charger.

D

So what I'll say if I'm doing to go in place.

D

Okay, once again, I haven't shared this one.

A

B

So, while that's running, can you show us the shipyard file.

D

D

Yes, and in the meantime, I can show you. We should get file that I'll use.

A

A

B

This one doesn't have a quality gate in it. Does it.

D

No, no, let me show you the another one that has started itself.

D

A

D

So here we are performing some tests.

D

On in the query, and based on the test that is performed um here, will be more sickness trigger. Okay, so.

A

It's pretty easy, oh.

B

So is that um on line 36 a manual approval, then.

D

So if this is successful, then this will automatically go into production, but if not, then you can you have like one API wherein you can pass an event which is like approved in the particular sequence that is basically manual approved approval from uh for the energy students.

B

C

uh So the shipyard is where you're defining your different stages right.

A

C

So Dave q, a production, very nice ah you're, making socks suck shop.

C

Oh, that's very cool.

D

C

That's what you see in the UI right when you, when you're on the UI, now that that code, there, you see Captain, Captain, yeah,.

B

So, let's see if you can show us what you um the running, you just did.

D

Yes, sir, so let me just speak with this.

C

And your captain is managing this. um This application right.

A

A

C

Very nice, is it a demo site? Is it a demo site.

D

Very cool also a few more details, so this is what we wanted to do. We have defined two sequence: three stages, that is that production, and here, as you can see here, this is my project name. You can Define for n number of project assets. They have defined two projects on this or other applications in the same project. You can also refine multiple services, so here I was trying to run multiple services and but like for some reason, I was like they were running, but they were not able to.

D

You know some kubernetes confusion, something they were not coming up so that I'm working on like how I can um and bring up the microservices using captain and, let's suppose after production, we want to trigger some sequence. That is that that is totally doable. So right now, like um I, spoke to Dad today and what we are trying to do now is, and we are trying to integrate with r2base apis now.

D

So, let's suppose, a deployment, if you have deployment is green, then what I'm trying to do is, but you know like integrating any apis that you, you know as part of our previous API and then um look at the response that has been generated in postpaid, SQL or continuous.

A

D

One API works. We can integrate multiple.

B

Yeah there's um one of the things with the ortillas apis is kind of like a uh you have to do a login call first, and then you get a basically it's a cookie with a JWT token in it and then from there. You can do your your authenticated and you can do any other API calls. uh Using that token.

D

You know yeah actually like we have three ways by which you can you know if you want to do it, so we have one way of doing so is using a batch file which can be integrated in Catman, using a job executor that is being out of the box provided by captain, or we can go with integrating the web store in webos. They integrate Secrets as well. So we can give a secret like this. Is my authentication token.

B

Right now is, is the web hook um uh a cloud event uh based webhook.

D

Yes, yes, so let me show you: what are the Integrations, so, whatever integration that you see here right, they all are like cloud-based, so I'm using hand chat um integration. So these are the events that are listened by, uh has a job. So that is why when I say deployment.creases- or that is why, when I say, test dot quiggers, so this jmeter service will start triggering the resources that I have added got it. Everything works and we went here.

B

So would we would we have our um like uh io.artilius.event, you know um kind of we would Define our own events for ortelius is.

A

D

How it happens like um do we really want to do that, because if that is the case, we would end up writing a separate integration. Okay,.

B

All right just double check and see how deep we needed to go.

A

B

Yeah I know: Joseph was looking at the uh job executor um service and he sent me a link, but I couldn't open it, but um that's what he was diving into um and it sounds like that's going to be an easy way to um do. Our first integration.

B

Now one of the things, um so there is a python library that uh it's in the artillius I think it's in the artillius comp update repo, um but it's that python Library does a lot of the wrappering and because there's multiple calls that are needed, sometimes to like, for example, you want to create a new component.

B

um It'll go check to see if the the component already exists and if it does it'll update it. If it doesn't exist, then it'll go create it. um So that logic is in that python Library um and that may be the easier way to, and it does all the rest restful. Api endpoint calls, um but there's just some extra logic around um the restful apis that save you a headache from bash trying to do it with pearl.

D

Okay, so uh that executor is for which Step now can you just really relate to the you know, the actual right now that you created right in order sequence.

B

So the the job executor um service can call python right.

D

B

I think or or if you could call bash I'm sure it can call python.

B

um So that would be where we want to do. The the integration, probably is my guess instead of like, instead of instead of diving in and, like you said, getting uh doing our own uh event triggers and stuff like that, like uh the J meter service.

A

B

So um let me share my screen: uh real quick.

C

What is is Jamie to is, that is that, uh is that, like a testing tool.

B

Yes, a Java test, Tool.

C

Okay, so I made it the testing tool for API responsiveness right is it? Is it that.

B

I think so, um and I can't remember if it stubs out the the call or if it actually does a call to an endpoint I can't remember: okay,.

C

Is that something you've got to edit uh yourself? It doesn't come part of the kitten.

D

Like if you deploy Captain, there are two files that we can use, so we have one basic checks right and the other one is like a load so and load what they do. There is like um a static JavaScript and what they will. Do you tell them. This is my end point that is going to be deployed, and then they will, you know, push some load and test that okay.

A

B

So, um based on what we talked about today, the um you have uh this part figured out what the GitHub action um uh in basically um throwing a message over to captain now right now you have the GitHub action doing the docker build. um Will it be possible to have Captain? Do the doctor Docker build itself through a job executor.

D

um We can do that, but what Brad was thinking is to keep CI and CD separate as part of the standard of bit of practical okay. So he was like a strongly suggestion to keep a CI first and then do the CD stuff.

B

Got it okay, so our diagram here may need to um change a little bit um based on that. So right now we have Captain dealing with the the CI piece, and what we'll probably want to do is change that up a little bit so Captain's dealing with the CD portion of it.

D

Yeah only the docker thing is has been changed. The same okay.

B

All right, um I'm, talking to Brad later today, so I'll I'll, get some more insight into that. Okay, excellent um and we have the quality gate, looks pretty simple uh to be able to add those. In with the like, you said, the out of the some of the out of the box, uh kept in quality gate pieces um and I was showing everybody before you join on Brad's um events around coming out of Argo.

B

So that looks good, so I think what we'll just need to do is flush out what we need from the artillius side for the events and and which restful API calls that we need to make and I think going through. The python Library will be the most convenient.

A

C

Great work, of course, yeah.

B

Looking good cool.

C

I disclaimer, that's awkward, gosh I'd do anything I'm, just learning from what kush.

A

B

You're well Sasha, you gotta, say you're the person asking the right questions.

C

Steve, you said diplomatic wow, that's so skills yeah.

B

You're just asking the right directions to guide you cursed on the right path. Oh.

C

A

C

B

So that looks good um I will, like I, said: I have a meeting with Brad for the Australian working group uh today at 4 30 mountain time um so probably be really early in the morning for ukash, um so I will get that information and I'll try to pull it together uh and share it with everybody to so. We we're all on the same page um between the two meeting groups here: okay, looking good everybody, uh excellent job.

C

B

C

Fascinating I must say to see this in action right and how it works, and it's really super interesting. Wow yeah.

D

A

That's just this.

B

Yeah and and check out uh definitely check out that uh Cloud events.io um uh and look at the spec around Cloud events, so you can just get a a basic feel for what we're going to be Pat. How Cloud events are kind of you know. The basic building blocks are are put together, so um we can expand upon that uh because there's other tools out there that use cloud events which is like directive and there's another one.

B

I I ran across just recently I can't remember off top of my head that are focused on cloud events as well.

B

Oh, uh it's! The CD events project uh in the CD Foundation they're they're spec, that they're putting together um is focused on cloud events uh and the whole idea around. That is just to make interoperability. So if we want to um take Captain out of the picture and put in directive, we don't have to do a lot of uh re-plumbing.

B

Basically Plug and Play.

A

B

All right well excellent, thank you. Everybody um and, like I, said, I will uh pull together. um What I get from Brad this afternoon and um I will let everybody know what the next steps are uh once I get a a status of where he's at.

D

B

Sound like a plan sounds.

C

B

All right thanks, everybody cheers.

C

Everyone cheers.

B

B