DFFML Rolling Alice: Engineering Logs: Volume 0: Architecting Alice, 28 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Architecting Alice: Volume 0: Part 92: VSCode DERP Remote Exploration, Augur Applicability Confusion

Description

https://github.com/intel/dffml/discussions/1406#discussioncomment-3754932

A

All right here we are so uh you know what I just noticed check this out. So if you open github.dev, they now have enabled vs code for oh okay. You have to be signed it, but they've enabled vs code or all that basically in browser right. So the question that I have- um or rather that we have is, uh can we uh leverage um uh the in-browser environment and bridge that to whatever machines we have right.

A

So the first question is, you know they have existing SSH, connectors um and and container connectors for remote development with BS code. So what we're looking into is how to leverage the existing remote development stuff within vs code and potentially patch in support for connection to uh workspaces, directly managed via coder, which is something that manages workspaces. It's like vs code on your own infrastructure, so right.

A

So what we're trying to do is basically take the vs code on our own infrastructure, which is the dfl community, Community infrastructure and we're trying to which is, you know as we're using to develop our second right plug-in ecosystem and make it accessible for maintainers or for for contributors, um and uh so so what we can do, ideally, is um you can we will need to spend out a derp server um and we'll need we'll we'll? What we're going to do?

A

First is we're going to roll some containers, so we're going to have a container registry um and then we're going to uh deploy this coder thing I'm on top of kubernetes um and then finally, we will uh well we'll first. Actually this will be the deployment. But the first thing we're going to do is patch vs code um to take advantage of that. Well, actually, okay, we're gonna have to um so we're. Gonna have to run we're. Gonna have to do that.

A

We're gonna have to deploy coder first, so we should be able to deploy coder.

A

Where it works with us.

A

We should be able to deploy code or using this group stuff and then okay, we're forwarding fantastic okay, and this will let us manage our workspaces and and uh infrastructure with terraform um so and uh with so so this way we can, we can provide compute, um you know for specific uh use cases or we can enable the original uh thing that we wanted to do. You know with this whole thing is basically, you know as we're all working on different things.

A

We need to be able to like jump into different environments, so um so that we can review pull request and so forth right. So um this has come up with now, I think if you go back about like a year and a half in the weekly sinks or more okay, um so remote agencies. So let's dig in here so we we did.

A

We went, we followed this Rabbit Hole um and that's documented in the engineering log, uh basically search Source graph, um we weren't able to find the actual implementation via the documentation link for vs code. If you go to the plugin documentation link, it's it's, uh it doesn't link you actually to the code. It links you to a stub which is just docs. The code it actually looks like is contained within the vs code, repo itself, perhaps under this srcbs platform, remote browser- and maybe this browser socket Factory.

A

So if we can figure out how to connect these websockets um or if we can figure out what what's going on with these websockets right, so references new browser, websocket, okay, that's here, that's where we came from default, websocket, Factory, so browser socket Factory is probably what we want to look for there, foreign.

A

Agent service is values.

A

Press a different service, so where is the remote agent service configured remote file system provider, so remote agent service? Now, since you pass, we want to figure out.

A

Where does this thing get instantiated right and configured, because we had originally found this path by looking for the show terminal login, which is one of the config options right, so we started looking for config options because we figured okay well you're, going to consume them somewhere right, um okay, and then you know we should probably it would be good for us to actually just look at what is the remote provider being used here?.

A

From my exploration, GitHub.

A

Okay, so open remote, Repository.

A

Okay, GitHub remote repositorious extension right, so maybe we should Google for this thing or not. Google.

A

Does this have anything? This also has nothing right. So there's no code here, which is obviously problematic. It's like where's the code, foreign.

A

Extensions GitHub, GitHub Authentication.

A

Extension management.

A

Extensions GitHub authentication, extensions, GitHub.

A

Application foreign.

A

Okay, all right, okay, what we will do is.

A

A

A

Oh yeah there's this.

A

A

A

I'm curious, but.

A

A

Okay, so you can really do everything.

A

A

A

I've never done this.

A

Okay, so what I want postgres, which it has.

A

They have those grass, they do.

A

Okay, let's choose the same: Data, Center, I, don't know: okay, I! Think! That's fine, we'll spin this stuff up and spin this stuff down.

A

Okay, so we want to we'll load the data into the postgres database and then you know once again we already have the digitalocean API automation, yeah. So um we'll just add this database stuff for that. So because what we'll do is you know we'll Define and obviously we're just doing spin up spin down here, but you know we want to enable us to do whatever so I've, just just picking Digital Arts, because it's extremely easy so for no reason other than that everything that's just too complicated. So.

A

A

Create multiple replica.

A

A

Okay, it's a little past 10 cents an hour, very careful, okay! um Oh my God wait a minute! Wait! A minute! Wait! A minute! Wait! A minute! Wait a minute! What.

A

A

A

All right, okay, um you might as well request it right um so because what we'll do is okay, so, basically, okay, so for for four dollars.

A

The thing is, we don't need that long. We can just spin this stuff up and spin it down. So this is actually incredibly cost effective. um It's free these things can really crank at this feed. That I mean eight CPUs.

A

Basic notes, so let's do general purpose and CPU basic kind of storage up.

A

How much I'm curious? Okay? So it's okay! If you Max it out, if you do something stupid, then it's a hundred dollars an hour. Okay,.

A

Basic and professional Finance, okay, um I just want them to increase the limit in case we need it, add another node pool. Oh you can't. You can't have even more pools. Okay, oh they don't want you to just hang there and structure I, see excellent. Okay, um a dollar an hour; okay, oh wow! We can go crazy. Okay.

A

um We can really go crazy, okay, so the chaos so.

A

I'll Flash the gift.

A

So I found out uh we found out about this uh chaos, risk coordinate group today um which pretty cool stuff that they've got going on there um and you know chaos metrics of chaos, metrics for the chaos God, so they've also has a risk working group, which is which is interesting so, and this is around way to go yeah. So this is interesting um need to dig into this more dependency risk assessment. Obviously we have uh touch points there. So, where two points are quite a few of these places, it's like all of them.

A

Not you know we don't really we're not really dealing with business risk so much, but you know we want to enable people too right so um via context awareness, so I'm going to join these calls. So uh you know, if anybody else wants to join me, um that's I will I will be joining, um see, see if we can understand what's going on there and contribute and get involved.

A

A

Okay, yeah, okay! So then, beyond that.

A

Beyond that, what's going on so yeah I I would like to and then and then after seeing the risk death saw this uh auger auger, um so I, basically very similar right, so very similar to what we're doing so. What I was kind of thinking is that we could just spin this up run some scans and spin it down right.

A

um So they have this database backend ooh do they have containers they do they have containers published to GitHub? Okay, this solves our problem, largely Okay. So this is fantastic okay, so we can just wow. This is great okay. This is fantastic okay, so we can just start doing this right now.

A

I always say that Okay. So let's take this um and.

A

Let's go over here and, let's go. Let me go to my repo so we're just going to play with GitHub.

A

A

Actually, you know what we'll do we'll create an orphan Branch or no, it won't run workflows that aren't in main, will it.

A

A

Oh, but it doesn't matter because you can overwrite an existing workflow and then it will run it. Okay, wonderful, okay! So let's just go through here and do this so let's say: Alice should I contribute Okay. So.

A

Okay, so we should be able to.

A

On the back end,.

A

Okay, let's see if we can figure this out.

A

A

I think you can actually say.

A

I need a container.

A

A

A

Container service containers I think this is what we want.

A

A

I could have like.

A

A

A

um So what did we want? We wanted the service container.

A

Okay, services, redis, okay, so um foreign.

A

Five thousand.

A

Okay, so and then we're put.

A

To the documentation.

A

A

Jobs in a container so.

A

I know and then we'll add the engineering log in the commit trailer right so um should I contribute okay, so yeah. This is fine. You know, I, think repo URLs.

A

A

Or we'll just do repo URL for now um this is workflow dispatch, so yeah, okay, so let's say on push as well.

A

Dk so then this is what is this thing called auger right? Okay, so longer.

A

It's all pretty selfish, I'm gonna get rid of that, and then we want. So this is their compose file. So I'll actually put this here.

A

Put the soccer back end right, okay, so and then.

A

Okay, that's right, so we want.

A

These stocks earlier.

A

A

All right, so we should be able to spin up the whole thing within the job. Is the goal right and then we can see about expanding the different. um You know if we need to run into solution or something.

A

A

So right now we'll just select it to the uh GitHub.

A

Is okay? We actually went back and.

A

This actually should have been this, and so this should be database. It should be.

A

The back end is the back end. This road runs the scans, they said. Basically, it did it's not okay, it's big, okay, so Docker and buy.

A

A

Okay, so don't forget in the entire file Edition, why are we not providing it via the.

A

Maybe it's just because.

A

Scripts docker.

A

Okay, so talking bye and then.

A

Is this just gonna like put it in the environment, though.

A

Pnb file, so let's take a look at this.

A

A

A

That should be Docker compose a bad stalker compose and then it looks like they have separate Docker compose files, they're not actually using they're like have. They have separate Docker composed cells, um so they have database composing. So just pretty nickel thing I think just for the second trade required so um supports.

A

So this is the database. So.

A

A

A

A

A

Backhand default ports for each service are okay. Why is that so many Imports.

A

A

All right, great.

A

I think we can safely just uh grow all these environment variables in here.

A

A

With the database container so database compose all right, so let's check out that database compose again foreign.

A

A

Microphile context it looks like it should be good to go when it's created.

A

Let's take a look at the door: okay and they've already copied scheme right in okay.

A

Okay, so um foreign.

A

A

A

A

A

Postgres DB test user on your password, okay, Define the port, and we would open the port the container okay, so I, don't know what that would do, but I guess we'll run it. So if we did our derp stuff, then maybe we can Network a bunch of these jobs together.

A

um That could be interesting, so we could do workflow dispatch to start other instances. So this workflow dispatch would start the primary one. That also has the database and then we could do um like a. We could add. We could do the Derp server and then we could do a matrix, maybe to start more jobs. And then we could. You know the scanning that way. So I think we'll just put this so.

A

Okay, so that runs this in a container okay and then we should have.

A

Okay, so here is the.

A

Token analysis token.

A

We should already have python.

A

Okay, let's see if this should be Source from the repo that's. This should be.

A

A

State-Of-The-Art.

A

I, do it different.

A

We can't look across branches, okay, so we'll just update it statically to the state of the art. For now.

A

For what they contributing just I have the magic command.

A

A

A

A

Because we're not using enough droplets okay, so we just need to use.

A

What are you on about over here.

A

Oh yeah, this is the computer with Git. That's the problem.

A

Back here we did here.

A

A

A

A

Everything before it equals and we want to. Oh that's all there we go there's the latest and greatest, although that's not just okay, so because this get repos, bad or old,.

A

A

A

A

Okay, so now what we'll do is.

A

Okay, so then we're going to take localhost run and we'll expose the okay okay, so we'll do localhost run.

A

A

I guess the question is what is in the auger container? So what happens? If you run this right, so we want Alice to also run register metric right. So we're probably gonna add one of these Matrix to provide releases okay. So this is a query so.

A

This will be UI.

A

Workers with your GitHub worker, repoing co-worker,.

A

Worker get interfaceable.

A

Okay, so I still have API. So let's do anything that says data source.

A

A

Create a collection of those, so let's look for data source just to see what we have here already, it's a good lab.

A

Agro repository data, GitHub API.

A

I grew a repository too, and we want to figure out where we should throw our data.

A

It's not anywhere that says: I, don't think right, because we want to scan the repos themselves.

A

That could be interesting.

A

Or could get integration what's this th? Okay? So that's nice worker base. So this is very similar to the initial uh interface that the neutral metric collection that can go way back in the code base. If you go back to like when the first open source release of dfml and very similar uh concept, only the workers are called the metrics.

A

Yeah, it works works well, but you know we can throw the data flows under it. um It definitely. It provides more conventional structure.

A

But the downside is the Locking right um and.

A

The Locking and the lack of the ability to Overlay based on the data Okay so.

A

Thank you and then obviously this is awesome, so this is what the heck is. This okay worker kit.

A

And we can read the docs.

A

Workers make commands.

A

Workers creating a new worker, okay.

A

So if you're hitting an API on a platform like GitHub or gitlab, follow this pattern and workers Okay so.

A

This is what we want.

A

A

And then we can expose it.

A

A

Okay in the housekeeper block in the worker block, you need something like this.

A

Also, if you want to have those blocks installed within auger itself, when you do the pr you need to add them to the root I'll go root, auger config, py file, the recommended way is set a port range not already in use. It's kind of random variable range, like others, this is new worker.

A

Okay, we can copy housekeeper black verbatim when you add it to your own auger config.json from the worker block, and they can figure why it would look like this or your work report.

A

Okay, so basically they're just grabbing random reports, all right.

A

A

Switch variable toes worker variables.

A

I, like whoever wrote this this I will add this question you I, enjoy you, that's wonderful. You copy paste, bugger bug report copy paste, that's great and fantastic I thoroughly wholeheartedly.

A

I agree with that approach, Okay, so.

A

Pull request worker so.

A

A

Operations table update redirects jobs, delay.

A

To help your amount of contributor breath, not only she is very much like the old metric API foreign.