Dagster Dagster in Media, 31 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: dagster + airbyte + dbt + snowflake - parte 6

Description

Practica de Data engineer usando tecnologias open source

A

In this part we are going to now we are going to get into how to install tbt in our project, right.

A

? Let's use pips to install DVD and then we'll give it a better team. Version. We'd have it installed, DVD, I, don't handle it much because I'm, not a specialist in each of the tools.

A

Can, someone have how to use it, how to use it independently. People that we have to get to this must be Versa and that it be installed, because the plugins are here to start using DVD.

A

The main one is bt-core So. This is being like python, When I say like python. It is, for example, I have.

A

For example, here I have python installed And. If I give it python version, it tells me the version I am using, but in order for me to be able to do something with python, I need python code.

A

Where are the files where this code then, when it goes to running it, makes python and the file that is going to do, for example, the dot country and runs that code in a functional way of btn of bts install global, and then you have to pass it a bt project And that the bt project will see in another video right now we're just going to focus And how to install it.

A

The docker way, because I know what we like So. First, let's see install it says here we can pull the.

B

A

And we can't run it in a container But. We can control our own image Ok. It says they try a doctor Let's see here interesting Here. It is all I hope this, and here they are Everything that is needed to use it.

A

Ok says doctor: it.

A

Means that if we use this doctor an image for each one of these I think it is best to simplify a little more here. The code that we are going to use and that, where it is creating images, again, looks like And. So we are going to copy this code from the one that is not giving and we are going.

B

A

Take our visual I think that here we can create a folder that is called baby.

A

Really we want a doctor.

A

And we paste what we are going to use So here. There are many things that we do not need.

A

For example, we can remove this right here. We have I, have a variable that I was using how we delete. We can remove this from base. I, don't know, These are variables, Apparently.

A

Then all you have to do install the python environment variables.

A

Because it's slow and then we have all this,, which are the additional things that are going to be used.

A

Well, I'm not going to use this, but you can leave it,. We.

B

're going to delete.

A

Everything that was saying as a base is the part that is being used. This argument, you are not going to need This, yes, because here it is installing many interesting things. I am a key in Jeep and all of that we are going. It was very small for us.

B

A

B

B

A

Whatever you are, this is only going to Just install an image, but it still doesn't have it.

A

When we run it,, it won't work for us,, so.

B

Here it is interesting that there is one.

A

That is to see what I deleted. A while ago it was the Core that I was installing from git,.

A

Interesting then another one we are going to use from Go, post and snowflake, so we can perhaps.

A

Follow this dynamic, perhaps that dynamic.

A

For example of bts, but since we delete the variable.

A

That we can do it in another way.

A

Not to put it here to install no cash, and here he wants to install vtv. But how do we delete above the variables we are going to use it in the traditional way and what I do is always look for it in this way in python to see where.

A

A

This I could even give it the version I want and with it, Well I'm going to do it like this, but normally we could read a file box right, but for now we are going to do like this So. What else like as our stakes, we could also install diabetes, dessert.

A

Then you must touch Take cor. It.

B

Must be installed.

A

Well here are all the plugins that we could use So. We are going to do the same. Let's go.

B

A

Copy everything.

A

And I'm going to give it Go with I. Do.

A

N't know what version of the Core debate it should have.

B

B

B

A

We could install.

A

I, don't know what they.

B

A

B

A

B

Copy all this Let's.

A

See it doesn't copy it well, I.

A

Don't know what Right now version.

B

At point, 21 always always the same, one that you have here.

B

A

I think that, with that we can do it I don't see any other problem here. You are doing a Walking directory dbt and it is going to give you a volume and DVD and more or more That's it everything I, don't see.

B

A

B

A

Pum pum So. Now we would only have to add it here as an extra service. The truth, and here it would be. For example, we can give docker example.

A

The points and here it would be biote.

A

And the video So, the context of the Beatles goes to be our folder of our folder of and also the doctor that is going to be used. That would be such.

A

That would be in this folder. It was a file called.

A

Let's name the container.

A

I think they all have the name of the container. Not all of them have the name of the container. What they do not have the name of the image with some clients If. It does.

A

B

A

Does not have it the name of the image So we are going to put.

B

A

Image name so that it is not creating images for us.

A

What, if not put that name it will be creating images for you and it will name you all strangers, It's good. The important part is the network, the network to which we are going to join it. We take it so much the others that it would be. This.

A

B

A

Nothing else That is our doctor. I am going to add the image to those that do not have it.

B

A

To have a little more organized the code that has your image that has your image.

A

Ok Let's run this to see if our container is created for us.

A

That It is being created interesting ugly. There.

A

Is no such directory? It says because here, where I put lowercase and uppercase the file, I put it in uppercase docker.

A

And our image has already been created. Now we are going to raise the container.

B

A

We should see.

A

If they open sudo what text they came from and container raised doctor good first, let's test if it works local desserts, Look there, it is working with our example file that we did last time, and we start our examination, our bts container. Here it is, but it is turned off it says ex anyway, we enter it Ok in this Command what but is turned off So. Let's give a little explanation of why this happens, because.

A

Docker images are only kept active if there is a process running in this case. This docker file is not running any process in the container. That's why it exits immediately and the only thing it did was expose an entry Point. An entry Point is the default command that is run when the container starts,.

A

So in this case, when you put when the content runs, it is running from bt and it is getting this message that the What would normally come out when I look for that Command without any without any option. For example, if we run DVD version, it should come out not this.

A

The Command So. We are going to test it as this is the default that is running So. We are going to use cmd. We are going to send it to see how it works. This is The default command when an image is executed and what it does take as parameters. What is put in the command So in order not to be raising and raising this, we are only going to work with.

A

The doctor example service, because I can raise this service independently Only for the tests that we are doing. I can't give it to you here to see doctor with post, I, 'm sure, The, Command I think this is what compost and the name of the service that I want to raise.

A

So if I short short.

A

They will see that now, I am getting the cut version that was installed plus the two plugins, which is the same as what you would have Well here. They are turned off because, as you remember, Here it is This was what was run So. This is not saying that we already have dt installed on that image, but the image is It's stopping because it's a command that I know runs and dies,. It's just running this one and then a trick not a trick. Then how can we need it not to die?

A

We need the server there, so we can use it so that daxter can access it. But before getting to that part, we are going to do a little test. For example, there is a trick that can be done to teach you how it works is, for example, I can delete this command and the entry points is what will be executed: By default, when I run,, it runs the container,. So in this way I can run the container with a command, passing it an adbt, command., For example,, let's see,, this would be doctor and we.

B

A

Give the image a name,, we are going to see test and we are going to do it.

A

In using this image, and what comes after is the command that I am going to run in this case, I can send you version So. What is it going to do is going to happen, it will execute Command, 2, The, Tucker container will run it and it will to pass this as a parameter and it should receive it should see. The same thing here is another DVD. Command I can use it's the test and it should see I.

A

Can't use images that are already in containers that have already been used. So, let's delete it.

A

A

Correct and if you look at the error that I am receiving here, it is a DVD error that says that I am missing the DVD project. Job I am missing the profaction that it is a project of So. I can use this image. I am already this image. Practically It is ready to be used as a command line.

A

What else can we do if you want to test How? Would this server look without dying So? We can put it to see. We are going to delete the image. First, we advance.

A

And there is a trick called Cheo that this will make a process stay there without dying indefinitely.

A

So I think what is happening is that this is a parameter of this Command, but in this case DVD does not know what to do with these parameters. So, let's discuss.

B

A

In this case, we would have to serve him out again.

B

A

We are going to.

A

Run The Command again that I do not want to write it and there are.

A

Already many many people- even this- is still.

A

Here so, as you can see, the server stayed here hanging waiting because that process is pending and is not ending So. It goes. Let's see we can see here again.

A

She is saying that this container is active running this Command until I finish this Command, the container is going to finish So. We are going to finish it as it ends. I have to finish it from here for it to die.

B

We have already closed and We delete. It.

A

So now we are going to return this from here and we are going to delete this then.

A

Here we are going to go back to our image, to use it better and because we already have our image now we are going to explore a little is what is inside that image? There is a command called docker.

A

The container that I am creating is called Saúl test So, that is the one that I am going to access interactively.

A

What example is when I use the doctor with post put Saúl test, and here I can enter the image So this image, where I have.

A

Tvt and desserts installed, so I run DVD and a version script should see the version I have installed and the other test bt Test the series of the same error that it gave me the other time. I have the DVD project and no I have all that, I mean practically, who I'm ready for and if we explore a bit. The doctor says that the Walking deadpool was going to be a user up. Dvd So, let's see, what's there practically there, I am I'm in that folder they look, and there is nothing.

A

If the motorcycle cc, we go in the DVD folder. There is practically nothing because there is nothing, because that is where the bt files have to go that are going to be executed. So in the next video. We will see how to pass that file here and nothing and with that it ends in the video, because here it is practically already installed from DVD. There is nothing else to do here.

B

So, nothing here just.

A

They already know that when they run The docker compost, Command is going to rise and fall Why Because there is no command. That is there waiting. There is no process still alive in the container, because that's why it crashes But, that's not because it's bad, but that's how we have it configured so far.

A

Let's run it all again, like last test.

A

And in the next video we will see how to mount those files.