Argo Argo Workflows Training, 22 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Argo Workflows 101 Workshop 22 Sep 2020

Description

06:50 fundamentals
23:30 Anatomy of a Workflow
40:00 Artifacts
1:00:00 Exit Handlers
1:05:00 Workflow Templates
1:19:00 Cluster Workflow Template
1:20:30 Cron Workflows

https://docs.google.com/presentation/d/1ftwxelCtW-onnuxoMMAL-ZAqLOICh9YuAGzgo-dghZQ/edit?usp=sharing

A

Okay, hello, everybody. Typically, a lot of people join uh two to three minutes late, um so we'll wait a couple of minutes in the meantime.

A

Can you sign in here please and and mute your microphone.

A

Maybe I will set this to also mute everybody when they join.

B

A

Okay, I'm just going to just change my video share here. If I want to share the whole uh desktop, not just.

C

We guess the whole desktop.

A

Okay, so we will be recording this and that will be available on youtube later. If you want to share that with your teammates, it would be fantastic if you could uh sign in to the signing sheets like we do in the community meetings. Just it's really good to know who will be in this um uh session. Okay- and it's really good to understand what you're using argo for as well. If you want to add any commentary about that, there are a couple of prerequisites to this workshop effectively.

A

You need to have an argo installation to run so there's some instructions on slides, two and three about how to do this. um We for development purposes use the mixture of docker for den3s. You could probably also use kind or doc of a desktop kubernetes. If you want to do it.

A

The reason we use k3d is really just because that's what we use on our ci system um and uh it supports our back correctly, which docker desktop does not uh do, and you'll also need cube, ctl installed um to do the workshop, and then a couple of people have commented recently. The k3d have recently changed the parameters, for um I think it's version three, so we are still using version.

A

One and version three has slightly different and command line parameters you need to watch out for- and there is here here- are more complete instructions to get that running. This will install the quick start um manifests along with mineo to simulate an artifact repository like s3 and postgres, so we can play around a bit with the workflow archive, because k3 deed is, um I think it uses container d as its run time, rather than docker you'll need to set your container runtime executes the pns, and so this is the patch here.

A

This may actually apply for many other systems and may also just work generally for anybody to do that. um This will typically take about two minutes to start up, because it's pulling those and stuff those postgres, pods and minneapolis takes a little bit of additional time.

A

Okay, it's a little bit about who we are so my name's alex. I am one of the three engineers he'll be presenting today and we'll also be have a present some presentations from bala and simon, who are also engineers on the argo workflows team and most of us have been working on argo workflows between, I think about eight months bala. How long have you been in our go? Workflows, 18 months?

A

Yes, yes, 18 months, I was.

C

B

A

That as well, we also work on other argo projects such as argo, cd and um argo events. um We won't be talking about those today there there has been and I'll go events uh kind of one-on-one workshop. Maybe we'll talk about doing that as well.

A

I will go over some of the fundamentals as well during this workshop, trying to give just reiterate some of the important groundings that people often try and jump in at a little bit too high level, and I think it's usually really good to just go over some of those fundamentals again and talk about some of the terminology.

A

Understanding the terminology is very helpful to kind of understanding things in general. What to do? If you have any questions, um simply ask your questions in in the zoom chat. um I've got the window up on my desktop. I will monitor that and if somebody is busy presenting, then somebody who is not presenting will then uh be able to. um uh You know: read those out.

A

Okay, any any questions before we start.

A

Okay, okay, cool okay, so let's just talk a little bit about um some of the fundamentals here of workflows, um I'm just going to really cover just the most basic of running workflows and a little bit of uh a little bit of terminal in that is. um I have our first very basic hands on exercise and I'd like everybody to give the hands-on exercises a go. If you can do if, um when you've completed them, go into uh the, I think you go to the chat and give a thumbs up.

A

I think why can't I hide, I don't see. I don't think I have the option because I'm presenting to give any thumbs up so give us a thumbs up if you have uh able to use complete exercise that allows us to kind of control the pace of all the different workshops and the way that we're going to do the exercises are going to be in a kind of a call and repeat so we'll show you how to do that and then you'll just have an opportunity to to complete the exercise yourself.

A

Okay, uh the first exercise is to submit a workflow uh that um present uh the print argo work. Hi argo workshop to the user interface and the way that I'm gonna do. That is I'm going to go into the user interface, which I've got. uh I've not got listening because I've turned off the port. Forwarding make sure you have your port forwarding running load the reo, the user interface go into, submit new workflow and there's a tab and play it. Work flow out, console phrase: hi argo workshop.

A

I'm just going to copy that from here, and this is standard template. So I'm just going to remove most of.

B

C

I'm just going to call this.

A

Hello um now all I just got the most basic um layout, I'm going to use docker while say latest and then I'm going to run commands going to bash, actually not sure if this will work so we'll find out. I see and the args are going to be a little scripts.

C

Because we want that car around.

A

So this is the a very basic um workflow I like to call my entry point main you can call them whatever you want to call them, and I like to call my containers things like pods, I'm going to use a docker way I'll, say image, which is a very common image which is based on ubuntu and contains some kind of useful things like bash and echo. I'm going to save that and I'm going to submit it and then it'll. Take me to the graph uh the the graph view, and I can see what's happening there.

A

Taking a few seconds to create the cane container, because it's probably pulling that image down from the repository, sometimes it takes long a longer time to do this.

A

So if you want to start the exercise, obviously go ahead, you can follow up on this and I'll just show you another way of doing this and particular um exercise from the command line.

A

A

Open up a terminal and then I um need to create myself a workflow.

A

I'm going to cheat a bit because I'm going to copy this particular workflow.

C

Oh, it's got all sorts of stuff going on here. Maybe I won't want to do that.

C

B

A

I need to make that a script anti.

A

Now, what I've done here is I've used name um from this, but actually it's more difficult to use generate.

C

B

C

A

Go submit uh what's up.

B

A

There we go. This is another way to submit your workflow from the command line. You can see the yammer one while that's running, I'm just going to go check back on this workflow, so he was successful.

A

The green ticket still been successful. I can check in the logs here to see what's been printed out and you can see that it prints out high argo workshop. This is a bug that we need to fix.

A

If we go back to the console, let's see how that was that one's going. I can actually see the console one and the user interface or I can't so. I've made a mistake here.

A

I need to submit it into the argo namespace there we go now. It's working there's now a rogue workflow in the default namespace.

A

There we go and he submitted there. You can see that I used generate name and so he's got his generated name. Okay. So it's over to you now to do this bit of the workshop.

A

I think we've got a few examples of running it on slide. Six.

C

A

Does it change on slide.

A

A

And when you've completed this exercise, just give me a hat just give a um hands up or a thumbs up in the in the chat.

C

I don't know how we do.

C

A

And I said, that's a quite acceptable thumb thumbs up the word thumbs up is great. It's fine. The word done is fine. Let's wait for about maybe say 10 people to have done that and we'll move on.

A

Have you have any issues? Do I ask in the trash queries or questions.

A

If you can even tell me how I can do a proper thumbs up, because I can't seem to figure out.

A

A

Okay, I think we've got enough thumbs up so far, um so here's some additional instructions on slide, six which likes to change sliders. Every time I touch my mouse, um you can do this from the command line using um I'll go, submit. There's also a couple other useful commands here. Argo lists and lists all the workflows in your namespace.

A

Okay, uh that gives me a list of them as well I'll go, get or get you the workflow, and actually we can be kind of clever. With this command I'll show you a little bonus technique. um I'll go hello, you can actually get other additional information, so you can go for a wide output or you can go for a you know. A detailed yaml output like I get. You can actually also provide latest as a an alias.

A

And that will just get the most recent one there. It's kind of that's a convenience command. You have and we can see what the logs were for that one. As.

B

A

Hope this works. Sometimes we do these things and we we discover that we have some bugs that that typically happens here as well. um I've got a I've got a challenge here. I'm jumping in minus nine go if you've got a tool called kube ctx installed. You can change your namespace by default, I'll change the namespace by default- and I have that I actually have that on my terminal here. So I can see what I'm doing.

A

Okay, so I'm going to give you another exercise here to do as well. I want you to submit a workflow and uh wait for it to finish now. I'm not going to give you any instructions about this, but I am going to give you a hint if you want to do that.

A

I want you to do that using the cli tool and give us a thumbs up when you've done that or if you've already done that and in the meantime, I'm going to take a moment to answer a couple of questions that have popped up in the chat here.

A

um So augustine has said: it's not working um for kind. Okay, so the area you've got there is around docker sockets. You might want to change your installation to use the pns execute. So that's on slide three. If you want to reconfigure that one no problems and address sorry, if I'm jumping in how about go client for submitting workflows, is it widely used? So this is actually.

A

This is meant to be a 101 workshop, but, and this is definitely a 201 or a 301- um you know intermediate advanced topic, so we do have a couple of different ways to interact um with argo workflows. You can use the cli and you can use the user interface and you can obviously use coop ctl.

A

There is also a reps api you can interact with and there are a couple of clients to that rest, api and one in java and one in python that both have actually been very recently refreshed and not updated in good condition, and also you can interact using go and that's obviously I don't want to go to details to show you how to do that and go because that's a very go specific, but that does exist um search for.

A

I think I'll. Go client.

A

Chen there we go there, you go. This will just give you a list of the client libraries I'll drop this in the chat for.

A

A

Okay, so I've got a couple of thumbs up.

A

Who would like a bit more time or would like me to go a little bit slower.

A

Who would like me to go.

A

Faster and who would like me to continue at the same pace.

A

One faster two fasters okay, so we'll go we'll go a little, we'll go a little bit faster. In that case um uh I I'd say: the number of people have voted for slower, plus, faster plus. The same did not add up to 100 okay onwards.

A

um So I want to touch on what is in kubernetes terms of kind of a more an intermediary advanced topic, but for um um most users of argo workflows is a very important topic um so workflows. um I tend to execute with a service account chosen by the person submitting the workflow, so we think of the argo controllers installed using its own service account and it gets the permissions from that service account and the workflows itself are submitted using a different service account.

A

If you you're doing third party uh integrations, it gets even more complicated because you'll probably have a service capac for every every single user. Okay, now by default, workflows, are submitted using the default service accounts. I actually think you can probably see that listed here in the console or service account this users, so you'll get the permissions from that service account. Now, with these quick start ones, the service account is given a role called the workflow role.

A

We can get that role by doing coop, ctl get roll, and actually, let me just have the full yaml of that, and the role has a series of rules which say what you can and can't do. These are very important.

A

So there are things like the ability to get and watch a pod and also a patch it and the ability to look at the logs and the ability to and the ability to in this case create and get workflows, which is actually not what would be necessary for any of this workshop, that's useful if you want a workflow to be able to create other workflows, which is an interesting pattern, but often you don't want to submit it using the desal service account. You probably want to create your own service account, that's relatively straightforward.

A

You can create a service account using coop ctl, um create service account me, and then you want to bind that service account. So you typically always have three things. When you're dealing with service account, you have the role, which is the permissions that you're allowed the rules that you have.

A

You have the service account, which is you know. Well, it's an account for in a service, but like a user, and then you have a role binding, which associates a role to a service account. Basically, uh you know creates that linkage and the way you can do that is doing coop ctl correct role, binding.

A

um This is the name of this, so the countdown I'll go namespace and then it's called me. So the rock the role bindings cover me and the role I'm binding is called workflow role, but you might have multiple different roles, but you probably won't do to be honest, you're, probably jeff, one called workflow.

A

Okay and now I can submit my workflow using that and the way that I would do that is I'll. Do argo submits service accounts me and then I would just watch it and you'll see this has been submitted with the service accounts now, depending on what um so, we definitely recommend you have a separate service account for running workflows um to uh um you know, running the system itself. Okay, so the hands-on exercise there for you to do is simply create a service account and submit a workflow using that service account.

A

Okay, I'll give you a couple of minutes to complete that exercise.

A

And less time than last time, we're going to up the pace.

A

I'm going to give a thumbs up because I've done this one.

A

But people the people like having me going backwards and forwards between the slides or do they want to see the slides and the terminal at the same.

A

Time, okay people got through that quickly, so I can see that the pace the pace could be even faster, still, okay, so that was just the very basics. um Just a quick review. We talked about submitting a workflow from the user interface and submitting a workflow from the command line. You can also submit a workflow using cubectl as an advanced one I'll. Let you go and discover how to do that yourself and also via an api and sdk.

A

um So that's just the basics of it, but what's super important is actually the anatomy of the workflow understanding, all the different terms and terminology, and for this bit I'm going to hand over to the session over to simon simon. Are you ready.

D

Yes, I am thank you so much alex. uh Can you guys hear me.

D

Alright sounds good all right, so uh hi everyone, my name is simon. I'm also an engineer in uh on the argo project. I've been working on argo workflows for around almost a year now uh and previously to that. I've also worked a bit with argo cd, uh so some of you might recognize me from github and from slack so hi how you guys doing so today, I'm gonna talk a little bit about the anatomy of a workflow.

D

um This is not going to going to be a very a very specific, uh like run through through all the fields of the workflow. uh That's uh that's something that not only do we not have time for, but that's something that should be done through the documentation that we've provided, but instead I'm going to focus on just abstract ideas of what a workflow is and a useful sort of a useful example that you can use that. I feel like if you think, of a workflow uh in a specific way it will.

D

It will help you with designing uh better workflows that you can use. So uh with that, let's get started. um If you guys have any questions, uh please feel free to uh follow me on the chat and I'll ask alex or bella to read them out for me, since I don't have the window right now, uh so you guys can see my screen right. The uh the slideshow screen.

D

All right, so uh the uh the workflow has many uh different sub units um we'll be in different concepts. um Some of them are on the screen. We'll talk about these independently uh and the most important uh part of our workflow uh is a template. uh So templates are essentially how we defined uh the work to be done and how we call templates to actually do the work. So uh templates are essentially very similar to functions or methods in your standard programming, language uh and they're always defined under the templates field of a workflow.

D

So there is, there is a thing called workflow templates uh work, capital, w workflow capital, t templates. That is something different uh and bala will be talking about this later on in the workshop. uh But when I just keep in mind that when I say templates during this section, it refers to lowercase templates that you define under the templates field of workload.

D

There are two: uh there are many kinds of templates that you can use as we will cover, but there's two there's two main ones that you should keep in mind. There are templates that define work. That needs to be done, and then there are that's one type and then the other type is templates. That call other templates to actually do the work and perform some sort of execution control.

D

uh We will so if this doesn't make any sense to you right now, don't worry. We will run through a very specific example.

D

So when I first started on the team- and I went when I was first getting familiar with workflows, an analogy that I came up with that was very useful for me was to compare a workflow to a java class. I'm sure that most of you are familiar with java or some similar class-based object-oriented programming language.

D

So um I will essentially you be using java, as a case study, to introduce you to the structure of a workflow. So here on the left, we have a very simple java class with we we have. This class has three functions. uh Two of these functions are used to define work, which is what I mentioned before, and one of these functions is used to actually call on mark. So, let's step through it just to make sure that we're on the same page, we have these two methods that define work to be done.

D

One of them is pretty simple: okay, that's forward to an input, the other one, just prints out a a string. These methods have names that we choose. These methods also specify their inputs and outputs. So in this case uh the function at four will take in an integer uh which we don't actually know the value of yet, uh and then it will return an integer and say hello will won't, take anything and won't return. Anything uh keep it.

D

This is important to distinguish that these at this point, as all of you know, by now these are abstract uh arguments. These are not actually live yet then we also have uh a block in which we call uh work or which we actually perform work. In this case, we uh we pass in the integer two to our add4 function. Save the result perform some execution control on the result, uh and if, if it succeed, we call another function, we use the names that we've defined and we pass in live arguments and we perform execution control.

D

So here on, the right is the equivalent argo workflow. um I feel like, if you guys were, if you guys, if this is the very first workflow that you see, and you don't have the uh the equivalent java code to the left, uh at least to me, it was very overwhelming and you don't really know what's going on uh or how it will like how it will execute or what's the what are the um like, how do you pass in parameters or whatever? So I feel like this?

D

Is uh it's it's useful to just sort of compare this to code that you already know so we'll just run through the same steps again and this time, highlighting both the argo, workflow and the job class. So here uh here at the bottom, we define our methods that we defined in java.

D

We we add, we do we name them as as we please. We define the abstract parameters that we will use here at the top. We define the actual work that needs to be done.

D

We call these functions using the names that we chose and we use uh we actually in this case we pass in live arguments, uh no longer abstract, and then we also do some execution control.

D

So I will let you guys just stare at this uh for a minute or so just to get familiar with it before I continue. If you guys have any questions, please feel free to ask them in the chat and if alex could read them out. For me, I'd appreciate.

A

That we have a couple of bits of feedback- silent, yes, uh one is they like the code flow script, analogy awesome um and the other one is. How does argo know? The output is of type number.

D

That's that's a great question, uh so argo is actually typeless. uh So if you are a javascript or a python, programmer you'll you'll take this with better you'll. Take these news better than people who come from go or java like I do so when it comes to parameters. Parameters are always strings, so obviously the type of those strings.

D

If you want to treat them as integers or if you want to treat them as strings, it's entirely up to you and how your containers that run manage them, uh and then we also have another type of argument. Called artifacts and artifacts are essentially files that we handle for you. So essentially arguments are always strings and then you can manipulate those strings. However, you want.

D

All right, if there are no other questions, I will continue on. So, as we discussed earlier, we have two. uh We have two main types of templates. One of them are definition templates on the left. These templates define work to be done now. I've I've said uh that we have set the phrase defined work to be done, uh but what it really means is essentially it just executes the container, uh as all of you have used kubernetes before you know, you can just define whatever work you want to be done on the container.

D

An article will execute this container for you, given the parameters that you have defined. So we also have. We also have a script template, which is essentially a convenience wrapper on the container, so that you can define your own scripts in line with argo, as opposed to having to define them within your image.

D

We have a resource template. The research template is useful for actually manipulating kubernetes resources. So, for example, you may have a research template that will get uh that will get a config map and read it or that will create or delete a service. If you want to create a service for whatever and suspend, is a template that will actually pause execution of a workflow and will resume based on a condition that you have defined or will resume based on the timeout or will resume or will just wait until a manual approval.

D

So these are useful for steps and manual approvals or delays stuff, like that, uh we also have execution templates. uh So these templates, its only java, is to call other templates. um They may call execution other execution templates, but most likely they'll be calling definition. Templates steps um is a template that essentially defines a sequence of steps to be on.

D

So if we go back to our example, here see that we're using a step template uh what the double dash means is that we will perform uh argo will do our will perform this step, and once this step is done, our world will perform the next step. So it's basically just normal execution.

D

If so, this is the type of this is a list of lists. uh If, if you, if you were to have a list of many items, so imagine that this uh dash here is not here, uh then everything within that higher order list will be executed uh in parallel. So if you wanted to have these two steps run in parallel, you just get rid of this little uh dash here and then these two will run in parallel.

D

But in this case it won't work, because this template depends on the output of that template.

D

We also have a dag template, which is which allows you way more fine-tuned control uh if you want to have, if you wanted to find a dag for you to work with, you will probably be using a dac and we find that most of our customers uh for actual for non-trivial cases end up using dags we've covered inputs and outputs parameters. um So when I first started working with argo, I saw that there were inputs.

D

There were outputs and there were arguments, and I was wondering um what the difference between, for example, inputs and arguments where inputs in argo are always definitions as well as outputs. So these are always the placeholders that you will have and arguments are actually the live arguments that you pass in. So if you go back to our example here you see that in our abstract code block here we have inputs.

D

These will just be a name of a parameter to pass in and in our our execution code of block. We have arguments and arguments will actually need a value to be passed in so, uh and so because of this uh inputs will always inputs will only be used uh in container or work. Definition templates and arguments will be used when calling those templates.

D

Here awesome, so now uh I have a small exercise for you guys I've. Actually, uh I have taken the this workflow that we've been talking about, and I put it on uh on this link right here- uh ignore this first link. This is an intro internal that we use when we give our internal workshop.

D

uh So if you want to open this I'll, be I'll be also basing this on zoom. If you want to open this, you'll get the workflow that we've been working with, and your challenge and your job for the next couple of minutes. Let's say five minutes will be to edit that workflow, so that it uh corresponds to this edited uh java class that we have here. So all of the changes that I've made are highlighted with this gray highlight um so yeah.

D

Good luck, I'm going to open this and I'm going to link it on zoom, so you guys will have an easier time getting to.

D

It and meanwhile I can be answering some of the questions that have uh popped up all right.

D

uh So a question is: how does argo pass parameters between the pods uh mounted a special files, so argo actually passes the parameters using annotations. uh So when you use parameters, uh argo will use annotations from the pods to communicate between the uh the pods and the controller which will pass into the next parts because of that annotations and kubernetes actually have a limit. I think the limit is around.

D

I'm not actually sure, but don't quote me on this, but it might be around 200 kilobytes or something so we have had users who exceed that limit and when you see that limit you will be, you will need to use artifacts to pass in parameters.

D

We are currently exploring different ways to manage parameters uh so that you don't have to use artifacts if you don't want to, but currently there is, that limit uh is the core difference between script and container template that argo converts the script to file internally and then calls command of that file. Yes, that is exactly the only difference it's like, like I mentioned before. It's a convenience around a container uh that will essentially save your script into your container and then it will just run it.

D

uh Those argo consider apache airflow, a competitor. I feel, like this question might be better suited for alex for our pm mukulika. But I'll give my quick answer. I I think we consider ourselves to be a specific use case of airflow under my opinion. So if you are working with kubernetes, then yes, we might be competing with airflow, but if you're not using kubernetes and obviously our good uh won't help. Much but again, alex or mukulika might have more to say about this.

D

B

D

A

um So, to answer your uh austin to ask your question, um the answer is no, no, we don't really consider it a competitor, because we would look. You know there would be a competitor if we were solving the very similar use case, um but we, you know, we're a very general engine. um We do find that people migrate off airflow onto um our go workflows um for a number of different reasons, such as um simplicity of setup and so forth.

A

But to answer your question, I I don't think we consider it a competitor.

D

And thank you alex and then one last question any owner references by argo when we create a volume through resource and templates. So don't quote me on this, uh but I actually don't think that argo creates uh volumes for you. I think you create your volumes and then you point argo to them. So, uh oh.

E

Yes, yes, now, whenever the workforce creating a volume it will set the owner, I wanna reference on that wanna reference on that one. If the flag is enabled, I think so.

A

Awesome, I I actually think there's a couple of ways to interpret this question, because you could create a volume using a resource template rather than a container or script or dag or step template. And when you, when you use a resource template, you give the you give it um just some yaml to create.

A

But there is an option called, I think, set ownership reference on a resource template to that sets the ownership reference, which is, which is quite useful if you want, if that's, how you're using it to create volumes, which I guess is a way to to do that, it's not the only way to create a volume.

D

uh Can I ask how people are doing with the exercise? uh Can you guys give me a thumbs up if you guys are done uh or if you guys have, if you guys need more.

D

C

B

D

I see that uh I see that uh someone finished the exercise, so in the interest of time I think I'm gonna hand it off to bala so that we can continue the workshop uh but feel free to continue the exercise in your own time and to always reach out to me on slack at simon and with any feedback or any questions.

D

uh Well, are you ready.

E

Yeah thanks simon, say: hi, uh I'm bala, I'm also one of the engineers in argo team working with alex simon and derek.

E

You can find me in the slack and you have sarah bella 1979, it's a handle. uh Let me share my screen.

E

Are you able to see my screen.

C

Yes, we see input, we say input artifacts,.

E

E

This is big enough or I need to turn into presentation mode, because I like to switch between this is good. This is good yeah. Okay, thanks uh can I start or I I will wait for some to finish. The previous.

E

Handsome, okay, I will start okay, so simon was explaining, like you know, the different types of templates and input parameter and the output parameter. Those are like it will look for the simple data to transfer between one step to another step, but some of the advanced use cases you want some steps. Steps are generating the big file.

E

You want to pass it to the below steps or the one of the steps want to read the file from the some cloud storage, those kind of use cases you can use an artifact artifact, also similar to the parameter, but mainly it need to be auto artifactory.

E

You need to configure that repository, which is at the cloud storage like s3 or gcs, or oss htfs, all the things so optic nargo. The artifact is considered as a single file or directory.

E

It's a basically it's a key when you define it. If it is a single file or go, will download the single file to get it ready for the main container or if, if it is a directory, it will download the entire directory and mount it for the main container.

E

So the order will support the below repositories like a s3, gcs, oss, raw and raw mean. Second, you can just generate a file in the workflow itself and htfs and github and http request.

E

So in argo, you can define your repository in the three way. The one is like in controller level in workflow, dot, workflow controller config map level. When you, when you configuring the controller level, the all the workflows run on the particular controller, this artifactory will be substituted if the workflow doesn't have inline artifact repository or artifact ref.

E

The second way is workflow level you can have in the workflow level. You can configure artifactor poster in the two way. One is like a config map, which is a artifact repository ref, another one. You can just put into that input. Artifact and output, artifact inline.

E

In right side, you can see that this is the sample of artifact repository configuration that you can. You can give the this will differ for each each repository, like gcs, will be little bit. Different oss will be little bit different, I'm giving a s3 as example, so you need to give the bucket name. Endpoint is like s3.aws.com and whether it is https or http.

E

Then you need to give a credentials. The credentials will be a secret in your namespace.

E

ah Let me go to the next one, so here you know how you can define an input. Artifact repository. As I said, uh this is also similar to the parameter in the template level. You can, you can declare like instead of input parameter, you can define as an input artifact under the artifact name and which part you want to download that file for your main container. You can give the path and you can you- can give the artifact repository configuration like s3 with the end point and which bucket you want to create.

E

You want to access it and what is the object key and your credentials, then, whenever you are submitting this workflow argo will automatically download that artifact in that container.

E

So in that pod and get ready for the ready for the file for the main container.

E

This is the another way you can configure config artifactor poster in the workflow level, so you can con. You can have a config map which has like a multiple repositories that you can refer into your your workflow like a artifact repository ref, then key. So the order argo will automatically refer that repository from the config map.

E

Any doubt, or am I going a little bit.

E

Fast, okay, ah let me show the hands on then you guys can also work. Try with me, so I have a one example which is here. I can put this link also in chat.

E

So mainly, this workflow has a one template which has output artifact.

E

That means this step is generating a file, and I want to upload it to my cloud repository as a key as a hello world.

E

So if you see that I configured my bucket name so the menu is the s3 simulator in on kubernetes. Just it will. It is a s3 simulator like this, so I configure that one and secure- and this is my object key- and these are my credentials if you, if you see here.

E

My menu was running on my local.

E

E

And I have a get.

E

Secret, so you can see that ignore them. My menu credentials are stored into my menu credential secret. So let me submit some of the workflow.

E

So what this workflow will do that it might my step. Will my template will create a file hello world and it will upload into my.

E

E

So it's done if you go here.

E

You can see it here, it is uploaded to the new file called hello world.txt. The date is today.

E

I'll give a few seconds you can also try it.

E

I don't know whether our prereq has uh installing a minion.

B

E

That yeah, I think so because the postgres will have a.

D

E

Anybody able to execute that samsung.

B

E

Are any questions? Oh okay, I got it thanks. So so far we saw that output artifact now second use case is input optifine.

E

I want to download some of the file for my container to process it. This is mainly like a big data use cases. Hey. I have a big file in my s3. It need to be downloaded and process processed by my.

E

Container so I will put the link here.

E

Mainly if we see that I have a step which has the input artifact, I define the path where my artifact need to be in the power and where I need to get it from, and I I'm just accessing that artifact in my.

E

E

So mainly what argo will do that whenever the input artifact is there, it will add a unique container.

E

Argo argo executor which will download the artifact.

E

For the main container, so, for example, this is done. If you.

E

E

So which which basically downloaded the hollowed text, which we uploaded on the previous step, and it just just, did a cat on that.

B

A

Well, while you're there barla I've just noticed, did you ask this answer this question from william tam? He asked: does a workflow run in a pod and each template or step run in a container within a pod?

A

It's about it's about half correct, I think. Isn't it.

E

uh No, each each template has a pod each each template has a port.

E

Or workflow or flow workflow will be run on the workflow controller, then, from the controller each template will be span as a port and each con. Each template will be executed in the form inside the pod.

E

Can I answered.

E

If you see here, I can give an example: get workflow, so just workflow is a definition.

E

Resource will processed by the workflow controller part.

B

E

But if you see that every template is created as a pod, if you in your workflow temp workflow, you are defined each step. Template will be created a part inside your definition, like the template definition, whether it is a container or whether it is a script, will be executed in inside the pod.

E

E

Yeah thanks: uh let's go to the next one, it's a passing out artifacts! So so far we saw that one is like uploading, the artifact workflow and another one is downloading the artifact input artifact workflow. Now I want this is another use case.

E

The first step is generating artifact and pass it to the second step. If you see that this workflow it has a two steps which is running on sequence.

E

As a as you see that there is a double dash, the first step, which has a we'll say, container, will say container, which is producing a hello world text which I'm uploading.

E

You can see the one difference I didn't give the artifact configuration, which is speaking from the config, the workflow configuration centralized configuration.

E

Then the second step and just passing, is an argument which is a live as simon says that which is a live value for this step. I am passing as our argument. Artifact name is message and from the previous step that you can refer like a states, the step name and output, artifact and artifact name here is a hello art, that's the one I given, which will automatically pass through the second step as input artifact.

E

You no need to define an artifact repository here because which will be taken from this configuration, and here I am just printing that message.

B

E

E

So first step is executing which created a port and created a file and upload it. The second step downloaded the file from the menu and printed it. If you see that auto.

B

E

You can see that the green one, which is the first step which was created a file and push it to the menu, the second one, it's downloaded and print it again.

E

If you see in the menu you can see the.

E

Files here, where I am.

B

E

I think I'm using the same file- that's try. It is overwriting here, but we can. We can change it to the different files, because the same key was output output. That's why it's loading to the same place.

E

Any luck! Anybody.

E

E

So the questions are like I can artifact, but be a folder or does always need to be. No, you can be a folder. Also, then internally, the google.

E

If you want to zip it, it will zip and upload it. There is a flag called zip and unzip. You can enable and disable it. Otherwise, it will upload the entire directory and download the entire directory enter.

E

E

Yes, so the torrent on tour is inbuilt. If you want, you can enable.

E

E

Any thumbs up for the artifact for.

E

E

Okay, I can see few thumbs ups. uh The next concept is exit handler. Exit handler is very similar to finalizer. In java the exit handle you can define in two ways. One is like workflow level whenever the workflow is done, you want to you want to execute the the particular exit handler to clean up or notify or anything there's another. Another level is you can different exit handler in the step, step, level and dag level whenever the particular step is finished, you want to do some cleanup or anything.

E

You can define a step level exit handler or diag level exit handler under the exit handler is it's the same as a template. You can define a template and just mention the template name on on exit element.

E

If you see that this workflow has a two template, one is a vlc and another one is say exit so well c is my entry point which is normally executing in the workflow, but I can't fit exit on exit. I configured an exit template so when, when this was is done, that exit handler will automatically execute it and finished off the workflow I'll paste this link here, just you can submit it or go.

E

Submit so you can see that the first step is executing once it's done, the exit exit template will be automatically executed. For you see here.

E

E

Done so mainly that exit template, mainly you can use for notify anybody, am I on your workflow is done or your step is done or your dag is done or clean up like uh cleaning up all the files, all the things you can use it see here. This is a main step and this is the exit hand, exit.

E

Template uh can out output artifact, so in the meanwhile you are trying. I will read the sum of the questions: can out output artifact be saved on them with the k8 executor.

E

I'm not sure I didn't try that.

E

But sanjay I will answer it in the slack one side I will take a look. I haven't tried that k8 executor.

E

The next one can I make a warning warning: only template.

B

If it is fail, it shows the rest stop still.

B

C

You mean that, irrespective of the status of the specific.

E

Yes, okay, uh it goes ahead, so here you can, you can go, you can define it you you can different the sequence of steps and the sequence of the dag, and you can say that you can continue on which will just skip. If the step is fail it will. It will continue on the rest of the steps.

E

In the step template you can have like a continuum for each step. You can you can you can you can enable the continuum equal to true means? Even the step is failed. It will continue. Continue the rest of the steps, the dag. We have a dependency. You can define a dependency to skip it.

E

Yeah, I think simon is posted here. The dependency logics.

B

E

Excuse me: can we continue any thumbs up.

E

Okay, perfect, so next topic is the workflow template capital, w workflow t capital t template, which is nothing but reusing. The workflow definitions.

E

So the workflow template is nothing but it's a definition of the workflow staying in the cluster. So whenever you want you can you can reuse the definition into your workflows.

E

So this will help the user to define that all the templates and dag and steps which they are frequently using they can define as a workflow template and store it in the cluster. Only only thing they want to pause the param arguments to just execute it with the different parameters or different artifacts.

E

Which is similar to like a class definition in java, you can define the class and whenever you are creating a new objects, you can pass the new or new new constructor arguments to create new objects.

E

So in the workflow template you can refer only the particular template or entire template as a workflow. That's a two way you can refer refer the workflow template in your workflow. One is like you can only refer the templates from the workflow template for in your workflow, the second one you can just execute. You can just convert your workflow template as a workflow, which is the workflow submit.

B

E

Just I will open the handsome here, so I will show that sample workflow template.

E

This is very similar to that workflow, temp workflow, but only thing the kind is little bit scientists our floor template and the name is a fixed name in workflow. You can see that always generate a name. So whenever you are submitting, it will generate a new name and you can you can create your entry point and you can define your all the pre different templates here.

E

And there is a separate command in cli and ui argo template. You can create the template. So when you create it it will be created. It will be available for all your workflows here.

E

So there is a list of commands like argo template dash dash.

B

Hell, oh sorry,.

E

I put a keyword inside.

E

So you can see that there is a bunch of command. You can create the template. You can delete the workflow template, get the template linked under list.

E

So once you have a template, then you can refer referring to your workflow.

E

In two way, as I said, you can refer the entire template as a workflow, then it will. It will take the entry point in workflow template and execute it everything the another one. You can only refer the particular template which is defined in the workflow template using a template ref in your workflow.

E

Let me put this one.

E

So, basically in here I'm just referring the template, which I created consider as a there's a whole. My definition, I want to execute it, that's a workflow, so.

E

E

So, basically, what it's doing that the controller is getting that workflow template definitions and executing as a workflow, for you.

E

There is another way also you can submit it in the command line. You can do that or go submit dash dash fram workflow template slash that workflow template name.

E

Which also doing the same thing for.

B

B

First, step.

E

Type, you can see that the cli will automatically convert your request to this format of the workflow.

E

The second example I have it. I just want to refer the only one template in my workflow template. I don't want the entire template to be executed. It is just reusing. The function.

E

E

So in this workflow I am just having a template. Ref in the template trap, I am giving workflow template name and which template I want to execute it, and what is the argument values so this argument value will be passed into that template definition and execute it.

E

If you see the techno, the template is referred from. The workflow template.

B

B

E

So, which is printed seven because I passed as a three as a.

E

E

The next one, as I said you we can submit through the cli instead of creating a workflow ml file, you can just submit it this mainly submitting that workflow template as a workflow.

B

E

uh You can you can do the cube. Ctl apply dash f, but you should have a name not a generated name on the workflow or any any ml for any kubernetes resource should have a name instead of generated name.

E

Second, one the fail to create a workflow template.

E

Is there any extra characters.

E

Are you directly referring that raw url or you are downloaded and executing.

E

E

For the workflow template you can use a cube ctl applying das f. Yes,.

E

Yes, argo sale is just a convenient wrapper on the cube ctrl, which will give which will pass that out workflow output and give the nice pretty.

E

E

So can argo executor container configuration can be controlled from the workflow e m alpha? Yes, you can. You can define your resource quota, those things, but you cannot change the container. The container will be defaultly added added as a sidecar in the pod for each steps.

B

E

I don't know, what do you mean by controlled.

E

E

E

Any thumbs up anybody able to create a workflow template and refer that workflow template in workflow and refer a single template from the workflow template. That's a two.

A

Good exercise before we continue, I see there are a few questions in the chat and because there's quite a few of them- and I know how difficult it is to keep.

E

I was keep on answering that one.

A

You're happy you're upstairs. Okay, it's good! I just wanted to check. That's all because I know it's hard to. I find it hard to keep up to date, myself.

E

So the last one can we specify the exit handler at the step level? Yes, you can do that.

E

And I think the current function current exit handle. You cannot pass the argument, but we were. We are thinking of enhancing the exit handler to pass the arguments.

E

But currently you cannot pass the arguments.

E

Any thumbs up anybody finish that workflow.

E

E

I'm just waiting for single thumbs.

E

Up: okay: if, if you're finding any difficulties you can, you can put a message on the chat, I can answer it or you can find me the slack. I can help you on that.

E

The next one is cluster level, workflow template so far. The workflow template is a namespace scope. You can access it within the namespace. Some of the use case.

E

I want a workflow definition sitting on the cluster, so all the namespace can be accessed that use case resolved by the cluster workflow template, which is same as our flow template functionality. Just a scope is changed, so the kind to kind is a cluster workflow template to create that one. You need to have a cluster admin to create it.

E

And only the one difference whenever you are referring the cluster cluster workflow template in your template, ref or workflow template ref. There is a field called cluster scope. You need to make it as a true, that's all other than that, the functional device. There is no change.

E

You can still submit, submit using a or go submit from.

E

Cluster or flow template, then name it will be executed.

E

I will just try to find that examples.

B

Here the cluster of pro template examples.

E

You can find a all the workflow template examples here. The cluster workflow template examples here.

E

Whenever you are referring, you just need to have a cluster scope equal to true, then the controller will automatically understand this name. Is a cluster or float template type? Then it will. It will automatically refer refer it for.

B

E

um Okay, uh I think I'm done, I will hand over to simon father, chrome, workflow awesome. Thank you guys.

D

All right, can you guys hear me again.

D

I'll take no answer as a yes, so let me share my screen awesome. So uh this is just gonna, be a quick note of a current workflows.

D

uh So in argo you are able to actually schedule your workflows that you've defined to run on the chrome schedule, and it's actually very easy to do so, and I'll just walk I'll walk through how to do so. Right now,.

D

So yep chrome workflows are just normal workflows that run on the schedule. Converting one is pretty easy, so here we have a sample workflow. um Obviously this one is pretty small, but this works the same for any arbitrary workflow. The first step to converting a workflow to a current workflow is just to change the kind uh just name it a chrome workflow.

D

Then you need to take the spec of your workflow and just put it under a wordpress spec field like so the workflow spec, uh the chrome, workflow workflow spec field is exactly the same as the workflow spec field, uh the same types and everything, so it's pretty much guaranteed to work, and then you just need to specify which chrome workflow options you want, such as your current tab schedule, the time zone that you want to run and the concurrency policy that you'd like it to run on these.

D

uh These settings are modeled exactly after a cron job from kubernetes the standard crown job so things such as a successful job, history limit and bail job history limit work the same way as well as concurrency policy works. The same way, the only field that is not actually part of the crown job somewhat infamously is the time zone field which we do actually support ourselves.

D

Here, uh you can, you can go in our docs on our chrome on our current workflows, dock and everything is pretty well listed out there for you and like bala mentioned, you can create and manage your current workflows in the in a very similar way. So you can manage your workflow templates. There is the argo cron command, so argo chrome create our chrome list, argo chrome, delete and so on um another another thing I'd like to hint at.

D

Maybe you want to define your current workflows and have them scheduled with git with getups, using a tool like argo cd. I know some of our users do that where they define their chrome, workflows on the git repo and have our actual argo cd, pull the objects from the repo and then apply them to a cluster, uh so that they're always up to date, just some some interesting synergy between argo and argo cd that you might want to consider that I I know I like um and yeah.

D

I think this is pretty much the end of my short incursion to crown workflows. uh Does anyone have any questions.

E

There is a few questions, simon.

D

Yes, do we support dst and public holidays, uh so we actually do not support public holidays. uh If, by supporting public public holidays, you mean, if we keep a database of all the public holidays around the world and how they change over time, we certainly do not do that. uh Do we support dst? Well, we our time zone support is exactly the same as the uh go client time zone support.

D

So, yes, it will support dst.

A

It's probably worthwhile noting that some holidays vary massively from country to country and actually even from business to business. um For example, in the united states um there are a number of public holidays that are observed or unobserved and depending on the business people may or may not be in the office. For that, so I think that's very much. um It is on our radar, I think, but it is def. It's definitely a business decision rather than a decision we can make on your behalf.

D

All right so with that uh can user? Yes, so if the user can provide toilets, we do not have any interface for you to tell argo, which dates you wanted to uh to consider holidays. uh We do. We are aware that argo events has a similar functionality and we are considering maybe adding something like that. But as of right now there isn't anything uh on the argo workflow side.

D

There's a current workflow. We try if the job cannot be scheduled. That's a that's a good question, so what chrono workflows will do? Is it will just um create a workflow object for you? uh So actually argo workflows itself, regardless of chronoworkflows, will uh will retry. So let me be more clear. The only thing the only part of a chrome workflow or by extensional workflow that actually needs to be scheduled is a plot creation.

D

So our workflow object can be created and you can we can try to. uh We can try to schedule the pods, but if the pods cannot be scheduled, uh maybe for a lack of resources or some other reason. There are knobs that you can turn in argo to make sure that it keeps retrying. So the short, the short answer is yes, but you need to have it set up for.

D

That and another note uh regarding holidays, uh you can only you're always free to, uh as with anything in our workflows, you're always free to create your own containers and images.

D

uh That, for example, keep a database of the workflows that you want uh to consider as holidays, and if you, you can always have that, be the very first step and, for example, if the current date happens to be a holiday just fail that step by returning exit, one, for example, and then the entire workflow will not run so. There's always ways that you can sort of engineer a featuring by running it through your own containers.

D

All right I'll leave it open for a couple of seconds. Another question.

C

Does argo workflows create, like chrome, workflows, create crown jobs under the hood, or this is all hand over argo.

D

So this is actually completely handled by argo. Yeah. Are you a workflow? We we don't use the crown the kubernetes cron job or a biased crown job uh to do any of this. For us.

D

I wanted some insights about whether this executor can run with non-root docker. I think alex may be able to answer this question more proficiently than I can.

A

um So the reason I mean by by um do you mean privileged, or do you mean non-root.

A

um I think you can run non-root, but it requires work to do it. I haven't done it myself, um somebody earlier shared a page and I'll try to see if I can find it re-share it. That tells you all the pros and cons of the different executors. Let me see if I can find it and I'll pop it into the chat for.

A

D

uh So, with that I'll hand it over to alex uh to continue with the questions and to, uh I guess, give a conclusion.

A

uh Thank you very much thomas. We have. um um We have a few more questions in here about creating um resources using the resource template. I actually didn't know the resource template was um so popular.

A

um So I'm just gonna uh very sorry about it: you're making a bit of creaky noise, um so there's information on the different executors on on that page. The most popular executors are obviously docker and pns, and we we use pns uh ourselves, um but docker one requires additional permissions and privileges to execute um now.

A

But I can correct me: when a resource template runs um then the um then it basically runs rather basically it's very similar to a normal workflow uh pod, except it only contains a single container and that container's job is to create the resource.

A

um Okay, so next question about any plant supplied, non-root user for docker executor, um so we do have. We do want to make them more secure.

A

However, with the docker executed, the way that we get artifacts from the main container in to the uh into the into the sidecar container. So we can transport those to s3 on your behalf is by mounting the docker socket. So it needs to be that weight container that style car container needs to be able to do that. That, I don't think is necessary if you um amount of volume and have your artefacts on the volumes, and I think it can probably run as a non with fewer permissions there.

A

But I haven't really dug into that very much and maybe that'll be an epic for our next um next set of planning sessions.

A

Maybe I'll make a note of that now.

A

Any more questions, I've always corrected me say: volume doesn't support the resource templates.

E

So basically the resource template it will just take the the metadata and just to do the chip ctl apply in the inside container. It won't, it won't refer any volumes, so it won't. Do the volume mount on that.

F

Still in the resource, whatever we mentioned that gets stored in your, it should be getting stored somewhere in your executor container right. So in some temp file and you use this queue, theta create command, but that, if you are using that k8s api, then basically don't allow to execute the argo workflow, because there is no volume.

B

E

I'll private privately update this, uh I will take a look at the code and I will update you.

F

A

Here's another question about using retries on steps.

A

um Can user add retries the steps if the step fails retry on a given number of times before giving up? uh I think the answer to that is yes, um but bala. Can you confirm, I think, you're looking at this recently.

E

A

Okay, okay: let's uh I think we now need to draw this to a close. um So thank you very much everybody coming along here. If you have any more questions, uh then do come and ask us in the slack channel that we have or you can come and um you can obviously raise any issues or enhancements in github uh as well. Okay- and we finish a bit ahead of schedule. So thank you very much everybody and have a lovely day.