Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to Multiply the Power of Argo Projects By Using Them Together - Hong Wang

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How to Multiply the Power of Argo Projects By Using Them Together - Hong Wang & Alexander Matyushentsev, Intuit

The Argo Project contains three big software components: a GitOps continuous application delivery platform, a cloud-native workflow engine and an advanced deployment controller. All projects are evolving rapidly and have received a lot of new features in recent months. In this session, we are going to provide an update about project progression and demonstrate how to get even more out of Argo by using the projects together. You will see how Argo CD can declaratively define and manage powerful batch processing pipe powered by Argo Workflows and Argo Events. At the end of the session, we are going to talk about where we think GitOps future is heading and demonstrate how GitOps engine is leveraged outside of Argo CD.

https://sched.co/ekGL

A

Hello, everyone thank you for attending our session. Virtually today we are going to talk about how to multiply the power of argo projects by using them together. It's a really exciting topic for, since we have done a lot for the past year, my name is hongwan. My co-speaker is alexander mutinsof.

A

We are both principal engineers from intuit. Here is overview. We will talk about who we are the move to the project update and the last, but not the least, the most exciting demo. We are from intuit. We have 9000 employees in 20 countries serving 60 million customers all over the world.

A

We create financial products for our customers and we are the creator of turbotax, quickbooks and mint.

A

The argo project is a set of kubernetes native tools for deploying running jobs and applications. It uses github's paradigms, such as continuous delivery and progressive delivery and enable ml ops on kubernetes. It is made of four independent projects. I will go into individual ones. Soon we got a very strong community. The product has been recognized and used by a lot of companies.

A

We got accepted as cncf incubator project and have 10 000 stars 500 contributors and 170 end user companies. Recently we created the bootstrap committee with members from intuit, blackrock, redhat and alibaba to help with the governance. The idea is to keep the project stable and provide the direction for each product.

A

We are very proud of the current progress and enjoy being part of the open source community. Then, let's jump into the individual projects. We have all right. Let's start with workflow and events.

A

Argo workflows is container native workflow engine argo events is event-based dependency manager. We recently released the argo events 1.0, which was a huge milestone for us. Please check it out. It is more flexible, stable and scalable.

A

Regarding argo workflows, the core features are relatively stable, so we went deeper to help with more special use cases, for example, memorization, which helps reuse. The remember output when we run to speed up the progress and reduce cost and the mutex is helped guard the non-shareable resources like database.

A

The chrome workflow is another handy enhancement to replace chrome job which allows you run more than just a single pod. In a chrome manner, and also a lot of the user experience enhancement for the future, we will address more scaling, reliability and performance issues and plan to add the user interface for our events.

A

Arnold cd is very popular project and a lot of companies have used it as the backbone deployment solution to kubernetes clusters. So we have worked very hard on the performance and the related issues. Currently it can support 2, 000 applications, 100 clusters and also modulus repositories with the argo ecosystem.

A

Several sub-projects has been reached to a good level of the maturity and the string sends the argo cd story here.

A

Application set to support the application of applications pattern github's engine as a reusable library that implements core git ops, feature image, updater, which monitor docker registry and automatically update the image notifications helps notify users about the important state changes for the future. We plan to do the git ops agent, which will be a light weighted argo cd based on the github's engine and also the serviceability, is another area. We would like to put more effort as more and more companies are running our city in a large scale.

A

We would like to provide more tools to get insight around the system.

A

Argo routes is the youngest argo project as a declarative, progressive delivery and experimentation tool. However, it got a good momentum. More and more companies see the arcgap wish to find a solution in this area. Adp spotify, intune and many other companies have adopted our garage from the basic standpoint. Early routes supports two update strategies: blue green and canary.

A

However, it shines in the detail: it supports the fine-grained weighted traffic shifting with the integration of ingress controller and service mesh, the rich metric provider. Integration also gives the possibility to do a more sophisticated experimentation to find control the road progress. Okay, let me hand over to alexander. We can check out his amazing demo together. Thank you.

B

Thank you for introduction, honk and thank you for great update about every argo project and I'm going to talk about how you can get the maximum out of argon by using the projects together.

B

So argo projects are not really tightly coupled with each other. They are focused on just one use case and trying to do it as best as possible, but still you can use them all together and they didn't complement nicely each other and together all four projects combine a very powerful application delivery platform, and we can talk a lot about why it is true, but one demo worth 1000 words.

B

So, instead of explaining, why I'm just going to show you how- and here is what we are going to do so we are going to build a scalable and resilient system which implements a real use case and we are not going to code. Instead, we are going to use existing open source projects and each project again don't know much about each other. It just do its work and we are going to use argo to glue everything together and make them work together.

B

And, as I mentioned, we are not going to write a single line of code, except we will have to write some yaml and thank you for laughing on talk about yaml. If you are, but to be serious, I'm during the demo I'm going to prove you that that yaml provided us so much benefit. So it's totally worth to spend time. Writing it.

B

Okay, let's move on so a little bit more details about what exactly we are going to build, so we are going to create a web service so and web service is going to have a user interface which is uh here on that graph. So file stash is going to be our web user interface fast file. Slash is an open source project that implements a user interface to different storages, including s3.

B

So next component is our s3 storage, which is minion and I'm sure a lot of you heard about minion. So it's a great sd compatible storage, which you can run inside of kubernetes and that's going to be our infrastructure layer component. So argo events is another infrastructure layer component. It is going to listen to every event in.

A

B

And every time a new file is uploaded, argon events will will trigger argo workflow in background. So an argo workflow is going to download the file from the menu it will use face, detect, which is an open, uh open source project that implements all the machine. Learning magic so and phase detect will get access to that image and it will detect faces on the image and it will produce a new image which has all the faces, highlighted and uploaded back to stream, and then finally, file stash will show us.

B

You know the uploaded image with the faces. Plus we are going to have kind of two control plane uh components. One is argo cd and rollouts, so roll outs is going to manage file stash and cd. Manages kind of the whole thing.

B

Okay seems like, I think, we're ready to go so and by the way you can do the same demo by yourself. So if you open that url right here, let me open it, and uh so you will be redirected to the treatment page of that git repository and the readme file. Pretty much have introduction to this presentation.

B

It explains you know what we're going to build with what functionality we're going to get, and you really you can start from. Let's do it part.

B

And it's time to do it ourselves during that demo, so so so far we kind of talked about that system from user perspective you know top to bottom, so we talked about application, the user interface and then infrastructure and then control plane to execute the demo. We have to kind of do it in reverse order. So, first of all, we need to create the control plane and before we create a control point, we need the cluster, so it takes some time to create a cluster.

B

So I did it ahead of time and I already have mini cube, running and you're free to use whatever cluster you want to use. So let me go ahead and create argo cd first.

B

So before I create anything, I just want to show you that I have a brand new mini cup cluster. It was created 26 minutes ago and what I need to do, I need to execute two cube. Ctl commands to create argo cd. One is just create argo cd namespace. Let me make it a little bigger and second, I need to execute ctl apply and all that that keeps you apply. Just push the typical argo cd installation into my mini cube cluster.

B

So let's go ahead and do it- and this is it- argo cd is created in the background, mini cube, download images start containers so, and the next thing we should do is we need to configure our control plane to deploy the rest of the infrastructure.

B

So and, as you might know, argo cd provides user interface and the cli, so you can use cli or ui to configure arbo cd imperatively or you can use a declarative configuration which is really nice for for the demo and to apply the declarative configuration.

B

I just need to execute one more cube, ctl apply command, which is very handy, and I just want to go ahead. Do it right away, and next I want to jump back to the repository and show you what we just did so. First of all, let's take a look at the file which we just sent to the our minicube cluster.

B

So here is that file and it's just a yaml file which has several objects and let's talk about the subjects.

B

So, first of all, we are seeing here uh an instance of custom resource definition, called application, and let's spend a couple of minutes just to talk about what the application is. So application is an abstraction which argo cd introduces and it's pretty much a argo cd configuration to better understand what it is. We should just take a look at application spec and we really just need to focus on these two fields.

B

So, first of all we are seeing there is a source and destination. Let's talk about source, so source explain argo cd: where is your desired cluster state located? So it just points to a directory in a git repository- and in this case it's it's, the same git repository and there is destination and destination explains to argo cd where manifest supposed to be installed and in in this case we are going to install into the same cluster where argo cd is running and we are going to use kubecon demo namespace.

B

So once everything is created, we should get a container namespace and that's pretty much it. We have several applications here. Instead of you know, looking at these applications in this yml file, let's move back to the repository and take a look at the files in that repository.

B

So we have uh infrastructure components here and our web application that leverages infrastructure. Let's take a look at infrastructure first, and there is nothing. This is the least interesting part. I guess so here you could just see you know some minor tweaks which are related to installer events, rollouts workflows and menu such as pre-configured bucket and some are back rules and next more interesting part, but the web application which actually leverages that infrastructure.

B

So, first of all here we have a rollout object. Let me find it here: it is so a rollout, as you might know, is like a big brother of native kubernetes deployment and it does the same same job as deployment.

B

It creates replicas which create spots and eventually run some containers plus it supports advanced diploma strategies and in this case I'm using blue green. So and as you can see, this particular rollout eventually creates file, slash container, that's it and, in addition to rollout, we have two interesting objects. One is event source and when source is configured is configured. Argo events to watch for events in industry bucket and then sensor is the most interesting one.

B

It eventually produces the workflow, and this workflow literally consists of two steps, so first step find faces, and it's based on this image, which I created for the demo. Basically just triggers phase detect, and you know, couple parameters and finally, the second step upload results back to stream yeah. I think this is it. Let's we covered everything in that repository and we're ready to go check what our ocd did for us in the background, so I need to first of all start a mini cube, also start tunnel, so we can access all like services.

B

Okay, while it's working.

B

Let me just talk a little bit about like how we can do that, so we can use cube ctl to inspect our cluster, but that is kind of it's boring and not easy, especially given that we have argo, ui and argo guy provides us. A lot of information about our cluster seems like tunnel has started. Oh, I know why. So next we need to.

B

We need to actually patch argo cd service and change it to load balancer type, so it will be available outside of a cluster next tunnel wants me to ask to give it a password, so it can open access to port 18 and, finally, argo cd should be available just on localhost cost port 18.

B

seems like it's up. This is great. Next we need to login and built in a user. Is admin and password is a hostname, is a port name. So let me go ahead and use it and great.

B

So, as you can see that, okay, by the way, this is argo cd, user interface and what it does, it shows us all the applications you know configured in argo cd and as expected, we have five applications here for infrastructure components such as argo events, rollouts workflows and so on, and there is a web service and, as you can see, all the icons say green- and this is great because it showed to us that um all the components are up and running and running means not just created deployment, but also all the ports started.

B

Readiness probes are not failing, they are successful. All the services goes with ip addresses and pretty much everything is ready to use, and this is kind of demonstrate that argo cd is not not just the gitobs operator. This is also a very powerful, powerful, kubernetes dashboard and you can use it to learn a lot about your kubernetes cluster.

B

So um if you want to learn about more details about of your application, you can locate required application on that screen. Click on the element- and, let's use argo workflows as example.

B

So I just clicked on that element and I was redirected to workflow details page and it actually has a lot of details here and I guess the most interesting one is that tree view which uh show us all the resources which are part of argo workflows application, but in this demo I'm really interested in just in the service.

B

So let me filter service here and argo workflows has couple services and I'm interested in ergo server service, so it's available on port on localhost, because I use mini cube, and I really need a port from here and let's go ahead and try to open that url yeah. So here we go. We have argo, workflows, um user interface and we I opened it on purpose because we are going to need it for the demo and argo workflow. User interface is pretty much it's an operator console that operator can use to see uh background page jobs.

B

You know how they progressing in real time and then finally, we need one more url, so we basically we want to access our application, which is file stash, and I also just need a port support. Is that one zero? Nine zero, zero one.

B

Okay uh here is file, stash user interface and um as expected, it has no images here, because we have not uploaded anything here to me or so we're just showing to us that nothing is created, and this is it. So I think I just want to kind of take a quick pause and recap what we've done so.

B

We've started from an empty mini, cube cluster and we literally run just three commands: one to create namespace and then two ctl apply commands and we've got the whole system up and running and let's go ahead and prove it actually working and upload an image. So here is the upload button.

B

Of course I have image prepared ahead of time, I'm going to use that friends poster sorry, stranger things fans, I I like friends more. So uh what should happen now is that we uploaded an image and argo work. Argo events should detect it and it should trigger a workflow. So if we move to our workflows user interface, we can see that one workflow is running right now. Let's click on it, click on it and see the details.

B

So basically, this ui is going to visualize steps as the steps you know execute and you can use it that ui to even get more information. For example, you can click on the step and you can go to events and see what exactly is happening. So it was downloading image.

B

Then it started the container now it waits for container, and let's give it couple more seconds to complete.

B

Please come on.

B

Yeah still interesting, oh finally, it's running awesome and immediately succeeded. So basically that was fine faces. It hopefully found the faces and and then eventually pass it back pass it next to upload result, step upload result step is doing the same okay. So it's putting the image created. The container started the container and I suspect my mini cup is quite busy, but let's give it a couple more seconds to complete. So it's running, hopefully uploading data to s3 and it's done awesome. So if everything worked fine, we should go back to it.

B

File stash and refresh the page, and we've got a second image which has faces on it. So awesome and I feel like it's really excited and I did promise to talk about yaml a little bit and so yaml didn't just give us a fully working system and we would strapped it in one command.

B

In addition to that, that yaml actually encapsulated the whole application lifecycle and we don't have to reinvent it again and I'm referring to argo rollout object in this case, so argor allows object, uh actually supports blue green strategy and the the way blue, green is supposed to be done, is actually encapsulated in this like four or five lines of yaml.

B

So I think it's really excited and yeah. That's ends.

A

B

And please go ahead and try to do it yourself and try to upgrade file stash using instructions in with and see blue green upgrade strategy live, and this is not the end. I did want to mention one more component, which is kind of hidden, but it's also very important, and that component is github's engine.

B

So what is github's engine github's engine is a library that was kind of created out of argo cd. So what we did? We took out of argo cd, uh the implementation of core github's features and put it into a library, a reusable go link library and that library includes not just the basic.

B

It also has advanced githubs features such as sync hooks, sync waves and many and much more so, and that library powers argo cd for quite a while already, and what happened is that library is getting mature more and more, and we are really happy to welcome a new consumer that is uh gitlab so and I'm super excited to tell you that gitlab actually recently released an alpha version of gitlab agent, which is active in cluster component, that pretty much integrates gitlab and kubernetes, and it is powered by github's engine and why it is so exciting.

B

Is that because now we actually share the same code, so both gitlab and argo cd does githubs in pretty much the same way and teams are actively collaborating and working on new features and bug fixes. So you will see a lot of great stuff soon and this is it thanks a lot for listening to our demo and update about the project and please go ahead. Ask your questions. If you have any.