Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 18 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Guide to the Galaxy (Ep 58) | Argo Workflows with Pipekit

Description

This episode, we are joined by Caelan Urquhart and JP Zivalich from Pipekit to talk about Argo Workflows.

Pipekit is a control plane for Argo Workflows that enables massive data pipelines in minutes, saving engineering time and cloud spend. Data-driven enterprises use Pipekit to run their data operations on Kubernetes, from data processing and ETL to data science and ML model management. At Pipekit, we believe the future of data will be cloud-native. This is why we're a leader within the Argo Workflows community, contributing features and critical bug fixes for the Argo Workflows 3.3, 3.4, and 3.5 releases, while also convening end-users to share knowledge at regular community meetings. Internally, Pipekit uses tools from the Argo Project to run its own infrastructure and operations, while especially using Argo Workflows to manage customers' data pipelines. Learn more by visiting Pipekit's website or following on Twitter, GitHub, and LinkedIn.
https://pipekit.io/
https://twitter.com/pipekitio
https://github.com/pipekit
https://www.linkedin.com/company/pipekit-io/

A

B

Good afternoon uh hi good options: I, don't remember how I introduced this stream anymore I'm, so I'm done I'm fried it's.

C

Thursday and I need.

B

This week to be over um I, don't have any announcements today, so I'm actually just going to go ahead. Christian and kick it straight over to you so that you can get this show on the road I.

A

Thought I literally just thought, said: I'm gonna kick Christian I'm like this I.

B

Should just kicked guys, thank you yeah. You.

A

Just kicked me in The, Shins yeah, so um yo also tal says hello, good options, um so the so someone remembers their catchphrases, um so I actually never seen this uh spelled out. So that's pretty cool.

B

Wait someone you say that, like I mean first of all, I love and appreciate tall and his.

A

Yeah yeah tall, yes, but.

B

Also, like I, think it's more than one, it's probably at least two of our viewers. Remember that's.

A

Right, that's true: yeah, there's, there's dozens of us there's literally dozens.

B

Dozens of us, like half of.

A

Them- and they definitely know it yeah uh so yeah I I, really don't have any other announcements other than um uh I just got back from uh get Ops con, which was really really cool for me on a personal level, because I was part of the um I was co-chair of the planning committee, so that was a really really cool our first time as like being like on our own kind of well we're co-located with open source Summit but open source.

A

Summit is kind of like a series of of uh of um conferences, so we were kind of you know co-located uh with the CD foundation with cdcon. So you know kind of makes sense. We, um you know, get Ops CD Foundation, you know, let's, let's do something together, so those those folks are really cool, really really big, shout out to everyone at the CD Foundation. uh That was really really great. um I had a great time. I had a lot of positive feedback uh from a lot of people, so you know.

A

Hopefully we keep that train going, uh maybe hopefully in Spain we'll see um you know it's it's it all. It all hinges on the cncf so um and Linux foundation in general. So we'll we'll see what happens, but it was. It was a good time had by all great talks.

A

um I am seed the second day and I didn't mess it up, so that was kind of cool. um Yes, exactly anytime, you don't mess up, it's always really cool, so um yeah. So when, when uh check out my twitters um when you, um um when you see uh you know when the um when the recordings come out, I'll post those a lot, a lot of cool talks were had by there but, um like, uh like Hillary, said no really big.

A

Other announcements I like to um introduce our guests right or have them introduce themselves uh for me. They are here, I, don't know where they are in your screen, but yeah they're, they're right there um at the um you know Kailyn and JP uh from pipe kits. So uh you know what why don't you guys introduce yourselves uh tell us. uh You know a little bit about um a little bit of pipe kit and you know kind of your background.

D

For sure yes, I'm Kalyn, co-founder CEO I'm JP founder CTO yeah, and we are the founders of pipe kit. So Pipe kit is a control plane for Argo, workflows and running all sorts of data operations on kubernetes. So we help data teams scale up pipelines on their own kubernetes clusters. We also provide Enterprise support for Argo workflows the open source project, as we help maintain the project and have been contributing to the project for the last two three years now um yeah and we're calling in from Lisbon Portugal right now we had a great.

A

Time must be nice, yeah, great.

D

Climate Coop, County Youth decided to stay and work out here. For you know good chunk of Summer.

D

um We have a team of about 10 all over the world, so a good chunk of folks are out in Europe and we thought we'd yeah hang out here for a while, um as for a little more background about us, so I'm from originally from California um been working in Vienna, well, data space, that's at various startups in my career and that kind of led me essentially to work out a startup called kite, which was an autocomplete for programming languages.

D

So a precursor to the now famous GitHub co-pilot and chat GPT um and basically ran into a lot of the scaling challenges that a lot of startups are now facing and trying to leverage chat GPT in their products and got to realize, like oh infrastructure, is a big deal for doing machine learning and JP and I have been friends for a long time, got together and kicked off Pipe kit and I'll. Let you tell kind of the origin story.

A

Since you're in Portugal right, yes.

C

uh Yeah since uh or like before, working at uh Pipe kit I was working at Docker, where I was one of the maintainers of Docker Hub and.

D

Specifically, the automated.

C

Like image, build pipelines like as a part of Docker Hub internally, we were working on a tool that was called pipe git. That was a like a workflow orchestrator on top of Docker.

A

C

And when I left Docker, um you know, kubernetes was like the wave um and we weren't seeing as much like swarm development. Most of the workloads were moving to Coop and I asked some of my like data science and data engineer. Friends like.

D

C

Are you using kubernetes and Docker and they were all just like no like what specifically like we wanted to do, um but the tooling like really wasn't there yet, and so after chatting with a bunch of them um and like really feeling their pain, especially about like scaling up and scaling down like data pipelines, um you know, did some um you know Research into like what communities were like interesting places uh found like Argo workflows to be like the uh like Pipeline and like workflow orchestrator.

C

That was, um you know, built for and on like kubernetes and figured like hey, let's, you know just see if we can help like these data scientists and data Engineers, um you know adopt yeah.

A

You know what I I always and I always tell people like you know. What's what's funny, is that like when people say Argo now they think of Argo CD, but like not not too long ago, when people said Argo, they meant workflows right, like even the CLI for for Argo workflows is just Argo right, it is an Argo CD. It isn't.

C

A

Argo events or anything so that's it's, um it was one of the biggest um projects to come out. You know the most popular project for a long time. I think still is with respect to Stars. If you're gonna count.

D

Up yeah, so both are yeah I, think 13 000 GitHub Stars, which is yeah incredible to see um the growth over the last year, especially for CD and then yeah I mean workflows, continues to grow. Like you're saying.

A

Yeah, you know: what's funny people don't know that people don't know this to see. If anyone in the chat knows this, um but Argo CD was originally built. um If anyone knows this drop drop in the chat, but if not I'll spoil it right now, Argo City was originally written to deploy Argo workflows. It.

C

A

Yeah, so originally Argo CD was okay. Well, we have all these workflows like how do we? How do we get them to the cluster right, like okay, I'm, a data scientist right like I? Have all these you know I built all these. You know workflows or you know uh someone needs like you know, big batching, it's like well. How do I get them on the to the cluster and they're like well, they're, just crds, so we'll just use our and our Argo City, originally or original?

A

Well, it has its roots in basically just deploying workflows, so I just think I thought that was pretty interesting.

C

Taking off really just as like an application, delivery tool.

A

um Yes, yeah yeah, exactly yeah, so um so in um so in general. So in general, like people ask me all the time. I don't know if they ask you Hillary, but they asked me like what is Argo workflows right, we're from the red hat. You know side of things so like if anyone mentions CI CD um well, we're tecton right like we use like. Will you use tecton right, like yeah, exactly yeah the tekton cat there?

A

Yes, like these Techs on it, we're like well what about Argo workflows, he's like well, article workflows wasn't really originally created for for C I mean people use it it in in some capacity very successfully for CI, but you know: Argo workflows wasn't originally created for that. Yeah.

C

B

I will say sorry on the reverse side. Somebody was talking about using tecton in a way that is more similar to actually what Argo workflows. Yes,.

A

It was originally before.

B

And I kind of I kind of saw that on my team, I was like I yup I guess you could do that. Yeah yeah wouldn't have occurred to me, but.

A

B

Yes, you can and I think that's sort of how this has happened. It's just like oh well, yeah. You can yeah.

A

B

D

Use it anyway, yeah really generalizable tools, so that's I, think part of what's driving their popularity, yeah um but I guess to answer yeah what is Argo workflows for anyone who might be wondering? What are we here to talk about um yeah? You know it is a kubernetes native orchestrator super generalizable, so the original tent I think we're.

C

D

To was, it was for yeah data workloads, stepwise data workloads and then highly paralyzable data jobs as well. Taking advantage of you know, distributed systems um and really very pretty significantly from the status quo, which is airflow at the time and arguably still is a really popular tool, um but being deployed as a crd and just having a much lighter weight deployment a much easier time for if you're running, all your operations on kubernetes already, you don't have to manage as many dependencies um and has I think quite a lot of uh scalability um value.

D

Now, um so we were just working with a customer on benchmarking like yeah.

D

How would your your workloads work with airflow versus Argo on kubernetes, and we were surprised to see that we got like a 7x performance bump uh by running the same jobs on Argo, just because I think it's running a lot less on the cluster and the kubernetes API handled all the the paralyzed jobs that we launched um at a much higher scale than airflow could so I think that's like just a little bit of a glimpse of like what value yeah we're seeing like Argo adopters, fine now and what we think is like driving the popularity still.

A

It's it's always very cool to have like a generalized orchestrator, because, like right, like uh probably that's part of the reason why, like Argo workflows was um you got, you saw that performance is because, like like the orchestrator underneath is doing that you know doing the bulk of the job that maybe are like. You know um if you were using like like airflow or or something else where, like you have to write the orchestration part of it, um whereas in kubernetes, that's that just it's just an API, it's just there for you!

A

It does it for you, so you just so um it's just like it's just crazy. How like kubernetes, just in general, has it kind of allowed us to do um a lot of us not to think more like on the process and not think about orchestration, because that part's, just like taken care of yeah.

B

I actually was just talking and I that we ended up using abuse instead of tecton or for um for this project um that I was referencing a minute ago, but I was actually talking to one of the engineers on that project and um we were talking about like the principles of distributed computing right, and so it's kind of interesting as I'm hearing hearing us talking about this, like the performance business so forth.

B

If you think back like all the times, we all the time we spend writing into our our software, like parallel processing and like multi-threading, and all these other things right. That could kind of do the same things um or it's like, or we just have this one thing and we spawn it a bunch and we distribute that it. Actually, if that gets into what you were saying, is performance boost and resource usage actually tends to be like what is the most efficient way. I can use the hardware and probably continuing to do that.

B

Lots of times can give you boost versus um you know like having having multiple threads or multiple processes, all touching kind of some of the same things and causing locks and race conditions and trips, and all these other, all those other little finicky bits that you get into when you start doing it, um and so we're talking about I was like yeah so like the thing about kubernetes is like before we had kubernetes like you still had like computes distributed across. You know like multiple places and so forth.

B

It was just like the orchestration of it. Now we have that without needing to actually physically separate out our compute, because we've allowed us to like virtualize the whole thing, but like the principles and the foundations for all of this is real old, like if we didn't, we didn't do anything new here. We just changed the playing field. Where we're doing it.

C

Definitely it it also just feels like it's good to offload a lot of the like mental overhead of like hey I have to worry about. You know, maybe like setting a mutex somewhere or like handling like that parallelization.

C

um You know really like someone who is, like you know, a data scientist or a data engineer really whatever task you're doing like a lot of that is just kind of like boilerplates, and it's a lot of things where you can really hurt yourself if you're like hey, you know, we didn't set this a lot correctly and we want to like distribute these tasks and run them in parallel, so yeah being able to also load that to like the API layer, like that's such a huge value, add for people in general and really like this kubernetes wave and has like unlocked a whole new level of uh distributed programming.

C

That's just really interesting to see.

A

So I I, uh so I always talk about like the vast popularity of Argo workflows because, like it took it, took a while for our as popular as Argo city was it took a while for it to like catch up in popularity.

A

um You know what what do you see the biggest use case so like I, you know we kind of touched on a little bit like AIML, workflows, kind, maybe a little bit CI CD, maybe some batch process job patch uh management. So why? What? What do you see as the um the reason like the popularity for for ARCA, workflows like and also like? What's like the biggest use case, is it AIML? Is it the CI CD? What what are? What? What do you guys see in um with the popularity of Argo, workflows, I.

C

Mean just a great logo.

A

D

Adorable, the kids, yeah yeah right right and.

A

I mean, like you know like so speaking,.

B

Of people love it. That is why I don't have an Argo plush here is because my child will not stop stealing it and.

C

B

Right then, the the Intuit people they were so nice at a conference. They gave me two one for me and one for him. It's hoping to solve this problem, but now he just has this collection.

C

A

B

Them I've ever received and I cannot get it back. Yes,.

A

D

The logo yeah between my daughter and my dog, uh our ghosts couldn't yeah.

C

D

It's hard to find out where Argo is in our house too.

C

D

C

Logo, aside, like um you know, I do.

A

Think, like Argo as.

C

A community is like really taken off. You know: uh we've seen just a lot of involvement like within Slack. um You know the community is really what's like pushed it forward. You know kind of at the end of the day, as y'all were mentioning like these orchestrators are very like general purpose.

C

um It used to be like uh tecton pipelines or her tecton CD uh was like very much like a CI CD tool right and like the marketing around uh Argo was that this was like a data, orchestration tool um and we've seen. People like you, know, kind of cross-pollinate a bit both ways where.

A

C

People are running some like data jobs using tecton uh like CD, and some.

D

C

People are running like Ci, using uh Argo workflows, but I I I do think, uh like Argo workflows um and specifically like the maintainers, have done like a really good job of you know, keeping that like Community involvement, keeping people active um you.

D

C

Encouraging people to like post and slack- and you know I I- think that's really like a big piece here is just like that. Community for for leading to growth.

D

Yeah shout out to the community- and you know holler at us: if you're looking to get more involved, share, use cases get you know, use cases or use their stories from others tips we're we're in the slack to help you out you yeah, I, totally agree, I! Think it's like the community. That's been really strong.

D

um You know some sort of magic, I think led to that I. Think initial early adopters, like red hat into it, Bloomberg um yeah a New Relic, a bunch of these early adopter type tech companies that really know what they're doing helped a bunch and yeah over time. It's just kind of cross-pollinated and when you look at like the user surveys of Argo workflows, it's really interesting because it's still like very much a generalizable tool and um current demand and like new user adoption, is coming from still I think three big sources.

D

So it's split about like 30, 35 30, like pretty split between CI use cases, you know people coming for either our workflows or looking for a Jenkins alternative on kubernetes. It's a big one um want to pair something that works great with CD Now. That CD is gone, mainstream and Argo CD is so big and why not go with our workflows in that instance? So that's like a good like third of adoption and then batch processing is still really big um again, like Taylor respond.

D

Why would you want to rewrite all your orchestration business logic and just focus on what you need to do and and Argo helps you um get a lot of those like parallelization features out of the box. um Kron features out of the box and so for batch processing. It's still super popular.

C

D

Now, what we're seeing is more ml AI adoption, so not just because, like llms have become huge, but more so I think because, um as JD alluded to earlier, like data teams are now embracing containers, a.

C

D

The development and driving iterative um yeah just Cycles on on the development team, um like data scientists, I, think really Embrace Docker now um over the last few years, which is.

A

D

um And then the next question is well: let's go to prod right and- and you know, if you want to do that, if you want to scale up, serve your models in production, um it's arguably, you know no better way to do that at scale.

C

D

Using kubernetes and deploying and with the cloud providers now providing a lot of tooling out of the box for that for data driven, apps I. Think now we're seeing folks come and see. Well, we want to deploy models. um You know on kubernetes, let's start to orchestrate everything on kubernetes and what are the? What are the tools available and so yeah? You can run airflow on kubernetes, but hey Argos here, Argo workflows is here: it's Cloud native.

D

um You know with a few like a yaml template, you can integrate Prometheus right, grafana and off you go to like a production, scale, uh deployment and full on end-to-end machine learning like orchestration, so I think that's the last bit of demand that we're seeing or like new user adoption is now ml.

A

That's interesting like I would think so. Like you know, from from my naive, you know: 10 000 foot view um it's kind of like it. It's like data science is like I think rapidly catching up like being operationalized, because before like a data, science would just have it. Have you know their machine like right next to it right and that's exclusively right like it would have like you know, five video cards or whatever and like it.

C

A

Just exclusive for running and then and then it started moving into the data center. It's like well now. I have to share this hardware and it kind of sucks right and then it's like well, we can share like we can get more density like by using containers right, it's like well, we need to orchestrate that, and it's like okay.

A

Well, you have kubernetes, and now you have like it's just kind of you know it's it's rapidly being operationalized right, whereas the you know not even like I would say as little as like three to five years ago, like they, they I mean probably still even said: hey some data scientists. They have their machine right right next to them right. It's like why do I have to learn like this whole thing.

A

I could just like tap into my like my machine here to do the, um but now our things are getting operationalized right, like things are like being like: okay, let's operationalize, you know just make some some sense of all this madness right, I, think I.

B

Think the the order of magnitude on the data has also increased exponentially as well. So you really are not going to I'm just going to talk about money, you're not going to get your money's worth out of a machine that you can put on somebody's desk compared to what you can do with cloud computing, and you know just like you said, distributed computing and so forth and then, by, like you, say, operationalizing it we can.

B

We can abstract the data engineers and their concerns away from the wear of what their stuff is doing and just into the what so we're making them more efficient, we're saving the mental Cycles, we're saving them contacts and Paradigm shifts and all the other, all the other gains you get when you add containers and and container orchestration into the equation right like if you're, an application developer and- and once you understand the Paradigm of you- know, writing applications for containers- you don't have to start caring anymore about the whole.

B

Well, it worked on my machine and all the other stuff right. Your dependencies are declarative. Your machine is declarative like every single thing about it is exactly the way you told it to be every single time, um so we've we've Stripped Away, some of the like you know, Christian was saying a while ago, like you know, you can almost be like clip click Ops right in terms of like maintaining, uh maintaining a public, Cloud presence, or what have you right?

B

So we've we've abstracted these things away and uh save you know: saving people, the mental energy of doing that, and then the toil of doing everything right.

B

So when we had to that's like been a necessity because again the order of magnitude of data that we have and the speed at which we are expected to get value from that data, because raw data provides you very little value like all the value is derivative and which way you look at the data, you have to run it through multiple models and and have opportunities to play with it and iterate over those. In order to really understand what is this data really telling me which of the context of this data actually matter?

B

If you start looking into things like iot Telemetry and all of like the raw metrics that come through, plus all the labels, you would apply to those metrics in order to contextualize them so that you know how valuable they are in Trends um and then start looking at where we start applying all that data learning to which types of problems we're looking at, like um preemptive maintenance on systems we're looking at early leak, detections on systems, we're looking at um you know all of these types of things that you can start doing and then start predicting based on the existence of this data.

B

So I'm not surprised like the operational thing like it just makes sense. When you start looking at how much data a data scientist needs to look at compared to how much data and that they actually have available to look at.

A

B

Compared to what they would have had five years ago, two years ago, ten years ago, like it's, it's consistently growing at an order of magnitude. We.

A

Have worse law right, yeah, dumb.

B

Amounts of data, the the big project I'm working on right now um is a data project right. That's why obviously I was looking at airflow um in Rio. We have dumb amounts of data, and so it's like, okay, let's start doing real things with this data right and so that's and we have to operationalize that we have to because there's just no way to to do useful things on it, while having to also care about the entire stack that deeply like. Oh, if that were my life, I think I would quit, but.

A

C

A

B

Job is simpler, thank goodness yeah yeah.

C

um Touching on that point, a little further, like kind of when we were starting out I, get I, was.

B

C

Some like research and asking like those data scientist data, engineer, friends like how much they were spending on idle compute time at their companies, and that number was like hundred thousand dollars like depending on you know. The company size like this is for a company that's doing ML and what like 2019 2020. You just had that as Christian mentioned, like have these like big boxes that were like next to your desk and just keep them on. Like all the.

D

Time yeah we bought gpus um sitting next to certain engineers at kite yeah, that was, that was 20 circuit, 2018, 20, yeah 2020.

C

Yeah and what's nice is like now uh like, as you mentioned, um like kind of having that large data, like forcing you to operationalize from the get-go I, feel like it kind of levels, the playing field a little bit more. For, like some of these, uh like startups, who.

D

You know, maybe they.

C

Don't have like a big funding round or like.

D

You want to, um you know, develop something that is like.

C

Very data intensive- and you know they just want to like get going.

D

um They can do that.

C

uh Going from like zero to like scaling up keeping the same like workflow, keeping the same, like habits going, you.

D

C

They start adopting like that cloud native kubernetes Native approach from the jump.

B

And this is not finite. Financial advice.

A

B

Look to me for math stuff, um but right public clouds. A lot of them are consumption based billing, and so, if you're not consuming, you don't have to spend the money on it. So the idle CPU is not costing in terms of, like you said: General base electricity, um Hardware lifespan, right I, always get into this like uh why I care about certain things when I'm looking at software, um it's not Financial advice.

B

About you know the profiling of the software and so forth, because I know that my a lot of times, some of the stuff I touch, is going to end up on real hardware and if anything we do is going to end up over committing CPU well I'm, starting to go into the the position of shortening the lifespan of my Hardware plus, you know, making it run hotter, which necessitates more air conditioning, and this whole other like go to go to the listen to the talk I gave for um I was recently on the harness um ship, Talk, podcast and I actually kind of went into that deeper about like okay.

B

Then we start- and we actually were talking about the fact that we have all this insane amount of data. We need to start doing smarter things with. It was the other half of that talk, but anyway,.

A

Thank you for coming to.

B

The extremely condensed two sentence version of my.

A

Talk yeah, that's right! Yeah! This.

B

Is uh thank you thank you, um but yeah. So it's it's um again, not Financial advice, but you put in just uh investing in some person. Hours up front saves you the ma that monetary cost in all of the other things for longer periods of time, and then you only spawn. However many jobs, you actually need, um and you only consume.

B

However many resources you actually need versus okay, well, I had to buy a terabyte of memory for This Server, because every now and again I need it versus you know, consistently needing it and so forth, and so like that's all those little like the consumption-based model of it it does. It does play into to kind of your spend and your that startup plan that gets into your Runway and all your other stuff and and so forth. um I just I've been around this industry a little while I know some stuff about this.

B

None of this is financial advice. Of course, I think I said that enough times this is.

A

Very exciting we got a ticker now by the way, do not come to me to talk about how you're spending your money exactly exactly um so, it's kind of um so it. You know we talked you know a little bit about like okay, like we almost democratize.

A

um You know some of these processes with like Argo workflows and it's kind of cool, seeing like some of the things like come um uh that spawned out of it, like uh the uh the Hera uh tool set right. It's just like. Oh, like you know, let's, let's build a a library that you know manages you know Argo workflows, which is I, think that's! That's that's pretty cool, something that that spun out um and so um the my next I guess the uh the next.

A

um uh uh My next question or I, guess kind of, like you know, leading on to the next topic a little bit about so okay, you have Argo workflows. What does what does pipe git? Give you right so pipe kit. um You know you guys said you know you thought this was awesome. A lot of people thought Argo workflows was awesome.

A

um You know what what uh what made you decide be like. Okay, you know you know what. How can we further help people, um you know with our workflows, so.

C

A

You tell us a little bit, maybe you know you know, here's here's your chance right to tell a little bit what like what pipekit does and help.

D

Sure yeah for for us, like with pipe kit, you know our goal is to get you to Value as quickly as possible as a data team, and you might be coming from a place where you're a kubernetes expert, you might not um and really the goal is. How do you you know ship your models faster, run.

D

Your workloads quicker um use, Auto scaling to save commute costs as as Hillary was just getting into, um and for us you know we're saying hey for the for less than a cost of an engineer, we'll get you to production with Argo workflows in a couple weeks or we'll solve your hardest.

D

You know scalability challenges if you're already using our workflows and cell posting it and there's a couple of areas where people see that really tangibly I think the first area is like, as a data engineer, infrastructure engineer, your your users are data users, data scientists, ml Engineers, who might not be as kubernetes fluent so we make it really easy to onboard those users in a matter of minutes with our own pre-built rbac model, Fargo workflows. That means they're, not provisioning.

D

A bunch of access to your kubernetes cluster, it can just run through the pipe kit control plane and in the second big Advantage with us. Is you get a lot of like best practices coming out of the box without Implement, alerting how to implement, triggering and get Ops Integrations um all the sorts of tools and techniques that the the companies that have been using Argo workflows for now years? Come you know right out of the box with with pipe kit for you and the nice benefit is we're.

D

You know our SAS company, but we have an agent deployment model, so we let every data team bring their own cluster so that the compute remains on your servers. You know within your environment and only exposes metadata to the pipefit control plan, to help you orchestrate all of that in one Central convenient place, yeah.

C

The the goal is to help like platform and infra teams enable like self-serve for their data science and like engineer airing teams.

D

C

Like if you're a platformer, an infrastructure engineer, you.

D

C

To say, like hey,.

D

C

Scientist data engineer like.

A

C

Your area, where you guys can just go hang out like you know, either you have like access to like three or four clusters or a couple different namespaces Within These clusters, and you can just submit workflows without having to like get a bunch of like control like access divvy. That out and like worry, you.

A

C

Has access to like what resources so.

A

C

Like Central, like control point where you can uh like onboard your data, science, data engineering team- and you know, yeah I I guess like shift left- is the yeah.

C

um It's kind of like shift left for like data.

D

Science and data engineering, so that's really covering like how you get started faster um with running workloads on kubernetes, and so we work with both like startups and large Enterprises that are in that phase, and then we work with folks who have already been using Argo workflows for a year or two and now they're running into like Ultra scale challenges and what Pipe kit helps them with. There is now that you have a central control point for all of your workloads.

D

You can start to incorporate a multi-cluster architecture which is not something that's available in the open source. So that's um our last kind of value prop for for companies that are kind of matured in their Argo adoption is hey if we're operating across regions. If we run out of GPU in this region, we want to now run the same. Workflow same template, I, don't have to like cuddle into that region.

D

Let me just like ship it through the pipe kick control plane and get the job done, that way, that accelerates their ability to run workflows across the world and and get ultimately again just helping them get down. The jobs done faster.

A

Does does pipe kick, um it sounds like you have a big focus on like on AIML, but um it's also do you find end users using using it for for uh for CI as well.

C

Internally, we use it for CI. um We haven't really built it with like cm in mind, but it has been like a popular like question yet or you know, some users are using it that way as well.

C

um We designed everything with the idea that eventually, data scientists and data engineers I do really want to start like uh saying, like hey, I'm, going to start pushing this workflow to get and using git as like the storage of State, um that's a trend where it seems like data teams are starting to shift in that direction, but it kind of wasn't that way. You know like again like four or five years ago, um so we thought like hey again. A lot of these patterns look very similar from like the CI workload, but it.

B

Wasn't our intention.

C

B

um Bringing in the dumb amount of CI experience I have, in my background, um love.

D

B

um The thing that I would have wanted so much in my early career before it is before would have existed, would have been the idea of taking all of the tests that we write and Define and being able to say: okay, now I'm going to run them all X like not just x times but x times, concurrently or like x times in batches and so forth, um because you spend all this time writing and defining tests that really like fully validate the behavior, but only the behavior at that exact scale right.

B

So if you could take that full the full validation, they get a lot of a lot of load, testing and stress testing tools like you have to just redo the work in them in their way of doing it and, like you end up here with your test defined in this, you know in multiple places, I think those I think stuff. That's gotten better. Obviously, it's been a a few years since I've been directly playing in that space.

B

But yeah I can see why you would you would start taking things like this and applying it to CI problems, because it was like oh I can do. I can do almost evil things.

A

To my application.

B

A

B

Bad things to your software is my number one recommendation for how to make your software better. Do really bad things. Yeah.

A

Do bad things to it see what happens right, yeah.

B

A

What what's, what's kind of um uh it's a little funny, I think that with data scientists since they're coming into this, maybe a little new um have like a better, um a better I guess way of understanding.

A

uh Things like um like workflows, right, like our workflows or even um like tecton, or even something that's like Cloud native though, whereas like, if you so you get someone like me, you get someone like Hillary and like we have one of our CI processes dies or like fails for some reason, like our immediate instinct is like well, let me SSH into that server and see what happened and read the logs and it's like. Well, what do you mean that container is gone? I'm like well, it's gone, it got it got.

A

You know it failed, so kubernetes called it and you know that's it, whereas, like data scientists, like you, know, you're teaching them from new like they don't know that that history so I think um in a lot of ways. uh You know some of these. You know you know people now that are kind of just like learning. This new um new um way of massively orchestrating things uh have a little bit of Advantage, whereas some of us are kind of like we have to like retrain ourselves.

A

You know to think to think in in terms of, like um um you know, of I, guess stateless right kind of CI process, CI CD processes, um you know especially like the whole, the whole dag uh design of workflows and trying to um you know trying to like keep all that. In my you know, you can have massive parallelism, but I don't know I, don't know how many jobs you've ever killed or you've ever ran in like Ci jobs.

A

In parallel, like you can get pretty pretty massive I I, don't think I ran more like than five. Oh.

B

Yeah, that's that's about how many I I ran. uh We gave a whole talk about. This I showed.

A

Examples yeah so yeah this.

B

A

Wasn't that long ago, we're not that old! Our memory isn't that bad I I don't know 2021. That.

B

Is the daisy chaining right, the daisy chaining of Jenkins jobs is like the thing I like to famously reference back to and I say famously like famously like people have heard me talk about this more than like the one time I've said it, but yeah the daisy chaining of Jenkins jobs that we used to do um I I it's for me. All of these things seem like a very natural progression, because Christian didn't I I have accused all of these tools of basically just being clever ways of orchestrating bash Scripts.

A

That's essentially what it is: yeah or Python scripts, I guess in this case or.

B

Python python yeah yeah.

A

Yeah that do best like things yeah bash, like.

B

Things you can actually just execute bathroom Bash from python, although you shouldn't, for you know a best.

A

B

Security whatever, but just because you shouldn't do, it doesn't mean I've, never done it out of just sheer laziness. So here this is who I am as a person, um so yeah um I think that these are all uh going back with.

B

From my background and experience and like so much of my career being spent specifically on cicd I understand how these became, how these became the standards and the patterns and how we ended up in in overlaps, where set tools designed for CI can be used for data and tools designed for data can be used for CI like I, get it I'm, not sure if I can fully like Express. Why I have the level of understanding that I do because of.

A

B

Context but like no I totally get it no notes. This is fine.

A

So um I guess the next next thing: I kind of want to talk about Argo Khan, a little bit um Hillary and I. We we talked about Argo Khan, but uh last last episode. uh However, we were squarely in the CD rollouts.

D

A

um There's so for those for those of you at Argo Khan, like you know, if people think about like cicd people, think about get ups but like there's this whole other world of like events and um uh workflows that are um that have nothing to do with get Ops, have nothing to do with cicd, right and so I kind of want to talk about a little bit of argocon a little bit about um you know what you guys saw in in the workflows track right is. Is there um this? Is there anything I know Caitlyn?

A

You were involved a little bit in the planning of Argo Khan. um You know. Is there anything that stood out to you guys or anything that surprised you that that surprised you how people are using workflows.

C

Y'all had a densely packed room.

A

In that yeah yeah.

C

Like that was like standing room only like Jesus Christ, yeah yeah.

A

C

The attendance in the workflows room, but like good Lord, hundreds.

D

In the workflows, breaking thousands I think.

C

D

The CD side, so yeah props, to to offer organizing that side of it.

C

um Yeah on our.

D

End I mean we um I think there were a total of eight or ten we're close talks so pretty packed day. um I think what stuck out stood out to us is just how uh mature now um some organizations are getting with workflows um either for data platforms or even for CI. So there were a couple of notable talks. um Rob a bank I think kicked off the day, um which was an impressive talk about just like how their Journey went with Argo, workflows and shared a lot of great best practices um and was perfect.

D

Given we were in answer damn so that was cool um and then and then you know another kind of I guess a coincidence was JP gave a talk with another financial services. Company Bloomberg who's been using Argo almost since the beginning as well, and talking a lot about um the data platform and how they're using Argo what challenges they're running into now. Maybe you can collaborate on that yeah yeah.

C

I mean they. uh They have a lot of uh interesting things around like workflow template sharing and their like approval process that they go through.

C

um You know they spend a good bit of time, building out that uh platform to really enable self-serve for, like the broader, um like Bloomberg company, and it seems like the vast majority of their workflows- are being run, I mean just generally throughout the company, whether it's ml, you know what have you data science, um you know analytics, uh are being run on Target workflows and seeing that level of maturity is pretty cool and then I mean I, know they're also using a hero workflows. So you.

A

D

C

Give a shout out to Flavio who give another talk as well: yeah yeah.

D

I think the trend now we're seeing there a quick shout out to Yellen, who is your co-presenter at Bloomberg that was awesome um and then yeah shout out to Flav the creative.

C

And maintainer.

D

Of Hera workflows, which was mentioned earlier again just now, bringing more pythonic approach and Concepts into Argo land I think is more than welcome so first with our workflows, I think next, probably for August CD as well, and there's tons of pros and cons behind like why you'd want to do that, but obviously, from a data perspective, we want to meet people where they're at so we're big fans of our para workflows. We integrate and provide out of the box, support for hair workflows and pipe kit and I. Think.

D

What's um really what was really interesting about that talk? Was the share amount of scale that you can you know going back to like everybody who doesn't need their own GPU anymore, yeah, you can you can plug into hundreds of gpus now um on Google or AWS, Etc and flat talking about how.

C

D

um Now a pythonic approach to orchestration on kubernetes. uh He, he basically just hands off templates to his his scientists to run these massive bioengineering jobs and Gene gene therapy, modeling jobs, and it's it's something like um two three thousand node and parallel jobs at a time- um and you know this is like the data scientists running that now, um you know pretty hands-off from the data engineer that that platform set up I think that's becoming more and more common, so we're seeing I think big adoption now with our workflows among Financial, Services bio.

D

um You know mentioning internet things earlier as well. um You know observing manufacturing, observing like Battery Technology is the main use case where we're seeing as well and I think those that's shown during argocon um yeah. So those were a couple of the takeaways I think another big one was um shout out to Steph at Walt, who gave a talk about just using Argo to orchestrate a bunch of different tools as well and actually set up some load testing for ML models, um so that was that was pretty impressive.

D

You know integrating Argo with workflows with CD, with flight um Prometheus, again just showing how like the systems, like you, maybe start with just a basic dag, but in six months or even three months, you're layering on so much um so much more tooling. That really makes your system you know more resilient, makes your team more. You know, move more fast like faster and iterate, faster with your models and I. Think that's um again, why we're we're pretty pumped about our workflows and the growth we've seen.

A

Yeah I love technology that gets out of your way, which I think that's what technology should should do right. So sometimes you work with something and it's like you're, just fighting with it, um and sometimes it just works and you're like I. Just now, I can concentrate on doing my job. I can concentrate on doing what I wanted to do like you know, which is modeling, AI modeling data science.

A

That's so it's kind of cool that people are using it for analytics, which makes sense now that I think about it um to trying to like massively digest a lot of things and trying to make sense of it. um And you know you don't need a lot of you know long running processes. Long-Running servers is just kind of like you know.

A

Consumption like you know, like you know, using some like Pipe kit, right consumption models like let me you know, pay for what I use and if I don't use anything you know I'm, not using it. So um you know, that's uh uh uh I, think that's, that's always cool! You have to be careful, though I think what was that was a quote. It's like uh for your Cloud spin, you, don't you don't uh the most expensive part of cloud is paying for what you forgot to turn off or something like that.

B

A

You got to be careful about that. Yeah.

D

Again, why yeah having the right alerts, matters yeah,.

B

um Observability matters so much um say that is, that is the beauty of doing it as Jobs, though, because jobs complete right, those those pods, those containers they're not still running they, they do the thing they complete, they're they're done. You have the record of their existence so um like prune that clean up STD, that's.

D

A thing that you.

B

Gotta remember to do is clean up at CD um archive.

D

Your workflows, PSN.

A

There you go PSI, yes right, PSA archive your workflows.

B

Archive your workflows, yep exactly so, um keep keep that keep that happy! You don't you don't want to see what happens when NCD gets full. um It's.

A

Really not a good time Tails from the SRE right yeah.

B

By the way, Argo and tecton have both done that to me. I.

A

Guess the other customer.

B

Infrastructure so watch watch that one it's on. You end users. You must clean that up. You must they all have the operators have um they have like pruning like configurations in there like to prune the old crds out. Go, go configure that it's probably.

A

On by default,.

B

Now you're welcome. That was me yeah. um It was definitely me in techton, I I I'll take the credit for Argo too, but it's definitely me and tecton I have those conversations. I was like nope um but yeah. The um my open source contribution.

C

B

um I didn't actually do it I just made them. Do it.

A

We'll just do it, which is also needed right, like.

B

We just also needed I was like look here is all of the things that exploded, because you don't have this feature turned on by default. Please turn it on and they're, like, oh yep, okay, we're gonna turn that on.

A

Yeah we're gonna turn that on yeah.

B

Good talk that was.

A

B

um But yeah so um take. But yes, it's as much of these things get out of your way and do lots of things for you and it's great because, like I said their jobs, they run they execute they complete. They are they don't vanish so.

A

Yeah yeah as much as your garbage collection and not not like the job away. That.

B

Might blow up in your face, but like the real one, yeah.

C

I will do a quick uh product pitch like with uh Pipe kit. We do treat workflows as being like ephemeral and like once, they're done nice still even from the cluster like yeah.

D

Just of course, I get.

C

Hygiene, because you know yeah, we.

D

A

C

A

Yeah we love it. We.

D

A

So much that we're gonna prune it.

D

Yeah, exactly and now our workflows does have an open source feature um to garbage, collect artifacts, that's another great way to make sure yeah, you're cleaning things up um with old jobs. So we talked about that at our con um yeah I think a lot. A lot of these tips are the things that you don't think about in the first sort of yeah week month that you're getting your hands dirty. You know heads down, but um that's what we try to help new users with in the community.

D

Our own customers with is just hey from the get-go. You know, Implement our workflow archiving implant. Artifact, you see it's not hard. Here's here's, the the config!

D

um You know whatever works for you, obviously, but there's there's a few of these best practices that we try to get out in the community, either in slack where you'll find us or on our blog.

C

I feel like that's just generally one of those things recycled.

D

C

Feel, like generally that's just like one of these things with some of these, like Cloud native Technologies, it's like you.

D

A

C

Yourself, um you know if you've configured something the wrong way like stuff can't explode. You.

D

Can get like a real.

C

Expensive on you, um like you, know, it's important to figure out like what are those uh like really big, uh like levers, whether it's you know setting the TTL on like tecton or Argo, workflows or artifact. You see, as you mentioned, just to make sure that you know you're not saying like hey this Cloud bill. Just like got out of hand. Observability matter should be a t-shirt, I think yeah. You should take credit for that one. Maybe we can have like a gym. We.

B

Can we can add it to my you wouldn't have seen this probably I have a t-shirt store, but it's not a real t-shirt store, but it's where I put a bunch of Snappy one-liners up on t-shirts. Yes,.

A

Observability matters, so it actually really matters is a.

B

Good one I'll put it up there: nobody has ever bought a t-shirt from the t-shirt, store and I, don't encourage it either. That's not the point. The point is so that I can point at it and say: I put it on a t-shirt. No stop asking give.

C

Us a link I mean we might do it yeah. Do you shift off.

B

A

B

Admin and people are like, oh, my God, put that on a shirt and then so one day my husband and a fit of boredom bought me the domain, not your sysadmin.com.

B

Where I I keep thinking I'm going to put a Blog someday, but then I'm, lazy and I don't, but the most important thing that he did is then he had them make these pins.

C

B

Oh cool and, if you find me at a conference, I have them yeah and.

C

B

Give them to people if you know to ask for them.

C

Will you be at pargocon or kubecon North America? Oh.

B

A

Because that that you should definitely pass that out, yeah.

B

No people there are people wandering around with the pin. You will find them, um especially red Hatters. Have them um that's my uh please, don't actually buy things from there. I would feel.

A

C

But like just to go and laugh.

A

That it's there I've put things on.

B

A

B

To go on one of those I I owe that um but yeah. So that's that's the year! Next time you have a Snappy one liner. You want on a t-shirt without actually having to go through the whole t-shirt thing. We can add it to the list of of things. People will never buy and then laugh and enjoy the.

A

B

Really it's just a very long running joke, because I just think it's great just.

A

For the lulls for sure.

B

But uh there was something I wanted to say, and then we got distracted by t-shirts and I forgot it, but it was about um shoot. We were talking about cleaning up at CDE and the care and the feeding, and there was something on those lines. I'm gonna be so mad because we have five minutes left and I will remember it. When.

A

B

um F, my life: yes,.

A

B

Live rep free in my head. What was I gonna say.

A

Yeah yeah I have no idea so something about yeah something about workflows. It.

B

Was gonna be something well.

B

um Consumption and fourth and.

A

B

Know what we were talking about that made me think of the thing it doesn't.

C

Isn't it Cloud native Tech? Yes, it.

B

Was there was some stuff about? It was something about like getting yourself in trouble with Cloud native technology. It really was exactly on that um right, oh committing to get. Thank you. Okay, yeah.

A

It was actually a question.

B

Not a statement, so in my past life, um one thing that I did was I took the results of all of the previous of all the tests that were run and I actually committed the results to get um so, instead of having to store them anywhere, anything like that it, the scripts, run they execute, they commit their results back to git, and then we have a historical git is the source of Truth. What happened? What did we test? How did it do right?

B

Okay, so can, since you mentioned that you prune you treat it as ephemeral and printed as default, is there already a like template or workflow, or something in pipe kit? That would do that, basically capture the outputs or capture some important thing about probably observability data or whatever about the most recent run of the jobs and committed to get.

C

We do integrate with the git like check suite, um so you know whenever a workflow like runs successfully if you've like tied.

D

C

You know get whether it's like a pull request, a tag release. um You know Branch, what have you it will like update the check. I, don't think, we've um like actually gone through with outputting the the result of that workflow like when I say. The result I mean like the actual, like hey here's, this big, like Json blob and like throwing that back into.

A

C

um But I think that that seems like pretty valuable generally. So thank.

A

C

I guess for uh like farming, free future ideas, yeah.

B

This is, this is what they uh they pay me for is to.

A

B

Code and armchair troubleshoot and armchair software design I actually just spent I spent about 30 hours of my week on meetings, yeah yeah um on because I'm across several different different projects. That's why I have the title: Chief mermaid as I do a lot of swimming.

A

B

You haven't seen that it's that's a real thing.

A

Yeah yeah, that's right! That's right! Yeah.

B

Exactly anyway, um anyway, so all I do is armchair uh armchair coding aren't implementing armchair, designing.

A

You should have like that anymore. That's another t-shirt, armchair, coder, I'm.

B

An armchair coder, yeah right or just from this particular.

A

Yeah, yes, exactly so cool yeah yeah. We have a a few minutes left.

B

um Armchaircode.Com is available.

A

I know what I'm doing.

C

A

um So Nick yeah next time you go to armchaircode.com.

B

A

It's just you're parking it there so uh yeah with the last couple minutes. We have there uh um Kaylin JP. Is there anything else? um You know you want to talk about anything else. uh You know. Where? Can you find you guys, um you know best place, you know kind of you know, I guess besides.

D

A

Where else can.

D

We find it first place to go check it out. um We'd love, to hear from you share your demo um yeah, maybe next time we hop on. We can make time for that too, but this was great chatting with you all. um Besides, at Pipe kit, dot IO find us on Twitter, pipekit IO and then um the best place too, is in slack so in the cncf slack check us out in the Argo workflows, Argo contributors channels, um yeah. If you have future requests bug reports, we're here to uh to hear from you and.

C

D

Help out, however, we can um yeah also looking to build the contributor Community. um So aside from you, know, JP and I and and three other developers on our team who are active on the repo we're looking to build. You know more Community contributions going forward, so don't be shy to reach out if you're looking to get involved in open source and push code. We'd love to support you in that um yeah other than that we'd love to see you at con I think we're gonna make an announcement soon about when that's happening. Yeah.

A

D

Yeah we'd love to see folks and chat there and, of course, kubecon will have a booth and we'd love to hang out. We.

C

Got a good blog that uh Caleb has put together and I forgot to give us credit for it. So I got it awesome.

A

It's also might see our.

C

Infrastructure engineer: uh Tim who's like really active in yard, River, close slack.

A

C

Fun, guy um very British, great sense of yeah.

A

A

Cool cool yeah, so um so yeah, no I, think I think that's it I, don't have anything else, Hillary anything anything else. You wanna end.

B

On doesn't even know what our next dream topic is in two weeks, I was gonna, say that.

A

I don't know I need to look at I need to look. We actually need to meet and see what the next, what we're gonna do next, so we'll we'll. uh We have nothing, nothing to tease out.

B

Yet I don't think we plan on the second half of the year, yeah.

A

Yeah well yeah shoot from the hip yeah exactly.

B

We'll just do it on like door, metrics or something oh I, know I, don't have a link to it, handy because I'm not responsible or an adult. We discussed this before the stream started.

A

B

um Next week for those who pay attention to the um the women in Tech uh community, that I run um on behalf of red hat for um the wit Network, which is a um a non-profit um that that we that we sponsor um next week on that it was posted on LinkedIn. I was tagged, I'm sure it comes up. If you search it, um I am giving a talk about get Ops infrastructures Code, why they're different and how this all relates back to the whole concept of CI CD and like delivery.

B

um So you know it's basically, uh if you've seen the stream you're probably familiar, but if you have people who are interested in coming to something a little bit more interactive, a little bit less recorded um or uh women and allies um and others in your lives, who are kind of looking for that type of community, just kind of Point them to that thing.

B

That I do because then, like the more people who come the more other red Hatters who I can get to agree to speak on things like so we'll have some people talking who aren't me about kubernetes with that community in a future date. We did a couple really great security talks um in the past.

B

We did blockchain at the beginning of the year so like we do cool things there, we talk about cool things there and we get cool experts in to talk about their cool expertise, um so that is happening next Thursday my morning, I, don't remember what time I'm, not an adult like I, said I just didn't like last last time: I did one of these presentations. I literally was editing the slides as people were joining the call.

A

B

A

B

A presenter- and that is what you can expect.

B

I literally, there was four slides that was all like. I was like it's for.

A

Me, it's always the presentation's in here. It's just yeah I, just putting it on the slide. It's, usually.

B

The longest thing I do I'm, not a video I'm, not good at it. You would get like just blank slides if I could get away with it. Our talk at argocon I had architecture drawings up there, simply because I needed to practice making them yeah.

A

Yeah, that's the only thing yeah yeah we.

D

Find wine a bottle of wine, helps with architecture.

A

Oh wine, there you go exactly.

D

Yeah well you're in Portugal, so I'm pretty sure.

A

You're you're enjoying some of that right now, yeah.

D

To draw a lot of architecture diagrams.

B

Yeah, it's fantastic uh I will uh take you or take your advice and report back with my findings.

A

Yes, exactly awesome cool cool all.

B

Right um that said, we are past the top of the hour. um Thank you folks. So much for joining. We really appreciated your your time and your company on the stream. um Thank you as usual for watching and sticking with us uh for a little bit late today, um I'll. Let Christian decide yes or no, if he's going to say some goodbye other than what I'm saying, but if not um choose your technical debt wisely, and we will see you in two weeks goodbye.

A

Cheers thanks for having us.

A