Cloud Native Computing Foundation Kubernetes AI Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Data Science Platform with Argo Workflows - J.P. Zivalich, Pipekit & Will Wang, Bloomberg

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Building a Data Science Platform with Argo Workflows - J.P. Zivalich, Pipekit & Will Wang, Bloomberg

How do you automate thousands of data pipelines for a dozen data teams, each with their own pipeline requirements and use-cases, all deploying to multiple clusters in multiple regions? Bloomberg and Pipekit have taken their own unique approaches to solving data operations at scale using Argo Workflows and a multi-cluster Kubernetes architecture. Learn how these Argo users are creating reliable and scalable data science platforms that enable data scientists to self-serve their data engineering needs, from creating DAGs to analyzing logs and metrics.

A

You so much for the kind intro.

B

A

Foreign, thank you so much will um yep I'm JP zivilich I'm, the CTO and founder of pipe kit. We are a control plane for our good workflows, I'm, currently based in Paris France, where in my free time, I love to butcher the French language, all right.

B

B

Like doing model training, Works, videos and stuff like that, are like a officers together and as a result of that, sometimes teams are starting out there. Healthcare service.

B

B

As a result, we wanted to do.

A

A

Thank you will yeah. So when we set out to build Pipe kit, um it was kind of roughly the like 2017-2018 era, um and we spoke to a bunch of data scientists and Engineers about their problems, and they all mentioned, like Hey we're running into problems with dependency management and scaling, and the obvious solution to us was like well. Why not just use Docker and kubernetes?

A

That's like what those things do um and they said that the tooling wasn't there, the dev experience wasn't mature enough, and so we got to work on trying to figure out how to bridge that Gap.

A

We then found Argo workflows as being the workflow orchestrator that seemed to be winning in the cloud native like workflow, orchestrator, Wars and said, like all right, let's dive in a little bit there that led us to find that many people were building out these workflow orchestration platforms or these data science platforms internally and spending a lot of resources doing it. So we said hey, why not uh like use our expertise and start building that out as a service.

A

So why are your workflows you asked well, first of all is automated.

A

Software flagstone I guess a lot of open sort of FDA is available.

B

A

So there's a set of common requirements that both of our platforms had. um Obviously we all wanted to use like kubernetes for our compute layer.

A

um Both teams agreed that Argo workflows was the orchestrator of choice that we wanted to go with, because it's built um using like kubernetes native Primitives you're, interacting with like pods and all the same things that you'd be interacting with like if you're just like writing. You know any sort of like application definition. We also needed to both schedule jobs across multiple clusters, from a single control plane for Bloomberg's team. Having like a consistent rest, API was super important and for us on pipekit.

A

In addition to a rest API, our customers were asking for like tight Integrations with their git providers, such as gitlab GitHub, and also a CLI that ran on top of what they're already using for Argo workflows going along with that. Our customers needed strong role-based access control. So they could say what users have access to: what resources on what cluster, which expands like an extra dimension and than what you can currently get.

A

If you're running like a single cluster setup with Argo workflows and then one other thing that was kind of interesting, is we both wanted to set up some sort of workflow template versioning within, like a vanilla, Argo, workflow setup, you have access to workflow templates, which is how you like reuse code, but you only get like one version, and so we wanted something similar to what we do with like Docker images where you get like hey I've got like my ID service or whatever and I've got like a.

A

You know, however, million like tags that are associated with that. We wanted to replicate that for like easy rollbacks with Argo workflows, templates.

B

So, okay, it is.

B

A center platform, so we figuring alcoholic interface so that our users don't need to worry about which they're connecting all the time. We also want to have a favorite concerns, and it's been a little bit earlier.

B

Purposes so we want our users continue using those platforms. There are dedicated themes for dedicated purposes, uh so what we did, though, is we build a centralized way for communication between other coordinates clusters, and then we offer a library of possible content based generous way to go to safety here, because for.

B

We also valued stability, and so it's likely well, we've already won our user to run whatever they want. So we only allowed us um and then we want our touch base. Their names can follow a semantic version standard. So originally we need to name their local template. Template, like my own content like 1.0.0 and we'll take those simple equations with Evernote and terminal is very awesome, open source technology. It is a really easy way to register our vitamination and find any whiteboard.

B

And lastly, our local templates must be promoted into production environment. What that means is we don't allow local templates in product.

B

Teamwork in development environment first and then they need to use our managed API for copy, and then we added that sort of industrial recovery and first so wait in the event of one has ever been a white balance, hurricane or whatever, where you don't want to know your certain notice and people so, but the Argo comes, are until we use an internal source chronister to our constant and then. Lastly, we want to limit our selves, we're going so we'll get themselves and provision in many states and many different individuals.

A

Great, so we have some pipe kit, specific requirements and approaches that we needed to tackle when we were setting out to build our platform, so they can be filtered into like three broad areas. The first was ease of use. The second was security and then there's some just kind of like additional I, don't know, let's call them like junk drawer features like you know that, like draw in your house where you're like yeah I, don't know where I'm gonna fit like all the other stuff, we'll put them in that category.

A

So when it came to uh you know just like using the platform again, our customers had like three things that they wanted. One was like acli, where they could schedule workflows on different clusters from that single spot, essentially something that would be like a drop-in replacement for the Argo workflow CLI, which performs wonderfully for like a single cluster setup. Second, they wanted uh like an HTTP API. So if they, you know we're using like some sort of like web hook or something like that, they could integrate into that nicely.

A

Specify like a workflow definition, specify like a cluster name and send it that way, and then lastly, something um that we get asked about all the time. Are these, like git provider connections, uh so connections with, like GitHub gitlab bitbucket things that are going to be. You know like a dedicated, GitHub, app or gitlab app, whatever the bitbucket equivalent is uh so that we can read those like web hook.

A

Events uh like pull down the the code that might be like hosted, uh you know, run things and then update whatever the like checks are on each respective platform uh with a good. You know like check if, like the workflow, uh succeeded and acts if it failed so on and so forth, um and that just does a lot generally to like make the dev experience um like much smoother and easier to use from the security standpoint.

A

um Initially, when we set out to build the product we were thinking like hey people want, like you, know, a hosted version of the platform right right. Well, that turned out to be wrong. uh Like people wanted to bring their own clusters, especially if you're running like data processing, um you know you don't want to trust that, to like a a third party, you want to say, like hey, I have X number of kubernetes clusters under management.

A

uh I want to bring these clusters and additionally, I want to make it so that these don't have to be exposed to the public internet. So we were a little Limited in like say, setting up like endpoints on each cluster. We had to figure out what is like some sort of like pull based or like what we went with was like a queuing based system so that you know we could easily um say like hey.

A

You know we're going to limit like the security footprint here and then lastly, like uh similarly was like bringing uh your own logging provider so again, initially, when we set out, we thought, like hey people, will trust us with their logs. That turned out not to be the case. People want to bring their own uh like logging back ends and that's uh something that we support as well. Then circling back in on the workflow template uh setup, uh whereas Bloomberg and Company was able to go with that kyverno base solution.

A

You know using like a uh very opinionated setup and a mutating web hook that isn't something that scales for like multiple, uh like companies that are using like the single platform. We have to be like very flexible in what sort of like workflow template names and tags that we allow, uh whereas like Bloomberg, was very intent on like hey. We need to limit you strictly to that semantic. Versioning like we as a vendor, can't be opinionated on what like an end company like wants to do so.

A

What we did instead was say like we're going to rely on the git provider to give that back end uh there and say like those are where the workflow templates are stored, and then the get tags whenever, like a user cuts, a git tag for that repo or that uh like workflow template file, that'll serve as like the tag which uh the uh where Argo workflows like runtime will then have access to.

A

Hopefully, it'll make sense when we get to the diagrams anyway,.

B

B

It will send a request or deployment API in very first building include the Manifest algorithms, as well as some criteria for selecting a clock that you put an API with the resultant cluster and certain data Enterprise to the cluster. This is what an automobile kinky. So, basically, our camera work towards the wheelchair and.

B

B

Along with some metadata into our Supply Group studies, external training command cluster is.

B

Inside each cluster will have a filter that constantly representing with whatever team the activity of their closer with what? What's in the external truth. So any update to the resource index terms of the truth will be reconciled and updated in SPCA along the way from the center without a kind of country texture, and that study collector will also check the workflows inside the ICD and it starts with recruit. Based on the configure.

B

Historical and then a another important component of our romantic studies to our description.

B

So a schedule workflows basically can be opened them into straight Parts without them drawn while we're on Twitter and would have runs. So basically, each job is to schedule down itself and the call trigger is called and you're just remote and the royalties. So execution of the schedule. Job.

B

Here, where they can get another sort of patient-based report.

B

That calculates drop runs based on the particular sport in the external database, and then we have another attitude or even inside each password, foreign.

A

Thank you. So much will so. Our architecture follows like somewhat similar patterns, but there are some restrictions that we run into since again, we're not just serving like one company, but several companies so we'll break it out in terms of layer. So the top layer is how our customers are interacting with the control plane again. I mentioned that CLI HTTP API and then the git provider like backends next, each one of those is going to send what is essentially like an HTTP request to our hosted control plane.

A

Now, with Bloomberg's approach, you saw that they use like postgres as a source of Truth for State across their multiple clusters, and we use uh what is our hosted control plane, which is essentially just a postgres database, but with API layers on top of it right like if you're, connecting like multiple different people to like you know a similar database or the same database, you need to have good Primitives for authentication uh and things of that sort, so it wouldn't be safe to just say, like Hey we're going to open up like a postgres database, and you know let everyone like directly connect to it.

A

So the pipe kit control plan serves as like a like Central API layer for storing State uh for all of our different customers. Now next, we've got this message: clear, I'm, going to briefly jump into that and get into what, like the individual clusters, look like. So, whenever a customer like registers their cluster, they have Argo workflows installed on that cluster, but they also have the pipe kit agent, and that does a lot of the things that you saw will cover uh previously, such as like garbage collection and things like that.

A

But it also reads in our case from these uh like message: queues that get spun up one per cluster or I. Think now, we've done it with like multiple message: Keys per cluster right now we're using sqs as our message queue provider of choice, but we're thinking about moving to redis to have some other like nice, Primitives such as, like canceling in-flight messages and things of that Source sort.

A

So each cluster, then you know, is reading these message, cues and saying or listening to events that are in emitted by the pipe kick control plane, and once we get that event that comes in, that tells us like hey. We need to schedule like an Argo workflow on like this namespace and on this cluster, or actually never mind it's um in this namespace right, since each message queue is dedicated per cluster. So if a cluster says like hey, you know this is the uh the message comes through.

A

It knows I'm going to schedule uh like this workflow on this cluster and throughout the life cycle of that it's sending HTTP requests back to our like, hosted control, plane to update the the state of what the workflow is doing and so that each one of our users can have insights into what's happening across their entire fleet of clusters.

A

All right now, Let's uh bring it back to some like contrasts and comparisons between our two solutions. So the first problem that we both tackled was like multiple, our multi-cluster scheduling across different sets of uh clusters.

B

um But the problem is stored in a decent terminal sort of Truth, so we'll basically relying on other teams to manage those for us. Something would get around at Bloomberg. So I have dedicated games for databases, but that might help you always.

A

Available for everybody and at Pipe kit, we also went with an agent-based uh approach, the difference between being that we are relying a bit more heavily on message cues, given that we don't have as much uh like direct API control and control over the infrastructure. That Bloomberg does we're, also providing that like hosted, control plane with uh you know again, a dedicated API, Pi layer providing that source of truth. Instead of the direct access to the postgres database,.

B

And the other four times.

B

A oracle-time diverging problem: it's not available.

B

We have used and sell local temperature.

B

Will have to either either you and Twitter will have the clamp here. Both versions like e0.

A

Effective different kind of using anymore or would I have to use the process kind.

A

And a pipe kit um as much as I would love to force everyone to use Sunbird because I think it's you know awesome. We just can't be as opinionated as like the Bloomberg platform team uh can. So we set things up again um to just read: workflow templates from git tags, and that gives people the flexibility to you know make like tag whatever and just kind of send it um which again you know. I I would like to be a little bit more opinionated on that one, but that's not always realistic for everyone's needs.

A

All right. Let's get into a demo, so I'm going to go ahead and show uh importing a cluster in the pipe kit. Ui we're going to go ahead and create this cluster within the pipe kit organization, and we've already got one cluster running but I'm going to show you how to install uh like Pipe kit on my Docker desktop cool. So we went ahead and created this like cluster, manifest I'm going to go within our terminal here and apply the yaml that we generated. This is going to spin up our Pipe kit agent.

A

Currently, we already have like a functioning Argo, workflows instance running that you can see here right. We've got the server and the workflow controller and we've got a 30-second. uh You know Readiness check just to make sure that uh everything's uh working looks like we created our message cues successfully and in seven seconds this guy should be live yeah all right, that's gonna be left cool yep. It's live now all right. So now that this is live, we can demonstrate how to submit workflows to multiple clusters. We have a cluster, that's hosted remotely.

A

That's called like a runner uh cluster, and so first I'll show you like what uh yaml we're going to send to it. This is just some like dummy test. Yaml. You can find this in, like the open source, Argo, workflows, setup, it's going to log. You know a whale to the console, so first we're going to submit this to our like uh remote cluster, and we can see that we've spun up a new workflow.

A

It is currently running we're going to get the logs of that workflow available to us, and then we can see on our local, like Docker, desktop that nothing's there. Since right, we submitted it to that remote cluster. But next um you know when we submit another workflow. We should uh in this case be able to see like hey. This workflow is running on Docker desktop, so we went ahead and submitted it or not yet I'm going ahead and submitting it here and we're specifying JP Docker desktop right.

A

That's our new cluster um and boom I'm, going to refresh the page here and we've got another dag that's been declared here. We can see like this is the state of the workflow that's being run on the second cluster, that Docker desktop that we just imported, and so we have the logs coming in live and we can see like the global state of the workflow. And if we go to our like kubernetes visualizer, we can see on our Docker desktop that again, this workflow is running.

A

The pods are getting spun up and we have access to information about these pods right. So we can see on the first cluster the workflow has been completed and if I go back to the docker desktop cluster, we are completed as well.

A

I think yeah. That should be it all right.

A

A

I think do we have time for Q a and seeing the the 10 minutes in the back? Are we.

C

Gonna yeah: well, we probably have time for like one quick question, so get your hands up fast and then we'll do it.

A

You got to make this question yeah we'll go over! That.

C

um Yeah I love the approach by the way. That's really cool to see like um you know how, like a company, implements it as well as a vendor like alongside, and the compare and contrast right, I, don't think I've ever seen a talk like that. That was awesome. Oh man.

A

A great presentation: do you plan to integrate it with algo events uh like integrate with Argo events. Was that the question yeah um yeah as a vendor I, don't know right, that's something that um you know we gotta like just pull our customers figure out like how tight of an integration they want or if they want the integration- and you know, go from there. How about Charlotte Bloomberg.

A

Okay, thank you. How good.

C

Great um all right, I'm not going to keep everyone because I think it's time for a break now, um but obviously you know go and refresh yourselves and then come back. We have three fantastic talks lined up afterwards, um so yeah. Thank you again. Let's have a round of applause for um will and JP.