Argo Workflows and Events Community, 15 Mar 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Argo Workflows & Events Community Meeting 15 March 2023

Description

This Argo Workflows Community Meeting covered updates from three other open source projects in the Argo ecosystem:
- Numaflow - 8:30
- Hera - 49:50
- Argo Helm - 5:17

Community Meeting agenda, notes, and schedule: http://bit.ly/argo-wf-cmty-mtng

More Argo info:

ArgoCon 2022 -
All ArgoCon talks on YouTube here: https://www.youtube.com/watch?v=ZaQ95WNI-qs&list=PLj6h78yzYM2MbKazKesjAx4jq56pnz1XE&index=1&t=0s

KubeCon North America 2022 -
AI Day - Building Data Science Platform with Argo Workflows - Will Wang, Bloomberg & J.P. Zivalich, Pipekit
https://www.youtube.com/watch?v=rLPuTL60T4c&t=0s

AI Day - Building a Batch Processing Platform with Argo - Rakesh Subramanian Suresh, Intuit & Aroop Maliakkal Padmanabhan, Intuit
https://www.youtube.com/watch?v=JQr16laspSs&t=0s

Multi-Tenancy For Argo Workflows And Argo CD At Adobe - Srinivas Malladi, Adobe

https://www.youtube.com/watch?v=aTgINAFV3T8&t=0s

A

All right we'll get started, then hey what's up Flav I just saw you're here too um just so. You know, flap you're, up at the end, you'll be bringing it home today.

B

A

Yeah welcome everybody to the are the workflows and events community meeting um thanks for joining. We have uh agenda today, a few announcements to kick things off, and then we have two presentations updates on projects related to the Argo ecosystem, so I'm excited um to get those updates from jam and Derek and Flav.

A

um So all that folks join us we're continuing the announcements here. I wanted to make sure everybody could see um that the Argo con schedule is announced. So if you go to the link in the Google Doc and by the way thanks everybody for putting your names in um you'll be able to see who's, who's speaking um looks like there's two tracks so two rooms, so one for Argo, CD and one for Argo, workflows and I, think Argo rollouts as well, um and then there's a good amount of lightning talks as well.

A

So we hope to see you all in person, if possible in Europe, but understand that's not always easy, so uh we'll catch you online. If not, and um that is that I yes, one thing to note is the in-person event is sold out. So if you don't have tickets, uh DM me and I might be able to help you, uh but if, if you want to go good luck, we'll figure something out um yeah next announcement here is we have added to the killer Coda course. This is a free intro to Argo workflows course.

A

So, if anyone's new on your team or you're, looking to um yeah Refresh on the latest Argo workflows, we added a module here about using helm to install and manage our workflows, and then we also added a module. That's pretty handy to run all of the examples that are on the uh the open source repo, um so everything there is pulled in um into here and can be ran in the browser. So that's a nice way to you know quickly. Try a new example.

A

If um you don't have everything set up locally already and next, a quick announcement. uh Just a couple of shout outs to folks here: um yeah we've had some new contributors, so just want to give a shout out to Sandeep um for this quick fix he put in some nice work. Even A one-line change is welcome. So much appreciated. It's a great way to get started. I think uh Sandeep also was able to use code spaces on GitHub to run workflows.

A

So that's something we'll be trying to document a bit better for more folks to use getting started in developing I. Think Alex Collins worked a bit on getting that set up so shout out to Sandeep and Alex on that front, um shout out to Yao I, think you're on the call too hi yeah um welcome and congrats on your first PR.

A

This was a great fix and um yeah excited to work with you more um do Let us know if there's other PRS you're interested in working on uh so yeah, quick Round of Applause for Yao, who actually is on the meeting.

C

A

Yeah you're welcome uh another shout out um for a config Mac fix that got done. I think this might have been a regression, so thank you John for hopping on that and putting that fix in so the core teams didn't have to um and a shout out for another I think it was a quick regression in 3.5 that got fixed really fast thanks to Petrie um so yeah. This was also Petrie's first contribution so high five.

A

Thank you for putting that UI fix in, and one last shout out to isatha uh for getting a couple of long-standing books closed out um great to see these so yeah thanks, everybody who's contributing. If you um are looking to put in your first fix uh just, let us know open up an issue on the repo, a mentorship with the mentorship label and we'll Circle up with you and help you get your first PR going.

A

You can also always DM me and slack at Kalin and I can point you to someone who might be able to help in the part of the code base that you're working on all right. um Let's see so from here uh on the agenda I'll hand it to JM um for a Argo Helm update whenever you're.

D

A

D

I'm Jan Jason Merida, fully uh staff engineer: procore I've been a maintainer of the Argo Helm repo for a bit now with a Marco kilchifer, um we are doing our best to make sure that the Argo Helm charts stay up to date with the Upstream projects.

D

um It was pretty bad for a bit there. um We got it fixed up now, I think the longest we've had is a day where we fall behind an upstream project. The biggest highlights now right now we're trying to get oci formats released, so we can have those available to Consumers we're also trying to update the documentation and the readme's to make sure it's more easily um consumed and we're open to all PRS and feedbacks.

D

We've opened up the discussions board on the GitHub repo now so people can actually ask questions if they don't want it to be code related or more about process and we're open to all feedback. um We were originally trying to stay as much in sync with customize as possible, but Helm has its own quirks, so we're trying to work those idiomatic philosophies in place. One of the most common ones is: why are the crds inside the templates folder?

D

It's because we want to be able to customize some of the annotations and also the updates in Helm. 3 are not very friendly if those are, if they're not in there so yeah um again, PRS are welcome. All feedback's welcome no nobody's married to the code, but we do have room for improvement on some things.

D

Anybody have any questions.

D

And I know uh Kalin and the pipe kit folks who've been showing interest like they added the the killer, coded thing, Fargo, Helm and other stuff's potentially coming down the pipe so uh as much exposure. If you have companies or friends that are using helm and they need assistance, I've been trying to be very active on the Argo home, charts, uh slack Channel and if you all see anything in your Upstream uh channels that are Helm specific, please redirect them to our home charts.

D

We kind of got to start redirecting folks so that you all don't get overwhelmed with those questions. So.

A

Cool. Thank you for the lift with that and slack Jim yeah. Definitely awesome yeah any questions for Jan.

A

Awesome yeah thanks for the update, um I, don't know if there's a quick show of hands if anyone is using Argo Helm, um you know, let us know we'd love to see who's interested in getting involved there, but um yeah really great to see the project, uh the sub project, keeping Pace with all the other Argo projects. So thanks for the the leadership on that Jason and and team of folks all over the world, maintaining Argo helm definitely.

D

A

B

D

A

Next on the agenda was Derek Wang and I want to hand it to you for an update on Nema flower and Numa proj generally.

E

All right uh thanks, Kelly, thanks for killing them for inviting me yeah sure we're a player to be here. um Let me share my screen.

E

All right, um oh sorry, no.

E

A

Always do that too.

E

Okay, um so let me start with some self in reduction in the project background in production, my name is Derek one and I'm a software software engineer working for Intuit I'm, also a project leader of Argo event um and in terms of this project nimaflow.

E

um It's actually started like 15 months ago, and um the reasons for that is like um so, first of all, my team into it is like platform analytics and observation and from the name you know that we're going to do a lot of real data processing and real-time data analytics right, and so we were thinking about if, if there's, if there's a way to use um a tool to do in-class data analytics without going, you know outside of the cluster, to do some. You know external which, which has some external dependencies stuff and internal.

E

We use Argo workflows a lot, but you know our workflow, as you know, can used to do batch data processing, but it's not pretty good for like streaming data processing. So that's the reason we started this project and um so, first of all, uh it's a you know. It's a parallel. Real-Time data processing engine running on kubernetes Institute random uh to do streaming data processing. As you know, the overall definition for pneumo flow uh so features um first, one is the kubernetes native I mean um you know.

E

I, don't need to mention too too much on this like if you know about kubernetes, and then you do how to use numer flow, because we we wrap the data processing jobs in numberflow. As a crd, it's a query object, and then you get your data person's job running and lightweight, and the name of like mirror in kubernetes cluster. Also, um we ever try to run Nemo flow in a Raspberry Pi.

E

You know it's pretty interesting so um and next is so I can language agnostic so because a data pressing job running with pneuma flow is, is the actual prominence part. So you can write your data processing jobs with whatever language you want to use and we claim to be um exactly once semantics and which means there's no data loss during the data processing with email flow, and we also support Auto scaling out of the box without any using um without using any external Auto scaling strategy.

E

But if you want to use you know, for example, you want to use HPA it's doable and we also support. uh We have the watermark Autobox support.

E

The watermark status is used for um aggregation type of data processing uh for a data processing Pipeline, and we do like an apple reduced type of data processing uh in terms of use cases, um as I mentioned in internal oriented, we do real-time data analytics and or if we we do like even driven applications such as normal detection, monitoring and learning stuff, and yours can use pneumo flow to do a streaming. Data processing like do the data instrument or data movement. So.

E

um Yeah skip the left part of why we're building just to look at the right logic, diagram for the pipeline, so um purnima flow right now we support a line or three Dax for data processing, but uh idea that we're going to support a diamond kind of Dax as well. But you know it's not there yet so from this diagram, you want to see uh with Eminence pipeline right for each pipeline. We have you know each pipeline contains multiple. We call this noses vertex.

E

So each pipeline contains multiple vertices and there are three different types of the vertices. In a pipeline from the color, you can tell there's a source of vertex which used in ingested data from the external data source, and there are some user defined, a function vertex and where you run your own data processing logic and then after processing the data you forward data to some sync covert X. This SIM card actually used to write the process data to a real data single here you know from this.

E

The data will be read into Kafka or Pulsa right.

E

um A select, The, Logical View and here is a simple example to write a number for Pipeline and run it in a comments cluster as a spec. So you have a con. We have a new crd named Pipeline and you give a name- and there are two major sections in the spec. One is a vertices where you define all your data processing nodes.

E

So, for example, here for this one, we have a red X named in which is used to serve as HTTP endpoint to read data or you can you know you give some other type of data source like Kafka or any other type of source, with support and there's a processor vertex, it's a user defined function or a short name is like UDF, and you you give your a container, which is a you know, image you build to run the data processing logic and after processing the data, and we have another um vertex name out which is used to write the data to another cover topic and there's another section in this spec.

E

It's called Edge, which is used to find the relationships between those vertex. So for this one you have three notes: three vertices right and then the relation is like the flow is like from in to processor and or from press and then from presser to out. So that's a that's a way to define um pipeline as a crd object.

E

And uh moving forward I'm gonna um spend most of time on on the introduction about the data plane to see how it looks like after applying a pipeline spec in in into a kubernetes cluster and how it works there for those data processing and those.

E

This is a high level architecture of data plane. So, for example, uh just if we, if we apply the spec as I just mentioned to your kubernetes cluster in our controller, will generate you know some parts for each of vertex.

E

So, for example, this Source, um you know you know we are generating some parts for it to used to process data, to read data from from a data source and similarly for UDF, instinct and there's one major piece. I didn't mention uh here now is which is called uh Interstate buffer and short name is ISB and um in pneuma flow we, the communication between those vertex vertices, is not. You know, directly a call from a source to a UDF or from UDF to a Sync here.

E

So, instead we use a with something called ISP to transmit the data so for, for example, this Source vertex the pause read data from a data source it right to ISP, which is a in Walker queue.

E

It's a message queue and then the next vertex you read data from from the message queue and do do some processing and write to the next um message queue and which means for um for each of the two vertices there's a message: queue sitting between used to used for data transmission, okay and the logic and The Logical running in each type of vertex is similar, no matter the source, a UDF or a sync.

E

So the read data either from a data source or from ISP, and they the proper they're, going to do some data processing for Source there's in the data process. But it's like you know into the skip that logic and then right to the next ISP or right to data sync, and after that you do acknowledgment on from your data source or from you know, previous ISP.

E

So uh we uh abstract is logic for each type of vertex and that's a high level um data processor logic happening in Nema, Flow, overtex, Part, okay um and next is about the ISB I just mentioned the reason we want to have a ISP component sitting between each two, every two um vertex is. We want to manage the whole process of data processing. We don't want to detect the back pressure if we go back to the diagram, so data from this pipeline.

E

The data move from the left and to the right, but it is possible that you know there's some back pressure happening so, for example, for a sink there's there's some um um pressure to write the data data sync and then um we need to get this kind of pressure um passed to to you know to the previous Red X. To that we're going to you know, do some strategy to slow down or either stop writing the data.

E

So we need to detect this kind of back pressure and with this ISB a components in between we, we manage. We fully manage those um buffer uh buffer, the message we can detect back pressure and then we'll control all the flow.

E

um Also with this, we can achieve Auto scaling stuff, so our order scanning is actually based on you know. One of the factor is the back pressure step and that's the reason we want to introduce back an ISP component in into the data processing uh platform, so actually the ISP in its buffer.

E

um As I mentioned, it's a worker queue um so technically message queue technology having this kind of a requirement can be um use as ISB. Today we we support Reddit stream and and ingestion from nas as instant buffer implementation, um but you know, as I mentioned, this is the plugin. Technically you can use um as as long as you implement this kind of requirements you can use as ISP okay, uh this slides it actually is used for a deep dive for internally, so I'm going to skip some other slides.

E

um You know this is you know how the ISB worked internally, you know skip, and this one is the thing I just mention uh what the logic happening. uh In every vertex past we read from buffer or data source, with power, UDF or not, and or, and then forward to the next ISP or data sync and then acknowledge right, and we do a infinite look in the vertex path.

E

um You know, and then we have some logical to control. um So, for example, before we acknowledge a message from from a lost buffer or data source, we will not read data renew data, so this makes your um or the message will be processed and when we forward the data to the next buffer or sync, um we have some mechanics to make sure the data will be only written once and the whole logic here mix.

E

Here we achieved the exactly ones: delivery guarantee uh I'm going to skip this one as well as, like you know, how do you implement uh ISB stuff? That's something I just mention how we achieve exactly once quickly go through it and then next slide. So it's about the awarenesses we support. Today, as I mentioned, we support a source user driven function in in syncs, okay, um so first one in the source is quite you know it's easy to understand. The source is like used to read data or interest data from a data source.

E

Technically we should support uh any kinds of popular data source for Kafka sqss. You know, but the SQL is not there yet, but it's easy to implement actually and yeah. The source is quite quite uh straightforward and one one of things in the real map is which we will support um user-defined swords, so we're not going to support all the sort. You know sources, but we should. We will give you a chance or give you the capability to implement your own source um where we call the user defined source, but it's not there.

E

Yet in cinema, roommate um and next part is about user event, function vertex. um Today, the user function has two different kinds of processing like once for math one is a map when it's reduced. It's a map reduces a common um data processing approach in in the in the data World. um So for us we we do the similar things.

E

So the UDF is actually a player. It's a it's a user to find a function, and um you know you which, as a user, you use the user function to process. Your own uh process, a large process, the data and apply your own Logic for it. You can use whatever language you want to use in in your user function, and uh this is a simple diagram about how a map type of user function works so for for the vertices, if it's a source Instinct um the past generator.

E

For for this two type of the word, vertex, um there's only one container in the Pod, but for UDF they're going to be like two containers: one is the main container and otherwise a UDF container, the main container used to read data from ISP and also write that into the next ISP and your user-defined logic, user-defined function. Logic is actually running a a different container and the communication between these two container is is through grpc over use Unix to main socket.

E

So um here the the UDF is the Japanese server and the main Canary is a client today for the for the math functions we uh the grpc protocol we use, is a unary course, which means um you send a request and you get a response and the logic here.

E

As I mentioned, you know read from isbe and then the maintenance read from ISP and then send the data to the UDF container process data, get the response and the right to the next ISP and then the acknowledge message in the last ISP, and this is a sample code. If you write a single link, if you write your own data person launching goal and what the interface looks like, so you implement a function whatever we use a handle here right and then you're gonna get the data.

E

You know the data, basically the data in all the data parameters like the key and value so you're going to get it get the key of the data and also you get another option in the data within datum. There's most important things. A digital value is like a binary array and in this example, uh what we do is like we, this user function, just to do an even or odd process.

E

If it's even just the same to uh you, return message with the key, even in with the data, it's R returning, odd right so that you can do conditional, forwarding, stuff and and then use you know, use our SDK to store your um start, the GPS server and then wrap it as a Docker image and then put in you know in the spec. You get it wrong. We also support some building functions so, for example, some filter functions. You don't need to write your own UDF to process it.

E

You just use some building functions, um and so that's from map and for reduced type of vertex, um so reduce is like a you want to do some windowing operations. So, for example, you want to do um some uh do a sum or every one minute right. So it's like a this is like a fixed, windowing um or tumbling windowing or or you want to do some sliding window every one minute. You want to use um aggregation based on the past five minutes, for example, right. That's a one-day strategy.

E

We support today is like what a fixed or sliding- or um you can say- hey I, wanna, I wanna group. A key uh you wanna, you wanna, do some aggregation based on Contour code, for example, and then you can do group a key, and also you want to do some shuffling. um You want to do some parallel processing, for example, um you have in your throughput is very high.

E

You want to do compare the processing with sharpening and you can specify um the parallel folks I'm going to do five um concurrently right, so you can do that and there's something called PPQ we'll be introducing in the in the reduce operation. I'll explain um here so a similarly for the reduced type of vertex.

E

um There are two containers running in the pond. There's main container read from the ISB similarly from to the map, but um there's some difference here is: um after reading, the data from the ISB is going to pass the data to a pbq. The PPG basically is a story. Local storage.

E

Today we use PVC for it, so the data Written Letter, read from the ISB will be starting the lookup PVC and passive data to the user, develop function and then um let's say you do like um one minute window after one after one minute and we close the window and you got the response from from the user code, and then we write the data to next next ISB and then we we clean up data in the pbq and I'm. Sorry I forgot, you know mentioned the acknowledge.

E

The acknowledge actually happens when you start the Devlin pbq in and send it to the to the user function and name the acknowledge so there's some um processing logic, difference from the map, processing map and data processing, and the reason we we do that is you know. The aggregation operation is quite different from the map. So for map is you you send one request, you get one get a response per reduce. It's like you, send a bunch of data to the user function and then you get a one response.

E

So here um we still use grpc protocol over units Unix to main socket to to process the data between the two containers, but we use different um different mode. So here we use bi-directional streaming, so you send the data to the UDF container in a streaming fashion and then, after window Chloe after closing, closing the window, you gotta the response data in another stream, so there's some different there and for the user, the interface, it will be a little bit different from the map.

E

So here what you get is gonna is you're, gonna get one key and then you're gonna get a channel which used to pass the data from the you know from the main container, and here this example is like we do a sum right, so you get the data, get all the data from the channel and in the end you write a response back.

E

Yeah, that's a reduced operation, and next one is about the things since we we support some native, a native um type of things, folks in my Kafka logger or um any other. You know popular uh data data things which we will be. We will support those kind of uh data sources natively, but we also um provide some uh something called user-defined things, which means you can Implement your own data writing logic.

E

uh You know, for example, you want to write some write, the data to uh to a database, that's something we will not support natively, so you can write your own launch for that and um the data process logic. uh Similarly to the UDF, it reads the data, and then you pass the data to the ud sync container and where you write your own logic, to write the data data sync and then you give the response to the main economic data. Okay um and the user. The user interface will be something like this.

E

You get a you know, Channel, which, where you can get all the streaming data from the main container and after processing they give the response say you know you have processor data and they return.

E

um In next is about auto scan we as I mentioned earlier. We support Auto scanning Order Box and the auto scan is based on several factors: Penny messages or the we have isps. We depends on available buffer space in your processing rate and also we'll considered inserting back pressure so for most of the most type of vertex in in nimaflow. We support auto scan OtterBox, but there are some exceptions, for example, for HTTP type of source.

E

We we cannot do that because there's no payment message right, but for those two, those type of events or oh, this type of networks, you still can use um HPA to do it. So we support HP as well. um You can use CPU or memory or whatever or any other kinds of other scaling.

E

Yeah briefly, you know mentioning and correctness, um so the credit is the most important thing for a data processing platform.

E

So here you know for, for our Nemo flow is like and there's some data on out of other stuff, because that's the reason you know we have multiple for one word x with your model pass. So out of order is the it's a natural data processing platform and then, within uh with this kind of out of order, we still need to make sure um the data is correct. So we have some.

E

um We put a lot of effort on it, I'm not going to touch uh red deponent and the way we you know I'm going to skip all of this. It's like the way we make sure data is correct.

E

Okay and Watermark, as I mentioned um uh watermarks, are very, very important for for data aggregation today, when you do windowing no matter, it's a fixed window or slider window, even you say, I I on a in a group group are the key every one minute, but that one minute is actually not the processing time it's like. um So the time is is like a real event.

E

Time based on reading, for example, now it's eight eight o'clock or let's say 802, okay, if I want to do um like some every every one minute, so you cannot say hey no, it's 802 around the closer window from 801 to 802. Actually you know the data you're pressing, probably is still you know, 7 59 right. So there's some! You know, that's that's my natural data. The data happened in the past, but when, when it goes to your data processing system, it's not you know, um there's some delay.

E

So we we actually put a lot of effort on Watermark propagation as well to make sure the data moving to the system have the watermark correctly calculated and propagated, or the downstream, where this is and we use a watermark to make sure there's no late data to close the window correctly.

E

Okay, um yeah I'm not going to touch uh very deep on this as well and next part is about control, planning, I, think uh I'm gonna. You know use a very limited time to to mention this to introduce this because I I think everybody here are very familiar with the kubernetes related application development.

E

So um the control plane design philosophies. We want to make it as light as possible, um and um we only want the control panel to do resource, orchestration and management. It should not touch any data, plane, business logic and if the control plane is down um or temporary unavailable, they existing data planes should still work still be running without any issue. So that's the basic design Philosophy for control plane. It's a high level architecture for the control plane just to quickly introduce there.

E

So there's a there's: a control, plane, namespace or system namespace, which you use to install the controllers. We have three type of crds increasing in animal flow, one is called um High plane and otherwise the word X and the third one is called. uh As I mentioned, there's component neon fluid called ISP, so we have a ISP service controller, um so this ISP service controller. Today we support redis and just stream.

E

So um so, if you want to use Nemo flow and to to create a pipeline, you need to you know, create a internet buffer service. Crd object in your name, Service First, and in this object you mentioned what kind of ISP you want to use. For example, you want to gesturing you give some spec of chest stream and we're going to create adjustment customary in their namespace, or you want to use redis you either use a in-class or redis, or you want to just give it.

E

Some redis configuration in order to have thread is running somewhere. So you give this kind of crd object um to your namespace and then we're gonna either create the service for you or use your config.

E

And then you create a pipeline when, after creating pipeline, um we're gonna start some jobs, Commerce job to create those ISB within the innocent buffer service, and also we're going to generate some vertex object automatic for you, based on your pipeline spec and in those and then there's a Vertex controller based on the vertex crd object, we're going to create some parts for you and those parts will use to talk to the ispe, and you know: do the real data processing and there's a there's a you know: demand service.

E

um It's a department quoted by the pipeline controller, which used to do like data mesh query or some just some query stuff, so high level architecture is I. Think it's pretty easy for um you guys here to understand so I'm going to spend this time. You know uh very limited time on it and I think in this.

E

uh These are all the stuff I have in the slides I'm going to skip all the rest, slides, but there's something else I want to mention here is we we actually assume, as you see, um for the UDF using you're gonna, write some um code to implement the interface, so that can run in the pipeline. So we support some sdks.

E

Today we have the golden Python and Java support today, and it actually is, you know we didn't get a bandwidth to to implement other language, but it should be very easy to do that within with current examples and um yeah. Actually, I should do. I should show you uh some the real pipelines, we're running um oh hold on. Let me okay, yeah.

E

Okay, let me show you some real pipelines, we're running with the production data, so um this is a pipeline running in with the real production data has been there for uh running for for several months without any issue.

E

um This is a you know, pipeline only process, the map kind of logic. So you read data. This input is actually uh reading data from a tough topic and then there are three data processing vertices and doing doing some. You know, processing and and then in the end, it forward to another kind of topic, and you can see the throwback we we can achieve right now at this point is like about even 11k per second and um so for Nemo follow pipeline, our highest throughput. We can support today. So it's about a 15K.

E

um You also can see the The Watermark stuff. So basically, um this pass is processing the data. Now it's uh 10 15 43 right. You know, there's they're like one or two minutes delay for data processing um yeah and we are working working on you know, get to to reduce this kind of delay. Stuff yeah um there um I'm gonna, show you another another um pipeline. This pipeline has some aggregation logic there. So you also read data. Oh, it also has like more than 10 ktps.

E

We did a I think it's also from a tough topic and do some filtering it's a map operation and then you do a fixer upgrade aggregation, I. Think it's a uh windowing yeah, it's a group by and it's one minute one minute, you know a woman aggregation and that's you know, pbqs. We used as a storage as PVC yeah, and this vertex is followed by another um wondering operation and then in the end, right just two things yeah. So that's some real example.

E

I'm going to show you, okay, let me see what else do I have? Yes, I'm a router Maps. Today we are running, uh we have the 0.7 release and 4.8. We are doing some performance, enhancement and Improvement on, especially for reduce um and then 4.9. What are we going to do is like, as you can see today, we only support um around 10K or 15K TPS, but um that's our highest throughput. We can support today so want to um achieve higher throughput and the idea.

E

What we're going to do is like we do model partitioning for the ISP, and we already did some POC to to prove that you know with multiple partitioning we we can achieve higher throughput.

E

um Yeah I think that's all I have and questions.

A

Thanks a lot Derek yeah any questions from anyone. I see one in the chat from five. Do you want to jump in flab and mention or expand upon your question? Yeah.

C

Thanks for the presentation, Derek I'm curious outside of the potential for a lot of integration between pneuma flow and the rest of cncf products. Are there specific differences between pneuma flow and a combination of Pub, sub and Cloud events that are triggered by Pub sub events.

E

C

B

C

E

Yeah, first of all, the pneumo flow is that it said um streaming.

E

Data processing, engine, um uh the pops up or um our Cloud events is kind of data source and to understand right um so Nemo flow is you know so, for example, you use Argo Rock to do batch data processing, but you you there's no way to use our prototype streaming data processing, and then uh you know this pneuma flow is mainly for you know streaming data processing, it's not particular for one type of Event Source or you know what time data source is you technically have whatever kind of data source you should be able to use, use Nemo flow to um data processing.

E

You want to, you know, um do some data instrument or data analytics, and then um you know if our data to somewhere.

B

Yeah sorry, a similar question um as a follow-up is how uh kind of pneuma flow compares to other stream processing engines out there, like Apache flank.

E

Yeah, that's a very good question, um so we are not saying we're going to compete with Flink at this time uh there there are a lot of uh um gapsing in at least today, uh for example. Flink you know supports very high throughput as you as you um can see. We only support like the 10 15 ktps today and then we are trying to achieve higher throughput. That's one thing and the other thing is um flank is kind of a professional data processing which is widely used by a lot of data Engineers.

E

um The reason we want to, even though you know for internet, we we have link support. uh We have you know some platform. You know built on top of Link, but to use Flink you you know first off link only support a job or Scala right. So you only have you only can write your data pressing job by using in Java scale or in.

E

um If you want to use fling, you need to you know the process kind of complicated. You need to wrap it as a jar and upload to in the to to fling and get it wrong. You, probably you know, you know you do not have um lots of inside about what's happening there, if there any issue yeah and for us we we want to do like an in classroom everything in cluster.

E

um So, for example, you have something running your own cluster, but if you use fling properly, you need to, you know, uh get the hook to external Flink system by using some, for example, user type of topic right there topic and then you spring job to read from the topic right after link processing you you forward the data to another topic, and then you read it back right.

E

So that's uh it has a lot of expand external dependency and then we want to um use pneuma flow to eliminate this, this external dependencies, so that to make all the data processing in cluster um yeah.

B

E

Anyways, we're not saying we're: gonna compete with Flink, our our Focus will be. uh You know, real-time data analytics and you know at that data instrument stuff and also for new approach. We we have some some other projects, animal logical. uh We met a focus on AI Ops again, you know.

E

um Feature-Wise Flink has a lot of more features that we don't. We do not have, for example, data combiner. We don't support today.

C

E

um I'm not sure if the answer is a question.

B

Yeah, it does thank.

F

You yeah yeah I, think we can add on more things like uh it's like a Alpha version of Link in cloud like a kubernetes native version,.

E

Yeah, if link has been there for more than 10 years, yeah.

E

Any other question.

F

So just I was mentioned in the chat, with the fabulous like uh like so the complete the cloud function and Lambda all the stream processing, which is providing me any cloud provider has like a some limitations like uh the the processing time and resource utilization. But if you use it, it won't have any any limitations like as long as, if you have a your cluster, has like a resource, and the vertex will process that that I element with the and indefinitely the timings.

E

If there's no more questions in our things, everybody for listening and if you are interested in this project, you can you know this- is our um GitHub repository, URLs, anima flow and, and we have some good, a quick start. It can actually try this out I. Think it's very easy to start. We actually try. We have to test it. You can get installation in running two pipelines down in like 10 minutes, you're gonna see you know, you're going to experience all this uh stuff in 10 minutes and also, if you want to contribute.

E

Yes, more welcome.

A

Okay, awesome yeah thanks Eric um one question: I had really quick was if you could share more about any other users um yeah, like other use cases you're, seeing besides what you just shared settings hit for.

F

B

E

I'll see uh what's the best approach. I can share all this information.

A

Do you yeah, do you know if you're off the top of your head, like any other users that are experimenting with it so far, other use cases.

E

Yeah, um actually we have a slack Channel. um You can find it uh in our GitHub repository and we we have some. uh We have a use case from Toyota they're. Actually, you know it's very interesting they're using a buffalo to do so um uh autopilot and and navigation stuff. It's quite interesting. You can get those info from Snapchat as well.

A

Cool yeah I'll post the link to the slack channel in the community, Google doc. um I'll dig that up and if anybody has it feel free to drop it in the chat here. um Well, yeah thanks so much Derek. That was a really great presentation, um we'll be looking out for more pneuma flow talks at kubecon all right, but um yeah handing it off to you flab for an update on Hera.

A

um Take it away. Okay,.

C

Thanks Caitlin and Derek again awesome overall review.

C

um Hi everyone again I'm plav I've, been here before um I, always use kind of this. This set of slides to provide a brief update on uh on Hera. Specifically, this intro slide um to remind everyone that Hera is now on Argo, workflows and events, python SDK, um with its core feature and guiding principle always being and having been uh simplicity- and you know the main approach and or the main selling point is that you just inject your host and token, uh and you start delivering value through article workflows and events.

C

I have some brief updates on um on adoption. Last time, I was here we were on v420 with 36 contributor, 220 stars, and we went two versions up with some new features and we added more contributors and just slightly more popularity, and we have two new maintainers um Sam and Elliot, both from Bloomberg who have been uh contributed, contributing actively a lot to uh the new release which is um hero V5 and that's the main thing that I would like to Showcase uh today or and even update on so previously.

C

I mentioned um that Hera aimed to have feature parity with Argo workflows, but it never really had that um up to V4, because I think we abstracted away a bit too much and we kind of coupled we were implicitly using templates as inline templates, and it was resulting in large ammos that were not kind of reproducible with Argo workflow specifically, but now we've um made some changes, and now we are, we actually have full feature priorities.

C

So all of the objects all of the apis available for workflows and events are are part of Hera and in addition to that, um we've changed the dev experience and now, as part of CI, we actually um check that the um yaml files that are generated by Hera are exactly one-to-one with Argo workflows. Upstream example yaml files- and this is part of CI. It's part of um it's part of the local Dev experience as well, and it's documented in the contribution guidelines really briefly.

C

um I wanted to. Oh really, quick workflows are more mature than than events for now, just because they've been part of Hera for multiple versions, but we have upcoming um changes for events to make it more compatible with. With other hair objects coming from workflow, so you can put the two together and configure Argo events to launch specific workflows that you create via Hera and I. Have the auto generated RTD docs here and I wanted to show you a very brief example.

C

So now um you can easily you can instantiate a workflow and you, you notice that we now have. um We have separated out templates and actual um calls of these templates into the independent steps, tasks, dags HTTP data templates and things of that kind. So you can use all of these uh things and they're Auto generated the the documentation is actually auto-generated as part of CI, so that the Hera example is actually.

C

The yaml is also added to the same piece of documentation and the link is added to the Upstream Argo workflow example that you can see here um as well, and that is the one that's actually compared that can that is compared against in in in NCI, and there are a bunch of Upstream examples that are now part of the repository and they follow a similar style of implementation in Hera, the actual yaml output and the link to the actual documentation in the Upstream Argo Oracle repository, and there are also some extra examples um that are added that will probably go to unify with the Upstream one foreign we're still kind of working.

C

um It's still working progress and we're still kind of working on V5, but we're getting very close to um to the end and we're primarily focused on making things even nicer than they currently are, such as you'll see this contribution from Elliot, for example, that makes um templates callable similar to how you call a very simple function in in Python that you can then parameterize with the regular um Step or that task um template that you normally call that you normally you have to configure in in yaml and then, of course, you get the same yeah more yaml as part of the output of of Hera.

C

So previously, here, I used to use the currently published python SDK in the Arco workflows repository, but that is generated by open API and the service that is actually generated is very, very hard to work with, and we, we wrote. Our Own parser of the Json, open, API, spec of Argo, workflows and events, and it actually generates um API calls that are much much easier to work with and understand, um and it makes it clear about what you need to pass in just in case. You do not want to use any of the hair objects.

C

You just want to use the auto generated identic era models, so you have the flexibility to choose between the two and the models. Automatic are also accepted when you work with Hera objects, so you're not bound to use only hero objects. You can kind of use them interchangeably with uh with these services, and this is a similar example- it's just uh the one for workflows rather than the one for events.

C

So the current plan is to release rc1 this week and hopefully add more examples of the Upstream workflow and events yaml files, hopefully officially released V5 relatively soon and from there we're going to work on improving support for events so more featurization and syntactic sugar and making it easier to Define um and work.

A

C

Events in Hera, together with workflows and then we're going to start working on Hera CD, so that means we're going to have a an Argo CD client that will hopefully unify all of these different um Argo ecosystem offerings into one sample python SDK. That allows you to work with everything um in in Argo.

C

Of course, no! No, the not none of this, would be possible without a lot of collaboration and Community involvement and support and I want to mention specifically Sam and Elliot or pivotal in developing uh most of E5 actually and I'm very thankful for all of their their help.

C

um That's all I had just a brief update on current hair development and future of um of heronite B, of course, so excited to talk to anyone about era and usage at your company.

A

Thanks Vlad yeah, thanks for the update excited for the upcoming release, any quick questions I know we're at about an hour, so folks probably need to head out, but any any questions for anyone.

A

All right, awesome! Well, yeah thanks fly for the presentation. Thank you Derek, as well, um really great to see everybody and JM thanks a lot for the update from Argo Helm hope. Everybody has a great rest of the week and we'll get this recording on YouTube soon um catch you at the contributors meetings or next month at the next or actually next month will be arguecon. So we probably won't have a community meeting here, see you at articon, either online or in person.

E

Thank you very.

A

F

Thank you guys.