Cloud Native Computing Foundation Serverless Working Group, 12 Mar 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: CNCF Serverless WG 2020-03-12

Description

CNCF Serverless WG 2020-03-12

A

A

Lemons how's it going can hear me Clemens.

B

A

B

A

Let's have an show with my microphone earlier in the week: yeah.

B

I have failed to upload the documents, even though I I've been editing like while, but I just missed the deadline. It's just okay, oh yeah, I'm gonna, so I'm already I'm mark down with that subscription document. So I'm gonna go and I'm gonna upload this in, like probably not today, but tomorrow morning sounds.

A

Good to me, he's got.

C

Out from quarantine.

A

B

A

B

A

You to stay home or do they just say it's like your choice,.

C

No well, there better be a really good reason to come to the office or be within six feet of anybody. Interesting okay, yeah.

B

My world hasn't changed, but it feels more like a burger now.

C

A

C

A

C

Sounders game: it's a big sporting event here well soccer day and then the next game is is canceled because we broke the rules by congregating in 30,000 people, geez.

B

Yeah I went I, went to a football game on Sunday that that, just that so happened and and yesterday we had our big local derby against Cologne, and that was without spectators, which was weird. So we all watched but like I, have my stadium, friends and watch on TV, but with them here and now. The discussion is whether they're going to go and cancel the entire league we already did and Damon, and that is like Italy in Spain without football is I mean that's, that's a real state of emergency for.

C

B

And Italy has all the restaurants shut down and all the stores and everything it's quite grave. Yeah.

A

Hey John, hey Tommy, hey, wait! A minute was that Tommy did you actually speak yeah? Oh my gosh.

A

That's like a first I must have caught you off guard there for a minute. You weren't thinking straight. Were you yeah.

D

A

There you go okay, I'm, sorry to pick on you, but it says funny: Oh,.

A

Yeah they're extending my sons, spring break vacation from college and they're gonna start doing classes online, which means he's at home now, which isn't you know a bad thing unto itself other than he keeps two really really weird hours and he's a boy. So he eats a lot um so he's at like he's up at like 3 or 4 in the morning in the kitchen cooking stuff, and it's and he's a boy so he's loud. So he getting very little sleep these days and it's very annoying.

A

I can't wait for this whole thing to be over.

A

Hey ginger good.

E

Morning, dag.

A

Hey Mike good morning, hey Kathy, welcome it's been a while. Oh.

F

Hi good morning, sorry I was on mute.

A

Morning, Jeff, hey.

D

Good morning,.

C

Related to the previous topic, I think Doug will like this. You know a company's been doing off sites, but because of budgeting regions off sites have now usually been on site, but they're still called off site because it's like you should maybe you travel to a different office and then those got cancelled and they're going to be virtual. So now they are virtual off sites where it was going to be an on site, but now it's virtual, but it's still called an off site virtual off site. It's just amazing and.

D

Even if they would have had an on site, it still would have turned into a virtual meeting. So it's just another meeting yeah.

C

That's hilarious, it's a be on the conference. Call all Bay. Oh.

A

Man all right, let's see who we have on now, Klaus. Are you there? Yes, LuAnn, not you whoa good God hold on a minute. My screen scared. Yes, yeah, not.

E

Sure, if I can still it right, I'm, so I apologize.

A

Vinay, are you there, yes I'm here and mr. grant.

C

First time, unmuting: okay! Yes, all.

A

Right cool all right: we usually wait till three after you guys are in new, so give it a minute or two. Oh.

A

Thank you. Thank you. Grant.

A

James, are you there.

A

Hey this, isn't your first call? Is it yeah.

G

It's the first one, I made it this one I actually signed up the wrong call initially, so I figured that out. Okay.

A

um Me a favor. They paste the link to the minutes. They didn't just put your company name, there's a link to the minutes. I.

G

Want to be clear, I'm not officially representing my company I'm here more as we talked about around the book, but okay.

A

In that case, I'll just say see yourself: that's fine.

A

All right, it's three after see, if I miss anybody, then we get started. Hey everybody! Okay! Let's go in and jump into it, um okay, community time, anything from the community. People want to bring up this not on the agenda.

A

No I'm not hearing any STK, um Scott's or Clemens, or anybody else who made the call last week was there anything worth bringing up.

C

B

C

Provision from oh three for us, dkz.

B

Ceo three is being the one that we still support and then tossing it yeah.

A

But talk about that a little bit later, this PR down here we talked about that a little and I did update that based upon our discussions. So we can talk about that again down there.

A

Okay, any questions for the SDK team, okay, Cathy since you're on. Is there anything you'd like to update the group on relative to the workflow stuff.

F

Oh yeah, I'm. Sorry, um no really! Sorry I haven't really got much time to work on that.

F

A

That's fine just give me the opportunity. Thank you all.

E

Right DK I put a link in for the SDK so group next week. I have an elixir SDK and the works. That I would like folks to take a look at cool.

A

Wait a minute or two actually I'll. Do it later. Yeah I had a link to that information into here. Just google. It draws people's attention, so it.

E

I can drop it in. Oh.

A

Cool that'd be even better. Thank you all right. In that case, let's go ahead and jump into the delivery stuff. So tell you what it's Clements I think yours actually might be shorter. Let's go you go first and then we'll jump over to my GP. Are we.

B

So what we did is my story. So what we did this week is we actually didn't work.

B

This document at all, but we broke out into a so I, took this basically after our previous call, had some text that's here and put that into a word doc as an interim step to preserve the the formatting and then I meanwhile, have this done in markdown, but I've done some clean-up on terminology and I'm turning turned this more into spec language, but I have nothing like crazy, but with everything that's going on I, just not been able to good to to complete the homework so I'm, sorry and I will I aim to upload that Markel documents as a PR tomorrow, I think one of the things.

B

So one of the key things we did in terms of clarifications is since pull and push were contested, and you know with some and I and I buying some of the arguments. I'm, trying I'm trying to sting wish between effectively who's initiating the communication, whether the delivery and and really and like what is the delivery model, so you'll see a Turing subscriptions with customer. Let's start with consumer solicited delivery and subscriptions with subscription manager initiated delivery.

B

Those are effective, the difference between pull and push to clarify, where that mean there and and otherwise work. This thing will go as far as to define effective what the API a looks like subscription manager. Api looks like in the abstract, but will not yet have the be down to the HTTP level definitions likely, but it will contain what we discussed in previous calls like the filter, the simple filters effects, etc.

B

But again, so that's that's as much as I have I'm going to take a look at the at the PR sometime tomorrow and you'll find a link to the document.

A

Sounds good looking forward to it. Thank you. Any questions or comments for Clemens.

A

Alright, not hearing any, let me bring up mics PR, so you can talk that.

E

Iii apologize for not having much to say, I have not had a chance to work through all over the comments. Yet it is just crazy week.

A

Short and sweet. Ok, any questions for Mike, you don't know. Seattle is in a panic right now, so Jack on the Y, any questions or comments.

A

Ok, so just set of curiosity um and the question: isn't that's just for you Mike, it's actually for anybody, but it just using your PRS uh as I sort of a guinea pig here at what point do you guys feel like? We should just merge this PR? Obviously, once all the comments are addressed, but is there any like minimum bar? You guys have in mind it's. We can actually merge the PR, or is it pretty much all the initial comments or address?

A

They can just merge it at that point which is 2p ours against it, because it's like I need a doc in the repo. You guys have any opinion on that. Just from a process perspective I, just wanna, make sure we're not missing some step here. That people think we needed to go through before we actually merge it.

B

We need to get to be able to comment effectively. We need to have the base stock in the repo, so we can review on there and it's a draft and I mean it's the same process as a movie spec.

A

No I definitely I think I tend to agree. The only reason I ask the question is sometimes it's easier to work outside the PR process. If you have large-scale changes you want to make, which is why we started. You know the Google Talk to begin with and stuff like that, I just don't want to preemptively merge this thing. If somebody thought from a process perspective, we should hold off a little, but am I turning away to really jump up and objecting yeah.

C

It's fairly hard to collaborate if it's not merged.

B

The original groups are saying this is this is ready, then I'm trying to get for our group is that everybody wants to the. This is what this is our result and then we're gonna go. So the document I'm gonna give give you all on tomorrow. Is it's not that yet I'm still gonna go and collect feedback from the group.

B

A

Anybody else have any comments on that.

E

All right, yeah.

A

E

I think on this one I'll make sure to note at the top that this is a working draft of the document. Ooh.

A

Yeah, actually, you may want to I think in the past we used to say, like oh.

B

A

I was actually wondering, but a p.m.

B

A

But there was, this is said, like version 0.1 RC or something like that or because I think that's that's the pattern we used to use with, because it's not technically a zero one. Yet right, yeah.

E

It's like a pre zero one. I can put I, see one yeah.

A

That's a and I might put something in there like that, look at than other than that it looks good any other questions or comments.

A

Okay, easy enough and thank you Mike for the SDK um moving forward. Okay, here we go so I came here for sure other than I'm, pretty sure Clemens. You said you were gonna. Take a look at this PR to make sure there wasn't anything funky in here. Did you get a chance to do that or did anybody else? Look at it? How many comments.

B

A

Well, I'll give you guys and particular Clemens a chance, though it's just a double check and just refresh people's memory. The must here and the must here are technically breaking changes, but the assumption from last week's call was that we assume that that's what we meant to do. It's just more of a a syntactical thing at this point, because I was always the intention for these requirements to be there.

B

B

A

I just want you, people a chance to look it over. Okay, any other questions or comments on this one.

A

Any objection to approving.

A

All right, thank your buddy. Thank you too. Okay, this one! Well, this one is not ready to merge, because if we like it, if you like the general direction, I need to make the similar changes to the other transport binding, so I'm gonna select. Let me hide the comments, so just refresh people's memory, there was a little bit of there was lack of clarity and a specification in terms of knowing whether a binary message is a cloud event or not for structured. It's obviously very easy.

A

You can look at the content type, but by neri it wasn't clear. So this text in here basically says: if all the required attributes and I believers for them appear as HTTP headers, then you can kind of assume it's a cloud event now, based upon slinky developers comments.

A

He had me changed the wording slightly to make it clear in terms of what that means to to think it's a cloud event. You know. Does that mean it is a cloud event or you can just start parsing it hoping it's cloud event or whatever, and so what I did is I modified. The text here, where is it.

A

Where is it oh yeah, that's up here if.

A

Talk about it be about the weather. The message ought to be attempted to be parsed as a cloud event and then down here I talked about how um you know tough, all the four attributes are there. Then you could start that process. However, just because those fruits are there does not mean it's, it's technically a valid cloud event. It still has to adhere to all the normative language in the specifications, or example. There are musts and their relative to that I, use and stuff like that.

A

If they don't meet those musts, even if they are even if they do have the attributes, it's still not a valid crowd event. I don't go into what the implementation does. If it's invalid, that's completely up to it, because we start at state out of the processing model, but I think this at least clarifies whether someone should at least attempt to parse it by right. The presence or absence of those four attributes.

H

About question here, so we are referring to the attributes in the HTTP headers, to increase the probability that we determine the problem that I'm struggling with them, and please help me if I'm mistaken is it still doesn't give us confidence that it could be a cloud event. So what is it actually helping? Is it clarify anything or as a is it just confusing it? A little bit more I mean. Does that make sense? It's very saying, then it's probably a cloud event.

H

Why are we not able to unambiguously determine that if those attributes do exist, that it is a cloud event right.

A

I think in my mind, there are two reasons why we can't say for sure one is: our specification cannot mandate that people do not use our attributes randomly right. So someone could say hey those look kind of like interesting attributes and I'm gonna. Stick them in my message, but they don't actually know about our specification right, and so the presence of those attributes technically doesn't mean anything because they don't adhere to the full-blown spec. So we can't force people, not user attributes. So their presence in a message does not guarantee anything.

A

It's it's just the it's just a hint that it may actually be a cloud event now, even if they, even if the person meant for it to be a cloud event if they messed up and didn't, adhere to the rest of the specification according to the spec because they didn't. You know, I'd hear to all the musts and stuff like that.

A

um If it, if it doesn't hear to those, it is not a valid cloud event and all we're trying to say here is we can't be definitive one way or the other, but we're trying to provide some guidance on what people should look for to try to see whether it actually meets the criteria right so visible. If it doesn't have those attributes in there, then they shouldn't even bother trying to parse it as a cloud event, because it's not gonna pass cut it correct. So it's just a it's a wonderful, featuring.

H

A

Kind of that's why you look at it technically, there's no normative language in here right. This is just sort of abstract guidance and I write it in a lot of ways. I actually thought about putting this into the primer, but I thought it. This might be important enough that it should go into the spec, because I mean I got more than one eye. People have asked me this question more than once, thanks.

A

Any other questions or comments does this seem like the right direction to go and if so, I can make the changes to the other specs as appropriate, or the other transport specs as appropriate.

A

Okay, not her any objection and I'll make the changes, the other specs, and then we can man me preview it and approve it next week. Okay, thank you, everybody, um okay, this one! Technically it's an SDK thing, but I wanted to give everybody else a chance to review it. Now, in last week's call, we were talking about what are the requirements for what these aren't high requirements right? These are pretty much just suggestions, but we're trying to talk about what are the requirements for SDK authors and last time we talked about.

A

Everybody obviously should support the latest and an N minus one of the major releases within a major release. They only need to support the latest version or the latest minor version. However, we do have sort of the bootstrap problem of what do you do when the latest version is one point, though, what is the N minus one at that point? So, according to or the agreement we talked about last week, I had a note here. That says one point is a special case and in those cases people should support zero point.

A

Three, so Scott think lemons. That is that, consistent with what I think we agreed to last week.

I

J

A

Any questions or comments on this: oh, yes, that's.

E

A shirt and not a must right.

A

This isn't but technically it's not a normative spec, so yeah yes, either way. Yes,.

A

Sorry John I couldn't hear you say that again, I.

K

Was actually gonna point out the same thing thanks.

A

K

Starting a new UK right now is problematic right. If they're gonna go backwards to support, oh three, so yeah it should just be, should not a month. Yes,.

A

Yep I think I have a bigger question. We did talk a little bit about that last week as well, but also keep in mind this isn't an order spec, that's why it's lowercase should and could instead of capital should, but these are all just brick home suggestions.

A

Okay, any objection to approving that.

A

Alright, thank you.

A

Alright! So so I put this on the list, even though Clemens raised an issue with it today, so some Googlers are working on to make pops-up binding for for their offering and and Clemens you pointed out hold on. Where is it.

A

Where did you say that Clemens, somewhere in there yeah no I thought.

A

It's painful what Clemens said.

G

A

So you want to raise the issue here. I, don't think is any objection, but just hope you understand why things may change radically. We.

B

B

What is acceptable official?

B

What is not cool pops up is a proprietary product and therefore we have a document that is the product proprietary products link document that takes links, so this spec um should be hosted in a Google or posit or even pointed to car.

J

The rules- yes, thank you payments for pointing out it was my bad. I I didn't know what the rule hey I'm, trying to move this so repo and they're, not the link here, yeah, but actually I'm glad that I made a mistake, because I had some really good comments from twist of them. That will probably address and I think I'm.

B

Totally against having that, it's great that we have coverage.

J

J

Yes, oh great, thank you.

A

So expect the I think a fairly radical change, the PR, but obviously, if you guys want to review it and provide feedback. Obviously not so it's not just said you know more feedback because it appears to be welcome, so get it in there quickly before the tech stack disappears. Alright and.

J

One question there regarding timelines, so when we add these, this might be approved next. We need to wait for a week until the next, a working group meeting.

A

Technically, yes, technically only approved PRS during the week unless they're, you know typographical like things. If you can get this change in there by Tuesday night, then people will should be able to repeat it and approve it by Thursday. Okay,.

J

Sounds good thinking about mmm-hmm and.

A

Jim your hand was up there for a second did. You want to say something.

L

I retracted, I retracting, my thought, I mean I was gonna, say given what's going on with this proposal here, I. Is there anything we can do or say around?

L

You know if you're looking to create a transport binding, but it is a proprietary thing that you know feel free to run it by us before you just sort of throw it out there in the world, I mean if, if the intention here is that this PR would turn into essentially just a link to another speck, is there any sort of onus on us to make sure that whatever that link is pointing out actually works?

L

That's really what I was driving at yeah. There.

C

Was never right, like the the the link to a third party, was the best intention. So if the spec is correct according to that vendor, then that's that's fine yeah.

L

That was my thought. I know: yeah I just wanted to raise it, but.

A

Didn't we either talk about or put text someplace? That said, it's within our right to remove things if we think they're pointing to old or back informations, like that, um you.

C

Know I think we had a provision to delete links, but not really the spec. That's linked yeah.

A

But I think I think Jim. You raised an interesting question, which is you know? How does somebody just ask for our review? Even though it's not necessary, you know I'm trying to think I. Think most obvious thing is me be happy.

A

They can raise an issue just draw our attention to it and then once they get, you know a couple reviews they can close the issue, but I'm wondering more right whether we should add some text someplace and either this dot right here explaining that they people have that option to open an issue to draw our attention to it or it's to join our weekly call the draw our attention to it, or something like that. Do you think that would be helpful, Jim, I, I think.

L

That needs to be something front and sent into somewhere. That says, you know if you're doing a proprietary binding that this is what yeah. This is what it should do. You know look at these, you know take them as a template and then you know open a PR on the page, we're looking at to cross-reference stuff yeah because they think that's where you're, maybe SDK people are going to get interested as other bindings start showing up so yeah I, don't quite sure where you put it that that's that's. Why I'm struggling in.

A

The interest or in the theme of no good deed goes unpunished. Would you like to take a first pass at a PR dad some text that effect.

A

I'll teach you to speak up. Thank you. Jim and I.

L

Knew need I'd be dropping off in a minute, so you won't get any more bizarre questions that.

A

Was a good one that I could, because that more guidance we can provide a B would be good.

A

Okay, cool all right next on the list and slinky developer has joined us. So we have this issue that he opened and then you made a comment down here. Would you like to talk about what you're proposing that we do with this issue? Yeah.

M

So the fix that.

N

The paragraph about key it just points to that partition. He assaulted the partition was named partitioning extension, which is just confusing because he is a really well defined concept in in the Kafka message specification. So just saying the key extension must map to the Kafka message. Kids, it's far more clear than saying there is a partition, key extractor that the extra things and then put inside a key. But then you open the partition accession and it's called partition key extension.

N

So it's I think it will be better to just remove the link to the partitioning extension essay. A when you create a called event that goes on the kafka wire. Then you put me to put the Kafka and I need to put the key. The message as an accession as an extension name key to meet, sounds for more clear this way than going through the partitioning extension and the partition TX structure.

A

Jim, did you wanna say something I always.

L

I must say: I haven't, read the full text here, so I'm not entirely sure what the problem is. The partitioning the petitioning of cloud events is agnostic, two transports yeah so but I think when that extension was added, it was there to enable a caf-co transport binding to take advantage of it if it was present yeah.

L

So you are not quite sure why why we'd want to remove that that linkage, because I should be able to produce a cloud event and to define a potential partitioning key, irrespective of the transport it goes over and in fact you may go over transports that don't do partitioning yeah so, but we wanted that to be retained across multiple transports. So that's where you get this interesting yeah.

L

If you publish me then onto Kafka, and it goes through and into media and gets move to a different transport, or vice versa, so that you need that partitioning concept that I declared event level. We.

B

Might we might actually have a conceptual? There might be some miss wording because the the partition and that might lead to that confusion, the partition, key or partition ID doesn't actually show up in the in the message per se. But it's it's the partition that you talk to in in Kafka so and you talking to the partition directly. So that's a connection property, ultimately and and we're we're pulling that so this function here is effectively pulling that information out of the message and then and then making that something that's related to the connection.

B

So it's not a it's, not a it's, not a key per se, but it's really. How does how for Kafka? How can you tell which partitions are sent to, and that is by looking at the partitioning extension by default, but it could also be any any other key. That's within that's what the mechanism is meant to do. I.

O

Am a little confused by that, because I I would expect that the partition key would be have the Kafka partition selection algorithm applied to it in order to select a partition, especially since partitions can move around and such so. Maybe you can help me get I'm confused.

N

And I think and I think that's what my command does again. I mean that's that that's what I say in this issue that when you say partition key talk to a Kafka user, it's the key of the message or is the partition ID? It's not it's not really clear from the beginning and and also there is the typo that in the spec is it says it's a state's key while in the partitioning exceptions a partition key, so yeah.

B

I think I agree that that's confusing I don't have a solution for it, as you said that that is confusing, because because the the way we discussed is we probably didn't didn't didn't follow through and updating this correctly is that this key extractor for the key extractor idea, basically just stems from having to affect the assign a message to a partition and that there is a function that does that job and that, depending on what that transport is, and in Kafka being that there might be a hash function that you use for this or that you that you may want to go and specify the partition directly and so that that affect the partition is the partition is something that's kind of a flexible assignment and that the partition itself cannot be manifested in the typically isn't manifested in the event per se.

B

That's the thing we tried to. We tried some kind of decouple those things we. What we didn't want to do is this. You write the partition ID directly into the into the event, also because, if you forward the event into a downstream Kafka broker, so you have effectively you you use, let's format, if, for instance, you use mirror maker and you go from a four partition to a sixteen partition to with to a for partition broker.

B

If you use a key function, if you use a key, then the the partitions will distribute differently and if you use, if you use a stable partition number, then if you go from the four partition to a sixteen partitioning, Kafka install, you would basically just go and send you know from four partitions or four partitions and you would have twelve partitions that are idle. So the idea was here to have a way to dynamically in the client determine from the partition key.

B

What the partition number would be and of course the key attribute here is kind of shooting that in the it's probably it's not a proper resolution in the in the Kafka stack based on the discussion we've been having so I. Think that's where some of the the the confusion comes from and I won't. Think we've we've had some of those subscriptions in the in the PRS, so we probably have to go and dig back into the history and find.

H

Out so I had a comment about this is the problem the implementation of that function, that populates that attribute, or is it the naming of the attribute and I think it sounds like that should be? Those should be separated right so, depending on, if it's Kafka downstream, then you know the appropriate partition. Key function is factored, which has knowledge on a number of brokers and how the best implementation versus the overloading of I mean the attribute is key and how it gets populated are two separate things. Is that a fair comment? Yes,.

B

Yes, oh yes, it is I, think I, think we've been we've been a little lazy and wavy on this point where we we've basically left. We may see left this with this function and when I think we thought we had that solves the function. The function ultimately in this transfer binding takes by default. This partition key, that's what this this extract.

B

Key extractor function is, and it goes and determines what the pilot, whether if the partition ID only shows up in your usage of the Kafka API or if you want to go further a little bit further down into cough our protocol, but doesn't materialize in the message per se.

B

The key here, the key attribute I, think that's a that's a leftover that that we didn't drop as we were having that discussion, or we didn't clarify that here, so the the key, the yeah. So, ultimately, let me move it back out once the idea is.

B

You should be able to specify some criterion in the message that will then go and determine what your number is without that key or the partition ID having to manage be manifested in the message per se, because the the mapping of the of the message to a partitioning may be different as a routing that events for multiple kinds of kinds of infrastructures that are partitioned. Okay, thanks.

H

Right yeah actually can I just maybe suggest we take a concrete example like and then say what was that key being so I think we're all in agreement that the key should be I, don't know implementation agnostic, but from the key we want to extract some data that would indicate how it should be used by this extraction function to determine downstream choices. Right, yeah.

B

So make that concrete if I was if I wanted to say the my partition criteria is actually the subject. That's in in the in the event, I should be able to do so. So, let's say, I haven't I have an event that events that are coming where the source is a device of sorts and that emits all kinds of telemetry and then the subject is temperature and then other subject is whatever rotations per minute. The other one is so different criteria that are happening and they're all on the subject.

B

I should then be able to go and have a function. That then assigns those for two partitions, so I saw that function might be knowledgeable of the of the numbers of numbers of partitions in my immediately in the Kafka implementation. That's immediately in front of me. Let's say that has four. So the function is some kind of hash over the subject: mod 4, that's how I get the partition ID now I, pull that stream out and now I'm.

B

Taking the same, the same, the same streams like of the all the data that comes out of that Kafka customer and I want to go forward that now to a different color cluster, but then has 16 partitions, so I should go in and again run a fun that is some hash over the subject. Mod 16 and I get the and I get a different site, give different kinds of partitions. That's that's what I mean okay.

H

No, that's perfect so now we're. Where should the responsibility of the knowledge of that particular? Let's say Kafka cluster should be I, mean I. Think that's the point here right. We I think we all agree. It must be.

B

It must be in the forward inclined, it must be in the implementation that is, that is interacting with that. That is implementing this. This, the transport binding, look the way how we thought about this, like as we have evolved. These things is that, as we get into these scenarios, where we're actually forwarding all this data and forwarding is something that is so.

B

The reason why, when when by a forwarding was it was a project discussion in this whole time was that industrial scenarios, for instance, have this device, the device emits telemetry and then a telemetry goes to know dozens of parties who might be interested in that information for all kinds of different reasons. So there's there forward, there are those forwarding scenarios and in those forwarding scenarios we've we've talked about transcoding scenarios, etc.

B

So we always assume that you're taking a condiment off a transport, then the cloud event becomes goes back into its abstract form and then it gets mapped down to a transport, and then it goes through that if this leg of its routes- and then you pull that off- that transport, it goes back to into its abstract form and then you're choosing a different transport.

B

So for here effectively the the the client, the cloud in its khakhra client that is connected to a concrete cluster will then know how many partitions exist, and it will then go and calculate what the appropriate partition is for that cluster and correct somewhere else on the route. Another client may make it very different and the condenser yes, sorry head what one says, and that is why, and that is why the partition number right. The concrete partition hint cannot be in the message, because the decisions on the route will be different, correct.

H

I completely agree with you, so I think, and maybe you know when we talk in abstract it's hard to parties for me to grab my head around it, but that what she just said was perfect. So what what that meant to me was the forwarding client.

H

Let's say you had four kafka clusters, but there is a config which is at some point which, when the rubber hits the road needs to tell it, it has I, don't know for brokers, twelve brokers, thirteen brokers, and this is the extraction function, and it knows how to take a generate key and map it to when it determines what the routing needs to be. It knows how to appropriately choose the hashing function endured.

H

So coming back to our original point, which, in my understanding I, think we we should not bring in the partition key attribute, but just keep it generic as key. Is that fair.

B

I think this I think what we need to do is we need to make a correction here and basically say the key, a true we need to be. We need to make clear- and it's not clear here- that the key attribute really means. What is the partition we're sending to so we need to reword I believe we need to go and do a bug fix here, which is, is the partition ID, the Kaka partition ID is determined through partition, key extractor function, and we might also call this call this difference.

H

And which is independent of the name of that in the generic message? Yes,.

B

So so maybe we need to go. We need to do a box fix here. I think I can take. This I can take this as a homework item to to.

A

Propose a correction so so comments got a question for you: cuz I'm not represent sure I'm following this, but I think I should what you just said there at the end. But it's not clear to me whether something actually appears in the message related.

I

A

So so nothing appears in the message relative to keys or partition case, or anything like that. Correct.

B

And that's I think that's what the bug is, we're talking about a key attribute here still, but we actually talked ourselves out of it and while we were discussing this whole thing and then we left.

A

That in but does that mean that we should then propose to kill off this extension right here. No, no.

B

They partitioning extension that what that gives you is it gives you a partition key in the cart. A partition key is, if you need, if you can't, if you can't have a if you can't derive from the message content itself, what partition that ought to belong to? You have some artificial criterion by which you can go and order that so so you might have you a partition. Key is effectively an art to the artifice of correlation key of some sort. So.

A

You're saying this is obviously it's just. This is almost like a back up kind of thing. Well,.

B

It's so if you, if you can't determine only through source and and type and subject how you want to process that, because that's often what partitioning does right partitioning is, is slicing up your event stream in a way that the processor can deal with it. Well, the oscars can deal with it, which means that the publisher may give hints in a form of partition, key for how those events shall be.

A

B

To partitions and the partition key might quite well be a totally artificial assignment that the application logic kind of jammer generates in some way and how it groups, events and those groups. The groups of events may be groups of, let's say, may be groups of devices. You have a thousand devices which are all different, all different sources and sending different events, but you're grouping them together, like a production lot that all then emits the same partition key, because you want to have them together. That's why that partition?

B

That is, that what that partition key is generally for, and then that function that we're referring to here. The goal of the the job of that function is to look at either something that is in the metadata of the message or that partitioning that partition key and then affecting compute a partition ID out of it and computation man-made.

B

In the simplest case, the partition key may simply be a number from one to four right, may simply go and take it and copy it and then go and set that as the partition ID for the copper pot.

A

Okay, I think it's around this gel Francesco did you want to say, slang your hand and so yeah.

N

So if I'm the super glue from what you are saying, what you described here is Maura client and SDK behavior, then something that she put inside the event when I send the event and when I received it out. That's exactly because this! This is completely not.

B

Clear because, because the way, how those quiet so Kafka on the record itself, right doesn't even have that information can neither does neither does here our our event, information. The partition is something that the partition information is something that is outside of the event, and it's really only relevant for for the senders and receivers how they get the stuff out, but it becomes in completely irrelevant once you are forwarding that event elsewhere. So it's simply, it's really just a local function for a local local concern for that particular car cluster.

N

Okay, Alice Fink to you so after after you work on over rewarding this this paragraph, can you please check the implementation that we did in the SDK code? So you can double check if this, how the spec maps to the actual instrumentation yeah I will.

B

Try my best to I will look at the code and then make sure that whatever I write will make the code be compliant. Yeah.

N

Yeah I mean if the code needs to be changing. It's fine. For me, it's not about just just right. I just want to be at all distinct Islands, okay,.

A

So Kathy has a question: yeah.

F

Okay, so I just want to make sure on the key, if I'm here, maps to the key in Kafka, which the producer can put into his message and then the Kafka broker. Can you know use that key to map that message to a specific partition.

B

Okay, because in Kafka in Kafka in particular, the Kafka doesn't have a doesn't have a shared gateway. Is architectural remodelers the way how you find that the way how you talk to partitions is you connect effect directly to the broker, who owns the partition? Which means you need to know ahead of time which partition you're, selecting and that's where the clients are to actually do that job. Okay,.

F

But but they can also use something else to Mack the message to the yes patient.

B

Right and that's how the function is smitten with that function. Ultimately, you need to get a number out which you then can then.

B

F

This partition constructor is that function which person dies, yeah.

B

Okay, because we may have the partition key right, is it generic mechanism that exists in the event extension? There are other brokers outside of Kafka which all have partitioning models, so so the goal for that partition key mechanism is to be generic and then be usable by all the other by the other brokers, which also need partitioning. So this is a particular implementation here, and that should, and that should map to the partition ID. So we we. Basically, we made a mistake: I'm not cleaning this up when we when we.

B

When we landed on the solution with that with that function, we didn't we just do that right.

F

Yeah I just think this function name is a little bit confusing because it said: is it structure if that means except the key, but actually it's still the mapping to write and also this option might do the Maggie. You say not I mean no not based on the key right. It could.

F

Because otherwise, you know this name just means in fact the key always a key, but if it's a function, if it's a mapping from the message partition specific rotation, then you know it's I mean based on some other criteria and we shouldn't call it the key structure right.

B

I'll try to dig up from me. We we had several issues and PRS about this.

B

Call and if we did, then that's going to be harder to find, but I'll try to think that up and if I can find it, then I'll put it into the into the into the issue here of peer. The the PR here.

A

So Clemens I'm, just sorry sure, I understand this, because when I'm reading the text in here, you see there's to me that, assuming that, while the wording may not be exactly what we want is what's written here actually wrong, or is it just a little misleading because it the way I'm reading? This is that it says: there's an extractor function that people will use to figure out what partition to put things into.

A

But if that either isn't there or you want some sort of default extractor function, then we offer up this partition extension thing you could use, but really it's the extractor function, that's the main, the main way of getting the partition key. Yes,.

B

A

So it's just a more forwarding thing more than anything else. Is it true.

B

A

Okay, so let me show on the same page: okay, it's.

B

Not the key attribute it's the partition also, what's also weird, is that that's not really using using car nomenclature here so I hope the morning change will be minimal. Okay,.

F

I think it's also good to clarify yeah. This key is not the partition ID. It could be used by a matching function to map to a Cartesian ideal. It might it's obvious. All the parties might be function can use other attributes to map the message partition for partition right. So if we can all just say this key map to the key in this partition key map to the key of the conquer that maybe make it clear.

A

Okay, does anybody have any questions or concerns about the direction it seems? You're headed with us, I.

H

Think just the last comment is the whole point of a key and a partition key and everything is very Kafka specific right and it's it's and I think it was also mentioned I. Think by Kathy that you know everything else is extracting that using metadata in the message to extract the partition key. So it's something that is in the in the client and the client function right.

H

There's there is no key in the message that.

B

Is correct, there is no there's, no there's no partition ID in the message. There might be something that's usable as a key in the mystery exactly.

H

Exactly yeah exactly attribute.

B

The key attribute that is described here makes little sense of the message correctly.

H

Exactly and I think I would like to I think we should. My only point in my last point is thank you for this discussion. It's been extremely helpful. I think we should just mention that there is no key attribute in the message. The point is: it's it's symbolic, which is extracted from metadata and the message to facilitate downstream actions, but I'm gonna.

B

I'm gonna rename that I'm gonna rename that section from 3 3, 1, I'm, gonna, rename it into partition ID and then clarify effectively the first. The first sentence will probably be 2 and there I'm I'll be talking about the relationship between the partition, ID and the message. The.

A

One thing that isn't quite clear me, though, is as users are saying: if there's nothing in the message, what does that mean for this extension? So there.

B

Might be something in the mat if you really, if you want to have something in the message, if you, if you, if you, if you need to have an artificial key right, our official key to drive the the partition ID from then, is that ok.

A

So it ok, so it's like there's not a must into something but there's a mess of may okay.

B

And that's and that's why we that's why we have this as an extension, because there's not applicable to all cases, but in case you have a partitioning log or Q whatever. Then you need to have a partition method, partitioning key of some sorts and because that is not too uncommon. We made an extension out of it, but that's.

C

The partition key extractor can ignore key if it wants to yes.

B

But this what what doesn't make sense here is for for a transport binding to add stuff into the message like to add stuff into the event that that is already I. Think that's already a.

A

Foul: okay, okay, before we're at hold too much on this, but I think was a good discussion. Any other questions or comments.

A

All right cool, so Clements I wrote you is taking the AI to provide some curved line text on that. So thank you.

B

A

Right, you can never have too much homework all right. That's technical, the end of the agenda, any other topics. People want to bring up.

F

A

Is that is a Google specific binding and they are going to move that or they're gonna change that pull request so that it's just a pointer to their binding that we host it on their website.

F

Okay, thank you. Oh.

A

Here we go wait a minute Christoph are you there yeah hi, better late than never gotcha we're just about the end. Yeah I just noticed that I'd say it's time to do. Yes. I'm, sorry, I suspect a lot of people. Okay, um in that case, you guys wait. That gets us in into the call. Unless buddy is anything else, they want to bring up Vlad. Are you there.

L

A

I'm fortunate daylight savings times bit Jeff, so we're technically.

J

A

Paula I probably should have sent that uh note to remind people but I completely completely slipped my mind, I'm sorry.

A

Okay, anyway, I think that's it for the day unless I miss somebody for the agenda, we'll talk again next week. Everybody. Thank you. Thank.

M

I

M