DASH High Availability WG, 13 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: DASH High Availability Working Group Sep 13 2022

Description

Pick up w/SAI definitions on page 10
Q: will this be represented as a P4 model?
A: Sanjay - this could be challenging. Will look for a way to formalize and represent it.
Q: Guohan - would be great if we can sample code to call the api.
A: SAI API common config examples (Marian)

Change CP messages opaque vs common

A

uh This is, you know what we talked about last week, but you know we had more in depth on the um amd document and we had some conversation here around.

A

You know how it works and um things like that and we we talked about gohan, had a few questions here and I think I put these in the last meeting notes and so this next session we were going to pick up with amd and then, if gohan joins, maybe go into these topics here. So I'll stop sharing my screen and let you guys pick up.

A

Oh bulky's, joining okay, cool.

A

So amd team did you want to go ahead and pick up where we left off.

B

Yeah I'm trying to share this.

A

Okay, sorry- and that was my dog making noise. Sorry about that.

B

Okay, everyone able to see what I'm sharing.

B

We can start, I think we had stopped.

A

B

Yeah, I think somewhere here there we just finished the state machine is where we had stopped. I think so. Next, I think we can continue on from here. I think here uh I think, what's coming is we have some side definitions with given the background of the state mission we were talking about, uh so there are two uh aspects of messaging: that's going on for uh control.

B

uh One is. There is some messaging that is going on between the uh the state mission itself talking back to the sdk, so this is the psi sdk definitions that we have below and then there is another aspect of communication that is going on, which is the the communication happening between the boxes. So there is, uh there is some uh communication happening within the node, which is the psi sdks, the psi sdk definition messages, and then there is a messaging that's happening across the nodes, which is the control plane.

B

I mean on over the control plane channel and those are defined uh later on. I think we'll come to that section later on, which is the grpc uh definitions right so first to touch up on this isd case, which is this in-node communications.

B

I'll just start from the.

B

End here and then we can look at what are the different functions that we have and the attributes are defined on top right again, one disclaimer, not the still picking up on psi, and this thing. So if there are there potentially are mistakes here, which feedback is much appreciated.

B

uh So one is the registered.

A

Missions come from the um ocp or the linux foundation, or the psi repo or just little con, or are we making we're making these we're? Creating these? This.

B

Is made up, this is created. This is made up. This is something we are creating for now. Yes, okay, these are essentially new definitions that we are saying we would want.

B

So one is, I think there is uh if we look when we looked at the uh picture right. So there was this.

B

There is this channel that we were talking about between the dpu. These were messages that we are passing through to the. uh If you remember, we were talking about a stream, so there is that channel that we have to establish the first message here. The first call here is to register register that cp channel function.

B

So in this document I'm just suggesting we use a something like a pi uh named fifo kind of a thing so that it can be uh to simulate a stream if there are other uh uh ideas that the community has on what is a better way to represent a stream inside that, please do suggest right, but essentially this is to create a stream, a bi-directional stream between the sdk and sonic right, so that I'm representing here there's a registered cp channel.

B

But if you look at the actual definition of the function and the attributes in the function, it takes a named pipe right and it's a named fifo, which is being used for chan.

B

Then there is the get capabilities function, uh so this is for in the in the state machine. We had this where there was a capability exchange, but before for the capability exchange before it before the exchange happens with the node there is a capability get from the local hardware, implementation from the local dpu implementation.

B

So that is this, get capabilities function right and then there is. Maybe I can just quickly go to.

B

I'm sorry, I'm not sure how to exactly handle the code thing, but I'll try to sequence out this thing and you can. I think it's open any comments on the from a review of this thing. Also, it's mostly.

C

um Just a quick question on just say api right, so, as you know, you know for for dash overlay apis, you know. As a as a rule, we basically uh said we are going to derive these. I apis through a p4 model right, so we we also basically said that, okay, that c4 model is going to be the source of truth from we are going to auto, generate these api. Now for the underlying apis prior to these overall api that we had in the ocp psi generally, we we have what we call it.

C

You know some visio diagram really going through some logical pipeline and that becomes a source of truth from where we are going to drive the api right.

C

So so, with this discussion that we are having right now, are we just basically, you know just brainstorming here and then eventually we are going to have some sort of a model, whether it's a visio based or p4 model based, so that we can sort like you know, um officially have some representation saying that okay, this is how it's going to be. You know depicting it.

B

Yeah, so that's a good question. Actually I don't know the correct answer. One thing that I know is representing this as a p4 model might be a little challenging, um is my opinion, like, maybe so that I can think maybe I'll get some this thing on that vizio model and see if that is a better way to uh formalize it and represent it, but maybe.

D

B

Is yeah uh because this yeah this it's not a p4 behavioral thing, uh truly right! Some of these get capabilities. All these are not truly a p4 thing, so maybe we can do it. We can explore looking at how to formalize that, for.

C

You, you can probably see some examples in the ocp site, github repository and that.

D

C

You some sort of like a you, know, um examples of how you can show this thing, but all in all, I guess you know the the ways I actually represent. Anything is size. Just a is a is a logical pipeline right. So, however, you represent it. You know it just it's just to basically show that okay, we need to get get something done and then eventually you know something should be represented.

C

Eventually, you know what has happened is that even in the underlay apis, the pin spins project has really represented the even the psi uh so psy.p4 uh in the p4 as well. So those are good examples to see how, even though what you may think that okay, this may not be able to be representable in people. You may find some ways to represent the people also, but.

E

C

I think you know just just take a look at it and one of those two ways. You know, I think we should explore.

B

Okay, cool yeah, so I think for documentation. I think we can do that yeah, I think. Maybe now we look through. I think my purpose here was to put the messaging here so that we can actually visualize better or understand better. When we are, we were talking about on the in the state machine we were talking about. We do this uh handshake with the dpu. This is what we are going to send across to the uh the node etc.

B

I thought putting a more uh little bit of structure to it, so that it's uh more easily understandable, like we everyone reading it can get a mental model of what's actually happening behind the thing. So I think it's not probably the best way to document. I think next step we can see how to formalize it and document. This thing, like you, see.

C

B

So this was the register cp channel attribute. We were mentioning.

B

All I wanted to show in this peer capabilities is so these capabilities themselves are kind of. There are two sets of capabilities that we have talked about, so there is a sonic implicit capabilities and there are some capabilities which are very implicit to the dpu itself, so those will be probably something like.

B

Something that can be very pers, very, very customized to the dpu at hand, so the message also tries to kind of keep it uh through that this thing right. So there is a there is some some some amount of opaqueness to that, so that it can be extensible right to handle different gpus.

B

Then I think there is also, then there is the.

B

Yeah, so the attributes are defined here right. There is some which are uh this thing, and then there is one which is a sciulisti which is actually an opaque. This thing that comes through.

D

Are we going to offload the heartbeat through psy.

B

So that is one approach we were, I think, from the last thing we thought we will have another discussion uh marian. We are thinking the dpu right now. I think in this design, because we were thinking it can be uh pretty aggressive and happening uh this thing, so we thought we can offload it to the dpu itself to do the heartbeat.

B

So that way it can react faster also when there are switch-overs extra. But that is, I think, the last meeting we thought you should probably have another follow-up discussion focused on the heartbeat to see where we want to do it. We want to leave it in the sonic stack or offload it to the dpu.

B

A

From psi standpoint sanjay for the heartbeat, we will still need some means to configure the value of the heartbeat frequency right.

B

Correct that would be the northbound. I think the user might need to give input too.

A

Yes sure, but do we need to have a psi attribute for that to be able to program it if it's being done in the dpu.

B

So this this is what uh exactly. This is what I was thinking: yeah yeah this had the interval and the miscount. This is the configuration that we are pushing to the tp.

B

Then there is the process peer capabilities, so I think this one is the call that happens from once. The sonic stack has exchanged the peer capability, so what's happening is each local node is using the get peer capabilities, get capabilities to get the local capabilities and then uh exchanging it across. I think the exchange happens via the control plane message.

B

We will see that in a minute, but so there is an exchange happening between the nodes and once that happens on exchange each of the sonic stacks themselves, the sonic itself can actually do some uh do the first level of uh validation that they are compatible and they can actually peer up right after that. There was also this component, where the capabilities are hardware specific or specific to the dpu, the underlying implementation.

B

So for that uh this get process peer capabilities function. This is a psi api that gives a chance to the underlying uh implementation, to signal whether it is compatible or not. Depending on what was each of the dpo capabilities were passed.

B

Then there is the dpu control message, so this was the message that we said last time we said well dpu, dpu dpu to dpu. There might be some control messages that need to be exchanged which are relayed via sonic.

B

So if it comes so, this is uh if the dp2 current gpu to dpu control message comes via the control plane channel. This needs to be relayed back to the dpu to the sdk. So this is the dpu control message. Process deeply. Can.

D

You give an example of such a message.

B

uh So there might be, I think, when we are doing the bulk sync, uh there might be a flow control message or to control the uh what records are being exchanged with from dpo to dp right for the bulk sync right, uh for example,.

D

That would be a specific operation of inserting flows. Oh.

B

Right correct, so this.

D

Is something that so, why do we put it under the umbrella of just process, control message.

B

Oh no, no so, for example, let's say we're doing the bulk sync, and I think the example that we were talking about in the last meeting is uh each of these dpus may have specific ways which are more efficient for it to do the bulk sync. So, for example, if there were multiple threads or multiple slices of flows that we have, it might make sense more. It is much more efficient to get flows from one thread which is free to process flows right.

B

So there might be some message that the dpu can, uh but then the relay back to the other side to the peer side, uh so that those can be those are the flows and there's some number of flows or a specific slice of flows that can be pushed across on the bulging pulsing channel right. So this is a message that is going from dpu to dpu. So these are the kinds of messages that are.

D

Messages, proprietary.

B

uh Yeah could be proprietary if you wanted.

D

And what's the point of psy here if sai is just relaying the proprietary message.

B

We can say I think uh there is a there is a continuum right, so there may be things that are common. I think that is something that we can uh this thing that might be common across multiple dps. We have the same set of requirements that we could make it in this control message. We could make it this thing, but there can be some things like the implementation itself, how many?

B

Whether we have, I think one implementation may have this, where we have different slices for performance for the flows and we don't want to hold a lock while we are processing the other right. Another implementation may not have the same this thing or they may have a different set of requirements or restrictions or requirements or optimizations that they can implement right. If that is the case now, I think the question is: how do you achieve this when there is a.

B

There are optimizations that can be specific specific to each implementation right.

B

D

Don't have the setup, I believe that if you think there are such cases, we shouldn't try to use psy for that.

D

Because, initially, if I'm not mistaken, once proposal was to sync the already opened connections from active dpu to the one that is only coming up through the control plane channel, which would be universal. If there is, uh if you think there should be still some proprietary aspect to that, and it ruins the whole point of uh having a common way to synchronize.

D

The open connections uh between two dpus.

B

They're not not really, uh so I think the connection itself the flow closing message itself. We will see in the control plane in the following messages right, the actual flows themselves are being synced. uh The the floating messages during the bulk sync are being synced via the control plane channel completely.

B

uh There is no proprietaryness in that right, so everything is whatever is being synced. Is.

D

B

Is all defined exactly as it is this thing right now? The only thing that is uh that will need to be. This thing is what is gpu specific if there is any optimization to be had, which we see some optimizations, depending on the implementation, from our experience right, so there are some optimizations to be had by actually knowing what the implementation is. So if there are things like that, then you need some mechanism to be able to come.

B

You know, communicate those optimizations or you know some mechanism to make use of those optimizations, and since these optimizations are cross node, it's not within the node, but it is within. You know something that needs to be communicated across. This is that dpu control message, but the flow sync messages themselves. We will see what we are coming there. That is all in the open. Now it could be that you one might have. One of the implementations does not need to pass any of the dp control messages. That is fine.

B

It still works right. It.

A

Does not depend on this.

B

A

Depend on this.

B

Message, this is what I mean.

A

I see so whatever is needed that can be generalized and uh common across different vendor uh hardware is specified in the non-highlighted portion and.

E

The highlighted.

A

Function here carries opaque information that is specific to. However, the dpo wants to process.

C

It if it's correct.

B

C

A

Rename it a little bit saying it's an opaque information, that's being passed optional as well.

B

Okay, okay, we can do that. So I see this makes it more like all dpu control messages which might be misleading. I see the point.

C

Would you consider even those messages to be the controller.

B

Now that is uh not no, that can be. I think that can be made common. I think it doesn't seem like there is nothing it can mean we can distill it out to have one common approach for the heartbeat we can put up. I think we can specify the message format and make it into a common. This thing, too,.

C

Or you know you can have something like. Basically, even if you have control message just um you can have a control message function which can process all the control messages and one of the messages could be say. Com common message, such as you know, speed and.

E

Then there will.

C

Be some uh you know bits in there which can say okay if somebody wants to define it as a an extension vendor extension or proprietary. They can do this thing, so I think instead of perhaps I mean that would be.

C

My my you know, like suggestion, is that instead of actually doing this thing in the api level, api can basically have you know very clean interface, which is you are depicting right now, but you can, you can choose to basically have some sort of like a control at the at the message, format, level, where you can have certain bits to say that okay, you know you can have these are vendor extensions or something like that.

B

I see I see I I got it yeah. Let me just.

B

Yeah, so I think this yeah, I think you can add additional things here so right now. I think there is an attribute type and there's some attribute data and the attribute type I was thinking there will be one op. Whatever is this vendor-specific information? I haven't added other things, but I see a point there could be. I think in this thing, for example, if there is a heartbeat, the heartbeat could be here or something else which is more, which we find is a more common denominator across multiple implementations.

B

Those could come in here too right and one of them would be the opaque thing or the vendor extension. The data would go in here and the attribute type for that would be the this thing. I guess.

A

Yeah, that sounds fine.

B

Okay and I'll take an okay in that case. Would it be okay to leave it uh as gpu control message? You think! That's fine if we call it out in the attribute itself as uh yes.

A

If we can move the opaque information in the attribute, then this can.

B

Similarly, there is this flow sync message that I think this is the actual flowsync message itself, um so that was just the back back channel, so the actual floating messages are happening in the quote: unquote open. If you want to say that right, so it's in a well-defined format. So that is this.

B

Then there is a upper role status. So this is a this thing that we want to any status. This thing that we want to do- and there is a cpa, control message if you want to call between it's not dpu to dp but from the sonic stack to the underlying.

D

B

What's the last one used.

D

B

D

What's the last function is used for.

A

B

Yeah, so I think this one is, for example, marian here. uh Let me show the state machine and show what potentially, where we could this thing. So there is this. uh There are some some of these things where we are controlling to say to move from one state to the other right. So we are saying start sync right so that one, for example,.

B

Yeah yeah, so that would be whatever is that operation that we are requesting it to do.

D

Yeah but again everything is opaque. I see type un16t shouldn't, we make it all uh standardized like define the medium start, sync and so on.

D

You expect control plane to actually uh set this operation.

B

Correct yeah, so I think that is my uh ignorance marine. I can look at this thing and see. I think the intent is to make it as a enum and specify this here. Maybe I think, like I said, my uh inexperience. I think this should be.

D

Yeah, so I I think I I saw a comment from guan. He said that it would be good to have sample code that calls these apis. I think that will make our lives easier, to see an example, how it all should work, and we will find all the missing pieces if there are any.

B

D

C

D

Know usually, when we have psy proposal, uh there is an example how the apis are called like common.

E

D

B

Okay, I can look at that here.

C

And there one other part would be that you know in this attribute, where you're defining the enums for different types of messages. You.

E

C

Split, you can split them into two right. You know, like you know, just just have. uh If you have 16 bits, you know, maybe higher or higher bits are basically eight bits you can deserve it for for standardized. You know messages and then maybe other bits be it for, for you know saying that, okay, you can either stuff or you can use for for vendor extension attributes right that way, what happens is that at least you have some means of really defining it, and then that becomes really opaque for any implementation sort of speaker.

B

Got it okay, I think yeah, okay yeah, I don't see it in this, at least in the cpu control message. For now. I didn't see any vendor specific uh this thing, but it will be a good thing to keep it that way. I think I think, over time we will realize there might be some vendor specific things that we want to do.

B

Potentially. So that's a good idea. I think right now I didn't see anything vendor specific in the cp control, the last one that we are talking about, but it when we, when I look at making it into that enum like marian, was suggesting and putting it here.

B

Maybe in that we can make sure that there is extensibility towards what yes.

B

Cool that is this thing, so I think uh uh yeah. I think I take two comments from here. One is uh yeah the uh the enum thing. The second is uh sample code on how it calls, I think I need I I need some we need. This needs some work in terms of probably making it more of a psy compliant. uh This thing I guess but yeah the intent was to just draw a proper mental model on how it is working, the definitions.

B

I will look at making it uh making docking and documenting it in a more accurate fashion.

B

Okay, move to the next section.

B

Yeah I mean, if you look at each of the attributes and find something I'm not having a nice uh process of getting feedback for each of the messages. Please do look at the messages on your own time and see if you see any gaps there or you see anything that glares or please do comment on.

B

Okay, so moving to the control plane message. Definition again like I was saying this is the message definition that is, uh that we are talking about here, is here uh in the control plane channel right. So this in this, this thing is defined as a bi-directional grpc channel.

B

So this is the jpc, generator are talking about. So this is a bi-directional stream, and uh each of this stream carries the sync message. The sync message can be either a flowsync message or there is a control message right. The control message is one of a tpu control message. What we have an equivalent that we were talking about there, it should be. It could be a compact check.

B

There is a compat results message and then there is a cp control message right, so you will find, I think, very similar equivalents in the psi that we were talking about. So the compact check and the compact results um were tied to the capability exchange right. So there's the cp control message, which we are just talking a few minutes back and then there's a dpi control message, which was the dpu to dpo control.

B

So this is one of these control messages. One of those.

B

Right so in, I think the dp control message here, like we were talking, there is a type and then there is some data, so the this thing could be. This is that vendor specific option here.

B

Then there is the compact, uh this thing itself the message that was there. The compact check has two: I mean one piece which is the whip info. There is a vip id, which is the two pips that we have there's an ip address. There is an admin draw and then there's a protocol matrix. So sometimes, if we, when we need to control either the bgps and something like that that we need to there is a metric that is needed.

B

So that is the protocol metric and then there is a dpo info which is uh specific to the this thing. So this mirrors what we saw in this high side, where there is a heartbeat interval and a miscount which we saw in the dp capabilities, and then there is the opaque uh capabilities themselves right. This is the vendor specific capabilities. We were talking about right, so there.

A

A

um Do you have a version number in the lessons that you exchanged to prepare for future extension.

B

uh You mean uh this, is I mean you mean like a message version or a version of the software.

A

Message version.

B

Message version yeah. So since this is proto protobuf it's usually you don't need that because it is extensible. But we.

A

uh No, what I mean is, let's say you run an older version where you don't support this, uh this extension of the new message right. So what would be the behavior? So that's what the versioning is for, fortunately, for the message itself, so that you know what information coming so.

B

Usually yeah yeah, so one way that in pro, when you're doing when you're, defining on grpc and protobuf, one way that this is handled is the the protocol. It's a prototype itself handles these uh versioning, so you could have two peers which are working off two different definitions of the message right, so some of the invariants, for I think this is standard thing, which is uh the message id doesn't change so whenever we extend the protobuf on uh whenever we want to make any extensions to the protograph, there are some uh rules or guidance.

B

This thing right so where the message id doesn't change and the prototype of itself will handle. If there is a if it does receive a message which- and it has a message id in the which it does not understand with the local definition- does not understand- there is a way that the prototype handles it right when it unmarshals it.

B

So product gives support for those kinds of extensions right, but yeah. I think there will have to be care taken semantically to make sure that if the other side does not support, how do we communicate that etc? In a sense?

B

In the sense to be able to handle, when you get a message from the from an older version, for example, which does not is not specifying a parameter so to have a reasonable default uh to be to be able to work with those things are things we can do, but specifying the uh header version kind of a thing right. A message version is not strictly necessary with protobuf.

B

If you see a necessity, I think you can see. There is a specific. I.

A

I feel like the protocol in general is a semantic right, whereas I'm asking I'm more concerned in the context of the actual h, a handling between the two pier whether they compatible you know if they support all the message. That's expected.

D

A

Functionality itself yeah yeah, not so much about.

B

The semantic- oh yes, okay, so that I was thinking we can be able to capture in the oh, I say in the I say we need yes, I I think I misunderstood, so I think in these, in the compact check itself, we need something else to say uh h a protocol version or something like that is what you meant.

B

A

Another question I have is um you seem to have this whole thing going through the sonic deck um at what component in sonic using running the control channel protocol.

A

Where, where do you and sonic stack.

B

Yeah, so that I think we should the ha the state machine itself. I think that is uh we need to decide which component will run the state machine. I think it would be somewhere where the state machine the same component, that the state machine is running.

E

Yeah yeah, so so that's why I think um you know we we we probably want to implement this in the zinc d uh in the zinc d process. You know where, where it's, uh I think the reason is being that uh you know the first. This is a generic sonical feature right. You need to be implementing in a sonic and then the second is that the sync d is directly linked with the side.

E

So you know basically, there is almost no overhead to get those flow info and then use it to you know transmit to the to the other side right. So you compare with the occasions so therefore you know my thinking is that um you know we we can get it in the zinc d container.

E

Yeah, let me sing the synthetic process.

A

Okay, so in this case this would be a different thread and a new thread go ahead, so that we're not impacting the programming of the gpu, why the syncing is happening.

E

A

E

Yeah, whether it's a new thread or you know uh other things, I think uh yeah, uh I think you're definitely open to that.

E

That approach. Yes,.

A

E

A

This is a large amount of data that we're sinking across right, so yeah.

E

Yeah, I understood okay.

A

E

Oh, but by the way I you know, I I made a comment: don't see if this is right, so is it? Is it possible that we, you know, usually a company with a you, know the side proposal? um People will have a sample code to see okay, how how to invoke those site functions right. So you know generally, that will give you know people more idea how those api used and then people can walk through that code and the singapore and and that will be the basis for the you know, for the actual implementation.

E

So you know I I think yeah. So it would be great if you, you know, add those sample code. You know how to call it right. So you know, I think, if you look at other site proposal, for you know some, you know big proposal like this. They always have this uh sample code, but you can look at.

E

A

About five minutes, oh.

E

All right, okay, sorry! I missed that yeah.

A

It's okay, yeah, that's fine and um they took a note to try and figure that out and yeah.

C

One other thing you know I wanted to mention about this packet io, how a packet I o processing is done so far in sonic. Also when we get the packet to the cpu.

C

So here also, we are essentially getting packets with the cpu right.

C

We are processing them um and it's it's going to be a channel and then also we are literally you know getting things through grpc right, so we do have some implementation of the grpc as well. um So we have to see whether we want to really utilize that grpc implementation that already exists in sonic and how does it really tie up with the stream channel that we are talking about here?

C

But we also take a look into that that one as well.

E

Which grpc you are referring to.

C

Yeah so gohan, you know I mean I'm talking about. Gnmi runs over grpc right, so yeah. So if we, if we already have a gnmi right right so um and then gnmi has a grpc, do we want to duplicate it? Or do you want to really? You know? Just have one implementation and then utilize that I think those are the those are the things that we need to really look at with respect to all the packet. I o that you want to really process.

C

Do we want to really do all the packet I o in the same d, or we want to have something. So all of those things are obviously some type of experimentation needs to be done in order to really realize that you know how how it's going to scale?

C

What are the other things that need to be done, and today today the packet io doesn't really come through simply right, it's it's a different way of. Actually, you know going through any packet that comes to the cpu and sonic.

E

C

E

Yeah, maybe can you clarify the package? Also, the the fastpass is not going through the cut control play at all right. So what what? What is the packet? I o.

C

I'm talking about all the control pack control. You know I mean today. What happens is that you know let's say bgp packets that are coming to cpu right. They are not coming through, say.

E

Yeah, but what is that? What is the packet you are referring to in this.

C

This flow flow pack, you know closing packets or control packets that are coming in, and then this was the actually. The discussion that you know even lisa was basically asking in terms of.

E

Do you mean the.

C

E

Data pricing or the control plan sync.

C

um There is only basically this you know: flow sync and control plane messages right. These are the two things that we are talking about.

E

Right so my understanding that they are not going through the control plan at all right, the dpu process that is sent to the other side right so that that's you know those packet doesn't go to the control panel.

E

Are you referring to those packets or some other packets.

C

I'm talking about like bulk sync right whenever you do the bulk sync, you cpu is involved.

E

Yeah but but there's no package right so you're just you know getting getting. You know flow record. You know, which is the five tuple from the using the side. Pi, there's no packets there right so.

C

But eventually they are, aren't they encapsulated into something and then they are sent out to the other side.

E

Oh, that that is a purely data plan, functionality right, so you know there were two part ways the box sync and otherwise I don't remember what is it called? It's like an incremental sync, those incremental sync, those uh those thing pack, I think, is being encapsulated and sent to the other side directly from the dpo. Those packets are not going through the control panel.

C

See: okay, yeah, um depending upon whatever the rate at which it is. We just have to be careful in terms of how much we want to really expose it through the sindhi versus how much we want to over some. Some other other means to handle it right.

E

Yeah yeah yeah, so so you know, maybe this is related to the other question we raised. You know, for example like can we reuse the gmi? You know container and those kind of things, um so I think that that might be a a little bit. You know the you know as you as you. As you know, the rice, you know the overhead might be a little bit. uh The past might be a little bit longer right because it's a different container.

E

You need some kind of inner process uh communication, so you can send those flow records to those gma container and then from there you go you go to the other side and then you know get it back to the sink d. You know, then, to the side right, so that part is a little bit long right. So therefore you know it, you know. Probably you know it's better to you know uh to shorten that path so that we can, you know directly, you know sink in between those uh um two components right so.

D

Therefore, you know.

E

uh We were thinking you know, just uh uh you know, use the thing d because it you know basically link to the site library and you can get all those messages from the site. Using your native api calls and then you know, have a grpc channel to sing. That's probably the lowest overhead. We can think of.

C

Yeah clearly the this may be the first place. We can try it, but just keep in mind that you know what might basically entail right or for any or the other things that are already other dynamics within sonic that that might be run into, but you're right. Actually, we can try the first thing and see how it performs and then then we try to see you know the reusability or scale or whatever be the case we have to really. You know we keep in mind about it.

A

Yeah you're always the one who's, mindful hanif about the performance and overwhelming the hardware.

A

Yeah, hey guys. Sorry, I keep parachuting in and out. I just used today.

A

Maybe this was discussed earlier.

A

Is there acknowledgement uh in the process of bulking.

B

Is there yeah? um This is: uh there is no specific act unless you want the implementation need something to say that okay, we have processed this. I need to get to the other one, but there is an there is an acknowledgement at the end, which is the bulk sink done.

B

You know there is a marker to complete the pulsing, but there is no act per flow record or anything like that. If that's what you mentioned.

A

I meant overall for bird sync.

B

Yeah, so there is in the state mission. If you see there is a bulk sync done, there is a there is an exchange that says that okay bug sync has done that there is a marker at the end of the stream. That says that this was the last flow record.

A

I see and the acknowledgements uh for bulk sync: are they handled and processed in the dpo itself or are they.

B

A

Sonic doesn't get involved.

B

uh When you say acknowledgements, did you mean uh what did you mean by acknowledgement bulk sync.

A

Bulk sync: completion acknowledgement that the you know it passes.

B

Through sonic it passes through sonic sonic can handle it, uh but it needs to pass it back to the dpu too, because there is some state on the dpu that we will potentially need to change. For example, you know like we were talking in the last meeting right once bulk sync is done. uh The way we are syncing, uh the dp sync messages right for the syncs for the flows that have been synced across already. There might be a marker. We may need to put that the bulk sync is complete for this.

B

So for that we need to pass the message to the dpu too, but it passes through sonic. Sonic also has visibility that bulk's done.

B

So that was the compact message that we talked looked about and then uh there is this uh the equivalent to what we were talking about last time. I think this is what uh marian was also bringing up. We need a enum. I think I didn't know the right way to put it inside, so I left it as a un-16, but here in the this thing, like you see there is it's a well-defined set of operations right.

B

This is the mirror image of or not mirror, but the analog for what we saw in the side of uh you know. This is for controlling the state mission on either side other than this. There is the flow sync message itself. The flowsync message is a certain noise. This is a batch of uh flow info messages right, it's a repeated, it's a batch and each flow info has a key and then the vip id which it belongs to there is a key and then the policy results themselves is carried in the metadata right.

B

So this is what we said: the whatever the active had set up. The policy results from the reser. This thing we pass it on to the standby, so we don't need to re-evaluate everything. So that is the metadata.

B

So I think, as you might expect, the flow key as the five tuple and I think the pi tuple is defined here.

B

Right other than the fighter.

A

I know we had some preliminary aha apis. I thought in the repo I'll try and find a spot where they were so. I can show them I'll. Send them to you offline.

B

B

Right so that is yeah give me one, so that was this thing, so we looked at uh uh the flowsync message itself. Then there is the control message, the different incarnations of the control messages. What we saw so I think, since we are running about six minutes uh remaining and just yeah, try to see if we can complete the different message flows and then that way we can, uh you know, gone through one round on the list, so the message flows themselves. I think we can add more flows as we go.

B

I think one one flow that I have on my to-do list, which I have not added, is this preemption flow. I will do it soon and notify, but other than that, I think we have the note pairing and bulk sync.

B

We have the unplanned switchover and the planned switch over okay, so the so the note pairing and bulk sync. I think this, these floating messages, I think, is to clarify how I think this state machine that we saw uh had a node centric view this. That was more of what's happening on one node right in terms of state uh this. uh The these ladder diagrams are these pictures and message flow pictures are uh supposed to give.

B

The intent was to give a two node view on how each of them are moving in relation to the other side to the peer side in this picture. What was this thing is on. The left side is once one node. The right side is another node in the left. Side is a is the node that was already booted up and existing right and it was in standalone primary mode state and it is actively forwarding traffic at this point in time the left node comes up. There is a peer connect.

B

This is where the on the heartbeat channel that we see a heartbeat- and there is uh this thing at that point in time. uh The sonic state mission- this, I think from our discussion, is syncd. Wherever the state mission is running, uh there is a the control plane channel the grpc channel gets established and then once that gets established, there is a this thing to notification back going to the sdk to say to register the cp channel that we saw in the psi sdk.

B

After this there is a compatibility exchange and there is a compatibility result that happens here right and then there is a notification from this is being pushed from when the newly coming up. Node is ready to accept this thing. So there's a start, bulking message that is sent to you know owned over the cp channel, and this is related to the to the implementation.

B

And then you see that there are a bunch of these floating messages that are being exchanged right uh to sync, the channel optionally. There could be uh there could be messages, this gpu control messages that are flowing in the reverse. Now, if you needed to do any of the.

B

Flow control or anything like that there could be these optional dpu control messages that we talked about right and once this is done, uh the active site or the primary does the primary side of it or the one that was already active.

B

That knows that the bulk sink is complete and then the bull sync notification is passed back or relayed back back through to the all the way to the other side. One thing to note here uh here is: I think we have the black messages, all the black, the dark. This thing that is there, those are messages that are happening on the cp channel. I also tried to represent the orange lines, so these lines are actually dpsync messages that are going on.

B

I think it was just to illustrate that, as we are doing the bulk sync there there there will be dpsync messages, also that are happening between the two nodes between the two dpos right, so this will be for any new uh new connections that are getting formed or updates that are happening depending on what state- and I think we discussed this two weeks back and what is this.

B

So this is the note, pairing and bulk sync any any questions here.

A

The heartbeat also has to be all the way into, and just like, the yellow.

B

A

B

Yeah yeah, I will yeah that's right. I haven't haven't tried to cover that in this picture, but you're right.

A

Well, if there's anyone with mad drawing skills who wants to help out yeah, please do.

A

um So we're one minute away: um gohan, prince maria, not everyone, I'm not sure how you're looking on time. If you want to keep going or save this pairing for the next time.

A

A

Chris marion.

D

I guess, let's finish for today.

A

You wanna, you wanna finish now and pick it up next time.

D

A

Okay, thank you for taking us through this sanjay. um It looks like we'll just need uh one more next week.

B

Yeah, okay, I think that should be sharp. I think we just have not much. Maybe a couple of pages that'd.

A

Be great and I ran the spell check on the document and got it fixed. I've got the words added things like that, so I did that and so great. If anyone wants to read it, throw a pr this way. Thank you very much I'll. Stop the recording and share it like. I always do and um guys I'll have.