.Net Foundation Design Reviews, 19 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: .NET Design Reviews Brotli

Description

-02:-12:-07 - Approved: Add Brotli Compression to CoreFX https://github.com/dotnet/corefx/issues/25785#issuecomment-352921246

A

All right, because the important matters I.

B

Used to have the link.

B

C

Dim the light, oh, we.

B

It up, you know, I, think that's.

C

B

Same 184 that you don't like this is getting better. We all just want to go to sleep to go um so I can think that's something we talked about this. We basically said like, ideally for the stream. um You just take my seat deflates to anymore. That's.

D

Exactly right, sis.

E

B

E

B

Thing is like you have these guys here, the buffer size, the other one doesn't have talked about that yeah.

F

So that I we could take it or leave it I'm, not sure how much value it's really gonna have. So, if you want a smaller API, we can just knock those off so I.

C

Have a question: could we make it into a property yeah like what will happen if you change it? What would we do if somebody changes it after? The stream was yes, so that would be the problem. I.

F

Guess then, we would have to flush the buffer essentially to the underlying native implementation. If we reduce it such that we have curtailed the data, otherwise I mean that would be doable I'm, not sure anyone would want to do that, though. Well.

C

It's not that whether somebody first like what what would we do, not whether somebody would want to do it but like we would have to have some reasonable behaviors. So, let's say, let's say: I create a stream and don't use it yet I just created an instance.

C

If somebody now says the property, would it be reasonable? Would you have to do anything with flushing the buffers and copy? No.

G

Not if there's.

C

No data in there, no because you don't have to create the buffer. Somebody starts using the stream. So when, when the nice test is created, it's empty, it's just like you know it has the settings, so you could then they could freely change the property and when they started to use the stream it would all work. Yeah.

E

C

Would just have to come up with something to do when they start using this stream, then they change the property yeah.

G

I know: what is that you.

C

G

To do the flushing? Well, there are types in the framework that don't allow you to change, they'll, say Lydia yeah,.

C

Or we could do an exact, because our.

G

Probably easier.

C

There's only easier because you see like that would kind of solve the problem. So now we have like six constructors, we would have. We would go back to four and also if it's a property like, if you don't care, you don't see it versus constructor parameter, is kind of interface. Once you like start looking at the constructor, it shows you intelligence shows you. The signature is.

G

It only used for compression or also decompression, I've lived for both.

F

I think probably the best way to go about it would just be to copy deflate streams API, give it a buffer stream for now and then later we could add the property or constructor or.

F

The plate stream as small as possible I like.

C

F

Yeah stream done now.

B

For the longer one, this is everything had I mean. This is the first thing, I think the only thing we actually probably before me for the meeting when a long time.

G

Yeah I also uh asked earlier, but you weren't here to answer on what are the semantics of flush on this ADR and on for.

F

Bradley stream or for the Reilly encoder, either.

G

Wolf shrink since that's what we're working on right now so.

F

For this stream, it's pretty much the same as deflate streams, it'll act, the same way. The only thing about Bradley, which is different from deflate, is that a brightly stream isn't really completely inflatable. Unless you have the finished fight so like for the flight stream, for example, you could do you, could encode half of it and give it to an inflator and then call it a day, but for Bradley stream you need to do the finalization first. If you don't do the finalization, then it's not going to have that final bite and the inflator.

F

Google search bratli that one will actually throw if you give it like, partially compress data, so it's as similar as it can be, but you still need to do. It dispose I.

C

See so basically dispose is the way to tell the API I'm done yeah.

F

Our finalization.

F

Other than that, it's all the same, it.

G

Just removes all your lines and some Adrian's, so here's the particular scenario that I'm thinking of say that I'm writing a large document to the output stream and halfway through I, say flush. Then I send that off to a receiver somewhere. Can that as that receiver runs it through the decoder? Is he actually going to be able to see all of the document that I've compressed thus far, or is there going to be some type of buffering that you won't be able to work around?

G

As long as you haven't hit the end of the data that you want to encode, then it should be doable yeah. Let's say that I said I sent half of it. Yes, I ran half of it for the encoder I said plush I got some help and I just sent it right across the wire yeah.

F

And we've got some tests for that too, because.

G

That's what I mentioned an email earlier about, like real-time stream scenarios, yeah.

G

E

B

All right, so this will be happy so far. Our disagreement should not come with any surprises. Do we want to do the way.

F

Then yeah that's right, then.

B

The base trim is the same as both muffling gzip, so let's just exposed a puppy.

A

Is the buffer size I thought it was supposed to be a property right or no I was.

F

Thinking we should leave it off for now and add it.

B

Later, oh yeah, mr. Park, oh that's inconvenient, to set it up, it was yeah. um I see the usage is okay, so then the decoder so.

F

We have some internal discussion, we had an email thread floating around for changes, so this has changed pretty recently, I updated it and I added some design questions we were having at the bottom sort of like three names.

C

So do we already have operation status in depth?

C

E

F

Sort of like conditionally approved.

C

B

It in there so it's not condition.

C

They're proof, interestingly, recently asked on discovered that they started a type called operation status in the framework.

F

A

Results or something like that, but he said it's operation status, so operational status didn't go through this API reviews all right line, oh yeah, so even if it's in my system member, it might be the like yeah, we are working on it. I'm, pretty sure that.

C

It went when we reviewed base64 80s for about that was interesting. Well, it was in the signature of the API so and then there now, oh.

C

C

um The name renaming it, but it's a separate review and I forgot about that. It's already there so oh and this IP I would depend on system memory. Yes, okay, yeah, cuz, I was kind of thinking.

C

If this needs to be lowering the platform, then system memory, we cannot use operation, service, yeah, but I think we are good because you're gonna be on top of this one you're, not adding it.

F

C

I simply doesn't go into this.

F

Is its own right now, but we could put it it's just amount of pressure if you wanted so.

C

Yeah, so if we put it in systemic compression, then we have a problem with dependency.

F

So right now it's in system I have a compression that bratli. Why do we have.

C

Well, because this operation status.

A

Is in system memory yeah? What kind of cannot be a Dutch dependency like IO compression is not lower under system memory right, so you won't get.

E

A

C

You want to lift it up or that's the reason or well. Today we don't. First today we don't have a dependency from system compression to memory and we would be adding dependency and I. Don't fully understand what it would mean like we already like it's a general fear, which is fine, that we just.

A

Understand basically,.

C

With the open in Platt.

A

How compression is special, because it should you know on the side as well, so it was as a pure hope. So, let's bureau up and it's built.

C

Into the dotnet framework, which makes it it's the combo, it's already like I think that maybe we should.

A

F

Mean it's perfectly fine and it's not assembly well,.

C

The initials not being a few okay, so I think we can make it work if it's purely on top.

C

So what's the plan for networking stock? So if we ship it on top of the platform, networking stock cannot take dependency on it correct. What do you got.

F

Atop the platform, you means not a part as a new, so the pair of different hours.

C

F

Put it in the platform, so it'll be a part of my subnet forever we're not thinking we want to move it, especially because it has a native dependency. All.

C

Right, II more, you need to decide whether it's how we do it I'm already miss.

F

What what's the benefit to it,.

B

So I would say: let's first look at the api's and then let's talk about whooping, so you make into the API sure the thing movie name is better in wing how we ship it is like not really walking you right now versus api's kind of are so. Let's.

C

Finish that maybe first, but the point is not benefit from this, who being or not to be, is that system memory that DLL is a loop right. So if you have a dependency on system memory, I assume you are. Oh, you.

C

See this is where I'm gonna get use. Now, since the mostess idea.

B

That HP net1 see who's Wally, so it has to be into one.

F

B

I mean expand is in to one right so like that, doesn't prevent that he doesn't have to be just because spans. It just means that the inbox version you know you can have.

G

An inbox version than the.

B

Underlying Spanish line box, Sydney Toronto on framework like wow, that's that's a whole different, Yale Park. This is the other thing, any likely again can't change physics. They want to run in 461. Well, then, yeah! That's.

A

What Adele and memory targets like you know, four six one as well can right so I wouldn't be shocked if they won't play in the same boat. Well,.

C

Everybody wants everything, it's fair. Let's do the API, but I think it would be good to talk more about where it ships and the relationship with various existing yezi's, because this thing is super scary, yeah.

F

I talked about this with Wes and Josh separately, in sort of a we thought would be, the best route would not öbut unless we have a fully managed implementation. It's the native dependency dependency makes things complex, so we would just require a Stinnett to upgrade.

C

So and now, if we don't a little bit.

C

F

C

F

Would it fail, will develop packaging or detail building but well? Where is the problem now.

C

And if components.

F

Cannot depend on up components, do.

C

You agree, I, don't know.

A

C

Yes, so if system memory is oh, this thing has to be Oh Biff, it depends on it. In other words, in box cannot depend on, things are not involved in the neck. Is physics involved? What API you can call, but an operation started so system memory is like a hybrid component like part of it is in the platform. Part of it is just oh yeah, I think operation status is just like. Basically, we don't have IP is in the framework that returning we only moved into the platform span, read-only span and kind of its dependencies.

F

What, if I, don't feel that operation status should modify our decision on to vs. Nadu, because operation status is just the need of, we don't need it. It's not like for this. I have brought Lee resulted. It was the same same thing. Well,.

C

Anyway, let's let so, let's go through IP I person.

B

Then we can I, send it some minor clean of the status. The bottom so is easier to see the.

C

B

In the static non-static use sure, so my first question would be like you should probably follow the novel dispose better in here where you would have I guess it's abstract this matter. Yeah.

B

F

Variations down below in the issue also mostly just naming stuff, but so when does decompress fully returned phones if it has too much data to put in the destination span or if it fails, because the input data is invalid, so will.

G

Never program ever it won't throw numbers so.

F

It should be called try yeah, that's one of the variations down below try, decompress foliar tried decompress, singular, try create decompressed data or try decompress, if the above so.

B

Before we go there, like my prescriptions, disposable instructs, so what does it happen when you copy destructor well- and you call this pause one of the instance then.

D

The ones are basically defunded.

B

What they all get disposed, yeah a point well, but the problem that you have today is when you have. Let's say you have like two classes, so you have one class right if I call this Folsom that that guy can basically register that you call dispose another out any pointers and then effectively when you call another method just so it's not supported because it is posted. If you have two structs and it's a whole day out, they hold on to native memory.

B

If you suppose one of them well, you can clean up the pointer for that guy, but the other one still points not to memory. They don't exist. You crash next up.

F

Sipping Olivia this one called.

C

F

So there's sort of the difference there is that this is a struct, but it has is not entirely not okay, house these days, gonna.

B

Move inside of it so.

F

B

F

B

What ends up happening is when I call disposal one of the structs and call methods on the element. What had one in the baptists can like object is photo.

F

An object is published so try to reference that safe handle is not a visual test for that yeah, oh yeah, so.

B

F

B

You would expect it yeah.

C

So how do I, let's say, I'm using the decompress method, not that decompress fully check. How do I say that I'm done dispose?

C

Yes and that's the only way to say I'm done, yes correct. You think.

E

You're, like flashes.

C

Well, mainly, I was just asking like how you do it I'm, not sure whether it's so in a lot of other AP is a Levi. You kind of worked on similar where you know you call like decompress, first chance, second chance third chunk, and then you call decompress final or, like you know, an API that is called final and then it makes it very explicit. Like you look at the API- and you know here, it's kind of like you have to know that you, you should call this pose. So now, maybe.

F

It's okay, it's a little bit different too, because Bradley knows when it's done, but there's there's bytes in the decompressed data. That'll say this is the end of the input. So oh yeah you're right. It's.

C

Only for the encoder yeah.

F

Is it one for the encoder.

G

F

G

Sort of flush or flush, parle, okay, okay, like in I, think in what you've done here is you never actually have to call dispose, except as a perp optimization, because, as you mentioned, the decoder always knows when it's done anyway, and the encoder requires a caller to call compress final at some point. So I see I.

F

Mean the way I have it. Written right now is to require dispose, but we could certainly do an actual dispose whenever the decompress is done with it. Wait you require : dispose yeah, that's how you get rid of the native pointer. It no.

G

But I mean it'll it'll get juicy, though. If you oh yeah yeah, it's just that's that's the pattern. Yeah yeah well,.

C

It will get GC, then the final eyes around the safe handle will be girls.

G

That want a yes, but what I meant was that it's your you're, not relying on the call to dispose to generate the correct string. Oh.

F

No because that'll be done by decompress or it'll, be done by compressed vinyl.

B

Right so then I guess becomes decompress decompress fully. Is that fully assumed everything fits into the buffer, yeah and.

F

Then you don't need an instance of the decoder.

B

So what did we end up with the other ideas? We ended up, calling them try, decompress, fully I, suppose right or whatever the other ones were on GP.

E

B

Like in other places like he had like it, you know decoder or something we call this fully. Try decode by this is would be try decompress, fully I suppose, but.

C

I so I also I commented it in the email, and maybe it's a good thing to talk like I I. Think that fully suffix is not necessary. I mean we don't like most IP is, unless they say otherwise, they do yeah what they say. So I will just.

F

Call it try to compress I think there's enough distinction there between the static one at the non-static. One will just try to compress. That was my worry for adding the fully well.

G

You won't in intellisense, you won't see a static extension method when doing dot on an instance, so that will.

D

G

From developers for the most part, assuming that we're relying on the tell us.

D

C

Do for discoverability that's primary way. People find api's, but if even if not like, when you look at this API I mean the static method and it takes source and destination and like what else could it do? It has to decompress everything and source of everything destination of it? There's no.

B

And then the contract pretty much this allows continues: usage, I.

C

Would you know like if we worried about people not understanding I would rather add the suffix to the first one like to like basically say like the decompressed chunk or something like this? That's good idea, that's another one of these options. Every.

B

C

Places that they had moved, we don't have, but.

B

C

Had so, we have actually I think in the encoder space already in framework IP is we have existing IP? Is that basically do things in chunks? And then you call find out this one is slightly different because you don't have to call find out on the at least decoder side, but what do we have on it and on the other corvex laughing for oh okay,.

B

So we don't have challenged our yeah.

C

Because we don't have, we don't have a final step like you know, or do we base64? What did we decide for the basics? Person is final.

D

C

In parameter yah, stroke, okay,.

D

C

But you see this one like the decompress. Doesn't it doesn't have the concept of find out? It knows when it's done so there's no reason to do it. I mean.

F

The compressor does that, like yeah.

C

F

One of the options is instead of having compressed and compressed file. You can just have compressed and then have an is final bull, yet mm-hmm and.

G

The behavior might be a little bit interesting to work around just because this is a stateful type, Santa statements type.

G

Because could you run into situations where the compress or decompress routine? Because it's trying to do an underlying buffer flush doesn't actually consume any source but still writes to the upper buffer, yeah yeah.

C

But I think you know it may be easier to write software if you have just one method, because you have a loop and basically you just call compress and in the last parameter you passing whether remaining and whatever you passing is, is the same. If they are the same, it's the final.

E

Block and then you have to keep following that or look where this final offer.

C

Otherwise, you have to have an if statement and then call two different methods and two different paths to different size of parameters. Is there like the loop gets more complicated.

F

The implementation I have now is basically what you just said like I just have one method and you just pass parameters for whether or not you want to flush or finalize or just compress. So maybe that's what we should do and.

C

It would be similar to the basics before decoder yeah.

B

That's what I'm, thinking or negative look it's not really going to be super similar just from a naming standpoint, even.

F

Have a flush done now? If we didn't just an is final boolean, so we would lose that functionality. That's what's making me nervous unless we have two boolean, but then flush capability, so like not finalization but intermediate flushing. Oh.

D

Then missions really useful for real-time sharing, where we should just add a method code flash does it take input.

E

C

It flashes whatever it has.

G

And you keep calling that in a loop until about response. No.

C

No right, you call it when you just usually look at it once you call it once, but how big of a buffer you provided. You don't provide a buffer it flashes, whatever it has inside to where you have.

D

To you have to have a destination buffer, but for.

F

A flush, so in this case you know yeah what we could distract. No that's. What were the push up? You look like I mean we have I mean it's the same as what we have right. If you compress final, except it would be called flush. Oh.

C

Okay, so this is the thing that I also kind of commented on that I'm, not not sure how to design it, so operation status, returns, destination too small or need more input. Why can't you keep calling compression?

C

You know, keep passing my destination instead of concurring flash you mean without in yeah, it's basically equivalent to your flash that just takes the destination. Roy is planning to this a compress final, the underlying.

G

Implementation could just choose to keep buffering data until it has a full water, compress yeah.

F

That's the problem. You need to actually tell it to flush if you want it to finish that block, because it's not an issue of not having enough destination down for why doesn't flush it just is waiting until it has the optimum optimal amount of input.

G

Is it wants to compress as much data in a single block as I can yeah, because that's like use of the most optimal.

B

Malati the compression works, you have the sliding vid, oh yes, yep and at some point the window is full and then the comparison magic ship recreated at last look well, there's no more data, so the window, never against folks. You said the system, somehow I'm done.

D

Please Riley correct and so let's define now so normally that's what is finalized for, but you disallowed intermediate flushing.

G

C

And intermediate flashing, so it may be what I'm getting confused is. It seems like there are two different reasons to do: intermediate. Flushing one is for some reason: I want the data to start flowing and then the second one is I have data in the input buffer, but there was no room in the output buffer. Now it's really just the first I think there will be a.

F

Scenario: it's just if you had that intermediate data that you want to then has to a decoder.

G

Somewhere because, as you mentioned earlier, when the applications in the steady state, it's just going to be calling this method in a loop right, it's not really going to care how much data is in the interview output. It's it's just going to do some, you know offset picks up and then just wild trigger Walter, all true, but yeah as soon as as soon as you run into a situation where I'm working on a real-time chat, application I need this data to get to the caller or to the recipient.

G

Now, at that point, you need to say the guy.

B

But what I mean I still not quite sure like how does all this body work around the car? So let's say you flush your current window. Does that give the receiver of their private of information will decompress everything up until that point.

F

The only problem with that is that it depends on the actual implementation for whether it wants to then parse that normally or for once, to treat it as the end exception. If it doesn't have an invite right.

G

That strikes me as very odd because normally.

D

G

Stream doesn't have this problem. This room does not have that problem. Deflate doesn't have that problem in general, so but if you're using a bratli, but if you're using a deflate decoder to get to decompress a bra extreme I thought we were saying that that.

F

Was awfully not rattly, that was the precursor to Bradley, Auto gotta go google abandoned so Bradley and deflate are not interchangeable. Okay, okay,.

D

So there aren't.

F

Very many bratli decoders, there's just the ones in browsers. So how do you / today it compressed finalists is what you're calling flash? Yes, it's both flush and vitalize, depending on the state, depending on the is final. So.

C

We don't have a intermediate way to flush intermediate, though that.

F

Is that is intermediate? He.

C

F

Find us in final drew all compressed file is finished us it's not a good name. There's a parameter parameter to compress final.

F

Oh it's in moon. Can you expand option section down below.

A

E

It doesn't have the.

F

A

Let's compress acceptor source.

B

Laughs, the other ones I just copied it literally from the github like there was the one that I copy yeah.

F

That's one of the design questions for whether we want to allow an immediate flushing flush first is finalized. Oh.

C

I see so here's the parameter is finished. Yeah so basically is finished, is to control between intermediate flashing and I'm done correct. Okay, so could we have an IP eye that basically just has a compress method and flush? Yes,.

D

What the signature flush, what would you have finalized, I think you're, just advocating renaming crossfire. Basically,.

C

No, no, no, no so I would add a parameter to compress with a boolean is final. So it's exactly like the base64 encoded API. So.

B

That means most and ours would only call.

C

This API there are clashes on.

F

B

C

And you have some special scenario: we have a method flash and it basic because but like what are the parameters to flush because it's just be destination and then light switches destinations by selections. Compass.

F

Or data when you flash so possibly also that's I, think the one after that allowed me to put the flush finalized, but I mean the underlying API allows you to give input to a flush or finalized. It's just does that make sense from an API standpoint us how.

G

Do you know when the when the flush is done, writing to the destination buffer flush would return operation status, hold it in the loop that seemed strange today? It is a little strange I.

F

Mean that would also happen like if you give it to small the destination buffer, it'll return needs more output or whatever needs.

G

More output like for regular compressing areas, I can kind of get that, but for a stateful API, especially like a spongy API like this I would expect certain values of operation status and never to appear ever anywhere before input needs more input with Neverland never appear on flush. I would expect that number to your.

C

So if you don't have enough data in the window to flash, it would be just bytes written zero network.

F

If it for some reason, can't write a whole block, yeah.

C

So so they said it is a scenario for operation status, not enough data, not enough input. They not.

G

Even wash yes, that doesn't.

C

Make sense just cut.

G

It off yeah, but the definition of flushes whatever you have, has to go out right now,.

F

But you don't have any. Not today is the keys.

G

More input because.

F

In most cases, all the cases I've ever seen it will flush and give you an entire block, regardless of how many bytes pass unless it has zero bytes, which pointed to know all yeah, well it over chinese-born, but so, if it gives one byte, it can there's.

C

F

Time right there isn't it done. There's a success.

G

Well, success at a zero like success seriously like there was. There was no input so with successful partials. We.

F

Can change that but the way it is now it will return each one. Implant.

B

So if you have one bite, either buffers this flush, which is basically the decorative addition of the firework is you have like you, don't have to pass in the buffer, so I like the X chemicals onto the on buffers such just means, whatever you hold on here on offer just wanted to the underlying thing, my dad's one, if the definition is finish the current window, so that the receiver of the thing has enough data to become totally start decoding bullets, which is a very different thing, I think on that point of view, but because flush in your worldview based, it means you know right like they finish the current window.

B

There's somebody reading they can read the entire window or not like it like have to wait until the middle to get more data yeah, in which case I would argue. When you talk about the flashing of the window, not enough data makes complete sense because you have to be have enough data to finish the window versus in the normal flush.

B

State like there is no doesn't make any sense, because all you're saying is that whatever you're holding on to yourself just but it's an underlying thing and that that makes no statement about whether that's enough or not. It just means you know, stop holding on to data which kind of means to like I think flash may not be the.

C

Best term for me, because I think it's I don't even fully understand. When would you want to call it like you wash.

G

So you're writing. You're. Writing a chat application of web-based chat application right, like maybe you're, working with, like customer support on a website somewhere. Now you you, as a chat representative like ask a question to the user on the other side. That question like the ASCII text, ends up through the broadly encoder, like the ASCII text is only going to be this big right. The buffer is huge, so you hit Enter. When does the question? Let you type actually make its way over to the customers when.

E

You say it's fine.

G

C

It's fine. You just.

G

Go stream itself.

C

Is still oh sure,.

G

This API cuz it has nothing to do with streams. No no I mean like the the underlining the underlying network stream is so off and as far as a browser, so she said, API.

C

Is not tied to streams, it doesn't take streams, doesn't code streams, so I don't know.

B

C

That you mentioned.

B

I, don't know what Rolly encoding looks like right, but presumably you have a header and that content right. Yes, so when you know read a stream, do you expect to see another header of another set of content? Do you think you only say he had a once and then continuous.

G

Content because what I'm concerned about is you mentioned that at the end, that's at the end of the final encoded block there is a finish there is. Is that basically, like an end-of-file marker, it's.

F

Not really end of file, but essentially it says that this is the last block you're going to get in this set of data. But if you have a decoder ya, can't quit the decoder decode more data after that or basically depend on the decoder.

B

D

Will are scanned.

F

In google, scamp's so I would say: no cannot and not so.

G

I guess what I'm thinking my mental model is, if I'm using a web browser that sees one of these bytes come in at the end of a question now the browser says session disconnected.

C

F

Disconnected to just reopen another instance of the bratli decoder, then, if it's expecting more input data, how does it know to expect more in potato? It just received a marker, say.

G

A little bit from the network.

F

Stream right, if it receipt, if it receives more bytes from the network,.

C

These IPS are like so a very low level and kind of almost stateless, if not for the fact that they have some like broadly specific state. They cannot be participating in the networking protocol. No, but I think when they see top stars.

B

And when it, when.

C

B

Maybe the better model, so what you have here is basically the the situation where you have a you know you, basically you compress data right now. The thing is, let's talk about how the phone of the data looks like you have a header and then like chunks of encode, the data you got blocks. Yes, yes and there's a header at the at the at the very beginning, always like every chunk the same. There's.

F

A header at the very beginning and then blocks which each other own header and, at the end of the final block, you've got a couple bites to say this is your last block s so.

B

Now this next, the question was like: okay, give it a given. This information may have like header, and then you know some amount of chunk.

B

Like that, should the lady I allow you to basically do a thing where you, where you are able to you know, have chunks in the middle, where you're saying you know, the chunk is not complete for the window size, but I write it anyway, so that the concurrent decoder never has to deal with another set of headers to.

C

Code, their livers I, don't understand so the current.

F

Decoder doesn't have to finish essentially and then start over again. I said: first header I don't take the cost, for that would be high, though the letter is not big. Nobody.

B

But this goes back to the interaction you guys just had I mean because if you say you have to do that, then no imposes limits on the consuming side of like. Oh, you have to reinstate the same well.

B

Do you know what to reinstate this thing versus if you basically just have to open connections, and both of like one of them is just putting there to any other one is just pulling data out, then the only logic you need is to say finish the current window, because the because the consumer can just decode that one chunk and everything also just works out, and these guys have never have to resume.

B

But they have this illusion if you will offer of one contiguous stream of compressed data and the stream itself just may stop at some point, because it's waiting for somebody to time to check message, but then, as soon as this step, machines is over. While you decompress that chunk over, but.

C

I so I still, it may be I'd understand one thing: so there's this native structure alcohol to when does it? When do you write to the destination?

C

Well, do it and this destination buffer for you right for encoder, so.

F

For the encoder, what it'll do is it pointers for both of them or whatever spans you pass them and then it'll pass both of those to the underlying API? Another lying API will take the information from the source pointer to do its internal style and put it in the destination port, but it does.

C

It do it eagerly yeah, then what does flash do in addition to what it already is? Having only my.

G

That's meaning.

C

Like if I want to flash I just keep passing destinations and in.

F

The 90 days ago, if you get it, say, 10 bytes in source, yes, but it needs a four this lot. If you keep giving a destination to compress, it's not going to do anything. Okay,.

C

So now maybe answer so flowers does not mean right when, if you have enough data for full block right it, it means right about right, don't wait for more data yep and it's different from a final blog because it doesn't put the final correct thing is at the end: yeah I.

E

See I understand.

F

Yeah, that's that's a bit strange. Maybe we just do compress for now and then it's people ask for flush. We have flush, see if we're missing anything, it's easier to put things in later than take them out. How.

G

Would you put it on a string, though just know off it at the moment for flush or Bromley stream yeah you.

F

Still do flush the way we have an app but.

G

You just said that the underlying encoder wouldn't support it. The API wouldn't exist partly.

F

Encode, a broadly decoder wouldn't, but we could just hit the native ones right now. What we have is Bradley. She was built on top of Rayleigh encoder Bradley decoder start once does not have to be. There's no particular reason why it should be well.

G

It generally should be just for ease of maintenance. It is more easy to make yeah.

B

So you know I guess the keep on is like. Is this thing existed or is this like pretty much yeah? That's the question because then the question is messages you need to differentiate where you want.

F

To just finish this one versus also writing this one personally I'm in the camp that I think we should do the finished block at the end of every set of data. So we don't have a flush. We just have a finish because I was looking around. Some of the decoders are always going to expect the EOF equivalent. They don't like it when you don't give it at the ELA. They.

B

F

When you don't have.

B

It at all or it so basically it would be illegal to say you have this right. Then you sent this and then you send another chunk. You know n plus one essentially here. Would that be legal, then? Or is that expectation? Oh, you said any Oh F. So you're done so you know that is the expectation.

F

All over again with header, it's sort of data, it's sort of the corollary to the encoding flushing right, so the D compressor will have this internal buffer that takes Chung's. The text owns the fixed chunks and it's not going to actually flush that data to the underlying destination until it sees that AOS. So the.

B

Basic means that another question for us is: do we care about being able to? You know you have enough? Let's say the buffer size of the window. Size is 2 K per, but you want to say: I only have 80 characters. I wonder I want to write a chunk and I don't want it add to transmission yet yeah. Well,.

F

B

F

Care about this another media, API Provenza, but then the question is on the other side of things, the decoder. Does it see that chunk and do the whole thing or what it has a bigger bubble? If it buffer is allows two chunks, does it wait for to fill the buffer or do they just get one chunk, you're halfway, finished chunk and then what I'm out of the jungle but I assume they have like.

B

Some sort of micro header that says you know I have they're like I know you kept a basin area, I guess from the decoder standpoint. They should be able to decode the data that is entirely in this chunk and then well. If it's like a checked client, then the expectation is that this is just text. So let me draw that this is a unit of work. It was like a JSON document and then you're like a bit of a JSON document. Then whoever you know.

B

Basically, you know the Costa polymesh will not be able to do anything meaningful, but because it's still waiting for the rest of the tracing to arrive yeah, but so, like that's more like then then you're talking about the data itself into the underlying data itself is an extreme enabled if you were good, but this is just talking about whether the compression itself can be stream in it, which is to me I mean you never care, we don't I mean I, don't know what their bodies use, such as an.

G

Ira is either we're.

B

Saying if it is, then maybe we should have the distance not.

B

Let's compress and compressed file or, like sorry, I've also analyzed.

G

B

With the basic C form style, holding it at the end, in which case what you call this guy, you call this guy you passing through and then, if you want to do it again, whether yourself, you.

C

Know all over again, so what's the cost of creating this structure like all the native machinery like? Does it make sense to reuse? Let's say you have the scenario that anybody was talking about my Kamchatka enterprise and you know. Basically we don't provide flash. So you have to like say: I'm done, I'm done, I'm done yeah today.

C

I might after you say: I am done. Yes, as the attend we make. Is it expensive to recreate? This? Has.

F

A run actual purpose give you the dance, because.

C

If it's expensive, maybe we need an API to basically be able to say yeah I, just called down now.

G

The the performance of that, because I'm sure we can always improve that. My concern was just I, don't know if the browser will understand it correctly, because I don't have enough data and I'm reading through the stack and the stack doesn't really talk about this. But.

C

What I was saying, maybe I'm, not by the way what I was saying. It has nothing to do with a browser. This API has no title browsers to streams. It doesn't do actual flashing for you. It just moves data from one span to another. It's up to the consumer of the API. To do the right thing for the browser from the browser's protects.

F

That's his worry, though, is that maybe the browser doesn't expect this elast method. If it sees an EF and says everything's done now yeah, but we want to be conformed ative to as many implementations as we can. The decoder will.

G

F

G

Than only scenario for right now is use inside of a web browser yeah.

F

That's the only that's the only place you can find a properly decoder, it's web browsers. So basically.

B

All about differentiating between finishing the chunk and writing AF and finishing the chunks they're, not writing us right or like the question is yes,.

G

What do consumers.

B

Do when they see anywhere and like that.

C

B

C

May not what you want so another thing we could do is kind of reverse it I'm now brainstorming, because there are some issues, I just noticed, but we could have. You know a final means just rush all the data, and then we could have right EOF method to basically everybody has to call my UF is EOF always the same. Is it like a constant payload or it varies well.

F

It's not it's not exactly an EO f, so much as it is you're telling the underlying native implementation to finish the block with those EOF bytes, so I mean you wouldn't want to do like a flush and then another AOF. You don't want to finish this block and then put another block there with just an EF in it. Why do.

B

You write two blocks just one: what.

C

I'm, saying is, if it's a constant, we could have that agent find out. So you, basically, you are now done with the native permutation, and now this API, just as a when I said, it's like the chunk itself has.

B

Some meditator in it that says this is the last chunk. Yes, so there's no e way, F marker, it's just that the chunk itself is written differently. So if you make it two things, your end up writing an empty chunk, adjusted and the last guys teachers write more later so.

C

What happens if you flash, and then you separately, want to write yo F, but now you don't have an able data. You have to write a whole empty block. Just yep in it is our API. Will our API actually do it.

E

G

Looks like purpose fact: there is a shortcut like a something that you can just stick straight in the header of the last block that says I'm an empty block, oh and by the way, I'm. Also you up yep.

F

That's it just that's love, you know, I can't do the whole full flush and yeah.

B

I mean could literally I mean I mean the the most straightforward translation, much sure whether we like you know, but the most transport thing I can think of like we just had two walls, which is basically finished chunk and finish. I don't know is that its final block so.

E

You're saying we could an enum yeah.

B

Because you don't want it's, not two independent votes, it's basically, it is either just like normal, which basically just fill up the chunk. The second one is finished. The chunk and the last one is finished chunk, and this is final. So.

F

Internally, it uses a datum. It.

B

F

Enum that has either finalized flush, compress or metadata block I think it is embedded in block we don't care about, but the other three.

B

See the problem that I have these kind of things I mean that's I've not really made up my mind whether I think it's a leaky. Abstraction was just part of the ceremony that you cannot really abstract away from the caller, but it seems a little bit like similar to base64 with same discussion right I, guess this final boxing of the memory maybe leaked the way, the form it looks like well I.

B

F

I, don't really think we should use the enum I think that's probably one of the worst droughts I mean would much rather just have compression the compressed file with the pool for flusher, don't flush or flush or finish.

B

Well but then the question will you expect like? Do you think it's easier for the college to reason about when they call which API rather than when they.

B

Because I mean like you, have two methods and then you have to know which one to call one right if it just until you called the same thing yeah, ideally the names just get longer or something see and then there's some default. Then it's fine, but yet so.

C

I I would almost say that, like maybe I'm wrong, but the flash is kind of rare correct, like you should listen Alice for everything without the line above yes, I would say like the first compressed method should be the regular base. 64 like pattern being.

D

C

Or will not work if you use Broadway compression all.

G

Right, why would it not work signal arm when you're using a server server side? Events depends on flash existing hit.

C

But I I it so the first method is, you know it doesn't flash it just I'm final and then we just have a flash method. First, the specific case, specific scenario and the flash method is, you know, I kind of also find it strange that it takes a destination. But for me you kind of have to.

G

F

I mean that's pretty similar to streaming.

G

Interface right, that is really because stream is always guaranteed to consume all of its data, whereas these encoders are not, and that actually makes flush very hard to use because say that you call compress in a loop and compress returns needs for it, but because it didn't actually consume all the input. Yeah do you, and, and now you want to do a flush.

G

Do you call compress again is a guaranteed not to return me to our input like what what happens if compress returns needs more inputs, but you, given it all the input you want to give it before our flush, yeah you're you're dead in the water. At this point, so.

F

Why exactly do you, then the water? Why.

G

Can you is this under the under the person that you don't have a flush, then I'm, saying? If, if the scenario is that I've put one bite into the and I want to flush, that out of the network again, if I call compress and pass one byte is that for the source? Is that guaranteed not to return needs more data?

G

So if it returns hands returning for it, if it returns, needs more data.

E

What am I expected to do as a color.

C

White with the whiteness compress return need my mother data ever yeah. Well, you know III may have zero bytes or one or two or three or five there's no like a minimum size of the input to compress.

C

Do you know yeah so.

F

C

F

It doesn't have to return need more input. We could just have it return success, but when does it typically.

C

Lines with beer.

F

Into the underline, or to the destinations.

C

Man yeah: what do you say to dangling you to return each more data for the case where you basically holding the output yeah for.

G

C

More yeah I, don't think that's what it Merck needs. More data should be used for I. Think the intention for needs more data was the format is invalid. If you don't pass me more data yeah, so there are some formats you know like base64.

G

C

On whether a encoding or decoding you need to have the chunk size is four or three, you cannot pass one and say that you're done so that would give you each one but exact.

F

Okay, so yeah, that's that's a pretty different behavior than probably okay, okay, so.

B

So I guess what you're saying is that you're returning for later when the window is full yet right? Is it fair yeah, that's the idea, and then you basically so what happens when I, when I basically pass in just enough data that finishes the window, but not more than you return to success, I suppose what.

G

Happens if I give you more.

B

Then, what's remaining in the window and more data, then you, then you busy fill up the current window, write that out and take the remainder put in the beginning of the me, not enough data I, suppose yeah, yes, so.

C

You know, maybe we need to add one more values to Dana a value called something like buffered, so people are not surprised that they call the API. It returns success, but nothing was written. That is kind of surprising, so new case. Isn't it isn't it fair to say that each monitor is almost like the.

B

Argument exception, like the method cannot work, no, no, no, in which case in case going okay, like it like it again. Okay, like in this case you could. It could argue that there's nothing the caller would have to do. We got just just throws in data and at some point you just write it out and thanks buffering you don't care whether they buffered or not. You just want.

G

B

In but in your case, you're saying, I need I need one two or three characters: I cannot take. Four cannot take what.

C

You gave me so far is involved. The reason it's not.

B

C

Is so imagine that you like basic C, for encoding right and a you encoded, this chunk and this one byte here yes, say: 32, it's space right! Well, you don't want to throw because there is additional buffer here in the pipeline. So if you pass just this chunk, it cannot do its work. It says, need more data. What you do is you take the next segment from the pipeline. You stitch them, and now you pass the you know the four bytes yeah.

A

I guess what is the end of the next buffer s again 32, so what you re going to get there? What do we owe you? You stitch together, the next buffer? Yes, at the end of the it is hard to do again because it's you know it continues. Well, the.

C

Base64 encoding, I PR, it will not like say, needs more data. I didn't do anything. It would actually start encoding as much as it can. Oh I sense is for for the encoder, not for a decoder. I couldn't I couldn't encode everything that you passed me I encoded as much as I could, but I need more data to encode a few more and detail people.

G

Data, just it doesn't mean I consume nothing. It just means I couldn't consume everything if, if it might mean I consume nothing in some cases, but the intent dessert. So within this case in the browser case, consumes it all. It.

D

Just doesn't write it all yeah, so if it consumes it all, then it should return it, sir sus so I think in general I think it's the only denying on value.

C

It's basically it's like buffered I consumed all the data. I didn't write it not because the output is not large enough. I didn't write it because I I bothered pulled out all the.

F

Excess, a questionnaire, how you determine whether it's buffered or it's been written, because what? If it's only partly bit exactly what it is both because.

A

You finished 1-1 chunk and you started the next one. What you're gonna return.

C

A

C

For the house later on so basically.

A

C

Most of this API is there as input and output, and it either gets moved from one to the other, or does it.

C

Produce special, it basically operated piratey partially moves the data from input to output, but some of the input is source for later you're always going to have buffered. Unless you have the exact.

F

Size of that part that you need so what's the point having it at all. Well,.

C

So it's any done. We need to return a value, yes or a the the one that you said is success. I think we call it done that success or maybe it was renege, but it was called down, meaning part.

F

Of a sign downward operation status is success versus down or not really edge, so.

C

This one is it's not that correct yeah, it's not that! Well, it doesn't need more input and it doesn't need more output. It needs you ten, it's not an invalid data. So like we need a, we need a new okay. What does what this tells me.

G

See none for a stateful API. We could just have it not return. Neither.

F

In tennis, but daily.

G

That there's no UNAM INSEAD worth well, but then keep in mind what you just proposes that we add buffered as an enum value. To that you know now what happens if a stateless API returns father so I think we're.

D

Having great should never happen, but that you don't have to be defensive again so well,.

B

Well, I mean I think in general, these API documentation because I don't think they're like I. Think if you use the API for the very first time, I, don't think any of these people will be successful. You know, like I, mean just looking in the Tennyson's I. Don't think you.

F

Can be well so here's another option is not have average status for compress at all and just use a pool. Since we only have two operation statuses, then right we have needs or output or we have we've compressed some stuff and we may or may not need more input. I wouldn't.

G

Just return void.

F

Or whether or not you need more outlet know.

G

If compress return void- and you had another method which was right to compress data to this destination buffer- that method possible video but committed.

F

I mean there's even an API phonetic one, but.

G

Compressed at that point would just be guaranteed to consume all of its source and do something.

C

Well, so what happens if I pass an enormous buffer is a source and a very small as the destination is the broadly internal data structure? Can it store very large input, buffer.

A

Very low it's fan o we.

C

Know will compete, I, passed a tool here, visit, I plastic to geeks or span, and one bite out out yeah what happens.

F

Destination, and vice versa, that's what the purpose of the bytes consumed is: there's only take what it needs to finish that block and then, if it doesn't have more space that will stop and then it'll say bytes written is your one byte compressed when the bytes consumed is one by right.

C

And it was the Destiny: what's the operation status that you return, all that the designation too small it needs more needs, more output, it's more potential destinations, you small yeah. Ok, then, therefore, if the method cannot be void, yes for.

A

You need to talk to the you know the color. Like you know, you need to pass me more output examples, so it cannot be void, but it can be. A bull could be.

B

But then, in hitting the pattern, man yeah, it's just a question of whether the operation status pattern makes sense. Alright, so we're not optional. We could just rename done to something like no action needed or something because basically done for the color is irrelevant.

C

It doesn't matter, it is not. Action is needed, so you started to talk about. What would the algorithm do if it's buffered, you cannot like now, stop it's not the same as that you. Basically there is some data that was yet not flashed. You have to somehow code.

B

That demand that is controlled.

C

By you, because they city, if you, if you pass in true.

B

For the final block, the API should never return buffer that should return whatever they wouldn't have done it because you said this is the final one yeah intermediary.

G

B

It they said it's not the final block, you don't care what it buffered or not, because you're not done yet anyway yeah. So for you, there is no action needed for buffer right. Whatever this earth.

B

From I'll be right, in which case is like it's just money, then I just say well: destination too small.

C

They call acid in something well.

B

C

Unfortunate that now it depends on the on the parameter, as you said, if it's, if it's final, yes I, do you well.

A

The parameter like it's more like you know, saying, like you know, hey on you know, this is the final, so you should.

G

Finish everything right, you're telling it explicit the basic. If flashing, that's how base64 works today, it's it returns a different operation status based on the value of his final yeah, because.

B

It's either needs more data or its or it's invalid yeah.

B

The only difference is that, in this case, is never another which is waiting for more stuff. That.

C

Fine I think maybe we could repair the bullion, but right and now the question is: is it worth it? Why would not you I.

B

Will return operation status, I purchased her major consider we need unto something that implies. Less of this is the end. Well,.

G

What was wrong with success like.

F

Success I think success makes sense. You said.

G

That was your original name.

F

C

Right of the ìiî result.

F

Natives enum that has either success or done I, don't know which one I don't know whether success are done, but then operations status has the other one. So.

C

I think in operation started, we didn't call it success because it implies that the others, the others and that success they are per se done means. You know there was an operation was.

A

C

Of things you have to call, and now it's it's over it yeah.

B

C

B

Argue that yeah.

B

F

Will never work creating this buffer there's also the possibility we might not be able to use operation status, so that's probably less important, yeah I would.

B

Like to avoid it because to me, one of the bases of this API gives use that the same you have like 20 enum members at some point, but then the enum is horrible, but if you end up with like four states- and you only have to learn about what these four states on and it's just some cases where one of the states are unreachable, I think it's much easier to understand from a caller sign then to understand each and every like kind of way.

B

You can sample bytes from one to the other, and then all this guy returns bool to indicate that it half of the other guy, which runs. You know false, to indicate that all the destination was too small and then it gets really more complicated because then either then, basically you have to deduce the arrow states based on you know the relationship of the out variables, which is what we had originally like that.

F

B

That's another idea to do crazy, math that never works I'd like to avoid that ok, 309 now I think we should push through, because we are close, I mean.

F

I think I think doing the compressed with this final than having a separate flushes, probably the way they go and before we decide on flush. I want to do some more research, but the browser's do when they video F, because it's possible that they treat this as equivalent to collect this or the possible they decide that the stream is done. Everything's done, Ella takes it, so I want to find that out. First before we decide if flushes necessary I think having compressed with is final, is good either way. Okay, yes,.

C

So did is so now I agree, but we saying that for now we are returning operation styles, still from the from the API from compress. What about flash no I feel like.

F

C

F

Have one because you still have the success state and you have the needs more output, data state or needs more destination to small destination, to small, so there's still at least two states and possibly error, though I think I still have operation status.

C

Well, error, if you hire like, if there's an error like you.

A

Can you can encode anything right so input that I will not happen or out of memory or if.

F

Fedex traffic, like you're, not doing it explicitly another one.

G

Would be if you already.

F

Finalized it and then you try to flush, but again that's something we can change I see we can make that but yeah. But that's a section. That's.

A

An exception yeah exceptional case: you misuse the NID ice yeah, okay, so for the early student. So.

C

For flash to and for compress how many states do we hear you need to read.

F

The how many states many possible return values we're cautious.

C

F

Return very so.

C

F

Capacités error its then success error.

C

Because of what all error is an exception correct this error as an exception destination too small yeah.

A

Because you get the two gigabit two gigabytes and you have only one byte in the output. What do you do just four more whatever buffer.

F

It would be or success either one.

C

But then, in this case I'm basically concerned, then.

F

I bastard I wrote.

C

Whatever I was about I needed I right yeah, but.

E

The game is myself that I used it you're.

C

Good there's no invalid data, they see.

A

E

A

Second, that you mentioned, might be that you didn't write anything, but you just like you have stuff buffer. If you in the meantime variable, you know successfully wrote something else. That's combination to that I.

F

Think that's good.

C

So it seems like maybe it's an overkill, but if we return operation status from both of them, it's kind of there's a bit of complexity, because there are values that are not ever returned. Yes,.

G

C

Simplicity because they all kind of work, the same you learned as emo said you learn the four values and from.

A

That perspective I agree. However, it's gonna come I live for it myself, so many times that you look hey did return something out. What the like. You know, how many values do I have to kind of watch over right. You know how do I have to protect myself. What do I do if it's not okay from some exception, that's basically that code yeah, it's I, think we should have the algorithms be super puzzles, nor corollary, just want to make sure that for.

C

Users I agree that there is the trade-off. It's just. The alternative is boolean's.

A

Are bad you can.

C

Make the you know the second one boolean so.

F

You see the second.

B

C

E

B

C

Oh, correct boolean, basically means I'm done, succeeded.

A

Or not or yeah so.

B

I'm gonna be a little confused by Isaac. What are we designing right now? So, basically we just said it's like now. We take our compressed API and the Eddie's final block here right, yep I. Think that's good! You should be voted to true because there's a basic C for that I think it's weird but like I, don't know.

C

Stated one doesn't syndrome: let's do the same thing that basically told us yeah true. Is it true, you know Israelites our guidelines, but maybe it's fine. Why other people can I say that defaults of properties should be what the default of the type is. So the default of boolean is false.

G

All in eyelash- and it is my final block truths at all anyway- the Chrome browser once it sees the less metadata marker- will destroy the decoder. Yes,.

F

The question is whether it starts a new decoder if there's more information after that in the stream heck, you know.

B

C

F

Have to destroy it either way having.

C

Said that the guideline is kind of silly, but I think we basically should do what base64 does I might a question is why should this be defaulted at all? Oh.

B

Because you have to do right, yeah, like I, mean you're, not yelling. Back to me, you, if you make two different calls and you're holding it wrong, I would I would argue. You only have to track in an expression somewhere. That's the last book or not I.

C

B

I think basically.

C

Let's say you just want to call the method and be done with it. Well then, you would use the static.

F

Right exam is compress fully or try compress it. This compressed fully.

C

I, don't see it. Yes, it's.

F

Right under compress, we have a track decompress and try compressor press foliar press adder as well. Is it in disguise.

C

3100 36 37 Oh.

F

Compress data Jannik naming is not great okay, so if we only want to do with once, let me just do the static ones. Compress data I see, try compress yeah.

E

F

B

To be consistent, so that would make sense so.

A

Then, in that case,.

F

I think there are the most people.

A

F

The try compress is a static thing where you don't have to make a broadly encoder, you can give it source to give a destination and you're done and it works or not. Okay, I see return, a bool succeeds like Forge it. So then, if we have try compressed, then I think is final. Block needs to default to false, because you're going to use compress with is final block. Whenever you have like you, don't know that you'll be able to finish in one iteration like you're gonna loop.

C

Yeah, this final block needs to be consistent across all ideas. I mean I think it would be very, very bad if, like the default is different depending on API, everybody would get confused because that what I'm saying is we only have two options.

B

That you, like I, think it's okay to say this one is not defaulted cuz. We.

C

B

Yes, the thing is: okay, other.

C

Things are defaulting, but I think.

B

C

I was just saying we cannot have between base64 and this one differents default with the same parameter. So I see one rather than saying to me defaulting.

B

This makes more sense to begin with, I agree and then, if we have to be for that, I think you only made them for sensible. Does.

F

It make sense for.

C

F

B

Block yeah yeah. Why.

C

Does it make sense for base64 because basically, for only has.

B

Aesthetic, but it's not like the whole API is basically much if it doesn't have this state. Yeah. Okay, see you're, combining these two guys here right. This is effectively.

F

Say let's just leave, this final block is non defaulted, so.

C

F

C

Yeah they have to supply it. I think we also need to rename the compressed final. What we basically does not completely it doesn't flow.

F

So from the discussion earlier, compare spinal is now flush, yes, so that we ever need it. If we decide that we need it depending on what the browser's do so that one's tentative.

F

So that's why? Why does this have to take any data? Why would it not be just what does it take data? It just has its destination, because otherwise, if you flush data, where does it go? Oh, okay, okay, never mind yeah! That makes it technically. You could take source, but I, don't think we should do that better. So.

C

Should we have this final block in flash I, don't.

F

Think so one flaw should only be for the case. We have intermediate data, otherwise you have to api's. Then they do the same thing: utilization.

A

Damn you can send zero bytes and call it, which is fine. A book trivia see yeah, okay,.

B

And then my last question is: why are these methods about properties because it seems counterintuitive give.

F

Me methods they can be properties that they can be constructor variables. You.

G

Can change these wall everything you.

F

Cannot so that's the argument for why there should be compression for constructor variables, which is what Christoph wants to do yeah? That's what I would do a good I mean the reason I have them like that, instead of being constructive variables is because it's a struct, so I kind of wanted to have the like the default. Struct declaration and dot set quality dot window, but you you can still have default.

E

C

Just do the bracketed thing you just instructs cannot have just parameterized constructor, yes, but you can have a default one and parameter yeah. I.

F

Have that in the issue too, so we would just have a default and we have one that takes a quality in the window. It's.

C

Just a default when you like, create destruct, do you do anything in the constructor like now I? Don't you just store? Basically, whatever will pass you? That's all you looking everything is lazy, yeah.

C

The only problem is we don't run constructors when you create, we don't run the default constructor. When you create like an array of charts, we.

B

Don't want it at all. That's.

C

The problem- that's Jessica, Claire this guy.

B

What kind of declared is that? Let's try, if you have no, you cannot. You cannot define a constructor that takes off and on this part it.

G

Would be narrated yeah like there's no code, basically, the very first time you call compress this thing's going to have to allocate a save handle in commerce, which is what you were doing. That's what we're doing now so, but all of when you create an array of these guys, like they're all going to have defaulted to tea, but they're all usable. It's exactly look where the quality.

B

In the window, okay, so basically.

G

Whatever values.

F

Equality and window.

B

Would be presumably field which would be 0, so you have to be all not the case when it have been at 0 yeah. They give you blow up because you say: oh, they have to be some way over then you're, defining it struct incorrectly yeah. That's that's not the way it works. 0.

G

B

In which case these guys will be gone, and then this one we have the same I think it's the same, API that they had on.

B

We called it, get max decoded from utf-8 length and get max encoded to YouTube. Anything too.

F

Wordy but yeah.

C

E

We can remove.

C

I am um getting.

F

Back yeah get maxed impress on that spot. Where should this take? One room probably should take long and.

G

Protract Island, if you I, mean a loved one, had one good argument for int is that you can create an array where the number of elements isn't it and then you can just create a span around that as a destination and pass it in to compress but I don't know. I could.

B

Go either way on this, but we don't support spans enlargement for gates, you know, and they only have any nexor. That goes to long time.

C

Because they are used for chunks, so that definition, but this one is kind of the total size yeah well, except.

E

C

You use the API in the loop, you kind of don't want to call this one. It's completely irrelevant like calling this API. It's only useful if you want to complete a one chunk, mush.

F

C

That point you are limited to span of bytes, so.

E

C

If you, if you use this API in the loop, you would never call get max compress the size because you're doing it in chunks, so don't only.

F

Matter what the maximum press sizes right, if.

B

You yeah- and this is the only BC you only care about this number because you want to like allocate a buffer. That is sensible, in which case you don't care what the total is, because never hopefully ever have to allocate that okay yep.

F

Okay, it's useful for the try, compress situation, because it gives you exactly how much space well not exactly but another bound on how much space you need for destination. So you know it's not going to fail because you don't have enough space because you're not calling it in the look anything.

B

I gave me nothing prevents it from passing in, like a general is in large thing right there, which is always.

G

It will overflow.

B

In Android yeah.

G

Like there is a check in that method, such that, if, if the value that you pass in plus whatever the header length is, is greater than in 32 max value across.

D

What that will mean you cannot be, you cannot compress a 12 terabyte file and run not in a single shot and with try compress now I'm.

B

Saying, but let me admit what I'm saying it says this API here right. So if I wonder, if I want to estimate essentially how big my buffer has to be, then what a person is input size is how much I'm going to buffer, like how big my chunks are right. So let's say this: like eight kilobytes yeah.

F

It doesn't work that way. I.

B

Want to I want to estimate essentially how big my storage on the file chair needs to be, if I'm running broadly over the files over here and the pods over here at 12 terabyte, so I can still call the API of Long's and get sensible answer. So.

A

I mean they can face.

B

It it's the same thing. It's just that if you call in the loop you, you probably don't pass in things in a bigger than utility to write their.

C

Eyes, and instead, when I'm.

F

Not certain if it even works, if you do it like that, I think it has to have the entire compressed size. I, don't know if you can just pass it chunk and get the maximum decompress their maximum request size of that chunk.

G

So the way that this API works under the covers is it's. It has some mystically assumes that the data that you're feeding it is not compressible at all right and it will assume that it's writing out uncompressed data plus X number of bytes for headers. So the answer that you get back from get maximum press size is always going to be strictly greater than the value that you pass them always now, which.

B

Kind of makes sense is basically doing a pug ball yeah. It's an upper bound of yeah.

F

B

This is a terabyte file over here and I want to estimate how much side I need over here, but.

C

If you have 12 terabytes, you don't need to estimate gigabyte.

E

C

Did you get byte? Why would you need to estimate what would you do with it? So it let's say it returns, 13 gigabytes. What would you do now with this information, for example, check that the output right at that size? So I, don't like one of this place helper to the operation? I, don't think that's super reliable operation!

C

You still can write a you check and then why are you compressing somebody else's writing to the disk and, like you, screwed like if you're, if you're, like approaching the disk being exhausted, like it's so difficult to write software direction,.

C

Experience is better than surf surfing.

B

Later as well, and what I would.

C

Do is I would basically say whatever is the long times. 1.2 is good enough estimate like we don't need API for it. I honestly, this API is for that for the first one. For the instance method I mean sorry now for the for the Tri compressor and for the static. That's the primary is what I like I just want a a if I don't pass enough destination for the for the try method. It fails yeah. It doesn't actually like right. It's not like the first one to try it's partially. It just doesn't right.

G

If, if for whatever reason, we decided, we need to add in the future sure at an overload, so the only not.

C

Was an infant yeah.

D

C

Long, no, it wouldn't have to be it's. An input parameter is.

F

The argument that get max compare size doesn't have value specifically in the Tri compressed state that we don't want to have. It is.

C

The reverse: it has value in try compress so.

E

We should leave, it ended, correct.

F

E

F

Is but it's an int because.

C

The try compress once sure they don't even deal with data size is larger than it.

F

So, what's the cost in having it a lot instead, a minute just to accommodate? Maybe the terabyte case follows: is a long back to cosmetics, yeah.

G

C

Well, that's why you could do overloads if you really want, and then you know and if it returns it long that is longer than then in 32 max, then you cannot create the span that large. So like it's useless little yeah.

C

We make it long, but you cannot use the long value in.

F

The tractor mistake: yes, that is very cool, so in today's.

B

Demo you're happy for the API shape is that I'm hearing.

E

C

World world, okay, so basic so far in kata, so a basic difficult for encoder this property. This method is called get maxed decoded length, and here we call its size. Should we call it like.

B

We call it get max decoders from YouTube it length and get max encoder to YouTube. Anything I mean it is a mouthful so I.

C

Don't know, but it we don't have to add utf-8, because nothing.

B

Wrong with you, TFA father, my.

C

Point is that we I think we basically decided I, don't know if it was conscious as accidental to basically use her a terminology which is length not size. What.

F

It's worth I'm, not sure if it's worth anything in the underlying native implementation uses. This exact name get max compressor size, not.

C

Super interesting.

F

So I preface it with you what.

B

F

Worth I'm not sure if it's worth anything I would.

B

I would say: I would base it, go to sale for race for just off of spans and its best all day, yeah.

C

Exactly and it's used to instantiate the right size of span or array, whatever you using, which basically means if I find my pressure again. I would probably say: let's call this and what is the parameter code?

C

Is it in basics, for coffee bootleg or the codec bytes, but I find that very day, meaning us sorry.

B

All the college is.

C

B

Okay, that's much and.

C

Not they we, why are you referring to yourself in the third person because passed me, you want to know usually.

B

Does via things that don't make me kind upset that I know that personally well, so.

G

To answer by the way your earlier question about: what does a chrome do if it gets more data after the last you know, then I found the truckin, so it was an explicit yeah, because I was looking through. Their uh I was looking for their bug reports.

G

Apparently it was hitting an assertive, so they out of a uh they added a check to see. If there is data coming in at the end and they just destroyed, it doesn't.

D

Mean we need an API, because we cannot handle this way I. Don't we can be? We just leave. We leave the flushing yeah.

E

Yeah everyone and we.

C

Are golden all right, so then this is the avi shape me.

A

E

Then that's this.

F

If didn't find the default, constructor umbrella, encoder, sorry.

C

One more thing: do we want to discuss for multi, but we want to go home.

C

A

C

You start with being okay.

G

C

Suggested this thing where we are safe handle, but this space struct there's a negative to it. Okay, the negative is that if you box this thing, you kind of paint because twice what, if you bugs this chart, so let's let's imagine that this this type roughly encoder, was a class.

C

It like you, always pay the cost of creating an object on the heap just like today, you always pay the cost of creating object on the heap because you create create the safe handle.

C

We could just make this thing. A class implement our own lifetime safety, which is seventy right to take a funnels object, yeah, something like this or we implement manually like. There are many ways, and then you know you don't every box it! Oh yeah, you have just one object on the heap always and always.

C

Or we could not use safe handle and keep it as a struct, and then it's not super safe, but at least it's cheap I think that we ended up in like this hybrid world. That is neither cheap, North I mean it. It is safe, but it's not super cheap, so I'm somewhat.

B

C

It's the benefit make in a class, because then you can. Basically you don't ever box it. You always have one one type on the GCE I one object on the GCP yeah.

B

What one another benefit that you have is, if you ever have to add any state to it that is not represented by the safe handle. You are not screwed construct, you're, always going to screw right.

E

B

C

Five copies that.

B

Having consistent configuration.

C

E

See like a mutable.

C

State that you'd yeah is.

B

Maybe you get for freedom in the bar and then you also get boxing for free. You could start to have virtual methods and right, if you ever need to like there's other things you can do with classes yeah, but on the other hand, it's kind of I. Don't.

C

B

C

B

Like about this, is that is basically it's telling somewhat of a narrative that you're not really up, for it looks like an aura locating.

B

Which every part I would prefer to be a class, so we see.

C

Basically, directly allocating okay, so let's so now, let's get! Let's imagine that we do it instead is it possible to create non allocating, probably encoder, API this safe? That's.

F

A safe know even have to just have an importer internally if.

B

You want to IV.

B

mmm What we could do, what Jared are called crazy, which is less style, borrowing or linear, tied to something you can enforce it? That structure some coffee to run so.

F

I guess I should add a caveat to matter leaves technically we could do that if we rewrote the entire manage implementation based off the see elimination, what would it be? Backfire I'm over the buck is attacked by just on a red ball. Alright, there we a.

D

F

Of questions there and I'm not sure I'm, not sure it would even work out, but it's ostensibly possible also.

C

Another thing: so it's not about abs, it's about handle recycling, correct issues, there's no possibility can detect the day if.

B

C

With the management limitations.

B

Of the same issue you have to use after dispose right, not.

C

Know so if the handle is closed, yeah I assume we can detect it already there buddy, but.

B

He just said is: if you move away from an implementation, we make sure we manage right, there's no hand, man I.

C

Am talking about the kinetic limitation, so it's not a Navy, there's no use after free or anything like this. If the handle is closed, we can detect it because the moment you pass it to some underlying API, it will basically return an error code. The problem is that the handle can be recycled. Correct is that the safe handle is secure against recycling. Oh, if.

G

You take the in pointer case, oh yeah, yeah. That's a terrible yeah.

C

I know you push for there and you know they. You think it's there, but I'm saying like what are the implications? It's it's not an IV and it's not a it's.

G

Basically e, so in the one one thing that we've seen before in asp.net when we've had stuff like this, is you end up writing someone else's response to a different network constraint so say that you and you are both coming into my web server I've accidentally I might have an int pointer. That I think is pointing to you that is actually pointing to you. I'm writing data to it and now all of a sudden you're getting the information that was intended for you like. That's why we moved away from them insisted.

C

Web whenever we could and as I understand how it can work for a network handles, you know, how would it work with this one, because.

G

If, if the end pointer is the buffer that I'm writing the data to- and you like in and basically I've swapped in pointers under the cover, so that you to basically think that you're pointing to the same structure, that means that whenever I do a flush for you or right right, your data to the destination shrink up sorry to the destination.

G

Spam I'm actually writing his data, because that's what the end pointer actually points to, even though the spam that I'm passing to the API is destined for you like, because what I'll do is I'll I'll get an array pull or stack alloc or something I'll have coffee, but data into there and then I'll pass that span over to your networks. Train, okay,.

C

So so another question related. So as we talk, those are three old Windows handles correct, correct they're, just as.

G

Their memory addresses yeah yeah and that's what makes us a little bit more insidious because real handles you can kind of detect like memory addresses, are reused all the time right, because that's how malloc and free are optimized under the covers.

D

B

How do I handle is in Windows right now. How do they know that the value is valid? They have a hash table on the side. That basically says this is how how to save how to work. Well, no windows handles himself, I mean the passing the random and they don't crash right. So.

G

B

G

Do they do have their own map of which handles are valid, but they do sometimes Reeves numbers in that map, which means that you can get to the point of writing. Yes,.

B

Recycling but you never have a B issues, not passing amending. They do like.

D

This depends it.

G

Depends on what API your.

B

Home yeah, okay,.

G

Like if you're, if you're calling a kernel API and passing it a bad handle, it won't Aviva LaVon check the machine, but if you're calling like bcrypt, for instance- and you pass it a bad handle yeah you're going to give you because that handle is actually memory address.

C

Yeah I guess this is basically the same reason. The way we have bombed memory but.

G

Against a safe handle is, is fine, for this cassette handle has its own record. So if, for whatever reason somebody tries using one of these after it's indisposed, save Han will will kick in and throw a proper exception and directed, and it's all threads apeshit now I think that, if account, it has its own separate, ref count posit and last until.

A

Finalizer basically yeah.

C

Okay, so I think now I thought about something, so this is very similar to how memory of TN owned memory works and the reasons for their existence on memory. Basically, the class, the traps native pointers such.

G

C

You when you pass memory of T around you, don't end up with the same issue and memory of TSS trap. We used to have a feature in memory and on memory that lets you reuse the internal type on memory, so you don't pay the cost of allocations. Okay, so here's how it worked when memory of T is created for owned memory.

C

In your case when protein encoder would be created destruct for given state like they're handled, we would give it a quid and then, when it's disposed and reused, the internal object, we would give it a new grid and now, if you're struck, tries to access the class and it weeds don't match it throws.

C

Basically, this is how it detects the canary almost so we decided not to do. Basically, we remove this feature from member of the and I now regret and we removed it because we were optimizing performance of Kestrel and we were just removing everything that we could, because we had problems meeting there. Ultimately speaking.

B

C

I mean like what 16.

B

Bytes, for instance, or something would.

C

Well, so it's one sec, okay, so initially it was a grid and we just put the long and then Vance said we could even make it like an int and it would be so even a symbol to get that's like your.

G

Birthday paradox: like you, only need 65,000 instances before you get a collision with an in 32, so.

E

65,000 2 to the 16, but why first place paradox: no, because on both sides.

C

Yeah sure, but it so for at least for memory and on memory, it's more like a I guess, maybe there's some security aspect of it, but it was more about, like developers not shooting themselves in the. But what are you gonna do what fail fast? If you see oh, we say: object the spot just like we say today like if you own memory of T.

C

If you disposed underlying memory, we throw object, discover if you, if you throw wall of web request to screen.

G

Shows an error page. What you haven't actually fixed the bug in the application will eventually manifest itself.

C

Come on can reuse, it other appendages that misuse it not it. Okay, but that that that is current behavior like you, you can dispose the memory and it if you try to access it, it will throw or do this both there's.

G

About if it makes it a little bit easier to get into this, if we start we're using the objects, are in the same zone, not.

C

Why, if you dispose down memory, all memories of T today will throw object, disposed yeah, whether you then subsequently reuse, the on member. It doesn't matter like they would all throw I thought you were talking about.

G

Reusing and maybe I misunderstood with your scenario or some. If you reuse, then some of them might not throw no.

C

They will because they have the alt width so.

A

If they try, but.

C

If you get collision.

A

On the grid, yes, it is going to happen one into 64 K. Well,.

G

A collision bomb in 32 will happen. What will happen like once, every 65 thousand times a collision on quits going to be a lot harder to get into yeah.

C

So anyway, but the but what I was trying to make is you said something that not whether you have a collision, then it somehow impacts security.

G

Of a web server and items I wasn't making a security argument. I was just making a reliability argument like this. We don't want to be in a situation where a bug could manifest itself, just by an exception, being thrown and swallowed that the developer wouldn't have really seen while he was debugging or better. Yet we would probably want this to bring down the application if it occurred, because it means something's gone horribly wrong, but.

D

This basically.

G

Like me to say it again, what would you would not want to happen? So what I would not want to happen is that the behavior right now, because we actually have like it, sounds like we have an actual object which is disposed under the covers. If we try using that, we know that the objects disposed, because we probably have an associate and we can jump, we can throw an exception and we we take that request, but we haven't negatively affected anything else.

G

Aside from that one request, but we start reusing these instances instead of having a disposed field. If we just use like a canary or something like that, then what happens is we will eventually get a collision which means that where we expected an object disposed exception to be thrown instead, the code will succeed and have undefined behavior sure.

C

But I think what? If you have a code, that if you have code that, like that's something with this exception, like it's a bag? Basically, it's a bug. Detection mechanism is.

D

What I'm saying.

C

Is that, like the problem, is you can get lucky and.

D

You might only ever see under water or under stress, well.

C

The exception yeah, okay, so, and so what so? You see.

D

G

Yes, on the under load, yes, but this means that your your application, you have 65,000 requests over the course of a day, 64 or 65 thousand minus one fail. Yes, and one of them succeeds and does random things. Correct.

C

But I'm saying, if you have those, if you have that 65 that don't succeed yeah you either your server crashes or you have some code to handle an exception which is just a bug, but.

G

Nor normally that exception handling code is just like to learn or show show an error to the user and maybe file an automated ticket with ID. You basically just cut that particular request. You don't fail. The entire website yeah.

D

Oops I would I.

G

Would prefer to see, like literally an environment I'll fail fast eclis uninteresting.

B

Instead of objects, suppose yeah I would I, you seem do the work. If you want to get a crash down, you want to get it literally at that point, nor.

G

B

After if somebody caught this guy but.

G

C

Probably fail testers or thought about because I basically have it. You know.

G

C

Said I like it doesn't change that, like you know, you don't have a fundamental problem with the canary search, then I think we could use it for some ideas like this to optimize for performance. If we cared I mean for a memory allocations, if we cared enough, I mean.

G

Why not just have the application poured full.

C

What instances of the encoder and decoder? Well, then you have don't? You have exactly same problem that you're trying to evolve pull instances of the encoder and decoder yeah. You have exactly same problem. No, why logical operations can have now register the same.

G

Which is fine, but the application developer has made an intentional decision to do that, whereas if the class design just assumes you create the encoder, you encodes and stuff, and then you dispose of it and if the application developer really wants to go crazy and woman, he can but he's responsible for lifetime. Editor.

A

Right cool anyone, yes,.

E

B

Class of problems well.

A

We'd be provided as an API, which we make it even easier for people. No.

C

But that April is not good for it, because array pool is a pool of arrays and you just want to pull individuals. You.

D

Don't want an array code, it's not a normalized. Whatever do.

G

We not have like a regular object, cache of it. We don't, but we should.

B

When Wilson has their own, like whoa,.

C

I think there is something in WCS if I remember correct, there's everything it's.

A

Safe to say that.

C

Whatever that looks like we don't there's some sampling idea, probably any fish would be my efficient for WCF purpose, which is already.

B

Like to do so, yes, I'm.

C

At okay I buy that we should play with either here I'm. Just somebody cares enough. Maybe they should in this booty, so.

G

How that means we have to help the encoder and decoder be reusable if they want that over. That's.

F

Not really doable, okay, everyone! uh Well, it carries a lot of state. Once you write the final block, you can give it more data so for.

C

Compression or decompression, even so, but I think this, like. Maybe that's how the underlying native actually.

F

C

But why didn't? Why couldn't it work the following way? You write a bunch of these chunks, you say final, it writes the final now you can start writing new chunks and then belong to a new kind of sequence, and then they can have final at the end, then you can repeat the cycle sure. No. This is not.

B

The way Google population.

F

B

Should file it well, my neck I mean: what do you do contractor you go to the native API and you get like an init or something you get like the route object right, yeah, just replace that so.

D

B

Just say when we write finally, you basically freedom line memory and then remember that you don't have anything. So if people ride you just go back to the implementation and ask for the next day. That's.

F

Yeah, that's not something we could do I suppose.

B

F

To just restart the whole object, yeah.

A

That's what we're saying? Okay, that's good! You.

F

Could reuse the NATO.

A

Okay, our object like there's the cost associated down right, but you would have to pay either on the track class. Doesn't matter.

F

What is the cost of the object, though? It's a struct right? It's got.

A

One field application endpoint about no allocating.

F

A

Like if there's a native cost already but you're allocating again yeah.

C

A

Thinking about the using everything we can't reuse.

F

Everything but okay.

C

F

Want to make reuse just.

B

Consume because it's a shark yeah.

C

Yeah yeah that doesn't make any sense unless we can reuse the native memory which we cannot today, apparently mostly, we can, did you actually try it or you what's.

F

Already checked but I'm fairly confident you can't cuz.

C

That seems like not a very Google design, I know, I mean come on, like I would assume the structures are kind of expensive and you have to allocate them and they probably.

G

Have their own custom alligator, you.

C

Have to you have to allocate a bunch of native memory because you wanted to write EOF like that sounds not super frugal, but.

G

Have I mean have we measured that these allocations are even that impactful of performance when just the sheer CPU cost of compression outweigh ever it's kind.

F

Of what my results have shown, yeah I have it where we have the not allocating one the struck, whether just using it pointer and I, have the one with the safe handle and then I have try, compress, bully or shrine compressed, try and decompress or whatever and they're all very close.

F

The fastest is the try, compress try decompress, followed by the alligator.

C

You know what's the size of the native structures that is allocating and see some.

B

Point eight bytes: yes,.

C

F

Like this, many objects.

G

Keep allocations that we wouldn't see but does yeah yeah. So we see maybe a 32 byte structure, but there it contains pointers to it.

F

Does it allocates several times for the for the windows I.

C

Mean, and also like so measuring performance differences seem like allocating nada, okay, P Isis hard, because if you, if you don't have lots of threads and a lot of pressure and memory, yeah yeah they're, not gonna, be very different yeah the pressure all right, it's only starts showing up. If you have like a taking power, you know benchmark that, basically, like does things like crazy.

G

C

Every single taking powering request, like a million per second, you are locate like some massive internal native data structures and basically, if you have a stateful object that you cannot reuse, you would have to do it make it also do try to dress and avoid.

F

That problem, how it doesn't create the Dutton underlying data structures, underneath it doesn't create nearly as much I, see oh really, why it doesn't need as much at the DOS. It's only going to do it in one pass, which means.

F