W3C WebRTC Working Group, 14 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WEBRTC WG interim 2021-10-14

Description

See also the minutes at https://www.w3.org/2021/10/14-webrtc-minutes.html

03:24 The Streams Pipeline Model (Youennf)
44:11 Altnerative Mediacapture-transform API
1:28:53 Mediacapture Transform API
1:49:07 Wrap up and next steps

A

The recording has started: that's the first words of every recording.

B

Okay, this is the w3c webrtc working group meeting. We abide by the w3c patent policy, and only people in companies listed on the page are allowed to make substantive contributions at this meeting. We're going to cover some issues with what wg streams and then talk about the media capture transform api so a little bit about the meeting. These are links to the specs. The slides are up on the wiki.

B

We do need a scribe. Do we have a volunteer.

B

You don't have to take down everything, just the major decision points.

B

Any volunteers for note-taking subscribe.

C

And take care of it.

B

Okay, thanks dom and as harold mentioned, the meeting's being recorded and it will be made public.

B

Okay, just a note about the code of conduct, we operate under the w3c code of ethics and professional conduct, while we're all passionate. Let's try to keep the conversations cordial and professional a little bit about tips. uh We're asking you to we're going to run a queue like they do in the itf.

B

So if you want to get into the queue type plus queue and then when you want to leave type minus q in in the google me chat, so we'll be using that, of course use headphones and wait for the microphone access to be granted. Before speaking, uh it's also helpful to state your full name just for the recording, although we can probably recognize you by now. I don't think we'll be doing any uh poll we'll be using the google me pull mechanism, but we may take polls in the chat.

B

Okay, just a few notes on document status, just because something's in the wpc repo doesn't mean it's been adopted by the working group. That requires a formal call for adoption and just a reminder that editor straps don't represent working group consensus, but working group graphs do and we can merge brs without consensus. But uh as long as we have a note, okay, so here's what we're going to try to get done today, uh uns going to talk a little bit about the streams pipeline model and some of the issues that are under discussion there.

B

We'll then get into an alternative uh media capture. Transform api proposal by you went in yanivar and then harold will speak on the existing uh proposal and then we'll try to make sense of it all in the wrap-up and next steps um we're going to try to keep pretty rigid time control.

B

So we'll I'll give a warning two minutes before time is up and- and we will move on once we get uh to the timeline okay, so uh until 8 30 um we'll have u.n present on some of the streams issues that he's found in that around their discussion. In the what wg streams working group you and you have the floor, sure.

D

So yeah, as bernard said, uh this um presentation is about uh topics and issues we discussed uh partially with univar uh when we dived into uh exploring.

B

D

Stream is a good fit for media pipelines, so we, the goal, is to identify blocking issues and which may allow us or not to adopt uh streams as a founding api for a media pipeline. So before um maybe next slide.

D

So a media pipeline is basically a way to connect sources with things. So that's why there's the diagram there, where basically sources, are readable streams? Things are relatable streams and you can.

B

D

Javascript to connect sources and things or you can use transform streams, so the example shows shows there is like the angle that everybody would like to to have, which, which is go from the camera up to the network, with just uh streams. So um as we as we'll see, we will just focus on a video pipeline here and we we want to discuss the issues around threading and video frame, interaction with streams, and so let's look at threading first, so next slide.

D

So the first question is where to run the pipeline. Given the real-time operations here, the best place is probably a dedicated background thread.

D

If you look at web audio, for instance, an audiograph is very similar, it's roughly the same as the strings pipeline, except it's a graph but other than that. It's really the same.

D

But in the web audio case, uh the spec clearly states that the audiograph is set up in the main thread, but it runs in the audio thread and the sec is very clear about that and all implementations are uh abiding to it and the nice things is of course the audio thread is a high priority thread. So you get very good performances there in our case uh we're using a generic approach, streams and the situation.

D

There is not clear, there's, no concept of any other thread or main thread whatever, so the safest assumption we can make is to consider that video frames actually flew in the setup frame.

D

If we wanted to not assume that we would need to optimize pipe 2 and pipe through operations and maybe next slide, we can illustrate that it's it's not easy. So the example one there is uh the case where you're doing the funny hat thing from a camera to uh an html media element. So let's say: there's a native uh transform there and you can pipe two, the media element in some ways. So there's no javascript uh and there you have a call to play through and a call to pipe too.

D

So from reading the spec from uh this example, we don't know whether, uh where will flow, uh the video frames, it might be my infrared, it might be a background thread, it's really up to the user agent, and uh that makes things really complicated from a web developer. Basically, that actually wants to get guarantees and the worst.

B

D

B

U.N, that would be true, even if this uh code example were to be written in a worker. You think it still might flow on the main.

D

Thread, uh oh yeah, sorry, yeah, it's running the safest assumption is that it's running where we call pipe through and where we call pipe okay,.

B

D

So if we call pipeful in the main thread, uh we don't know whether it will be infrared or background fed, but potentially we can assume it's main thread. If it's called in worker, we would say. Oh, it's probably called in the worker thread.

D

uh If we go from example, one to example, two, which is the same thing, except that this time we have a js transform instead of a native transform, where we are almost sure that uh it will flow on the setup thread and uh not on back and forth, because you need to execute.

D

If you go with example, 3, which is using t after the native transform and using pipe 2 there, it's really unclear where video frames will will actually flow. Will it will it be optimized by the user agent? Or will it not be it's? We cannot predict that. So that's why. I think that the safest assumption we can consider is that the thread wears down the setup is also the thread where it's done.

D

The processing- and this is by the very nature of streams being a generic mechanism that is not but is applicable to more than video frames and more than real-time processing next slide.

D

So one potential related idea is to transfer the stream uh to worker, for instance, to get rid of main thread issues and uh currently to transfer stream. You it's specified in the spec. As uh you take the stream, you want to transfer you pipe to in a special identity, transform that will actually send the content over to uh the worker thread or the worker context.

D

Let's say um if you just basically implement the spec like that, you you don't avoid a main thread blocking, because the source is still in main thread, so you have to do optimizations like uh like the clever ones they they did in chrome.

D

The third issue here is that this optimization is uh not standard and it's really difficult to make transparent web developers as discussed in the github issue. The links are in the slides.

D

I also actually tried chrome last week and the current implementation around. That era is not compliant on a number of points as well, and the second issue really is that light pipe 2 and python. It's very hard to predict whether the optimization the transfer uh optimization will actually kicking on or not, and the fact that it kicks in is really important from a birth point of view. So again, this next slide for some examples, so example, one is the typical case. Where chrome will optimize the thing um you have a camera for a stream.

D

You transfer it to work. It's working fine. Now, let's say that, for the purpose of the example we add a native transform between the camera and the stream will actually uh transfer there. We don't know what will happen, maybe it will be optimized, maybe not uh example. Three is again the same. Let's say that you have a camera, readable uh stream. Let's say you t and you transfer one of the teeth. What will happen in terms of optimizations? We we don't know, and the same applies to uh non-camera streams.

D

Will peer connection, get display, media or get viewport media uh be optimized or not. We we don't really know it's. The spec doesn't say anything and let's say you transfer uh a media stream track like get report media to another frame, and you then take a stream that you transfer to worker will be optimized. uh We we don't know and from a web developer perspective, there's no way. You know whether it will be optimized or not. This is very different from web audio.

D

Where you have a spec, you know it will it's guaranteed that it will run in a background thread in an optimal thread. So uh that's why next slide? If we, if we look at the idea of using transferring streams for our purpose, we we see that it's really a generic tool like strings are it's designed for flexibility and it's working fine, but we we cannot guarantee performance.

D

um The good news is that we have media slim track transfer, which is a dedicated tool to mediate track and there we can design it so that we guarantee uh performance. We can guarantee that the uh the transferring will be optimal and uh that's the proposal. I think we should go.

D

We should go with, which is to build on mediation, track transfer, and then we we have no hard requirement to re, resolve the github issue to actually unable to uh have a spec, compliant uh optimized, video frame of optimized stream, video frame transfer mechanism, and still, if uh with that, we have like uh optimization like type two is optimized. Python is optimized. It's it's a bonus, it's great.

D

You have more, but at least you, you you're sure that you, you get a decent level of performance because you are in a worker and uh it's as if uh french will go in the worker thread, instead of being blocked by the main thread.

D

um So that's it for.

D

Let's look at pipeline buffering, that's the next slide. So as we uh discussed in the in the past, um buffering with streams, so it's specific to streams happens at each transform, stat step in the media pipeline.

D

So in the slides, there are like two examples and the typical stream pipeline is the one on the top where the pipeline is filled greedily, which is uh great when you want to process all chunks like, uh for instance, when you have a big document and you unzip it progressively and you you transform it previously, like you, you're doing utf-8 or whatever the processing can be optimized so that it happens, sort of in parallel, it's up to the user agent and you're also sure that you have the slowest delay possible between the compression and the renderer.

D

But in our case, where we have a video media pipeline, uh it's sort of different uh first, uh you might not want to process all frames if there's a one second or video frame, for instance, you might prefer to uh skip it and get the freshest frame, for instance, and that's what media stream track processor is is doing. Actually, uh if you do not process frames fast enough, it will drop frames so that you next time you will get the freshest frame and uh to to have that and to also reduce buffering.

D

uh The second pipeline uh on on the silver second below is the one where uh we are doing: processing sequential, which has a cost of course, but it has also some benefits uh in in our case, and um I think that in general we it's safer to to get the second pipeline model. For me for video frames, and the first issue is that the second pipeline cannot be built currently with streams.

D

You can only basically uh build the the first pipeline next slide, and so this is the real issue because, uh as I said, uh syncs are really hoping to get. The freshest video frames and also video frames are big and scary sources, and also it's not very clear from the web developer perspective. What happens because the buffering is uh sort of sort of hidden to to your application?

D

So we really need a solution to turn off the frame, so universal issue- 11 11, 11 58, which and chances are good high- that we will get a solution there and the solution uh displayed here might be uh using a watermark highway to mark of zero and calling an api.

D

When, when you need it, um I I don't think the plan is to have uh noble frame by default, meaning that what we would like to get as a default, which is the safe, safe behavior, will probably not be what streams uh will be doing by default, and so uh this requires web developers to learn about that which might be good, but also, if they use streams, they will end up into the first model, which might have side effects.

D

So it's a bit sad that uh there, if we use streams, we we end up with uh not using what. What is the default uh behavior of streams.

D

Next slide, and it's it's also a bit a bit sad, like generally for streams, the buffering you you don't really care about it, because the idea is that streams will handle it for you for back pressure. But in our case for me, the pipeline, maybe some limited buffering, might be good. uh If you have two long tasks in the pipeline, like you're doing encoding and uh funny hat thing, then you might want to have like uh doing the encoding and funny hat thing is sort of important to to reduce delay, but nothing else.

D

So it would actually be good to uh do some allow some limited buffering, um but it's not really easy with what probably, what working with streams to to do that. First, the buffering is done in their own queue and it's mostly a pack opaque by design.

D

So if you ask as a developer a question like how much buffering am I doing in the pipeline, it's not really easy to to answer the questions there with streams and also the the queueing strategy is very static.

D

So you set up the pipeline, you set the high water mark and that's basically it if you want to change a little bit, because there might be some other constraints, then you basically need to uh redo your pipeline. So you redo your setups so um that that that's that's not great um plus. The idea.

B

D

Yeah you're, using like non-default stream approaches, which are not easy to understand. um I think what streams might be able to uh cover the use case, but in terms of complexity, I'm not sure that it's uh it's a good trade-off honestly.

D

um So that's it for buffering. Let's go with uh teeing now.

B

uh Question uh I think there were some questions on the buffering issue. um Is that what you wanted? We're in the queue for you on avar, so.

D

um So how should we proceed there, because I I only have like until 10 minutes? Why don't we finish the slides and then we'll we'll take the questions if that that works? Yeah, okay, so let's go with t. um As we know, mediastreamtrack have built in support for multiple consumers and t is the way to do this. Multiple consumer thing with strings a typical example in in this.

D

In the slides there is we we do some effects on the video and then we we key the the end results to do both rendering and encoding in parallel, and the use case might be doing analytics in parallel to rendering, for instance, and t is valid api it's already used for the stream, so we should be able to use it and we should. We should not be in a place where we would say. Oh no, please not your st. We will find something else. We we should be able to use this so uh next slide.

D

As we discussed this uh a couple of times. We know that t is broken if a stream is steered in branch, one and branch, two, the same object, so the same frame f will be presented to both branches, and if one branch is closing f, then the overbranch is only getting a frame which is already closed, so cannot cannot really use it.

D

The potential solution here is to use a structured clone as discussed in issue 1156, and uh it should be fairly straightforward to adult I mean uh if you compared to the other case with buffering there. It's stretcher cone equals true and we, I hope we know what structured clone means. So that's that's kind of uh okay.

D

It's a bit sad again that the default behavior is the focus one and there's a misalignment between the default streams, behavior and uh and the behavior we actually want. But at least there are there's a clear path forward. That can be understood reasonably well next slide.

D

So are we good uh not really? We need more more chance, more changes to key than just structure clone, because if you apply structure clone, uh then you create hidden buffering and if the branches are not consuming uh at the same pace, you end up into issues. The typical issue will be that the camera will run and run out of buffers and branch.

D

One will starve of data because range two is not consuming enough of the data and the typical solution in uh in typical uh real-time processing flow of video frames is to drop frames as needed. Like media frame track. Processor is doing that way. Branch one can continue at its space and branch. Two can get fresh frames as well.

D

um Understanding of the issue right now is uh that there's no clear solution to that issue, in particular streams by design, are not expected to drop frames internally and, if you add external mechanisms outside of streams to read as fast as possible, then you end up into issues with potentially with pressure as well and- and that's that's really a bummer, because it's very specific to what what working group screens as we said, uh as we saw uh in july gs, promise based callbacks.

D

There can solve this issue in a couple of lines, so it should be possible to uh to solve in a in a good way, but since streams have a model and the model is not aligned with the goal there, it it makes things more difficult than than it should be and uh yeah, but that's an issue that we we need to solve. Somehow if we want to use strings there.

D

Let's focus on video frame life type management, which is the last issue. I think we we're planning to discuss today.

D

So the expected use of video frame is to call close as soon as possible, because video frames are scarce uh resources and usually they stem from a buffer pool and if the buffer pool is exhausted, then you're not in a good situation, and we do not want to rely on garbage collection there.

D

But streams are relying on garbage collection, and that makes the overall approach uh not great. Let's say, for instance, that we're using a stream wrapper around video encoder. We need to decide whether the video encoder wiper will close video frame.

D

We call video frame close or not, since video encoder does not call close, it calls internally clone that media stream track generator is calling close and so that, if you pipe two medium track processor to the stream track generator, then you end up with calling close automatically so we're seeing that we have two different models here and streams could be used both ways, there's no uh uh way to enforce, like the case where a writeable stream will you always call call close, for instance,.

B

D

It's not done that way and from an api perspective, it's difficult, an error prone for a web developer to actually understand where they are in which world they are and which whether library will go in in world one or well closed or well, not close and up to you uh since there's no api contract and um I'm not quite sure how we can solve this.

D

um What possibly one possibility would be to go with some form of uh dedicating handling in the stream of video frames object like a subclass or something like that where we would have built-in video frame memory management. Let's say that you call uh you enqueue a video frame, then uh the video frame will be cloned and when you remove it from the queue from the internal queue, you will call close.

D

I haven't heard any work in that direction, so it's uh something we need to solve and if you look at the pipeline, if you start to uh change the pipeline, you will need to cancel things, and if you cancel a stream that has some video frames in its queue, then you rely on garbage collection to get the video frames, and this will happen if you, for instance, tear video frame and you one of the key video frame will have uh buffers in its queue and then, if you cancel one of them, then uh you can only use garbage collection.

D

So we need a solution. uh I don't know what we can do there, but it's certainly something that we we should do, and I have two minutes for the conclusion. So that's good next slide. um So yeah next slide, so we we need yeah. We need to solve those issues, uh buffering t and lifestyle management specifically for video frames, a stream studio frame.

D

So if we do not do that, then uh things will be difficult and it's unclear. I mean progress has been made and that's great, but we need more and it's unclear to me how how far we can get, and maybe you can go to directly to the last slide conclusion free.

D

So next slide yeah, so yeah and my point is that we need to solve those issues or have a very good confidence that we'll solve these issues before selecting the api model, because if we select a model, it means that we we want developers to go into that model. So we need to be very sure that it's a good model and also since, if we pick a model, we won't, we are sure of it.

D

We if we use, if we select streams which has its own benefits, uh we should exam streams, integration with exiting a new apis, and that means video decoder, video encoder. I mentioned barcode decoder and face detector, which are also existing apis or at least proposals, and it does not seem that we are going into that direction.

D

So far on webcodex side, I haven't heard of any desire to use streams. In fact, they decided to not use streams because it's helping for the error handling and also, as we saw video frames specific model, might make things more difficult for web codec as well.

D

So that's it for uh my presentation and uh there were some uh questions.

B

Yes, uh we're gonna open it up for questions now for uh yannibar.

E

uh Yes, sorry about that so yeah. I just had a couple of comments on the uh slide.

F

E

The controller release back pressure, uh I think the goal there is you and mentioned that that won't be the default behavior, but I do believe the idea is that, specifically, if you create transform streams which have a readable and a writable, uh if you put a high water mark of zero there today, the pipeline just stops so yeah.

E

I think the idea is that if you specify a transform stream, you should be able to specify a transform stream with a zero high water mark, which means no buffering, and then the transform stream uh will automatically call release back pressure when uh something downstream from it reads from it. I think that's the main. What we're trying to solve is the concept of bufferless transform streams, which would be helpful, um and the other thing was about dynamic buffering.

E

um It's true that the the convenience feature of providing high water marks is a static one that you set up at the time of the pipe, and you have to tear down the pipe again, but nothing prevents javascript to write, transforms uh to implement their own buffering, which could be dynamic and they can even drop frames. They just can't have a zero high-water mark to be totally buffer-less today and that's what we're trying to solve in that issue um and on the third and last on the stream reliance on a government collection.

E

I think that pops up as an issue in error situations- and I would suggest that we might be able to solve that by having the the producer of the frames own on those. uh So there might be some so error. Situations is uh a narrower scope of problems, but we still need to solve them.

D

Yeah, I'm thanks. I'm I'm not totally optimistic in solving that at the source level. Honestly, I'm I'm looking forward to the solution there, which, uh if we have a solution, then I I'll be happy to remove my concerns, but uh so far I do not see how it will work. So my my understanding is that uh and more generally with uh lifetime management, uh there's no api contract right.

D

If you can use you don't know whether close will be called or not, and it's depending on the api, and but I mean I like consistency and I like that we would go with just one model if we, if we had like, uh if we would implement our own stream based video frames, we would uh probably when we queue a video frame into the internal queue.

D

We'd probably call clone, and when we remove it from the queue we will probably be called close and things like that, and we we would define this memory management and we are not so in any case, we we are not on solid grounds. There.

B

Okay, um I had a few questions you when, um in the current model, where we don't have high water mark of zero in the transform frames, have you noticed a significant delay building up because I've been trying to construct these pipelines with up to like five steps, and I haven't haven't, noticed enormous delay.

D

um It really depends uh what what happens so, what you should try is, for instance, on, uh like let's say you have a render, which is a transform stream. Let's say it's at the end of a pipeline. Let's say that you add a one second delay and you look at you look at which video frame you will get and so on, and you will see that uh things will will not be will not be great there. It's true that uh what what you happen to have is like.

D

Currently, if you have like five steps, then you will have five five frames, each one in it in its own right.

B

D

So, trans transform- and uh if you, if you're processing things very quickly, uh things will be good if one uh step is taking on some time like encoding, for instance uh it it might not be uh as good, especially if the delay is like changing if you aren't cutting an iframe or a keyframe things like that, so in and also um the issue is that currently in the camera pool you, you might have like 10 10 video frames right and if you, if you build a five steps pipeline, you have five frames that will automatically be uh allocated and it's a fixed.

D

It's totally fixed. So now you only have five remaining uh slots and you need to be sure that the encoder which you do not have control with, will actually release natively uh the frame so that the camera can can.

B

D

To it and if you're adding that to the fact that maybe it will go out of process and so on, uh maybe five is not enough and it's on devices that have 10 frames. Maybe maybe some devices will have a lower capacity and will have a smaller number of uh buffer buffer frames, and then you will end up in two uh glitches where uh video frames will like the camera, will stop producing until you release frames and you will have frame rate that is decreasing and changing and that's that's.

D

That's not great, uh because you actually just did five uh transform in your pipeline and yeah. We should not be invested.

B

In that issue, so one thing I noticed by the lack of streams integration within uh web codecs is that right there are actually two cues here: there's the encoder queue and then there's the streams queue, and so you kind of have to manage these two cues yourself um and in web codecs. There's no is really the the cue is not uh transparent at all. You have to kind of keep track of it yourself with the pending output counter.

B

So that's another weird thing like you can, in my own code, I've been seeing up to 30 frames, potentially queued in the encoder, so you have to kind of keep track of that. So that's another interesting delay.

B

I have seen effects on the browser itself. When you get errors, sometimes it looks like memory. Management has been significantly disrupted until you garbage collect the entire encoder and then things kind of go back to normal.

B

So I think there's something there and the memory management that's pretty significant, um but it's not that there is this mismatch. You wrapping streams is not entirely satisfactory. I don't think particularly in our cases.

D

Yeah and I want to mention uh the promise.

D

A

I I I really like to have more minutes for more people, so you and you already did your presentation.

B

D

B

Okay, uh harold.

A

Yeah, so the discussion is discussion, not just questions. uh I will notice a couple of observations. One is that uh the web codex did have a stream based api for a while, after we developed the media stream check processor and made this change track generator. They said we don't need to have another one, so they dropped them so, and we have had actually very few reports on people having having trouble with these so-called issues.

A

I mean the the discussion of uh iran 24 the the issue of first of of sarah buffering. Oh.

A

I mean the no no, the the issue of transfer was a bit depressing because uh otherwise, uh because it started out with a few messages from people who understand street streams very well, and then it got the couple of back and forth between me and y'all nevada and then nothing so I'm eager to see that discussion come back, and my impression is that the real problem here is that uh the streams model has been somewhat somewhat uh confused, with the act with the streams shim implementation, which has these problems well.

A

What we should be doing is to have a model, have a clean model and the issues are with implementation of the model, and so.

A

On the comment of t, I had had a fun experience of reading the the cl that added the tea to the specification and all the people, all the things that people were worried about with tea when they added it to the specification were in fact the same thing. We're worried about t is a bad design and it's fairly trivial to write like this much javascript and have the tea you want, and the tea you want is actually quite dependent on on your application.

A

One of the nasty pieces of tea that you didn't mention is that it doesn't respect high water mark on the on the downstreams, which stunned me when I discovered that so t is bad. I realize, on the contract point I'm just taking trying to take off the point so that we get to it within the next five minutes. Are the contact points yeah?

A

I think it's natural to say that a downstream either have to call close or pass the thing on to something that the frame onto something that calls close for video frames and and and that and that they shouldn't depend on upstream to do anything. We do have a problem with uh disrupted pipelines, as you say, in that disrupted pipelines, don't seem to have a good method of letting the user define what should happen to what's in the pipeline when it's being disrupted.

A

We should solve that. Yes, but my conclusion on seeing these issues raised is that.

A

There's some of the some of this is issues with with the description more than the implementation, and some of these are issues that we need to solve, but uh they're not fatal, like the t thing is just because it's possible to do something badly is doesn't mean that uh that that all usage usage, so this is bad.

A

So my personal opinion is that we're already in the place where, where a stream spaced api is superior to a callback based one, because the kodak ways one but just means that you have to build everything on top of it all over again. So that's my notes from the conversation so far. Another question.

D

Can I answer or is it too late uh no time.

B

uh Well, uh three minutes sure.

D

We have two minutes: two minutes um yeah. I agree with you that tea is bad and uh we can try to salvage it. It will be difficult, and this is this- is due to the use of strings. As you said, you can use javascript to do your own key and it will be done in a much better way and that's why promise based callbacks is. uh Is the right? It's right approach.

D

If you actually want to use t, because that's what we will use, you will use promising promise based callbacks uh internally and you will asynchronously uh iterate over the stream. So in that case, why should we even use things on the closed case and on other issues that you that you were seeing they're, not fatal?

D

uh I welcome all the transfer streams as well. I welcome uh changes and I welcome uh proposals that will uh fix these concerns.

D

If we do not have uh proposals, then what should we do? It just means to me that uh we I'm not confident. uh I want proof that reads will be solved and I want proof before we actually decide. Otherwise we're not in a good place, and uh all of that is because we select streams which has a defined model which sort of match, but not entirely, and I want it to match entirely and so that it's the perfect, perfect thing and then I'll review.

D

If we fix these issues, uh I I'll be happy to to use streams because they they offer some some things that are great to have, but they currently offer also very bad things, and I want to remove the bad things before we go further.

D

E

I want to disagree with harold a little bit first, that I think all these issues are filed on get up they're public and that they are issues with a model uh and there are numerous.

E

uh At the same time, I wouldn't say that I would also disagree with u.n, but I don't think the issues are huge and I don't think we should block picking an api surface uh over these, because, uh frankly, one implementer is already shipping an api, and I think we need to pick uh and standardize my api as soon as possible and so and with promise callbacks you're right that the web codex lifetime model for video frames makes it tricky to keep track of things with streams and promise.

E

Callbacks may solve that very easily because uh you, but the problem is that a promised callback also doesn't work. There are areas where streams work better, for instance, with audio which we're not talking about yet where, uh for instance, you encode some audio that doesn't mean there's a one-to-one between encode and getting a callback. Now streams can handle that, but at the cost of uh losing track of it's not a simple input output, one to one.

D

I will I welcome pros and cons between uh promise-based callbacks and uh streams callbacks, it's true and streams. It's true that there I only uh showed the issues I have with strings and I I'll be happy to also get a list from you, our geneva or lab about promise based callbacks and to identify what are the issues and then we can compare issues of both models.

B

Okay, I think uh we've got to move on to the next presentation, which is yanivar.

B

So yeah you have the floor.

E

And unmute all right. uh Thank you for that uh setup. uh You covered more threading than I anticipated, so this is great. um So the first off. uh Why are we here? Well, uh threading is a big issue, and I wanted to highlight the build on what you and just said that today, the status quo in in specs and the cross browsers is that the real-time media pipeline is off the main thread.

E

That means that the midi stream track interface is purely a control surface on the main thread, but that media flows in parallel either on the dedicated media thread uh or other threads multiple threads. So when you call get user media, you get a track back and you attach that track to appear connection you're pretty much done.

E

The browser just takes care of it for you and if you apply constraints to that track to reduce the uh resolution, uh that's an asynchronous call that, uh where the actual change in the bitflow habits off of the main thread.

E

So, as I illustrate here, you can have a media thread uh where camera frames move through a downscaler to an rdp sender, and it goes out on the network and the main thread is not doing anything it doesn't get in the way of that I mean that's important. Media delivery never ever blocks on the main thread next slide now.

E

This is true, even in webrtc encoded transform, where we recently added apis to the sender where javascript can modify the encoded data and uh and there's a bit of a deja vu here, and that chrome originally uh released an api that was on main thread and we standardized an api that was off main thread.

E

uh So here again, the media thread has a send where the bits go: sender encodes it and it sends it to a javascript encrypt function, that's only exposed on worker and it sends it out over the network and again, nothing blocks the main thread and that's for encoded video, not even raw video. So since we recognize the importance of this for encoded media, we should be even more concerned about exposing raw uncoded media to main thread, because it's a lot more data next slide.

E

So the premise here is that the main third is bad, and so how do we back that up? Well, uh chrome dev summit in 2019 had an excellent presentation by cirma, which is basically the main thread is overworked and underpaid, and I'm quoting literally here, there's a link. If you can follow the youtube uh where therma says we are setting ourselves up for failure here, we have no control over the environment. Our app will run in the main thread is completely unpredictable.

E

What takes uh two milliseconds on a modern flagship phone might take 20 milliseconds on the next low-end phone. How can we escape this unpredictability and later in the presentation? The answer is web workers.

E

uh He also outlines the uh the frame deadlines that we have uh under 60 frames per. Second, for instance, we have 60 milliseconds to complete all processing of a frame before it got goes to rendering or it's gonna jack.

E

The video is gonna stutter. So on high-end devices you can have you know, 90 frames per second 120 and even some devices with 144 frames, whether we're less than seven milliseconds of a deadline. To finish what we're doing uh and then so. The point here is contention on the main thread is common and unpredictable, and that leads to intermittent that can lead to intermittent missed deadlines and video stutter.

E

Unfortunately, things may run fine and controlled environments on the developer's machine in the developer's browser, but not in all browsers and all systems, for instance, when the users actively scroll their page a lot. This becomes a web compatibility issue.

E

In contrast, contention on a dedicated worker thread is relatively non-existent because it is a controlled, dedicated environment next slide now, so we mentioned that web codecs recently had a decision to expose to window and here's what they wrote. They wrote there is consensus and we agree that media processing in general should happen in a worker context.

E

But then they said not all web codex use cases require this and they mentioned non-real-time transcoding, and that was their main argument for exposing this to window and they say expressly that this decision should not be interpreted as a precedent for allowing the proposed video steam track processor in window and that we need to discuss and consider it on its own merits, which we're doing and they also inspected.

E

There's a best practices for authors using web codecs when, when working with real-time media are encouraged to ensure their media pipelines operate in work, to contact entirely independent of the main thread.

E

E

So where is this working group uh on on this? We have, unfortunately, a media capture transformed uh document that says it's not an adopted working group document.

E

That means it's non-standard, because there's no consensus and google, unfortunately has shifted in m94. So there's a time uh limit here. I think for us to uh try to standardize something, or this becomes the factor standard. Now. Why wasn't it adopted? My my thoughts on this? Is that the problems I had with it? I should say.

B

E

It exposes real-time pipeline on the main thread by default.

E

Tracks stay on the main thread, while their readable writables may be transferred by the readable may be transferred or not, and this is also fails to encourage using workers both on javascript users and browsers implementing this stuff, because it requires extra optional work by web developers to get off main thread and it relies on non-standard browser optimizations to fully get off the main thread.

E

Also ergonomics-wise. These apis are thread couple to main thread, which is very unfriendly to workers which I'll show in subsequent slides. So we need to standardize an api without these problems and reclaim the url so- and this also was designed before media stream track was transferable, which we just had a call for consensus. That was positive, so we can rethink the api next slide.

E

So, um since media stream track is now transferable, we need to go and look at first principles from a worker point of view. The question here is a worker encounters a track. How does this access? How does it access its data?

E

Does it post message it to the main thread to ask it to create a media stream track processor and post message to readable back, or does it access a readable on that track? The former makes no sense. It shouldn't have to go back to main thread to get access, and this also doesn't necessarily get us off main thread in all cases. No process next slide.

E

A second question: a worker has video frames. How does it create a track from it?

E

uh Does it have to post message to ask main thread to create a generator and post message that writeable and maybe a track phone back, or does it call some api to get a writable and a track again? The former makes no sense that workers should be able to create a track directly without having to ask main thread to do it, and this points out that mediaseam track generator is a weird object.

E

It's a track stuck on main thread: it's not transferable, yet all its parts, the writable and track clones are so if we ever extend it we're stuck on main thread. So I think this is backwards, and it also warps code organization for developers to be main threat, oriented.

A

It's also untrue.

E

Let's not discuss that. Excuse me at the end, yep I'll get back to you.

A

E

um So these questions are getting a little harder, but apologies. So this is very. This is a more difficult question, but very relevant one that that euan already posed. How does a worker send processed video frames say over web transport without and also maintaining a self view that has high resolution, even though the transport might need to drop frame rate? So we mentioned t already and all these issues.

E

Another option would be to clone the original track. But then, if we're doing like a video replacement like me in the sky here, you have to do that processing twice and that's undesirable, or do you post message, constraints to the main thread, create a generator apply constraints to a clone from the generator create a media stream track processor? From that clone and post message, the generator is writable and the the producer is readable back to the worker.

E

uh No, you would want to have these apis available in the worker, um and this is also an argument. I've heard is that mediastream tracks they need to be on main thread anyway, right because you're going to assign them to video source object or add them to a peer connection.

E

So here's an example where uh send track in this case uh is solely created, so it's a track clone that never leaves the worker and is used there to natively downscale post-processed frames, instead of trying to drop frames or down scale using a transformer or using t.

E

So this is an example where tracks are actually useful in workers next slide.

E

And I also want to talk about that sentence. I I rushed through here, post messaging, uh immediately, retract processes, readable or a gender is writable back or transferring them at all to a worker, and that this actually violates the intent of what wg's transferable streams which uh which have tunnel semantics they're a special kind of identity. Transformer me, which has a the writable side in one realm and the readable side in the motherboard to implement uh cross-thread transforms.

E

E

Let's look little a source in thread, a that has a readable transfer that readable to a thread b that could then read from it.

E

They only transfer one side of the stream, basically the readable or the right one uh and they create exist tunnels on purpose not solving not to solve creating them on the wrong thread in the first place and without spec breaking ui optimizations.

E

That means media is doing track process where readable remains anchored to main thread on one side, even after transfer which uh to me makes mediastion processor a broken api built on broken assumptions and next slide.

E

So an alternative proposal and here's a link to uh issue 1559, which also has a spec document and more details you could look at. I would encourage you to look at uh after this presentation since I'll be going through the basics here, the goals are to align the api with transferable medium tracks, for simpler api surface and remove the aforementioned blockers to standardization, by exposing real-time media pipeline to workers and not main thread, and also encourage user workers by making it simple friendly to workers and the default, and also discourage usage on the main thread.

E

This proposal also starts with video to in order to reach an agreement and because we're trying to present an api to the working group that will be adopted, and we do so by removing exposure on the main thread, which is what mozilla and safari want. We focus on video for now, which is safari, has been interesting and we're also using we're still using streams, which is what chrome mozilla prefer.

E

So you might wonder uh we just had all this. You unpresented all these reasons why streams are bad.

E

I don't think I think that's orthogonal to this proposal and I think those streams problems can be solved and there are ways to actually where a readable stream can actually give you promise callbacks if you want to- and I have a slide on that if you want next slide onto the api, so here's the proposed api and I direct your attention to the blue box in the corner, which is level by dl, we're basically exposing a readable attribute on the midi stream track, but only doing so in dedicated worker contacts.

E

That means, if you see a medium stack on main media stream, track on main thread. You won't see this attribute, but if you post messages to a worker- and you look at it there- you will have this readable attribute, so I'm showing the javascript only the worker side of it here. You went showed example slides earlier, where that includes the post messaging, so here a worker. This is a sort of read-only example where we're just reading and we're sending it over web transport where we get a track in the worker.

E

We await we basically just pipe it's readable through some kind of a wrapper for web codex, and we also have to transform it to serialize it, which also means chunking it and stuff for uh you know, and it this. This might not be how you would send things over web transport, but this is a simple example right, so it says uh sync and that's it. uh The key here is that the attribute is only exposing workers and that keeps data off the main thread next slide, so a little more complicated.

E

Let's say we want to read and write. We want to basically have a self view here or show local view only and direct your attention again to the blue box in order to uh so the previous slide was basically the stem track processor, and this slide is made into a generator where we need to uh put data back into a track, put humpty dumpty back together again, so we exposed only on.

B

E

A new interface called video track source that has a writable attribute and a track, and also a boolean muted attribute, because sources in the midi stream track model you can mute, sources can mute themselves and all that means is that downstream tracks uh get muted events fired on them, so here again the worker receives a track.

E

So what we want to do here this is an example from web codex that I modified it's basically a crop example where you use offscreen canvas to crop some video, that's basically the simplest example of modifying bits that actually works. So we, the worker, gets the track. uh We create a video source object and we post message the sources track back to main thread where it can be assigned to a source object, and I know what we do the same.

E

As on the previous slide, we pipe the readable through a transform stream, which is shown below as a crop, and then we pipe it to the sources writable, and that's it that's it. This aligns better with the media capture main spec, which separates sources from its track, and this makes it easy to extend the source interface later. The source critically stays in the worker, while its track may be transferred cleanly to the main thread, and this is also extremely clean and simple, with uh how interacts with track clone uh and uh even structure cloning.

E

uh That's actually a flaw with me to subtract generator. Is it's sort of a pseudo object? That is a track, but it's also a subclass of a track. So if you call track clone, you don't get another major stream track generator. Now you get a plain track, so this solves that next slide.

E

All right uh process video and apply constraints, so an issue we have is to, if we're going to do any kind of processing like say we're, cropping a video uh you might want to have a self view of that. That has a high frame rate, uh but you might also want to lower the frame rate depending on where you're sending in so here's an example uh that, where you're sending over a peer connection, uh which is basically you send it to the worker, do the processing send the track back and.

E

Sorry, uh this one: yes, so you assign the so when you get the track back to main thread, you assign it to you admit you take a clone, basically and assign it to the source object and then on the track. You're sending you can now apply constraints and you can have a lower left resolution and lower frame rate on what you're sending so you can send, for instance, 30 frames per second, while the self-view doesn't start to stutter, but it's still a good 60 frames per second. If that's what the camera source was.

E

That's great uh next slide.

E

Now that works great for peer connection, but what, if you're sending over uh web transport? Well with web transport? We really wanted to encourage you to use open the web transport in the worker. So this is the same example basically, and it shows how you can uh how media stream tracks now are useful. Also in the worker. Again, we receive a track. We get a source from it.

E

We clone that source and we send the original source back to main thread for self view at 60 frames per second and then now we have the clone, which is called send track. In this example, we can apply constraints to it. Do anything we want, just as if we were on main thread and now we can open a web transport connection and we can read from that send track.

E

So what we're actually reading from we're doing two things here: we're reading from the track readable, sending it through the cropping and piping it to the source writable and, at the same time, we're reading from the send track readable, which is the processed downscaled data and sending it out to web transport I'm using pipe through here, which is a bit clever thing an earlier example. I had a promise all, and these are really two operations that are separate, but it shows you that you can natively downscale and that medicine tracks apply constraint.

E

Is it's very useful as a workaround, basically to needing tea, and so the question here for me just imtrack generator? How would you do this? How would you transfer media stream check generator to the worker, and I don't think you can so next slide?

E

So this is the summary slide here um benefits that we see the simpler api that takes advantage of transferable media's gym track. There are fewer new api objects to learn and it's friendly to workers.

E

uh It satisfies core principles from a worker's point of view without main thread and tangled apis means that the worker has a data object. It can do something about it and critically.

E

It does not block real-time media pipeline on the main thread by default uh and the source stays in the worker separate from his track, which gives us clean transfer, apply, constraints is available in the worker, there's uh parity uh and more with media stream, track processor and generator and both features and brevity, and I have a link to a javascript fiddle and that this highlights.

E

I think harlow's also going to show that the differences between these apis aren't that uh drastic as far as the number of lines of code and uh it's more about whether it's allowed in workers or not, and whether it relies on transferable streams, ui optimizations and there's a source muted attribute which we threw in and that's it.

E

I also have some appendix slides that I might have time to go through since I'm a little early. uh If we look at appendix a um I might as well go through them. uh There was mentioned earlier that we want promise callbacks, not streams.

E

Well, you can actually do that because every readable- and this applies to any streamspace proposal- uh you could use 408 on that readable and you're, basically being called back you're getting uh your async function is going to be called then, and that means that you can basically operate. This is basically a promise. Callback is what I'm trying to say and uh here's. What uh un wanted to show you think is that you can now do operations like encode and you can guarantee that frame close will be called.

E

um And appendix b, the next line and there's also, I added the slide only if people want to have more than one readable from a track but uh enter so far. It seems like people are more than happy to suggest cloning. The track for these things, which came up to avoid t, for example. So I don't think this is that critical, but I want to pull it up, throw it out there and I think, that's it. I'm going to skip the last slide and we can move to discussion.

B

Okay, so we're have discussion until 9 30.- uh I guess okay, so we have harold and then uh in the queue yeah. So.

A

I kind of like the proposal, first of all, because it's almost totally equivalent to mstp and msdg I'll return to that later. A couple of points I want to make on the presentation.

A

The the examples where you have posting a message to the main thread to to do get a message, stream check, generator and uh and get that to get generate the stream and post it back. That's just wrong, I mean obviously uh mstg and nstp were designed to be available in the same context as the trackish content. It is available in so as soon as so as soon as track is available on workers, msdp and msctg obviously need to be there too.

A

I was not very happy with the quoting of uh chris cunningham's message for on on the on the web codex decision, because one of the heavy uh things he said was that the availability of related apis is a strong reason why developers need to be able to.

A

Use these apis in main thread, and- and so I don't believe that, there's an accurate representation of what he said.

A

uh He did mention that his decision, they did not uh did not foreclose the discussion on msdbs and mr g, but so it doesn't foreclose it either way, and the third point was the the picture of uh transferable transferring streams where you said that it's a pipeline between the region, context and- and this is nation destination context- that's completely true, but you imply that the sword contacts this the main chat, which is not true.

A

If you have a camera, the source of sort, a camera that generates threats, the source of the of the threat of the stream is the is the camera not the matrix.

A

So these are just things I would say, misrepresent the con. The context uh they're not direct comments on the actual api, we'll revisit the discussion about whether or not they should be available on matrix, but the shape of the api. I like it it's very much very, very, very similar to what I propose.

E

Okay, I I I apologize. If I misrepresented, I see that uh the mediastrip track, processor and generator is exposed to dedicated worker, and I guess I missed that. But however I don't know that they're transferable.

A

No they're, not they shouldn't be.

E

Okay, so you would have to create them in the worker.

A

E

Which raised the question of, um but um so I would still say, though, that uh the model that people seem to so it's your suggestion that people should so. There are two ways to do things now right, so you can now either create you have your track on main and you create a media stream track processor there and you transfer it's uh readable to the worker or you transfer the mediasting track to the worker and you create a major stream track. Processor. There.

A

Right, the the.

E

A

The moment is that we can we people can implement the first one now and uh because we don't yet have transferable media stream tracks or availability of mediums interact on worker. We haven't worked through that yet codewise and so right yeah. Once we have it. Yes, you have two ways.

E

Yes, I think that's bad stream.

A

Streams are not going to go back to being non-transferable so or okay, but obviously in some browsers we don't have we don't they haven't implemented it yet. But so then transferable streams are not going away.

E

No, I agree, but they should remain what they were designed to do, which is to create tunnels between threats, and uh we wouldn't have to implement these these magic uh optimizations in order to get things entirely off main thread, which is only a.

A

E

With the first model, not the second.

A

We have, we have a difference of opinion and what that timeline actually represents. So right.

B

Probably listen to you.

A

D

uh Sure can we go to slide 37.

B

D

Yeah, that's this one uh where you say t is bad and we should not use it. I agree.

D

Currently, I hope we will be able to use it, and the reason is that uh the the small example is not creative, either you're losing back pressure, and uh I know univer you're a big fan of back pressure and you're you're losing it there, which is not great, and uh I believe that we might be able to make it work, I'm not quite sure yet, but we might be able to add it to either uh uh two medium track, transform maybe or we can fix t, but uh we should be able to to have back pressure and so there's an issue there.

D

Definitely um in general, I think that, in terms of api shape, if we assume that we think stream is the right way to do, um I think the api shape is is good. It's correct. That's that's solving some of the issues that I had uh with the prior proposal.

D

I think that in general, in media capture main, we have the concept of a source and we have a concept of a track and we have we're defining the relationship between the source and the track and the fact that we are introducing introducing a video track source. A javascript object that represents uh a track source is a good thing and uh it's very similar to readable stream.

D

You can have a native, readable stream, but you can also have a readable stream with the javascript source and the javascript source is also an object that abides to uh specific things, and so I think we should go there. It will be easier to extend things if we think if we think so, we remove some edge cases, so it will be much cleaner.

D

So that's why I I would tend to go with this way. uh As I said, there's no surprise for me to prefer uh not relying on transferable streams there. I think that we should rely on media stream track transfer, which will be more reliable, which will be a typed way of transferring, and since it's typed they are like uh requirements that that will be fulfilled while with streams uh you don't know whether these same requirements will be fulfilled because things are generic objects. So.

B

Okay, thanks ewan um yeah.

E

So I can respond yeah. I may have made a mistake in the example of which track I cloned, because I uh I chose the clone to be what you would apply constraints to, but I think uh it might work if we flip it around right, so that the we send the original track right.

D

I don't I don't think so it I don't think it will work. um The thing I would like to to get to is uh like, if you, if you drop a frame, if you have like a source- and you have like a five pips track and a 35 tracks, then you should you should get uh some back pressure that says hey.

D

I actually want 30 frames per second to the video track source and we have no guarantee there there's nothing there and it's due because there's no back pressure on the writable stream and if we start to introduce back pressure on the writable stream, which we might want to do, it would be a good exercise to do uh then we're in a good situation, because uh promise-based callbacks have that and uh we will not be able to have that with. uh We may just interact as uh video track sources, so we should be able to so.

D

Javascript will need to handle it. uh So I think we can make progress there if we really uh want to go with students.

A

Cannot self frame it.

D

I don't understand.

B

um So uh yaniva I had a question about several of.

E

Sorry bernard, uh can I just, uh can you jump to.

B

E

47, uh since I ended a little early, uh this.

B

E

B

Hold on answer.

E

B

uh Which is uh appendix.

E

B

E

I think uh un is uh concerned about back pressure and is right, so I did actually write a fiddle that I got working in chrome, even uh that uses uh t, and so, if you use the t solution, uh one of the things it it does for you is, you do get brushed back pressure, but at the cost of solving the t problem, so uh I was able to solve it um by by polyfilling synchronized through structured clone uh thanks to uh matthias bullins, and uh the only thing uh odd is the um that's a crate frame.

E

Dropper, it's the four lines from the bottom or five lines from the bottom. uh You have to have a transform string. Basically that drops frames, so that would be a way that uh of solving the t problem. But this is why we're here. So yes, so it's true that uh using a track uh to and apply constraints is a workaround for this. Basically, if we cannot solve the d problem.

E

An argument question.

B

Yeah um yeah a bunch uh looking through these um and that these didn't make sense to me right because, basically you would, I think you wouldn't post message to the main thread, because you basically create the uh track processor on the main thread and and transfer the stream or and what you're saying you would just transfer the track. So I don't these slides, don't matter yeah, I don't think they're distinctions between the two apis right.

E

Well, uh yeah yeah. I apologize for that. It's a misunderstanding that the I wasn't aware that mediastim track processor was.

B

E

In in the worker environment, but it's still true that you could do this and we would need to handle it.

A

Yeah I I added the feature last week.

B

E

Don't even think you would probably.

B

E

B

Yeah, I don't even think you would do this without transferable media stream tracks, uh because, basically, it created on the main thread. You wouldn't like have a situation where you have something and you and you need to like post message to make the main thread to do it. Yeah right.

E

So harold is right there, a lot of similarities between the apis, but the difference is that uh mediastimate processor creates a new class on the producer side and it uh where, as uh mike on yuan's proposal, does not create a new object. In that case, we just add it readable to the track, which we think is sufficient and simpler, because a readable already has locking semantics and once it's locked, you know that it's being consumed, we don't need a new object there.

E

Other than to to add, if you want to add some uh options that you couldn't add to the track itself,.

B

Yeah, I also wanted to add a comment on the webcodex decision, the the actual.

B

I believe this is a quote from chris needham, not chris cunningham, uh who is a working group chair and, as harold said, there was a discussion of the where we are with respect to workers red support overall, um and I think one of the big points, as harold mentioned, was that, uh for example, un has recently proposed rtc data channel workers, um which is, is not widely implemented, but um that's an example of something that today is only on the main thread, um and so as as harold mentioned, uh the big issue, a lot of the things that were constraining people to the main thread was lack of consistent worker support across the api set anyway.

B

So this wasn't increasing chris.

E

Needham well, chris needham is the chair of the uh the chair.

B

E

Web products, web codex working group right.

A

Apologies for mixing up my christmas yeah.

E

Anyway, so yes, so so- and the other thing I want to mention is that media stream track generator. It's a bit of an odd duck in that it's um it's just. It's also a track and that's where you and I feel like that's where we should have the separate api, uh not on the pro producer side and that's a cleaner model that fits neatly with the sources and syncs model in media capture as far as main thread areas where main worker apis are not available.

E

Yet I think that's actually a good use case for transferable streams, because once you have set this up in the worker, you can, if you need to transfer these streams using tunnels uh back to the main thread for those few apis that haven't been uh or those browsers, that haven't implemented uh all apis in workers. Yet, and that's that that also does not require breaking the transferable stream semantics and I think that's the proper use of.

B

D

I don't know who's on the queue uh at you plus myself, but I'm not sure I.

B

Think it's harold and then.

A

Yeah the uh that this was a response to um that. If you have uh have to tell some place up the stream that your desired frame rate is 30, then back pressure cannot carry that information right right, because back pressure cannot tell the difference between I'm slightly late, and I have to wait for one other frame and I only want every other frame. So then, a frame, dropper or separate tracks with separate uh separate frame rates is the way to go.

A

We need. We need those kinds of signals to be carried. It's one of the areas where I that I have always said that we needed to to to work on with the with the with streamspace proposals this the the singles that have to go in the other direction, and we haven't gotten around to it yet, and that worries me.

B

Yeah also harold, they have, they don't just go between one stage right. They made up to hop like the encoder might be two stages away from the thing that has the feedback yep. It isn't just from one stage to another. It's all the way through the chain.

D

I I tend to agree with uh with her that it's not always desired, but there are cases where it really depends, whether the source is pull or push and they they might. We are mostly concerned here about sources with pull, but in case uh it's pushed, then it's actually useful to to do that kind of stuff, and I wanted to mention that uh need to propagate uh things up to uh the source.

D

You might be right that in some cases we should not use back pressure, but we certainly need to provide that information in some way. um You mentioned the overweight uh that we removed, and I agree with it. We should work on that and fix that um related to uh bernard. You mentioned that lgbt channel is only available in main thread and that's true in some in some browsers.

D

That said, that does not uh remove uh the possibility that that does not remove the idea that going to work is a good thing, because you go to the worker to get video frames and the video frames you will not be able to uh push them into data channel. You will need to do some processing in any case, and it's better to do that. Processing in the worker. You set up all the things and sadly, maybe the last bit uh will be done.

D

You will need to go back to the main thread, but still you have some gains because you're doing like the heavy things like serialization, packetization and coding, and so on in a worker uh which, which is which is better so and you will be ready whenever uh people implemented this international uh transfer to workers to actually embrace it and use it. So that seems good uh from that particular uh use case. There may be other apis that are missing uh related to media um processing.

D

We definitely need to to look at them, but uh my understanding is that uh we we're we're in a good place there and uh we should. uh If, if the issue is uh implementation lagging, then we should push implementers to do more things in workers and I'm.

B

D

That media stream track uh transform. We will will be a good good way also to speed up things in workers in terms of api.

B

Okay, uh so there's nobody left in the queue. um Do we want to move on to harold's presentation a bit early? It could leave some more time for discussion. I think that might be good.

F

If we have a minute.

B

Oh hi queer sure go.

F

Ahead yeah, I wanted to add that the issue of main thread explosion, and so on. It's for on one hand it's uh availability of many other apis that are not even specified to be available in workers. So it's not just implementation, but also specification, but also that that we have first-hand uh requests from application developers that there are valid use cases that that want to do these in workers because of the nature of the application. So it's a use case that, okay, that we don't that. Okay, we have.

F

We can say that we don't want to support the use case, but I don't see any reason why we could make it a bit uh the harder, the that adding something to to the api so that by default it doesn't work if we don't want it to work by default, but but there there are use cases for this on the main thread that are not the uh that are valid.

F

These cases are that are valid, use cases that that people want to use, and I I don't think that artificially denying those use cases is is a good idea. Otherwise, the two apis are basically equivalent, except that just uh it's just surface difference mainly.

B

Okay, we have uh doms in the queue.

C

When you're saying there are use cases to examine on the main thread, I wonder: is it really about they needed to be on the main thread or just that dealing with workers is painful? The developer experience is not so great, and so they prefer not to have to deal with workers.

F

uh Well, first, the there are, there are use cases where the worker gets absolutely it's a net negative, because the worker doesn't give a. I mean what the worker gives. You is that if you have some free resources, the worker allows you to use them uh more efficiently to actually use them.

F

So, but if you don't need those resources, so the worker is just only introducing a cost and it's not giving you any benefit that has happened to applications are not like a big conference conferencing websites that are doing real, uh real-time stuff, but more like editing applications for composition for for, uh uh like uh dual applications where you're creating effects and- and you are editing them there- that that that sort of application uh doesn't benefit from the worker and and the worker only introduces complexity on ones on one hand and and and actually extra resource consumption.

E

So I would respond to that by saying that, even if there are such use cases, I think we're trying to protect a real-time media pipeline and that the media stem track.

E

Even though there might be some non-real-time cases like canvas capture that uh we we want to protect the parity with existing exposure and browsers today, which is to not expose media the main thread, except uh in the cases of uh some really slow ones like canvas great bitmap data and that kind of stuff.

B

All right uh so discussion period for this is over turning the floor over to harold. Thank you.

A

So I'm making the rather.

A

Unwarranted assumption that you have already already ordered the api already so I'll. Just recap: it shortly: media steam track processor is a destination for a media steam track and generates a readable stream.

A

Media steam tech generator exposes a writable stream of media frames and implements the media stream track. Api no fails no extra apis.

A

This was the salinas as lean and mean as we can make it.

A

So next slide implementation usage. Yes, we shipped it in chrome, 94, it's actually actively used in products. Now people writing new features.

A

The link that is on the on the screen is to hey one, someone that made a made an open source thing based on it with inserting is that inserting talk he had it center slides, and we have very few problems reported on this. Surprisingly.

A

A

And we believe that the threading model is something that app developer needs to pick and the platform developer can encourage by making sure that what the app developer needs but trying to dictate them is not the right approach.

A

Streams are transferable objects, they can go wherever they need to go and we can do it today.

A

We have agreed that media stream track should be transferable and available in worker.

A

Therefore, we added worker availability for media stream, track generator needed steam track processor.

A

I mean once you have the track and you need to make a stream from it. You should be able to do that wherever you are that's true.

A

We do need next slide.

A

To make sure that when we do work, examples.

A

Such as the one that we have once that we have on webrtc.give github samples, we need to make sure that the samples always show realistic stuff that will work in real time and when we don't need hard linkages to the main chat it, they should be off thread. So we need more. We always need more examples and improving the ones that are there.

A

A

There lots of things we need to do in order to make the apis as powerful as possible, better control of adaptation of the source, whether it's back pressure, it's setting rates it where it's adapting the synchronizing streams.

A

Experience with streams that don't come from cameras, the link I posted earlier was uh generating one stream from slides and one stream from a camera and mixing them. And of course those two streams are not synchronized.

A

That's not trivial.

A

Just just because it doesn't go, bang, bang, bang, it goes: bang, bang, bang, bang, bang, bang, bang, bang or something like that, and all this is stuff that we can work on further, but we have to get to a common base that we can start from.

A

So I did a comparison table next slide.

A

With the two com, the two proposals that are currently present frame delivery. Yes, we have streams, that's a good good point. We have the same objects inside the streams. That's a good point.

A

We have a sharp difference of opinion in whether or not they should be available on main, and we have some oddities of what the things are derived from what, when we did the stream generator, we decided that this was not the stream object. Not subclass of a stream object. Didn't have a relationship with it. That's because.

A

The ways we have of generating streams today generate the stream and we don't have any examples, otherwise, in other places of generating a stream from a stream clone isn't the same thing so having a stream generator. That was a stream was a new. I was a new concept and we didn't want to do that. So it's a separate class.

A

We did the opposite thing on the stream consumer, which is the media stream track with some other properties, and I'm not quite sure about whether that was the right choice or not in all other places. In the platform.

A

What generates an object? The actual source is invisible or it might be a completely different type of objects. Like you, don't have a camera object, you have a camera, but it's referred to. It's not made visible.

A

But this but the actual source.

A

And if it's, if it's a track, that's exposed, what does it mean to clone it? Are there not two consumers of the stream stream? The model doesn't work that way, it's a new object of a of a media steam track class or is it of the stream consumer class.

A

So I'm a bit iffy about having it having the stream consumer be a track. It could be that we should make it something that generates a track or can generate multiple tracks in the for the clone case.

A

We can discuss that, of course. The last difference is that we have shipped and this model for both audio and video.

A

Well, the proposal on the table from uh yaniva is to limit this to video only so we seem to have a very close similarity models and we also seem to have a very distinct set of cons, set of differences that we can discuss as separate issues that the underlying model seems to be gelling together in one place, we have tracks that go to streams, streams that go to tracks and that's a good thing so we're getting there.

A

We just have some points of sharp disagreement and I didn't find any reason to say anything more about it. So that's the end of my slides.

A

Now we have plenty of time for discussion.

A

And then thank you.

B

Okay, uh I have uh dom, and then yanimar or maybe dom was already there.

C

Yeah, that was fun before okay.

B

E

uh Yes, so um I took a couple of notes, so the one thing is that streams are transferable and that's true and that's web compatible.

E

However, implicit transfer of a stream source isn't and that that's the rub here and I think that the existing proposal, the old proposal, relies on this and that's what we want to get away from.

A

um That's the depending on your interpretation that the stream source is actually in the main thread. My conversation is that the stream source isn't in the main thread, it's off thread it's in its own context.

A

The stream source never moves. It's attached to the camera for the camera case, right.

E

But I was quoting directly from what wg's and it says that the purpose here is cross realm uh process.

A

Yes, we just happen to disagree on where, on which realm is crossing from.

A

Generator, you generate the stream that is sourced from the camera. You don't generate the stream that is sourced from from the main chat.

E

But an issue has been opened because the optimizations that chrome are doing are not technically allowed by exactly.

A

I have seen the issue. I have not understood why the claim is being made that this is not just the bad spectre. I think.

E

Well, it seems I don't find any support in this fact that uh that implicitly moving well that, uh even if you call the original camera source off main thread, uh the what lbd stream spec does not seem to consider that an optimizable path, and it also as yuan, pointed out. This would create action at a distance examples where, because you cannot optimize all the time, we can optimize some of the time, and that means that javascript developers might scratch their heads wondering well. This worked a minute ago. Why doesn't it work now?

E

So I think uh hopefully we.

A

Agree which case doesn't work, I mean the when I talked to adam rice about what was the problem.

A

He referred to objects that were already enqueued when you transfer the stream, which is, I think, uh side effect of describing the queue as if it was in the original context. In the in the in this. In the context they were transferring from.

E

In any case, the issue posted uh exposes that a lot of things that have to do with error, cleanup and stuff are still tied to the realm in which uh the object was created, and that would be main thread if you create it on metering. So I.

A

E

Agreed that yeah.

A

I'd like to see uh worked examples of that and specific pointers to the text I mean I I find it. I personally find this to find the stream spec almost impossible to navigate. So if you can help me find find the place where these uh these tie-ins are. I would be very happy.

E

Well, I mean, uh but as far as intent of the stream spec, uh I had a slide for that.

A

I uh and I and I disagree on your reading of reading of the intent, so I'm asking you to point out the language.

E

Yes, that was a direct quote from the spec, so.

A

Yes and that's, uh but the idea that the source is in the in in in the originating thread is an interpretation.

D

um I I don't. I.

E

Mean literally, it says: okay.

D

Yeah there are some algorithms here, but you need to point at that and these algorithms are run in in a given context. And so when you read and write there, you cannot say: oh we'll actually do that in the camera thread, which is not.

A

D

You you you do that in the in the context of the stream and that's how it's it's done. uh The thing you could you can do are like type 2 and things like that. You could try to and there's some leeway in uh in the spec from just from the start to actually optimize this. But for the rest, I I don't think that there's uh some allowance from the spec and uh adam rice uh uh I quoted adam rice- was a spec offer and said at some point yeah.

D

Maybe what we should do is to have a specific type of stream that we could optimize and uh and and.

A

Then I saw the pull and but that that bug thread is totally missing references, the spec, which means that means that I didn't manage to follow up.

E

You could bring up slide, 38.

B

Slide: 30 uh 38, okay! Well, yes, please! That's pretty far back hold on. Let me navigate there.

B

Am I getting close can't really see the numbers here, danny barr am I.

E

uh Sorry uh keep going.

E

C

I must violate what we do: transferable streams.

B

E

Got a couple more keep going down ah this one there, that's the one yeah, so the first uh paragraph there's a literal quote. It says a special kind of ide that transferable streams are a special kind of identity, transform which has the rightable side in one realm and the readable side in another realm to implement cross-realm transforms. I think you can't get more explicit than that that this is about uh transfer between realms, not uh to solve uh getting things that were in different realms on the same realm.

A

Well, uh which realm is the writable silo.

E

So you're thinking uh for media stream track processor they like. So if you create that. uh Well, if you create music processor on one thread, a and.

B

E

Transfer the writable, then, in that case the slideshow uses, readable.

A

E

If you replace it with writeable, the errors will go. The other way right.

A

Oh media stream track processor doesn't have a right media stream, check, generator, doesn't have a writable media stream. Type processor doesn't have a reader.

E

It's the same problem yeah, so so the the example I'm showing is uh a media stream track processor, but the same slide for uh media stream track generator would be that the the instead of source, it would say, sync on thread a and instead of readable, it would say writable and thread b and the errors would go. The other way.

A

My question again is: why do you think that the in this particular slide the source lives on the mainstream me in the main realm in the the in the same realm as it was, it was as in initiated like it's great, the creation.

E

Based on the language directly above uh that references, a writable site on the readable side.

D

Source is the readable as well.

A

Is the writable site accessible.

E

It's a if I follow the link, then it goes gets me to transform stream. So I think the idea here is that you would you would uh transfer transform streams.

A

uh You can't get the transform stream from a readable stream.

E

A

E

If you follow the links of the spec and I didn't write the spec, so I added links in the slide that follow the same links that are in the spec and that takes me to the writable side and available side of a transparent stream. So it looks to me like the intent of this working group, of the. What wg working group was that transferable streams.

E

What you want to transfer basically transfer streams that have both well.

A

Consider another matter: if you have three sources: three threads a b and c.

A

And you create a trans transform stream on a.

A

You transfer the readable side to b and then transfer the readable site onto c.

A

Is there a link to b anywhere? Yes,.

D

Yes, currently in the spec there's a link.

A

D

Can you point me write me to the spec.

A

D

There's an issue in the github, which is called the double transfer issue yeah, and this this is a known issue, and this is uh yeah. This is the limitation of students currently.

E

And I don't want to talk about the queue too much.

A

Yeah, so that's that's the problem we need to solve.

A

We have agreement questions that streams have, I think, have issues to solve. Yes,.

E

Sorry I had a question for harold: is that your stance is your stance now that exposure of raw video and audio on the main thread is okay and has that changed since uh we decided the working group already arrived at a worker-only proposal for webrtc encoded transform and why? Why is your opinion? How do you reconcile your opinion with that.

B

E

B

Okay, uh we're out of time for this discussion.

A

uh Just just quick, quick answer on the last one: okay.

B

A

Have a bug open for restoring the ability to do and call the transforms on mainstream maintenance? That should be enough enough answers from that. Please other people jump on the bug if you care.

A

I do believe that the my personal belief is that the working of the station to accept the limitation was wrong.

B

Okay, all right, so um we've now reached uh the wrap-up and the next steps portion of this, um and so we've left about 10 minutes, in which time we can get feedback on some of the distinctions that have been made between these two things, um and so one of the things we could, for example, ask for a kind of sense of the room.

B

It's not a formal cfc is on some of the distinctions, for example, support for workers only work versus worker, in name or audio, and video versus video only so that would be kind of my proposal to just get us, because the proposals are are very, very close on so many other aspects to just try to get some sense on these on the major distinctions uh between the two.

B

um Does that make sense just try to get people's opinion on on whether they on some of these issues? Okay,.

E

Well, I I guess I I if you're asking me since I just presented uh the proposal and and this meeting I would.

A

E

To get on a yes or no on whether the working group thinks that things, uh what what I presented is the direction the work group wants to go and uh answers could be yes, it could be yes if it also solves audio.

E

Yes, if it's exposed to main thread a deal breaker for me personally, but maybe it's not for others and or no.

A

I'd object to that formulation.

E

A

A

I don't see any reason why.

A

A

uh We have two uh starting points that are possible for the for the specification and we can modify from them. I don't see a reason at this point to say that to ask the working group to.

A

A

How they say it- and I don't see a reason to to ask the working group to assume that they're starting from yours.

E

Well, I think we had uh the same question when uh you had your presentation a year ago,.

A

Well uh normal. The conclusion from that point was that you and you n, were raising so many, so many points that we were awaiting a proposed, an alternate proposal from you, and now we have an.

B

Alternate proposal.

A

So we, if you, if you want the working group to choose between the two approaches as a starting point for further discussion, that's perfectly reasonable and making an assumption that we that that we, that the only only option on table is choosing yours. It's wrong.

E

Well, I wasn't assuming that I'm I was asking the working group if this is the direction it wants to go, because this is normal after you propose something is to turn to your audience and say what do you think.

A

Right yep, so if, if you, if you're okay with us asking with asking the two questions, is the media stream track processor generator a starting point for further discussion and is janeira's proposal starting point for further discussion? If you ask both questions, that's okay,.

D

Okay, I have a.

E

Problem with that.

D

Okay, can I add another problem, which is that uh it seems that everybody is taking for granted that all these streams issues will be solved, and uh I didn't see uh much progress on the difficult issues and I'd like to get a sense from the workbooks. If they feel that to be successful, we need to solve the issues or, if we just don't care, we will do it whatever or what?

D

Because I think that, uh um with the current state of the stream to work working with spec and features, uh it's not good enough, and I I don't I'm not I'm not feeling confident with any of the two proposals without a strong progress on the stream side of these issues.

B

E

My concern uh with what are all having two questions uh proposal of two questions is that I think media steam track, processor and generator has an unfair advantage here in that it's been already been implemented in chrome without uh working group consensus and it offers audio and video. So I I'm, I think we should ask a working group. uh uh What's uh what's the minimum api surface that we can agree on to start standardizing, and I would actually hope that we add audio.

E

uh Not everyone might agree on that, but we don't have enough time to stand up an entire replacement from the sim track. Processor as a whole. um So I think it's unfair to say that while we already have.

C

E

Non-Standard, you should accept it a wholesale or not.

C

If I may have what I understand to be the questions, yaniva is looking for and I think a question all of us might agree on. um There are two different api shapes one is using readable, unwritable and happens to be exposed on in workers, but not fundamental to it, and there is a generator and processor api shape which happens to be exposed on the main side, but that's not critical to it.

C

So I hear yaniva asking or a sense of the room between the two shapes without questions about whether it affects audio or not, and whether it's off or on main thread.

E

uh Thank you ewan. I think that that might be a way to uh if we can make the questions clear that uh we're asking about shape separate from what it might imply and then maybe a separate question for exposure to maine and whether to solve audio or not. So if we can have those be totally independent questions, I think you're on to something.

F

That was we're not original proposal except except for the stream issue.

E

Right and that's strange question as well so.

F

We can, we can have three questions instead of the two or you know, by bernard and the other one. That was the streams or not streams.

E

Right so api shape is one exposure to main two effects. Audio or not sorry include solves audio. It's three and uh streams are not s4.

E

C

So I think the question for stream, though, is- and there seems to be clear interest in streams in what it solves. uh The question that huan is raising is how confident the real issues that remain uh can be solved and can be installed in a timeline comparable to the one we want to achieve with this work.

C

So I I'm not sure that's so much a design decision as much as trust in the ecosystem kind of evaluation, so uh that one, I don't think it's necessarily a good sense of feeling as much as you know, more research needed more input from stream people needed, because the working group might have trust and be wrong or have not trusted me wrong. So I I would probably separate that question from the sense of the room kind of, but.

D

It but it would be good to have action items there.

C

I agree absolutely and.

D

I'd like to understand which action items should be we we can have to to make it happen. Otherwise, I'm not feeling confident on making progress.

B

Okay, um I have a draft of potential questions. uh Let me show them.

B

E

Are we thinking of doing a poll or are we gonna open it too.

B

I was just thinking of just doing it in the chat room. um The three questions are: should media capture transform be based on streams, uh question two: should it be audio only and question three, should it only support workers.

E

Well, you mean video only right.

B

I'm sorry video moment right, yeah.

E

And that's the I.

D

Would say video only for now there's nothing precluding to edit later.

D

Once we will have done more research.

G

So, just I guess, there's a lot on cuba. Just one comment on those questions like I. I could live with the answers to yes or no to as the answer to the to every single one of those questions like so I I don't- and I don't know enough about the details of the implementations- to have strong inputs so for some people on the call here this may be. uh I don't know I mean I, I won't provide any input, but it you know it's a complicated it.

G

We may be discussing a detail that maybe some of the you know there may be. We may need more things that help us understand the consequences to the answers. These questions before some of us can really provide very useful answers and, like you- and I totally hear that's very unactionable and does not make me feel very happy towards resolving these things either, but I don't know what to say to any of them. So there you go.

B

Okay, um well we're almost out of time, but we can send do the questions on the mailing list as well.

E

Could you change question two to uh should media capture transform solve audio, I think, is the real okay question. There.

E

And I should meet the third one I would say: should meet capture transform, be exposed to main threat, or I don't know. Maybe only support workers are better support, just say: support.

B

B

F

No, no, not that your proposal supports it. It's just that makes it uh annoying, because you have to create a worker. You can always transfer the stream to the main.

E

Yeah, okay, so exposed on main thread.

B

Okay be exposed on maintenance, okay,.

E

All right, I and I think I think, there's a fourth question on api shape, because I heard are all interested in solving some of these um correct me. If I'm wrong harold that you would you had an interest in the existing shape, even if we solved some of these.

A

Should the stream generate the bs, be it be a track and show this through the stream because you may be attract because uh that's uh that's an api shape. That's the main difference in the api.

D

Maybe uh to help kellen decide, I mean each of these issues should be, uh should have its own summary somewhere, so that somebody can read the summary and the person concerned decide because I feel like it will be difficult for people out of a room to to understand. Like the fourth question, for instance,.

C

Yeah and I think a way to frame uh this summary, this pros and cons is to explain what will change for developers uh when facing the different answers to these questions, because at the end of the day that really the kind of input the rest of the people in this room can give it's not. You know this will be terrible to implement in my brother. It will be this will make implementing this or that application more difficult or easier. I mean in the api shape thing.

C

There may also be a question about evolution of the api, how we might make it evolve over time, so that might be a consideration to bring in that summary, but otherwise we would suggest we formulated that way.

B

Okay, so I think the next step is: uh are we understanding this is to ask these questions on the mailing list, because I think we're out of time in this meeting is that where we go next.

B

And ask them individually, yeah.

E

I think question four is hard to to grasp um for someone who hasn't seen the whole presentation, both presentations so.

C

I would suggest the chairs meet and figure out the questions and the summary of pros and cons, because I agree right now. They are too abstract to gather useful opinions, so we won't do that in minus four minutes. So I suggest we take that offline.

B

Okay, all right well! Thank you. Everybody.

B

Thank you. You attpack.

A

A

Stop recording.