W3C WebRTC Working Group, 26 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WebRTC WG meeting 2022-04-26

Description

See also the minutes of the call https://www.w3.org/2022/04/26-webrtc-minutes.html

02:47 WebNN Integration with real-time video processing
29:20 WebRTC Extensions
29:40 Issue #95
45:47 Issue #100
55:16 https://github.com/w3c/mediacapture-extensions/issues/47 Voice Isolation Constraint
1:04:27 support for contentHint in Capture Handle
1:16:11 WebRTC Extensions
1:16:23 Avoid user-confusion by avoiding offering undesired audio sources
1:38:25 Region Capture

A

Okay, so this is the w3c webrtc working group meeting for april, and we have two hours today and a whole bunch of stuff to get through. So a reminder about the ipr policy we abide by it, and only people and companies listed on the page are allowed to make substantive contributions.

A

So here's what we're going to try to go through.

A

We will have a discussion on integration of webin with media capture, transform some issues in webrtc extension somewhere bridge cpc, uh harold, we'll talk about voice isolation, and then we have a lot with suggested content hint and avoid user confusion.

A

And then you, when we'll talk about.

B

A

Capture, if we time permits yanivar, has some slides as well and if time doesn't permit, we have meetings coming up on may 17th, the 21st of june and july 19th, and we can get to whatever we don't get to today. Okay, so the slides as usual are up on the wiki and we do need a note taker.

A

Do we have a volunteer.

C

I can take note.

A

Okay, thank you very much, ewan, okay, so a little bit about the code of conduct we operate under it, we're all passionate about improving webrtc, but let's try to keep it cordial professional.

A

um As we mentioned the meetings being recorded. um If you uh we, we are trying to manage the queue so type plus q and minus q in google. Me chat, and I don't know, is that harold uh anyway, harold can you manage the queue.

A

uh I can do that. Okay, thank you very much and please state your name that will help us figure it out. So a little bit about document status. We just try to tell everyone that, just because something's in a w3c rebound doesn't mean it's been adopted, we've been having a call for adoptions on this, uh and editors drafts don't represent working group consensus working group drafts do um and it is possible to merge prs that black consensus.

A

If you attach a note indicating that okay, so here I mentioned what we're going to try to get through today, it's a lot of stuff, so we are going to try to keep time fairly strictly and I'll. Try to do it. Maybe harold we'll see if we need to keep on track, though all right. So first item is m. Webin capture transform integration.

A

I know: uh uh do we do you wanna present this? I I added some slides just for background. uh Maybe I'll present those, I guess dom. You opened the issue on november 10th and the idea was to build a prototype, integrating media capture, transform with tensorflow.js background blur and then measure performance, and the focus was on video because that's where we think the the issues would crop up, if there any so um so this was done, and this is what a discussion of what it all means is coming up.

A

So if you look at the thread, there are several issues that were raised. One was garbage collection. There is a pr that's in progress about that. It turns out that it helps if you report video frame external memory as released when you call videoframe.close, and it apparently improves the garbage collection and doesn't create as big a spike that pr is not merged. Yet so you wouldn't see that another thing is copy removal, and I guess we'll discuss more about this.

A

If we understand it- and I guess dom- you thought that probably a bunch of copies going on from the gpu to cpu. I guess we'll explore that more and the third issue was webinam web gpu integration and whether we're using the right, textures and shaders and all that stuff to do this.

A

Okay, so I guess uh there is a sample app up there, where you can look at things uh and get your own idea of how things perform.

A

It integrates breakout box with various transforms, including webgl web gpu, stuff, like that, and I guess you do have to for the webinar. You do have to go in and chrome flags and enable that specific feature or it won't be available.

D

So that that's that feature is of course released now: yeah doesn't work in a worker, yet.

A

Right, uh although I think what we're dealing with here may not need that yet, although that was one of the goals okay, um so why don't? Why don't we just get into the do you want to uh talk to this uh slide and then we there's some other slides we can get into. I guess there were questions about the details.

E

F

uh Yes, I'm allah.

G

Yeah so yeah, so my name is ningxinhu, uh I'm working for intel um and now I'm participating in the double swissy, where machine learning working through with uh nc there, um I'm a co-editor of the our webspec there. So yeah thanks for the introduction uh bernard, so I think for uh yeah. Actually, this slide is is a like a high level uh pipeline and some components there to build up this video processing pipeline specifically for the background blur feature there.

G

So actually the boxes here actually are marked with some um some like uh gpu texture or a wix, so that uh actually this is a web gpu version with uh weapon capability uh weapon graph interface there. So, basically, this is a web gpu, plus webm processing pipeline and in the screenshot burner just show. uh There is another pipeline that using um webgl for the processing and the forza pipeline's uh segmentation, and here the the ml graph infer um steps in the pipeline is a tensorflow such as webgl um webgl, um backhand there so yeah.

G

So so, basically, so um this pipeline is running, we have two um versions: one is running in the mainstream. Another is running in the walker, so the screenshot and the other link just shared is for the worker but the best case pipeline code and the steps are same so um yeah so in uh in the transform uh callback. So basically we got the video frames and we uh have the first step is import, that's a value frame.

G

So basically, yes, import here is like a just a function, but uh actually uh underlying a current uh a demo. We at example we use uh two apis. The first is create an image image b map performs at the video frame. Then we use uh in the in the web geocache way. I use uh the the texture 2d. If let me see uh yeah the text image 2d to upload that video uh image beam map or to a web geo uh texture and in the web, gpu plus weapon case.

G

We use a copy external image, two texture api. There, then we have a gpu texture so um in in the gpu texture. Basically we do a tool. Processing one is just a blur, the whole image and- and that takes a video frame and a texture and and apply that into another um gpu texture. If we focus on here the uh webgpu pipeline uh um yeah, so uh so um on, another pass because we need to uh figure out which pixel is the background, which pixel is the foreground right.

G

So we uh first we need to run some machine learning models here, so we use another shader, for example, in this english pipeline to to so-called tensor tensorize, this uh gpu texture into uh into a tensor that can be taken for the um for the ml uh execution. For example, they have some some some requirement for the input for that uh machinery model.

G

So we need uh that, for example, do some normalization and some uh turn it to like a float point and do some like uh normalization, the something like this so yeah, then we turn into all this pixel um in the texture into a gpu buffer. Then we fill that into um the webm graph here, as bern mentioned, the way depend on the interoperable api between webm and the web gpu, so the webm graph there can take a web gpu buffer as an input.

G

Then uh that graph will infer that graph, for it will output the so-called segmentation map into another gpu buffer. That basically tells you which pic which object of that pixel.

G

It is, for example, it's a background or it's a it's another object, because we're using a deep lab of history model that can tell you um that can classify that each pixel into up to 20 21 um classes. So if I remember so yeah correctly, then uh another um shader I'll take this tool um together, like the um actually the three inputs together, the original video texture, the blurred video, uh the gpu texture blur the texture and the segmentation map and based on that to um to create to carry out the output.

G

That's like a pixel uh pixel, um a pixel from from the origin origin, texture. If the pixel is uh like a foreground and take pick up the pixel from the blurred texture. If this pixel is classified as a background, then this all seems well be drawn into our off-screen canvas uh at the last stable that, after this all done so uh this example will and create a video, a new video frame from this canvas and uh feed that into the controller that passed from the transform callback.

G

Then that's the best case, uh the the pipeline yeah, so yeah so uh yeah. So, as I mentioned that the the the webgl pipeline is basically similar, but the segmentation is by is done by a webgl shader, that's implemented by tensorflow.js webgl backend, and there is also to do that. We want to also do this for the web gpu, because tensorflow.js also has a web gpu uh backend, but due to some um gaps there. So this isn't not done yet uh will be down at the next step.

G

uh So far we tested like the pure web gpu a web to your backend and there's a web gpu plus webm execution, so um yeah today yeah.

G

So that's the the high level uh description of this uh um this pipeline.

G

Yes, uh the uh some uh bullets here uh share that we basically uh profile the um the the sample we yeah also don't address that there um looks like some, um not so very uh low, cpu usage there and no, no matter, you run it on menstrual or in a walker, so yeah. Basically after profiling, we see that there are like a 35 um percent of cpu usage in this uh in this sample. Actually, I spent on the cruise image map and another 20 um spent on the gc, so yeah so yeah.

G

We are wondering whether we can uh improve that by. Like uh you know, the image map is used for the video uh frame import to uh uh to a gpu texture, so yeah.

B

So so, actually there.

G

um There are I I was told there are copies there, because there are some like uh color convention or other um other um processing that turned to turn our video frame into a texture uh with the cop with a copy, so yeah, so that that um maybe one area we want to improve. uh I was told, there's some new api like uh import external image to texture, something like this we'll be introducing web gpu as an extension or so yeah.

G

So we will test that out if it is in place and and also on the on the cheesy part, so um yeah burner shared that there are some effort to improve the video firm side. Because you see there are video friends and we got a video friend and also we created a video frame as and right, and also there are some like some objects allocated to um to to run this uh web gpu pipeline.

G

For example, we we will every a video frame, input to architecture, we create a view and also uh we um rebounded the tools uh to the band group of the web gpu pipeline to run the shader so yeah. So probably there can be improved. That's uh to reduce this kind of uh dynamic objects allocation during this process, so that's another um area and probably can be explored. So I I I just myself, I'm thinking on the high level whether we can have some like some some configuration or a setup before the pipeline running. That's.

G

For example, we said that where is the video frame uh to import or, uh as all the like the buffers the textures uh be used for this pipeline, then when this pipeline runs just yes to to run it as a setup instead of creating something uh in the fly when the processing uh in the processing pipeline inside itself right, so that may be yeah, so some areas can can can be think about yeah, uh the the the last thing but another um uh yeah.

G

The last thing is about some some some performance issue I observed during this development. I observe some entry-level gpu that should run the webchild-based pipelines that actually put a high pressure on the gpu for this like for this pipeline, so it may cross the whole browser's browser's hand without like uh have some mechanisms that can control this like like flow control, um because, for example, if this process is too heavy, it probably can slow down the frame feeding to this processing pipeline. So yeah. So that's the thing I would like to share so yeah.

G

So that's all I'll pause here.

A

Yeah, so I had a question you mentioned, the import has a copy and we can see that- and I guess, there's a proposal to go directly from the video frame to the gpu external texture. How about the output? Is there a copy there as well?

A

um The you know the to the off screen canvas and then to the export to video frame.

G

A I'm not sure. Actually so I need some education here or, as a web, gpu expert can yeah come charming. Yeah.

A

I'm just wondering if we need to do a.

G

A

What both ways are just one way.

G

Believe a two, a both ways that we needed.

B

E

um uh Sorry, tim pantan was in the queue before you.

H

Yeah we've got got two queues, uh I'm probably in the wrong one. uh So just I know this is very subjective, but what's the um was the performance acceptable like do? We really have to be pushing down to get the ex the performance massively better or or is this like in the right ballpark, and we just need to tweak it a little.

E

I think that's for you.

G

Okay, so yeah, so my take is uh um probably because this is the long run. um Sim, probably we need to measure that is uh like a power or a battery uh impact on this yeah. So I think uh this is a like a one, as I know, is one um like.

G

I see people are interested that how this is like a video uh processing or video conference um application, consume your power or or drawing your battery right, so so that that thing we need to figure out, but we don't know that yet, but uh maybe uh dom uh highlighted the emphasis. The cpu usage is one aspect of that, so yeah. So that's my.

G

So my idea is to if we can reduce the power consumption as a less as possible. That would be um good to improve the user experience because that people- actually um I I heard some feedback from from that side, so.

E

Yeah I'll come on quickly on what we're doing with this prototyping. uh I think we are doing two things. One is checking what improvement we can get by using hardware accelerated neural network processing with lebanon.

E

The other thing we're doing is uh evaluating some of the roadblocks. Anyone that might be using media capture transform might be hitting when doing video processing, in particular, when trying to do full gpu video processing, because this memory copies that appear here are not specific to background blur or to machine learning processing. They are really anytime.

E

You want to apply some gpu processing to a video frame um and, more recently, some of the questions around, for instance, pixel format and color space conversion that are implicit in the diagram that nintendo presenting have surfaced uh in other places and I think, will impact our architecture down the line. So that's the two ways I'm looking at this prototype and its current performance results.

E

And I believe we have a un in the queue.

C

Yeah, so I'm looking at the gc, which is 20 and so on, and I'm wondering whether that's um that's something that is uh like that can be fixed by implementations uh underneath uh javascript. However, there's uh an issue with uh creation of uh a lot of objects, very repetitively like for every frame you you're, creating your gpu buffer and so on like and it's happening a lot and what you usually done with native implementations. Is you have a buffer pool and uh I'm.

H

Wondering whether.

C

This is something that should be surfaced to javascript applications or whether user agents will be smart enough to use buffer pools underneath in implementations, so that this uh gc and pearl issues uh are decreased. A lot.

G

Yeah, so uh to answer the first question that the gpu buffer- actually I allocated uh before the processing pipeline running so basically in in the setup stage. So all the gpu buffers are allocated, but there is like, as I mentioned, that there are some theme that's created for everything that is the way import our video frame into our gpu texture. Then we need uh to get this uh texture and vio, and the band into the uh like uh the uh band group of that uh processing pipeline.

G

That, I believe, will create some new objects there for for every frame, so that that's why I um have a high level idea whether we can have a like some something to um to bend this uh like uh before the pipeline. Then we just uh can like a developer, can specify. Where is uh video importance the destination?

G

Probably we can get it pre-allocated, uh buffer or texture. There then just populate that buffer without creating its every time. Right then, the gpu um I mean the uh web gpu processing pipeline could be like a cool lab could like be static without the change um for every frame. So I believe that will avoid many new objects, location so yeah. So that's my take so I, but I'm not sure that underlying implementation side optimization would help. So I believe so, but I don't have a many experience or source there. So.

E

uh Nintendo is no one else in the queue, I wonder. Can you say maybe a bit more about the next steps in that project? What kind of timeline do you expect for future iterations.

G

Hey so, oh, there are two um two items currently in my to-do list. One is like uh to enable the web gpu um tensorflow.js back and uh in this pipeline. So we have uh we we. We can understand that uh how this can work with like a webgl. We already have that and work with the pure web gpu processing, then with our web gpu plus uh weapons there. So that's uh um one thing, so we have.

G

uh um um We have um uh uh the pr uh in uh working progress that to fix that in the tensorflow.js webgpu backend. So um that's one item another one, uh as I mentioned, is that uh there is a uh so I was told there is a new website that can import uh the external brand. I just mentioned that to import a video frame as external texture to um to to to web gpu so yeah.

G

So I will try to um to use that and see whether we can reduce the um the um uh the uh image uh bp map um uh acosta, which cpu to see how how far we can go and yeah. So I that's the immediate uh two steps I think about another one, probably is. uh We can also measure that uh furnace shares that p, as there is a sale in chromium that to fix the video frame or improve the video from gc um scenario.

G

So probably we can also try out the ones that merge the so to to report out what we found so yeah. So that's, basically uh that seems uh in my mind so yeah. So I also want to hear feedbacks here so anything I missed, or we should find out more so yeah.

A

Chris cunningham, do you have any comments.

I

Hey y'all, sorry, I'm I'm I'm dealing with a sick kid over here. I'm kind of a fly on the wall, but I'm not paying a close attention to be helpful. Okay,.

C

um So I have one more comment, since we are talking about uh cpu efficiency and so on um so currently you're doing ping pong between main thread and and worker fred. So I'm guessing that it might be small, but maybe it's not negligible as well. So maybe when everything will be done in a worker, you will uh transfer the the mediastream track and do everything in a worker. Then maybe there will be a small uh perf improvement as well, so that might be worth experimenting when it's available.

F

G

I'm sorry so I I my understanding, uh we we, as I said we have a walker version, this of this one, so uh in zawa. Actually, I believe we already transferred the media track to that workers and all the like a process is happening on walker in walker, without uh transfer um objects between the mainstream and the worker uh in the pro in the processing pipeline. So I'm not sure is that uh you suggest that we we do or or we we already have- that in place. So.

F

C

I I thought chrome was uh did not have support yet for for that, but maybe it has already and you're using it. Then then, it's great if you're transferring the track. That's right to you: okay, cool, okay,.

B

Well, there may be transferring the streams, but maybe.

A

So I guess my main takeaway items is potentially look at that conversion. I know uh there was a a pr proposed for web codecs to do the conversion.

A

I think one of the questions was whether we knew exactly what the conversion ought to be. Are you saying that, if, if we did the conversion between video frame and gpu external texture, that would be used by webmining and and that's the one we that's the conversion, we want back and forth.

G

If the yeah so for webm, currently the interoper interface is designed to um take the uh web gpu buffer or web gpu texture. So if the conversion yeah already done that so weapon can take, I think yeah.

A

A

All right, I think uh time is up for this item, but thank you very much. Nixon.

F

No problem yeah.

A

And we will move on uh to discussion of some webrtc extensions and one of our tcp pc items. um Here's what we are going to try to get through um two items on webrtc extensions and then a bunch of simulcast issues that have cropped up.

A

Okay, so issue 95 uh was about potentially deprecating audio video enumeration and get capabilities in favor of the media capabilities api.

A

This is issue 95 in webrtc extensions, um and then we also have issue 185, which was in media capabilities, which is about retrieving rtc rtp code capability from media capabilities when you query it with type webrtc, so uh I think we've pretty much concluded how to do that, which is that you use the content type value to represent the audio and video codecs with stuff like profile id in the content type, so that it provides the rtc rtp coder capability, much as it would in get capabilities, and then you get the result, return from media capabilities which provides the rtc rtp coded capability dictionary, and you can then use that in as input to set codec preferences to set your preferred video codec.

A

So that's basically how how things are designed- and this is an example out of one 85 where essentially uh you put in the media config and you put in your content type in this case. The audio one is audio opus and you put in your channels and your bitrate and sample rate and then uh for video.

A

This one is vp9 with profile id one uh and then, as usual, immediate capabilities you put in the width, height bit rate and frame rate, um and then you, you call media capabilities in this case uh the decoding side and you get back a result um and your rtc rtp media capabilities is in the web, rtc codec member. So this is basically how how things are proposed to work.

A

um So uh there are a few differences, though, um which is that media capabilities currently returns information relating to classic audio and video codecs like opus, vp8, vp9, h264, etc.

A

Whereas get capabilities returns, a bunch of other stuff, like header extensions and the codex portion, returns things that you wouldn't think of as a classic audio video codec like dtmf, uh you know telephone.

B

A

Comfort, noise, uh ulp fec, uh uh you know uh other rtx red um stuff like that, so it's it's more stuff than is in media capabilities at the moment.

A

So here's an example of what media capabilities looks like. As you can see. This is a you have an example of audio red and then comfort noise with two different clock rates uh and uh telephone event, with two different clock rates. I guess one is for opus and one is for uh g711 or g722.

A

Here's what it looks like on the video side, where it returns: rtx, red ulp, fec and then flex fan flex vect as well.

A

So here are the here are the questions for the working group, so um you know basically the overall question is: is it a goal for media capabilities to actually encompass get capabilities and deprecate? It is that a goal and if so, do, should media capabilities for webrtc provide all information on all the codecs, so telephone event, comfort, noise effect rtx. I read all that stuff uh and then this, if it, if we were to do that, would the results make sense.

A

So would these non-kodak codecs would they ever be power efficient or smooth, um and then there's a question of uh would with height frame rate bitrate? Would that stuff ever affect the results?

A

So I guess we want to open it up for discussion.

A

uh Let's see who we have in the queue.

E

You're right anything, thank you. Okay,.

J

Why didn't you manage? Can you manage the queue down thanks.

C

um So my understanding is that media capabilities is really about audio and video codex and that's all so. I don't think it makes sense to expose cn or the tmf in media capabilities. uh So.

F

C

That should say in media capabilities about your staying, get capabilities, but all the uh audio video codecs, the real ones like uh ev1 and so on.

C

Metacapabilities.

B

C

Better place, so that's where I would like the web to move when you query audio or video codex. You go to media capabilities and when you have like webrtc specific stuff, you could you go to get capabilities and that will solve some issues that we have with uh get capabilities.

C

For instance, it is a synchronous api and it's it's okay with uh cn second term, because they're done in software and we we know really easily whether they're, available or not, but for hardware codecs, it's more difficult to do that. Synchronously and uh media capabilities being promised based it's a bit better. So uh that's that's my answer. So no, it's not a goal to deprecate it to fully deprecate it. But yes, it's a goal to partially duplicate, get capabilities.

A

I think florence you were in the queue.

K

Yes, so I agree with some of the points that you made. uh Media capabilities should probably not have um rtx fec and all of those uh non-standard real codecs, uh but I do believe that we need a place for to have views and at the moment uh it is get capabilities.

K

I agree that capabilities is not a great api, as it is not a sync and when querying it it, my results may be delayed and it doesn't really work with the synchronous api, and I think we should probably focus on fixing that, rather than trying to add um api, muted capabilities, which is uh made for a specific request about a specific codec, rather than listing all the different codecs that we have, if we tried, for example, to use, um if we didn't have any api for listing all the codecs and try to get similar uh used to set codec preferences as we have now, we would need to make many calls to media capabilities to get all the webrtc codecs in order to construct the array of codecs that we want to pass to set codec preferences, and that would be a big usability regression.

K

So we would probably need to address those issues as well.

E

I

Hey um sorry, I got uh a moment away from my kid for a second. uh I just want to say that, uh for my part, is the media capabilities, editor uh ewins and it seems like you and and uh sorry I forget. The name of the person who just spoke are mostly aligned um and I'm I'm fine with that outcome. For media capabilities to be focused on. uh You know real codex if you will and to leave the other codex to the domain of git capabilities.

I

There is an interesting question still about.

I

Do you want to return the rtc codec info uh from media capabilities? I think uh the the previous uh speaker kind of was uh circling that question. um If the answer that question is yes, uh I do want to ask that un and uh you know other folks on this- call, take a look at the immediate capabilities issue.

B

I

I'm trying to uh really nail down the steps on how to disambiguate the input from media capabilities to uh you know, reliably produce a rtc codec info. If that is a desirable outcome um uh you and go ahead.

C

uh Yeah, I think it's a desirable outcome. I think I filed an issue about that. So it's great that you're working on it and yes, we should be able to. We need to disambiguate uh the outcome, because uh it's useful for webrtc, but it might be also useful in the future for um web codex plus web transport for webrtc, like uh applications as well. um To get back to florence point about being able to list all video codecs or all audio codecs. That webrtc supports. I think it's a non-goal.

C

We want to move away from that. Media capabilities is really about uh you as an application. You have you're an sfu, so you're, probably supporting a few, a few codecs and that's the ones you actually want to uh to check with media capabilities. It will be a few calls for media capabilities and you have promised all that will solve uh most of the issues there. So I don't.

B

C

A usability trouble for sfu applications for peer-to-peer cases. I would think that you might not need set codec preferences at all, because um you want to expose all codecs and then vsdp negotiation to happen and pick the best codec between the two um peer-to-peer endpoints, so that that's why I think that uh not exposing get capable uh real codex in get capabilities.

C

uh It's. We are a long way from from that, but I think it's it's a desirable goal and not listing all codecs that are supported with just one call.

A

I think we it sounds like we got an answer to the questions on this slide, which is uh that uh media capabilities doesn't have to provide information on all the fake codecs and it's not a goal to deprecate capability, get capabilities or get rid of get capabilities, but just to provide enhanced info. So I think we can. We can move on to issue 100.

E

Wait, wait. You have a long queue of people that want to okay, uh yanivar, harald and flora.

B

Yes, sorry so I thought bernard had a good point that we could uh maybe move on and that we have answered this slide, but I think some good points were raised by un and I think uh we do have a problem where get capabilities is synchronous, and so it sounds like um it might be good to clarify that um if we do, there seems to be a still goal of if we deprecate real codex from uh center get capabilities.

B

uh That would solve the asynchronous problem that we have, and I think that sounds like a good long-term goal for me and if there are usability issues with using midi capabilities for this, we should probably look at that in the midi capabilities back to to streamline that there.

B

That's my input thanks.

E

D

I do worry that.

D

The we with all this uh stuff about the rtx, red and fake, we need to have that information somewhere, I mean if we keep get capabilities around, we have a method that has known problems.

D

That's the only recommended and only way to get that information- and I worry a bit about uh about to the leave it just leaving get, can build this in there because.

D

Because making it changes to get capabilities is actually harder than deprecating it because it will be a most confusing change to the users, and one of the issues with gate capabilities is actually its fingerprint fingerprint ability.

D

D

If we keep both around then.

D

And we kind of have a direction to say that okay, real codex go into media, can build this and.

D

But we still, then still don't have a bad design interface for the rest of the stuff, so we're teeing up more work that might be okay. There might be a need, a completely different way of thinking about the problem of getting that the additional information.

D

So in the long run it should be. I don't, I think, get capabilities should be deprecated and removed, but we need to have functionality that covers everything it covers.

D

I think that's all. I have all I worry about at this moment. Otherwise, I'm happy with the conclusions.

K

And uh I would say that, yes, as you mentioned, uh you enjoys two different scenarios uh where um people would use um subcorrect preferences. One would be when a peer is talking to an sfu, and in this case you want you know which codecs you would target, and then you can make specific queries to medicare capabilities.

K

uh If I understood you correctly uh in the peer-to-peer scenario, you might have a different behavior and uh I believe right now, if you're not able to enumerate uh all the codex, you will not be able to call set correct preferences. Really, you would need to hard code a list of codecs for that you would need to check- and I don't think, that's really usable and we'll need something uh different.

K

Is there a way to get capabilities to evolve in a different shape that would satisfy everyone?

K

That's probably a direction that I would like to see if it's possible, there's also a lot of usage of capabilities currently with us correct preferences, so duplication will be difficult.

A

Yeah, so my suggestion to everybody is to bring some of these great ideas uh to the github issue and or maybe fork another one. uh I would like to try to get to the uh issue 100, which sergio put in can can we move.

A

Okay, so this is issue 100, which is inactive by default, codex and sergio put in the use case here, which is he's adding vp9 profiles, one and three and h.264 high two of those, uh and then you get this enormous stp.

A

um So the question is: is there a way to have some of these codecs disabled by default, so they don't always appear in the offer, if you're not interested in them um and fitbo notes that you know they've had these problems before you can run out of payload types in the upper range.

A

We talked about issue 99, I think last month and and this is a very similar problem, so this was some of the ideas that were put forward to fix this and so kind of like to open this up to discussion.

A

You have any comments, sergio.

L

No, I mean, I think, that I just wanted to ride the issue I mean because I see the sdp was huge and I think that as people as phillips has said, this has very or similar to to what we have in the header extension. So probably we can reduce or use a similar api in both, but I don't have any preference of the over the solution that we could implement.

E

C

um Yeah in general, I think it might be fine. My main worry is: uh what are what are the defaults? Would would they be the same for all browsers and some sombra? Some codecs are now the default, but but at some point we might want to make them uh off of by default like legacy codecs and so on. So how do we envision this evolution and this this might create web compat issues as well?

C

So that's what's the solution there? What's? What are your foots there.

L

At least the one that I am using, uh some some of them, for example, are reciproli. So I think that these are is already handled inside liquid rtc, for example, the the s264 high profile, something like that.

L

I think that the ones that I don't know I mean probably just common sense- I mean- probably not everyone- is going to use vpn profile, one two, three or six, four high profile. So probably maybe those are off by default, but they.

L

The other ones say enable by default as for us for now, but I don't know, I don't have a strong opinion on that.

C

Just to mention that the case where what I'm worrying is the peer-to-peer case where, if the defaults are different between two user agents at some point, even though both user agents are supporting h264, for instance, they will not use it because one is off by default and not the other one, and there will be no available video codecs that are in common in vsdp. So that's.

B

C

Need to have like clear rules about how we're handling this yeah.

L

But I think that here at least my proposal was to have by default in the offer, but if you receive it in the in the offer, you can add it in the in the answer. So it is just what the codes do. You do you and you provide in the offer. So, for example, in in this case, if you receive an offer within, I will say it: bpm bp9 profile, 3, and you will be able to to answer that in the in the in the in the answer. But you will not offer it by default.

L

B

L

In the peer-to-peer you will be able to negotiate, even with even with a code that you don't have enabled, because um the enable the enabling is only for the ones for the codes that are going to go into the offer. Not.

E

L

E

We have harold in the cube.

D

So, looking at the way this is, uh you need two interfaces. One is the uh the list of co of uh codecs you're currently willing to offer, and one is the set of codecs that you can offer where you get, capabilities probably suffices for the first one and the and the one on the screen probably suffices for a second and offer by default is not needed when you have codecs to offer it's simply what codecs to offer in this class too, and as for the interoperability.

D

Well, I think we should to probably push our mandatory to implements up to mandatory to offer, even though they're all in enchanted, they are our safety net against non-interoperability.

D

Not saying that it's impossible to turn them off, but by default we we need to offer everything. That's in the mediterranean temperament.

K

Yeah I look at this interface and it seems like there is a lot of overlap with current set codec preferences and possibly rtb sender. Get parameters. Is there a way to just make minimal increments, to use apis to actually satisfy the needs? Instead of having one new method and two new attributes.

E

uh Sergio, I guess that was maybe for you.

L

Yeah, I I think that I copied that from, but it is already was what was already proposed in the for the header extension.

L

So I mean I'm fine to to to move it to the to the set codec preferences. But maybe we have to move also the header station there, because I think that what would be, where is to have different ways of enabling what codex and annexation to offer.

K

Right, I think the big difference between the header, extensions and the codecs is that we have set codec preferences, but we don't have set header extension preferences or something like that, so that that's why there's some symmetry.

L

Yeah but but I mean maybe we can just add the header extension to the preferences or something like that. I mean what I would. I think that it would be good is to have the same way to enable or very similar apis to and to set what codes are being offered and one header extension and not have two different completely different apis for that yeah.

A

um I think we've run out of time for this section. uh What I would recommend for this is that we people discuss in the issue and try to come up with a concrete proposal. Is that an acceptable way to go forward.

J

And then and then we can, we can bring it back or is it ready for pr like if people can? I think I.

A

Think it's ready.

J

A

Okay, yeah, so how about ready? Let's mark it ready for pr and then people can.

J

A

J

A

Okay, so we had simulcast issues but we'll we'll talk, we'll work on that some other some other day.

A

Okay, uh so we have voice isolation, constraints, harold.

D

So this is a very simple thing I mean we all. We all know that when, uh when we're in a situation where people talk, the important thing is getting the voice and when you set up a microphone, you get a lot of inputs, not all of your voice and.

D

We have multiple ways of getting distinguishing between the human voice and other things I mean frequency, bandwidth, analyzes click, elimination and various animal approaches. That came to be doing improvements so, but so next slide.

D

So we have a lot of stuff that works, but we have a lot of scenarios where those.

D

Applying those methods will produce a very non-obvious and very undecided result.

D

Try try doing human voice, isolation on an opera opera.

K

Lovers will not be happy.

D

So the important thing is not what exactly what you're doing I mean that's something that we need to live up for I mentioned the important thing is: what is the desired outcome?

D

What is what is it that the user wants to have happen next.

D

D

Just because we have a lot of things already, that we have a well-defined method of specifying that the application wants this to happen.

D

Let's define a new boolean constraint, either true or false, and if it's true define the result as attempt to remove everything, that's not a human voice may also include attempt to find the most important voice and enhance that which might be the the guy in the middle of the camera. It might be the guy who's tallest or the one.

D

The one who's been given the floor, that's quality of implementation, but we need to have a way to for the application to say: hey user agent help me isolate just a voice and everything you can throw away. That is not voice, whether it's doritos chips or keyboard clicks or the siren of a police car going by just get rid of it.

D

D

So the nice thing about constraint is that we have a nice way of feature detecting it, because we just called maybe get media check capabilities and you get the list of supported constraints.

D

We also have to make sure that we do the right thing with with noise suppression, I mean probably, if you're doing, voice isolation. You also need you also want to remove everything that is covered by nice, nice oppression.

D

So when voice isolation is true, we should just ignored the property of noise suppression, natural.

D

So this is floating the idea to the working group. If the working group indicates that it's happy we'll just uh make up a pull request and adding the asking for the constraint to be added and continue the discussion there so discuss.

E

C

um I think I think it makes sense, but we're doing that. um I think it's really reasonable to uh ignore noise separation so which would mean that you ignore it uh like before playing constraints or before selecting devices and so on. um There's echo cancellation as well. So all there are like various properties for, uh but which are bricks that you put in the audio pipeline and I'm wondering whether some of them are can be done serially or in parallel or not, and one.

A

C

Is eco cancellation? Does it make sense to do eco cancellation, but not this property and so on as well? I am not exactly sure.

D

So translation is in one way, rather different. It's a specific source of what might sound like a human voice or might not. So I would. uh I would consider that mostly orthogonal.

D

C

My immediate uh thought on this, so you, you would be able to say echo cancellation, false and uh voice uh separation- true, for instance, yes, okay, and vice versa, I'm guessing some implementations might might find it hard to have all the equations, like all four all four uh values, uh implementable and we don't have a good way of exposing these, these kind of constraints there, except by having applied constraints, uh rejecting which is which is a bit sad. uh So it's another uh issue. I have with constraints, I guess, but.

D

Yeah, that's yes, the reporting interface of applied constraints was gathered a bit because of privacy considerations.

C

um I'm not sure yeah well, we could. We could do things there. I guess, but still it's uh yeah. Well, that's like.

E

So um sorry, if no one else is on the queue, uh yes, there is an invite. Thank you.

B

Yeah, I'm just going to say I like this: we might want to remove the parts to detect that the voice is actually human, but but my real question was that what should the default be? And I guess we need to figure that out. There's some concerns there. Maybe.

D

I think I'll I'll make a pull request and we can discuss it there. My immediate thinking is that and conservative approaches to say that the default should be whatever we're currently doing, which means that it's probably our fault, voice, isolation, false because we don't do anything extra.

D

C

I'll go write, the pull requests.

C

D

C

As the resolution, as mark issue has ready for pr as well seems good.

E

Yeah we haven't done so, I think very consistently for constraints, but I wonder if, instead of a boolean, a set of string values- and that gives us more flexibility, one could be voice, oscillation, disabled and but for enabled values. I could be a single speaker, multiple speakers, and there could be variation on that I mean I I'm not aware enough of the various ways this could be optimized, but boolean has no extensibility strings, on the other hand, will bring us potential new values over time.

D

I do wonder about that.

D

I have to think about that.

D

Having strengths also creates lots of wonderful interoperability, ways of being non-interoperable.

D

But let's discuss that on the issue.

A

Okay, uh can we move on to a lot now.

M

I'm ready if everybody else is.

M

I guess we can uh next slide, please so hello, I'm going to discuss a suggested content handle it's a mix between two different features that we currently have, and uh first I'll.

B

M

You of the two features so number one: we've got content hint, so um in a perfect world, you always have enough bandwidth to send um at the top possible frame rate and resolution, and everything is just perfect, but we don't live there, and so it happens that we sometimes need to sacrifice one if not both and which one is preferable to their grade. First depends on the context. So if you know that you're sending uh text.

L

M

Static content, like text, then it's best to just have the highest resolution possible at the cost of uh frame rate and, if you're sending video, then it's a bit more towards sending higher frame rate or at least acceptable frame rate, even at the cost of relatively low resolution, and there is a mechanism currently that allows you to set which of those you would prefer to degrade first, so it's called content hint.

M

It's exposed on track, and if you set it to text then when the bandwidth is not sufficient, you first degrade the frame rate and vice versa, not vice versa and, conversely, for motion, you first degrade the resolution and only later, the frame rate next slide. Please now. The second mechanism we have is called capture, handle identity and that one allows a captured tab to declare its identity in a free, free kind of way, any string that it would like and it can expose its origin etc.

M

So I think that by now it's kind of obvious how we could marry these two next slide, please, namely when you capture something and you're just about to transmit it. If you were to know what you're capturing you would know how to best encode it, whether you want to optimize for resolution or for frame rate, and the problem is that unless you inspect the frames and even when you do you don't exactly know what you're capturing enter capture handle identity.

M

Now we could kind of contort capture handle identity, as it is right now to convey that information, because it allows you to declare an arbitrary string.

M

The problem is that an arbitrary capture would not know how to parse that string for specifically the content hint, so I suggest that we expand uh the capture handle, so that captured content would also suggest uh in a structured kind of way what content it believes would best be set on the other side.

M

So, for example, if right now we're in meet, if I were to capture a tab with slides, then the slides could say: you'd, better uh you'd, better optimize for text, and if the slide knew that it was going to embed a youtube video, then it could say promotion and there is a bit of a zelda code over there, and I think it's relatively clear, it's inside of the green rectangle. So basically the capturer uh says hey what is suggested and if I trust this particular.

M

Tab or if I don't care, if I trust everybody, then I just set that and uh that's it, that's uh the suggestion and next slide. Please one thing to keep in mind is that if we accept this, we probably also want to make sure that set capture handle config can be called more than once, which is a point of slight contention between between us and the reason is that content can change so, for example, slides sometimes can embed text and sometimes can embed videos um next slide. Please discussion.

M

And since nobody objects.

E

Urine is under cure.

M

I know I just get it.

C

So I think that the setting the track hint is unnecessary because if a capture is is setting the hint on on its side, saying hey, I'm a page, I'm text, then the user agent knows that it's capturing that the track is capturing a page that says that it is text, and so it already knows the hint. So there's no need for a capture to know to actually know the hint.

C

The hint might only be interesting if you're, using uh something like web codec, where maybe in that case the user agent um has not uh control of the the encoder and so might it might become useful. But I I don't think that for webrtc I don't think, there's a there's a need and it might actually be a bit sad that uh applications would have to say track content hint equals something it should be automated. It should be the user agent. That would say: oh I'm, encoding uh a frame that was labeled as text so yeah.

C

Maybe let's use the encoder. That is good at encoding text.

M

So um if I answered correctly, I heard two suggestions here. One of them is that you suggested that the captured content would self-declare, but that would have an automatic effect on the capturing side. Did I understand that correctly.

C

M

C

Not on the capturing site, the user agent might use that hint to uh optimize its the setup of the encoder yeah.

M

Okay, so that means that basically either the user agent always believes the hint the suggested hint or never right, because the user agent is not going to give differential treatment. Whereas if you expose it to the capturing application, it can decide whether it wants to act on this suggestion or ignore it, because it could be.

C

Yeah, the usual agent has access to video frames, so I guess but uh based on the output, it might say: oh no, it's not good, or I don't know but uh yeah. I think that it kind of makes sense, but you well, you have a point, but maybe I do not see how the capture would uh know better, whether to use the hint or not.

C

But I I still think that in any case, the user agent, having access to the hint to the images to the video frames, might do a good job at selecting the internet.

M

That's the second thing you also mentioned auto detection. As far as I understood so back on the uh how the user agent versus the capturing application, which one has more information, the capturing application can check the origin and if it's an origin that it allows listed, it can say: okay, I'm going to take this suggestion and if it has not allow listed it or even block listed it, uh then it can just ignore the suggestion and that's how uh the capturing application could have more information and for auto detection.

M

That does not always work as well as hand tuning.

M

So, for example, if I've got a video plane- and I know that I'm only going to play one second- then I am not going to change the suggestion, but the user agent might not know that all the application has some javascript code. That's going to stop the video one second after so it's gonna, you know oscillate between text, motion text and that might not be preferable. So in general I prefer to have more flexibility.

C

um Should I answer, I see people on the queue, so maybe bernard and universe should go and then okay.

A

um So I had a comment on the web codex case, so un web codex doesn't automatically consume content hints webcodex, considers content hints to be generic higher level information which the application has to translate into specific encoder settings. So it's not something that you could give to the user agent. The application would actually need to know it. The capture would need to know the the hint and then it like as an example.

A

If the content hit was text, we don't have this at the moment, but it's been discussed that you would like to be able to, for example, turn on screen content coding tools, but that's a codec specific web codec setting that the application would actually need to set in response to seeing the content and change to text.

A

So I I basically support a lot's original formulation. I I don't think this could be done by the user agent in every case in webrtc it might be able to, but in web codecs. Now.

C

Yeah, that's what I said as well. I I said that for webcodex uh exploding, the intuitive might be useful. Yeah.

E

B

So yeah, so I think um well, I agree with you that uh and the user agent is in a good place to to um at least short-circuit the part in the green rectangle here, but the your slides, you uh sorry, they allude to an api on the captured side as well that you're not showing.

M

Right that is correct, so we already have set capture, handle config uh and that one currently just accepts origin exposed or not, and an arbitrary string of the identity, and I suggest that we add one more uh field and that field is is called suggested. Content hint and it can only accept you, know, strings which are valid continues.

B

Right, so what I would propose there is that getting that information from the captured application to the user agent at least sounds valuable, and in that case the user agent could do this automatically, at least for the case where webcodex is not used and for web codecs. It sounds like we're talking about exposing metadata of the capture, so I have some slides which we probably won't get to today, but that has a capture capture controller that might be a good place to surface that kind of information which would be similar to um so.

B

I do see some. It could be interesting to expose that information, perhaps to as metadata of the capture somewhere.

M

Okay, so if I haven't sent you correctly, you're, okay, with exposing this to the capture capturing application, you just want to continue discussing where.

C

For instance, it could yes.

B

B

Okay, but I think we have some agreement that there's some need here, but the api shape is not clear yet in my mind,.

M

Okay makes sense to me: um I think it's been 15 minutes, so unless somebody else wants to to go, I'm ready to go to the next slide section.

M

Okay, so uh next topic um avoid user confusion by avoiding offering undesirable audio sources or basically, if you don't want system audio, don't ask for system audio next slide, please.

M

So here's just a wall of text that basically says currently, when you ask for a screen share, you can say that the web application can say that it also wants audio, but that's a bit too coarse, because sometimes it only won't stab audio but not system audio. For example, if a video conferencing application might decide that, okay, if it's another tab, then it's very easy.

M

I can just transmit that remotely no echo it works, but if it's system audio echo cancellation might not work as well as it wants as it needs, and it might not want to even ask for it. The problem is that currently we're kind of forcing the application to ask for both and users sometimes go and actually try to turn on system audio and when the application just ignores that audio and does not transmit it remotely users get confused next slide. Please so we see here. This is how it looks in chrome.

M

Basically, if you're on windows, both sides, both the one that says entire screen as well as the one that says, tab would show a checkbox asking the user if the user wants to share audio.

M

Ideally, we only want we want to uh to allow separate controls for whether the checkbox should prompt the user for that on the left and on the right independently. Next slide, please same thing. uh So, oh sorry, this in this slide. So here is one suggestion of how we can do that. Could you um could you please reload the page bernard.

D

M

M

One more slide.

M

Yes, my apologies for making it a bit harder by making the last moment change. So uh just an idea. It doesn't have to be exactly this api shape, but that's one api shape we can have, we can save desired sources and then the user, the web application can say. Okay, I only want browser and then it gets what we see here or, alternatively, browser and monitor just monitor, etc.

M

E

uh Tim is in the queue.

H

Yeah, so is this only applicable to? Is it really only relevant for echo management issues? If so, I feel like we're addressing the wrong in the wrong place.

M

No not necessarily.

H

M

It could be that an application is interested in uh recording, um for example, for privacy reasons right like maybe, if you capture a specific tab- and it knows via capture, handle identity, what you're capturing that it deems that safe records that everything's okay, but if you're gonna record more than that, it thinks that hey, there's potential privacy issues here, I'm not going to save this to a file and it just washes its hands of the entire ordeal.

H

I don't see your api addressing that use case because you you're, you need two sources of info to do that.

H

So if, if you're saying that you'll only so, for example, if you're saying I I will as an app as a capturing app, I will I'm only interested in capturing audio from google suite apps and I'm not not going to capture from any other origin.

H

Then I don't see that your api allows you to make that checkbox come and go in that way. I don't think you're exposing you don't think you're exposing the right thing to do. That.

M

uh You're right that this was not the optimal example, but in that case at least, you would avoid asking for audio and, if you're capturing the entire system, but you're right that this is. um It is not easy for me to come up with um credible examples of how you would handle it of other applications. That would really need this, except for video conferencing.

H

I mean, I think that there is a problem here and I'm not I'm not saying there isn't, but I think the problem is to do with our inability to manage echo and and when the echo is enabled and when it isn't, rather than to do with making this checkbox come and go.

M

So it's not necessarily just echo, for example. It could also be that you're inside of uh another room, so basically you're. Actually two users right, like you're in a room uh with your laptop and also the room, has its own speakers, so it can get complicated.

M

So even a good echo cancellation would not necessarily work there.

H

Right, but I my point was that we're this is this is an ah this is a requirement about managing echo rather than a requirement about the ui, and I think this changing this ui to address.

H

Echo feels like the wrong addressing the wrong solution, the wrong problem, but I mean that's just my gut instinct. I've been messing with this space for the last couple of weeks, so I'm kind of conscious of it.

M

I understand, and it could be that in a perfect world, where we did perfect echo cancellation, maybe we would no longer need this, but right now we're a bit limited and what happens is that video conferencing tools usually do not want to ask the you, or at least the ones I'm familiar with- do not want to ask the user to share system audio, but users are persistent and the only way to communicate properly to the user.

M

You cannot share system. Audio is to just not offer system audio. So in the future. Maybe this will be less useful. I agree.

E

Univising the queue.

B

uh Yes, so I'm actually supportive of avoiding capturing, I would actually prefer browsers, never captured system audio, because I feel that's a big privacy issue that users are not aware of. um So something like this might be desirable. I was wondering, but instead of creating a new constraint, I feel like we already have one that we could use here, which is called the display surface. I'm wondering if you thought about reusing that here and because we have a similar debate about display surface.

B

I have the same concerns in that debate, as I have here that we should disallow monitor so that this doesn't become an api to for applications to request the thing we're trying explicitly to discourage.

M

ah I would like that yeah. That would work for me. So basically, if I'm studying correctly you're saying exactly the same, so the code that we see here instead of desired sources, we write display surface.

B

Yes, and uh and whatever we decide on uh in our other discussion about disallowing monitors should apply here as well. I think I I would like to disallow.

M

Monitor specifically here yeah: well, that's, that's. That's not gonna happen because chrome currently supports that and wishes to uh keep on supporting capturing system audio okay, so.

B

Let me not tie it to that other conversation. I think specifically for this one. We should disallow monitor right.

M

I think that we can discuss the current matter currently and the other matters another time.

B

Right because I would have a uh concern with this api if it becomes a way for the application to request system audio.

M

B

You can request system audio.

M

It just can request all audio or no audio, so I think we're already right um steps forward if.

B

We just but the goal is to enable an application signal to say we don't want system audio. We shouldn't have to accept that this comes part and parcel with an api, an app signal to say we want.

M

System audio well, the way I see it if, if an application wants to say that it doesn't want tab audio, that's also perfectly fine. So long as you know, it doesn't make us ask for system audio more than we did before.

B

Okay, well, my support for this is contingent on not supporting, monitor.

E

I mean, I guess what I'm hearing is.

E

If we don't include monitor in the list of supported items, that doesn't mean you couldn't capture it. If you don't specify anything, there may still be the capture system audio option in the ui, but you can't ask specifically for system audio as a new feature that this api would expose. Is that what you're saying anywhere.

B

Yes, so um I mean chrome is still fine to implement it. The way it's doing now. It's just don't want this to become a web compat issue where applications are starting to depend on, let's say browsers in the future, want to change their default to not include system audio. This shouldn't be a way to request it.

E

B

We could we could take this offline. I think.

E

So, in terms of aligning with deeply display surface, um I mean in terms of name space in terms of value space. Sure in terms of name displacer, face and audio are really a strange combination.

E

So yeah when I maybe that's okay, but it's a bit of weird ergonomics to speak of display surface for audio sources,.

B

Well, the audio is still coming from, I mean you could say the same for the values right browser and monitor, also not audio.

E

I mean monitor is definitely strange. uh I mean I, I think that I don't think I feel strongly enough about it. But, uh okay, it's uh I.

B

Think! That's fine in that, because audio is thought of as a as a secondary thing in get display media at least currently.

C

B

Always belongs to a display.

C

Is there a use case for window because it seems there's a use case for tab and that's about it? So if that's the case, then we could be more specific.

M

I think that eventually some browsers are going to support capturing window audio as well, and in that case I want applications to be able to specifically say whether they want that or not. I don't see a reason to only go for tab. I understand about system audio and I think that it's a reasonable compromise to say that if you don't specify any desired sources, you get all of them like, if you just say audio true.

M

But if you want to specify I'm willing to say that you can only specify browser window or browser end window.

M

uh Jennifer, do we have a compromise here or.

B

I think we have some interest here, so I think we could resolve the details on github. We could probably make this ready for pr if other people agree.

D

E

D

A possible label, so you can mark.

E

That just so that uh it sounds like what we have in fact is not so much that you want to specify your display surface, but whether the audio source is strongly tied to the display surface. You've picked or not, and I don't know that it makes sense to say browser or window, because in fact you would only pick one of those display surface in the end. So it may be that what we are saying is audio source displays your face or audio source, uh anything which would open for system audio anyway.

E

That's probably something we can refine.

B

The weird part here is that the correct answer to specify this to avoid system audio is a bit convoluted. You have to say desired sources, or you would also you have to remember to always include browser and window just in case uh a browser in the future decides to add the audio window right, but it's perhaps a less ideal surface if that's the main goal, so maybe something more specific like avoid system, audio, true or false, would be.

E

Superior yeah, well that's an equivalent to what I had in mind, probably a clearer one.

C

I think that yeah sorry, it seems to me we we should continue this discussions on github. uh It seems that the scoop is not yet very clear as well, whether it's tab, whether it's system, audio and so on. So that's the first thing that should be done, it's the scope and then once the scope is clear and those consensus moves.

M

How do we, how do how do we clarify this scope, because I would like to you know, make rapid progress here and I still have eight minutes, so maybe you can resolve that question now.

M

A

M

How do you suggest that we resolve the question of scope.

C

um So I would animate the different uh approaches, so there's apparently uh avoid system surface through force. There is a display.

E

C

Which is yeah the display surface and whether to include monitor or not, and there might be a just restrict to tab. I don't know these kind of things, um so maybe there's uh one option that has consensus like is avoid system. Would you having consensus, for instance,.

M

I think that I think that this would be a bit awkward when we eventually implement window audio and then okay, do we introduce avoid window audio too.

E

Well, why would you want to avoid window audio, though.

M

um Application web developers are creative, some of the uh some users are going to.

B

C

M

Browser window there could be a lot of reasons for that.

C

So the good thing about having separate uh properties uh separate constraints is that then the user agent say hey, I'm supporting the uh this property, which might say: oh, oh you're, actually supporting a recording window. For instance, audio of a window. Oh you're, supporting a recording audio of the system, oh you're, supporting a recording audio of tab and currently.

B

There's no way, there's no easy.

C

Way for a web page to to do that, so if they are trying to convey information like information to the user- hey you- you might see this, uh so please please do that! Please click there and so on. They might miss some information there and so having different properties has some advantages as well.

M

So um my uh personal interest is mostly in avoiding system audio, so any kind of thing suggestion that would allow that that would be an okay compromise for me, but I think that you know going forward.

M

The most flexible option is the best option and that is to just allow the web developer to specify which sources he is interested in uh and we don't need to think about why, but that's my opinion. So.

H

So going harping back to my counseling point. Can we say something like um that? This constraint lets? You say you only want that checkbox if the source is capable of being echo cancelled,.

F

D

I actually think echo cancellation is a secondary concern.

D

You want to not capture testimony because there's a because they suddenly might come up voices from the next tab and disclosing that you have been having an incoming call from some person. You'd rather not tell the other guy about, but.

H

That's also true for um for window yep, but I.

D

Mean I I I take a point. There's.

H

A privacy concern as well, but.

D

It's definitely not about echo cancellation. That was my point and that's the ellen says. If we just offer the capability, we don't have to worry about. Why.

H

But if you knew that echo cancellation would work, then maybe you want window and if you know that it don't doesn't, maybe you don't want to offer it like that. The way this was presented, and certainly from my experience, the reason you don't want people to pick. This is because it sounds terrible, or at least in in large part. So right, I feel like we by not exposing that we're kind of in in in the constraint we're kind of hiding pretending the problem. Isn't the problem where.

M

It really is well we're addressing the problems that we're able to address right now when we've got imperfect, echo cancellation, that's number one and um number two is we don't need to do everything with one api right, so an application could using a different api, find out okay, but which sources would be echo cancelled well, and then it could request those or if it's got other uh criteria for making a decision about which sources it wants it can. You know, use both sources of information to make the decision.

H

Yeah, but I think that this is a nice place to put this because you're not offering the user, something that you're pretty confident won't work.

H

Like I mean I what I like about this, is you you're you're, stopping the user from choosing something that's going to go bad on them, and I think I claim that at least part of the kinds of things that can go bad and I take the privacy point.

H

But the other thing that can go bad is is er cancelling and the browser the user agent is conscious of what it's currently capable of, and therefore that information is already visible to it and, frankly to nobody else, and so I I contend that it would be really useful to allow that hint to cover this. The explicitly cover the can it be echo, cancel properly or not.

M

um I'm not an expert on echo cancellation, unfortunately, but the way I understand it, it's not a binary thing right. It's not you either do perfect, echo cancellation or not at all, and it is up to an application to decide whether you, um whether it's happy with the results.

M

So it I don't see how the user agent could make that decision on the uh on behalf of the application, given that it, whatever decision it makes, would have to apply to all applications.

H

I I'm well, I mean I don't know the um certainly my experiences. There are some like some things that could cancel well and some things don't and and and though that is fairly closely tied to whether it's in the browser or whether it's not in the browser at the moment.

E

We have yaniva in the queue.

B

Yeah, so um I think we have uh the res the situation that I'm concerned about is if this api surface, as described right here, would appear to allow someone to pass in only the monitor value and thus be able to sway the user to picking a full screen capture when they otherwise would have picked a tab capture. That would be really bad. So as long as we can have some language to prevent that, I don't think I care that much about the api surface.

M

Yeah, I think we've agreed on that. The nevr.

B

B

Alright, so then I suggest we bike set on github cool.

A

So can we move on to the last item.

C

C

Okay, so um crop target uh and region capture is a new new api and it's activating actively being developed uh in chrome, so it we have some issues that might have an impact on api shape and it would be great if we could have an understanding of the api shape as soon as possible. So there are like three different. uh There are four issues, but three of them are being discussed right now. uh One is whether we want to expose uh the creation of crop target either to elemental media devices.

C

The second one is whether the creation of crop target we we want it to be a method or an attribute, and the third one is whether we want to return the result of this method as a promise or whether it's async or sync, basically next slide. So, let's look at all of these, like in three slides.

C

So first issue is whether we want to attach the api to element itself or media devices. So a crop target is basically an object that is encapsulating a pointer to an element, a remote identifier of an element, so we could either uh call element.getcroptarget and then you you get your crop target or, as it currently done in chrome, port type. You would call something like navigator, dot, media devices, the produce scroll target and pass the element as a parameter.

C

So the advantages of for media devices option is that uh you're grouping the api to capture related stuff in a media related place. So it's uh some might say that it's using documentation and search ability.

C

The advantages for element is uh first uh in terms of documentation, search ability. We we already have partial interfaces so, for instance, there's element request full screen and request. Full screen is a kind of media api and it's it's okay, to put it directly on the element, because it's tied to the element itself and the second bonus is that the element and the crop target are tied together, but they are not tied to the media devices itself and uh by putting it at the element level and not the media devices.

C

We avoid some some corner cases.

C

For instance, media devices is attached to a single document, so you could call media devices for the scroll target on an element that is not attached to the same document as the media devices object, and this might have an impact. For instance, the media devices might be neutered if its own document is detached and then it would throw or it would reject the promise. Even though the element is perfectly fine and of course, an element might be added or removed from documents.

C

This is the dom, but media devices is not able to follow it's tied to a document and will it will stick to image devices?

C

um The third point is that meta devices is secure. Context and element is not, and you might still want to create a crop target for an element in non-secure context. That's still something that might be uh possible to use. um So my recommendation is to move to element uh because, as I said, there's not a lot of downsides and it seems easier to just call get crop target without any any paradigm fruits.

C

You any value on the.

B

Queue yeah. I agree. I think that uh the problem with putting it on media devices is that, then we have two relevant objects and we that's more than one what we need. So I think the media devices object here is actually an irrelevant object.

B

I would look for element.

M

Yes, um I'm voting for media devices and the reason is, as you have stated, that it is right next to get display media it provides nice encapsulation of related apis and right now, element has you've chosen. One example of something that is exposed an element. The problem is that you've got hundreds of examples. I think, because there are so many things that are currently exposed, an element, and I think that uh that is, it would be good to stop that.

C

um Why I mean co target is being used by mediation track, it's not being used by display media.

M

C

M

You say that again.

C

Quote target is used by media stream track. It's not used by get display media, for instance. So I do not see why media devices is tied to to do that and- and it's really tied to to an element not to its many devices. So I would prefer that you prefer to have some quirks uh for a benefit. That is uh what you're claiming documentation.

M

So uh you've made two claims here: uh number one. It is actually um related to get display media because you can only crop something that you got through get display media and that's why it needs to be adjacent in my mind, and the number two is that um there don't necessarily have to be any side effects to uh putting it on midi devices because call it on with one midi devices or call it with another.

M

It doesn't matter, you get two crop targets either you get the same one, because you know the the function actually put it directly on the element behind the scenes or you get equivalent crop targets. It doesn't really matter. That's.

C

Not true, currently on comp implementation.

M

That is true, but it is. uh It could be true on chrome's implementation if we take some time to fix it, and also there is nothing.

B

M

Us from you from implementing.

C

M

C

I believe that if media devices is neutered, so if its document is detached, you will have difficulty shipping anything but rejecting the promise, because that's something that is uh very natural and is being used a lot. So it would be counterintuitive to actually undo current chromatin chrom's behavior and do what you're suggesting.

M

uh It is not clear to me why the promise once produced cannot outlive me the devices.

C

That's that's uh usually what uh apis are doing currently on the web. So if you're, if you're calling uh getusermedia on an iframe that is detached, then it will reject and uh a lot of like all a lot of promises are, are doing exactly that and that's that's the web.

M

I don't want to dive too deeply, because I know that you've got a couple more slides and also more people in the cube.

E

B

Yeah, oh no yeah, I was hoping others would have. I just want to say that I don't see that the media devices object has anything to do with, uh what's being captured on the captured site right. This is where we're in uh this could be a page that doesn't even have https so yep, and so let's see here, stick your context. Yes, so this seems like a no-brainer to me, I'd like to hear how we would we would solve, and why are we not concerned about capture of non-secure context.

M

um I don't think that non-secure contexts are very interesting uh use cases. I think that those are disappearing quickly and I don't think that anything that you get from a non-secure context should even be trusted in any way. So when you get a crop target from that, I would just ignore it. If I were a well-structured application.

C

Thoughts from others.

E

I mean I I'm not very deep on the topic, but I I tend to agree that element feels like a more natural target, if only because what you're really doing is getting when you, you don't even have to pass the element as a parameter as a result and yeah right now. I guess I I'm not finding a lot of strong advantages to using media devices. I I think the ergonomic argument plays both ways so.

M

Yuan, would you like to proceed to the next topic.

C

I would prefer that we uh we tried to get consensus there. I don't know how we could do that. I agree.

J

C

You have an idea of how we can proceed.

M

M

Not really, but I will check to see whether it is something that we could compromise on, if, though, um so we can uh continue on github or get back to this. uh I also think that this is not the most important issue that we have here because worst case scenario. If we uh change the decision later, it's not a difficult transition to say: okay, we also expose it to an element and then to say a bit later and now we only expose it to an element.

M

So I see an easy transition there, uh if necessary,.

B

Would this be? uh Would it be helpful to ask the tag on this, because this seems like a fairly simple api question to me, whether you have an object with a method or a method on the different object, detects the object as a parameter.

M

Sure I believe that targets actually discussed this today and their minutes will eventually be published.

M

C

Okay, so I guess we can wait for the tag, input and see uh what they have to say about this, and we can restart the discussions on it. Stopping next um so next uh issue, um so in case of exposing api at the element level. So let's say now: we expose it to element. uh Api can either be a method or an attribute.

C

So there are two things there: if you're using an attribute, then you're somehow mandating that the symbol single crop target is generated by by element. So while for a method, the usual thing that happens is that you will get a different target every time you call the method, that's typical from uh for web apis, which means um so one thing to note is that it's already possible to generate several crop targets that will reference to the same element given crop target is serializable.

C

I I don't have a strong opinion there. Initially, I was for an attribute, but I would be uh okay with an element if some people really strong have really strong feelings about it. So let's say it's put at the element, which one would you pick.

L

M

So uh I would go with the method, and the reason is that there could be a cost to actually uh doing that and they want to dissuade uh applications from just inspecting that for no reason and the way that there is a cost for this, is that currently uh what we do is as soon as you uh meet the crop target. We mark the element throughout the rendering pipeline, and that means that the element gets rare data.

M

Rare data is a mechanism by which chrome makes sure that you know elements are relatively small and only those that need a bit more information actually allocate that extra information, and it's better if those are not created than necessarily.

C

um So I think you said also that you wanted to do lazy, surface tagging so that the creation would have almost no cost you you, your implementation is in current in this current state, but uh I I think I heard that you were hoping to to make it lazily uh tagging.

C

So in that case your argument uh would basically.

M

Like that yeah, the lazy tagging would not. Actually, uh if we put things on element itself, then lazy tagging would be okay, it's a bit of a longer discussion because it also depends on which document it is and then when the element gets moved. Okay, because basically, what you're saying is: if you do lazy tagging, then you don't actually take the element. You just put the tag on the document itself.

C

Or the map you do not need to make the element bigger if you, if you need to it's, just an implementation, detail.

M

um I would have to think about this, but I still think that um for so long as some implementations or some reasonable implementations actually have a cost, uh then it in the side effect. Then it makes sense if get crop target is a method just to make that a bit uh more evident to web developers.

C

I see universe on the queue.

B

Yes again, I also to save some time. I think this should be on the element. It should be an attribute. It should not be a promise to jump ahead of slide. uh I mean I'd like to cite the principle design principle of simplicity.

B

We should prefer simple solutions and the way I view simplicity is mostly to prefer uh the priority constituencies being the user and the web developers and all the concerns we've heard so far to pick something other than that have been implementation, concerns which I think, as long as this is implementable, the right way we should implement it. The nicest simplest api.

M

um I just want to say that I see the constituencies as a weighted average and not as a you know, first-past-the-post kind of thing. I think that, yes, it is important users, don't care here. Web developers barely care about these uh small differences. It's going to be equally useful for them anyway. So then it actually matters a lot for implementation.

M

uh I see how it is on the queue next.

D

I just want to make sure that I have said that I disagree with yaniva on every single point I think, uh get I think, uh messing with element is bad for the web platform. It's already it's already doing far too much. It's a god. It's a god object and uh the exposing. The fact that uh that these mechanisms have costs is a guy is a guidance to the users that to towards don't don't use them. If you, if you don't, if you don't want to pay that costs and doing the promise is.

D

This particular interface connects to methods that go between different processes and not necessarily get crop target itself. It's possible that they to do that elsewhere, but it's also reasonable to have implementations that go between processes and then promise is the right approach.

D

So keep it away from element because element is a bad place to be and keep it, keep it the method, because it actually has costs to run it and keep it a promise, because this that allows implementations that would be impossible.

D

If you do it as an attribute, and it's not that much, it exposes the cost to the user.

D

Instead of trying to hide it behind something that looks simple but isn't so disagreement statement.

B

All right I'd like to respond, so I basically disagree with what everything harold said. So the the cost here sounds like it's in chrome, there shouldn't be a cost. I think that's a bug, because the real name for this should really be to isolate properties. Here, the attribute should be really transferable reference. That's all it is because the element itself it wouldn't be needed if we could reference the element directly so and that should not be a costly operation.

B

I think that would be a bad implementation to if the user references and creates a lot of crop targets that should not be that should not cause any bad side effects, because and just for the other audience here, there's nothing that needs this to be a promise, because it's not like this would have immediately affect cropping.

B

This is just so that we have something to pass to a different process that might be calling crop to where the crop2 method is asynchronous, and so there's all the time in the world at that level to inspect these these elements, which they really are, whether you know them through a reference object like this, which is really what it is. So there's really no need for for a promise or a method, or for this to be on media devices.

B

So I'd like to deduct the need to be documented to not choose the simplest api here.

C

um I agree with yanivar there. I think that we, we identified some flows in chrome's, current implementations uh in terms of uh memory and so on. This has been discussed and I think there was consensus and there were some changes needed there anyway, and so uh that could change.

C

uh I I'm hoping that it could change your position uh on chrome side, um but but in any case, uh we should look at what what is being uh feasible to do in a natural way and what other apis have done in the past and other apis in the past have done have put apis on element because it makes sense all the apis in the past have looked at, whether it's synchronous or not, and if it's, if it can be done, synchronous, they use promises uh and the same for attribute versus methods.

C

So if you, if you look at what web ap how web apis are consistent, and if we want to keep uh consistency between these api and the other web apis, then we should use uh what other apis are doing and over. The good thing about other apis is that they are implemented, and we know for sure that uh it's feasible to to do without too much implementation issue dominique dumb you're on the view.

E

Yeah something geneva said makes me wonder if, in fact, we should be looking at something that is a transferable reference for the crop target. uh That's.

M

The full issue.

E

Okay, I mean: are there other api that would benefit from having this transferable.

C

Targets, that's a very good question uh that I asked on github there's a github issue uh for that and uh that's something we should. We should dig in because that would impact the name of crop target. For instance,.

M

Exactly actually, if I could just go a bit over time and answer this, and my argument is that a we've never not identified any such api, but even if we will, in the future, it's actually by design that crop targets can only be used for cropping, because right now you mint that and you pass that cross to another cross document right and you do that with the safety of knowing that it cannot be used for anything in the future that you did not think of as a web developer.

M

So it would be bad design to in the future, decide that everybody who did cropping is now also trusting the other side for something else.

B

You can check that in the crop to method. That's where you know.

M

B

Input checking.

M

No, no, no, because right now, if I it's the other way around, if I have a crop target and they post it and it exposes more information in the future than me as a web developer. Who currently saw that it was not exposing any information and I tested, and I thought I was safe in the future. Suddenly it turns out that they passed more information than I intended for information.

C

It might be used in places you, you might not have thought it would be used. Yes, and if we uh change the name and we change the the way we look at that, then we should take that into account. Thinking yeah. There might be new apis new places and it should be safe, uh that should it should be safe to use uh the same object on on the two apis.

B

But those future unsafe apis need to be written, so we we could make sure to not make them unsafe. Well,.

D

If we, the one problem here, is that you're suggesting inclu, suggesting and creating a completely general mechanism that can have many possible possible applications and using that as uh as uh and which can have lots of implications. That cannot be evaluated because they don't exist yet and using that to block a very narrow and specific interface that is going to be used. For one thing, only I'm not sure where we are the right group to and to introduce very, very wide wide-ranging interfaces like an element reference that can be used used across that process.

B

But but we are we're just talking about naming.

A

uh May I point out that we are over time now.

C

um How should we proceed, then bernard, because this should have been discussed for like uh one or two months and basically I think that uh in terms of implementers, we we are pretty clear about what people think, but we need still to make a decision.

E

It feels to me that there is a narrow crop target discussion on the broader platform integration question, and maybe we can start with the solution to the narrow thing and evaluate whether there is a migration passed to maybe the right solution. If there is a different right solution and working on the right solution, I think, as harald suggested probably requires reaching out to group. That would be the right place for this broader solution, but that again would require, I think, working through this other potential api users of this broader concept.

B

Well, I I think, that's a mischaracterization of one position here, though I think we're saying we're not saying this should be a broader solution. We're saying these are the standard web patterns that api hasn't influenced api design for a long time and there's no need to. We shouldn't depart from those uh standards without good reason, and I don't think we've heard good reason from.

E

Implementation so to be clear, my point was, I think, the the api shape until we discuss the name uh is definitely something where we should follow the call pass.

E

But if we are looking at broadening the scope of crop target, then I think we are starting a very different conversation, one that I find very interesting but uh very different one yeah. So.

C

That's one of the separated for this meeting. It was separated, it was api shape, and there was this fourth issue that we might talk later, but not for this video.

B

I don't think anyone's blocking on on the name.

M

Okay, by the way um in one of the emails, if I'm not mistaken, both call for consensus, the name was actually brought up as a blocking concern so uh and by u.n and dna vary. You agreed so yeah.

C

M

Hearing that you're not gonna block on the name now you and.

C

M

C

um As I said, I raised the concern that there was there were some issues and that they should be documented. That's that's what's uh my concern and I think we resolved it by saying hey. We are not sure about the naming and that's that's a good solution for me right now.

C

I think that uh we should try to solve these issues on time and I'd like to focus on these three issues there, which is think or not sync element or not, or media devices, an attribute or method, because these three are pretty uh specific isolated and we should be good. uh We will. We are the university working group that should uh make the decision there, and my understanding is that we have uh identified that.

C

uh If we look at the patterns that are used currently in the web apis, then we have a natural choice and that's the one I would push for using.

B

F

C

I'm not sure how we can make progress.

C

The tag information for the first issue, but for the uh second and third issue, uh I'm not sure the tag will give input by.

M

The way was it mentioned.

C

M

So tag already uh approved the entire design before and now they've been asked to uh weigh in on those particular issues and.

C

M

Okay, but let's say that they come back and the answer is not totally conclusive and nobody changes their mind. I suggest that um one way that we could proceed is that we can have this particular api accepted. We go with that and we consider migration paths in the future when it's when mozilla and apple have also implemented. This are ready to ship it and say hey. This is now um a synchronous for us, and so that's.

A

M

Another one is that you, you could always uh polyfill this. You know so it's much easier to polyfill one in one direction. Right like you can make sure that the um non uh the asynchronous uh I'm sorry, the.

B

Synchronous one: can you make it something.

M

E

And I guess we are way out of time, so the chairs will convene and propose a path forward.

D

Yeah, so we might be at this at the point where opinion polls might be useful because, given the given, that tank has already reviewed this, it doesn't seem to be all that chance, great chance of waiting for another tag. Review of those specific questions, but.

D

The one chair that does isn't strong, your opinionated on the mattress seems to have dropped off, so you might.

E

D

Take respect to the chairs.

E

And I think we need to adjourn having lost more than half of the participants. I.

H

Think the polyfill question is actually a place to attack it from next.

H

I think that that actually might result might narrow the the answer. Space.

M

Okay, one quick question: uh how would uh an opinion paul uh help here, given that we know everybody's opinions right like.

D

Well, well, there might be some there might be people who don't uh haven't spoken. You don't know my opinion.

H

D

B

H

D

As you can see,.

M

That is correct, but if you knew, for example, that you could have it tomorrow, if your opinion changed then like I'm not sure uh there are browsers right. Browser vendors have one set of uh criteria and web developers have another and different web developers might change their mind based on okay. How quickly can I have it, and I it's unclear to me how to evaluate such opinions.

E

Well, I mean I can give you the way this has been looked at in the tag before the future is longer than the past, and so, if it's gaining one month of variability against having a bad api forever or a bad api that is hard to move away from, then then I think the calculation points towards taking an additional month. But of course nothing is that black and white, but just to say that time to market is a pretty weak argument for rushing a design in general for the platform.

M

No, I meant that the web developer would want to have it sooner and therefore it would care, because in this particular case I think that the web web developers would not actually care about the shape right. The shapes are almost identical as far as a web developer is concerned,.

E

Yeah and that, I think, is a reasonable argument. I mean there is a cost in the mental model. If, indeed you don't, you can't use your existing understanding of the platform to navigate that specific api, but I, in this particular case, I don't disagree with you that the mental cost is pretty limited.

D

Okay, I think we need to end the call and with the chairs, we'll discuss further about how to resolve this issues. Okay,.

M

D

D

It's not recording.