W3C WebRTC Working Group, 22 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WEBRTC WG meeting at TPAC 2020 - part 2

Description

WEBRTC WG meeting at TPAC 2020 (virtual), October 22, 2020.

A

Okay, the recording has started.

B

Okay, so you have the floor: harold, okay, so.

A

The thing that everyone wanted to do once they prove that it's possible to do insertable stuff in the pipeline with the encoded media, is get something that works efficiently and simply for raw media.

A

Now everyone who's worked with media knows that raw media is slightly harder to deal with than most other things for your shuffle, primarily because it's so big, but I tried to design an interface that would be implementable, simple to use and admit the same power as what we had today and allow new cases.

A

So the picture to the right is a random advertising for google duo, which has effects, but like funny hats, but only on the on the on the mobile, so we'd like to do to make sure that it's possible to do this kind of thing efficiently on the web.

A

A

So everyone's seen this search right by now, basically there's the entire set of boxes that go into mrtc and what we're now looking at is not the stuff to the left, which is all the networking stuff that everyone loves, except the ones that who the ones who don't we're. Looking at the stuff on the right.

A

What goes on with use the media before it ever touches a peer connection, actually.

A

There was a nice picture. I think it was given an at an earlier talk last week that had like the media stream track in the middle, and then you could feed cameras. Microphones desktop captures incoming streams into it and pull stuff out on the right, presenting on a video on a canvas on a video tag on a recorder or media capture or something else, but the media stream track is kind of this central idea in the local media processing pipeline, and what we wanted to do is open it up.

A

I mean those of us who have dealt with communications for a very very long time will remember breakout boxes. They were what you used to deburg hardware interconnects the times when the standard plug had 25 pins.

A

Nowadays, people are using ethernet cables. How simple, anyway, a breakout box then permits you to disconnect some of the wires that run from one end to the other and perhaps patch some wires onto other wires, and that's the mental model that you would want to have in when making it simple to understand the break. This three stages of the breakout box, because the first thing we have now is a self-contained unit, the breakout box stage, one you can apply constraints to it.

A

You can couple it to sources and things, but you don't get that what's inside.

A

So media stream track breakout box stage. Two, you open up the media, you say: okay, any media that comes in goes to javascript. Anything that comes out from the javascript is interpreted as media can get gets fed back in sounds simple.

A

Perhaps it should be next slide, so as a track processor with the api, as proposed here, it's pretty simple, simple. To make. Let's say that you have a face: detector written, wasn't what whatever that tells you where a face is in the picture, so you take a video frame. You detect the face simple!

A

Then you have a function that adds a moustache, given that you know where your face is so now you call the atmosphere function on the same frame, bang, bang, but that.

C

Only gives you.

A

One frame, so you have to have a lot of scaffolding in order to get the frames to the right timeline right, not much.

A

You can just say.

A

I have my media stream track. That comes from my video input and I'm going to turn that into a processing media stream track that has this way of pulling out and then I can create what's called in streams parallel parents stream is such an overloaded word, sorry, a transform stream, which is you put data into it, and you get data out of it and what you're doing in the middle is call the add massage function.

A

So that's all you need to say: if what comes in is a video buffer, then add the mustache to it, and then you can say I have my track with the two wires dangling out of it, the readable and the writable, and we say, take the readable and pipe it through the red one.

A

That's it and now have a massage. It takes longer then grow it quicker than growing one right.

A

So that's stage two, but of course anytime you got two stages: you're, probably setting up the stage for stage three next chat and that's where we break the things totally open.

A

Where you say, when you have a media stream track, you can turn it into two objects that don't have any connection between them. One that takes a media stream check, emits the video data and takes feedback signals.

A

And the other half takes the media and emits feedback signals.

A

So the idea is that if you just slam these two things together couple the output of one to the input or the other in both directions. You have something that behaves, looks and behaves and can be treated exactly like immediate media stream track. But if you pull them apart,.

A

There is no connection between them anymore, except what is visible to the javascript. We have opened up all the switches on the breakout box next step.

A

This is whereby the alfred.

A

You basically have a check, processor names.

D

A

Welcome for bikeshare: this is what I thought of that takes a neater stream check the source and it has the attributes of the readable stream and the writable stream, one for video frames, the other ones to control, and then you have the check generator but behaves like the other end of the track, which has the streams and you can set them as sets the track generator as the source object on the video tag or do whatever mechanism you have for stuffing video tracks, media stream tracks into other objects, because it inherits the interface.

A

A

So when I'm talking to media people, they ask me: why do you bother with these control signals? Who cares about control?

A

And the fact is that most of the stuff we do has certain properties that are not just data flowing from one and to the other, for instance, you can stop a track.

A

If you have these two halves coupled together and stop the last one, what happens to the first one should stop if you have get feedback from downstream to the end of the track, saying hey, I I can't handle so much data. Can you give me something smaller.

A

That should propagate or give javascript a chance to to change it. So this is very, very, very sketch pad like saying that the objects you send is a control signal which behaves pretty much like an event.

A

It has a name and it has some properties.

A

The properties tell you more information about the events, unless you can, unless it's a simple thing so that you don't need them, so it's kind of like if you configure the downstream it should tell the upstream yes, well, here's my what I want as a configuration now can you can you give it to me.

A

One particular trick that I was thinking of based on other conversations, was whether we could allow control over the pixel format that is coming down the pipe because the two most expensive things that are overhead in video processing- it's one is copying the other one is format, conversion.

A

Of course, format. Conversion needs a copy too. If you have nv12 and your downstream takes rgb well, you've got to convert if you're downstream.

A

If your upstream is your v and you downstream is n with 12, you got to convert and every conversion takes memory bandwidth, so the less we can do the more speed we can get out of it.

A

The next step current status. We have an experimental implementation, it's capable of getting of generating a green screen for you. That is a.

A

Take figure out where, where you are in the picture and then take out everything else, and we hope to be landing, this experimental implantation behind the flag in chrome, 88 and after a severe boat with writer's block, I've finally managed to do a start at the specification in a public github repo that looks like it wants to be a wtc specification.

A

So that's the short version of the presentation of what I want to do and where we are so now is the time for questions or comments or whatever you.

B

Like I have a question harold, um which relates to the uh buffer handling, uh is everything necessarily in main memory, or can it be uh gpu buffers.

A

The we're reusing the video frame and audio frame for that matter that were defined for web codec and the web codec folks are thinking that they have found ways to handle a main memory object that is basically backed by gpu memory so that as long as you don't have to convert- or you can run the converters on the gpu- you can just chip around this. This thing with a pointer down into gpu memory.

A

So I don't know if that's going to work in the first version frankly, but uh it's definitely something that we want to design to be possible so so that we can either get it to work at once or get it to work in a later version.

D

A

For web codec and for breakout boxes.

D

If you got these handles working, do you think you could write like web webgl code, treating them as textures? Do shaders.

A

I think that this would require marking the buffers to be in the format of a webgl surface or whatever texture, whatever, whatever you call them uh as supposed to just the pixel 4 or the main memory.

A

Sometimes it should be possible, but.

B

I don't know, but the idea harold is that the gpu is mapped to main memory. So then, like web transport could transmit it without a copy.

A

Well, I hope so. Okay, I'm not sure how how access to web memory webgl memory actually works these days. Is it.

E

They're made for my.

A

F

I believe them you can use. um You can see these uh video framework products as cms bitmaps, which are structures that can be backed by the gpu memory and then you can you can run then I'm not an excellent detector, but I think you can operate with webvl on those objects, so everything everything stays on the on the epu.

F

That's totally what.

A

I'm hoping for guido is leading the implementation project.

G

So can I ask if there is, if these are strict transforms I mean, can I can I in effect write a an mcu by mixing streams? Can I write an sfu by taking a stream and sending it out to five different peer connections.

A

uh In the in the way, you should be able to write an mcu in breakout box stage 3, because then there is no connection between the two halves of the stream, so that you can take the take the buffer clone it and send it off five times.

A

We have a project down the list of things we want to do, which wants to do the same thing for the encoded side, which would would allow you to write an sfu.

A

But at the moment you can't write the necessary because the insertable streams for encoded media cannot create buffers. It can only either transform them or draft them.

G

So that is a strict transformer okay, so that one is a strict child.

A

Transform then the one we're proposing now will not be cool. Thank.

H

You so uh from the examples you've shown so far. This is all main thread. Correct.

A

um It would have been if the, if not for the fact that streams are transferable.

H

Right but the uh transform stream in this case was a main thread: transform stream yeah.

A

Well, it's sort of out there transform streams are transferable too.

H

Right, so to do it correctly, you would have to create the transform stream on a worker and then post messages to main thread, because media stream track is only exposed on main thread, and then you would have to pipe it in there. Is that correct.

A

Well, I'd actually do it the other way around saying you, you have the track on the main tread you take. You you take out out the streams. Then you post them to the to the worker. That's the way. I they just because that's the way I wrote my demo code for very encoded in certain insertable streams.

F

So you will have the track: the control aspect of the track and the main thread, but the the media flow and the transformation in the worker. If you want to.

B

Work and posting posting to the worker doesn't involve a copy right.

H

It might depend you're transferring ownership, so it's a little uh so you're relinquishing ownership when you transfer something right.

I

Yes, just to say that uh you your example is correct as as well, you could do what what you want to say what you want to do and harold as well, so it can be done. Both sides.

J

We have a colon in the queue.

C

So there's some- I I I think this is coming back sort of where tim was too a little bit like I. I get this like where this going. uh I'm thinking about the cases where we I mean if the camera is producing compressed media, this means we're going to decompress it to operate on the frame here then recompress it and and send it again.

C

um I think I'm sort of curious about the option of also having a possibility to do this with operating on effectively the the chunks of the compressed media, as well as the uncompressed media like you're suggesting here uh you.

A

Know that that depends on the compressor I mean. The current example we have are is hd cameras on on the usb 2 bus and they produce motion jpeg and no, but no sane person wants to send most to end up with motion jpeg, so handrick actually has some slides right at the end for dealing with uh or for avoiding motion jpeg.

A

But uh at the moment the easiest way to think of motion jpeg is that is it's another pixel format, just a very old one.

C

Sure um I mean most newer cameras are not doing motion jpeg anymore, but I mean uh for the ones that are doing h.264 or something like that. uh Yeah I mean I, I don't think it changes the story. Much like, I think the question is, is roughly you know, would you be open to having the possibilities of being able to get the compressed packets regard, regardless of whatever the format is um passed into? The you know exposed into the javascript um as a as an alternative? Well, look.

C

The main use cases like doing green screen stuff require uncompressed raw data. I get that, like I'm, not saying I don't want those use cases, I'm talking about additional use case beyond that which would facilitate things like sfus.

A

Yep, I mean, I think, that's an interesting, interesting question, and both I mean if we can can get things done without transcoding. It's obviously cheaper right, but that that might be, but having it in an encoder form that we're not going to decode might actually be more suitable for inserting the camera straight into the in into the encoded side, which is downstream of the of of the encoder and the nice thing about row frames apart from them being wrong, is that they don't depend on each other.

A

So you can process one one frame at a time and you can drop one frame without hurting the next one. When you do h264 things get a lot more complicated.

C

All right sure, so I mean I'm totally understand depending on the codec, so that some some are temporal. Some are not, but I mean like some. You know this would really open. This opens up to some of the stuff that bernard was saying from long ago that we ought to be able to have plug-in codecs right I mean particularly, if you think, about very low bandwidth, audio codecs or something like that. um I don't know. I think that there's a lot of options here.

C

If we could pass in the on both sides, we could possibly have compressed media.

A

Yep I mean one: one of the projects that is going on in another working group is web codecs.

A

We want we're trying to keep keep things in sync so that uh if we do the breakout box break out the speed we can the raw stream. We can feed it into the web codec and then take what comes out and put it into a paid connection.

A

Okay, so I don't think we have all the moving pieces there in any time at any time like this week, but in the areas who I want to be there.

H

Yeah, I'm just going to say a note on web codex is that they've so far rejected streams. I think for performance reasons. So one concern I have with this and I'm glad it can be done in workers, but uh since the api, the control surface is on main thread. I'm worried that this might encourage sites to try to do I mean doing raw media on the main thread seems like a horrible mistake to me.

H

So have you considered having this api throw if the pipe contains any main thread javascript and would might there be other ways avenues like uh the reason we have to do this post messaging is because the media stream track is not transferable. Have you thought about that as an option.

A

I've thought about making media stream tracks transferable and- and it's kind of it's kind of tempting to do that, whether or not we have this additional apis, because if we can just move this media stream track, then that makes things a little simpler to understand.

A

I mean the bits will be floating approximately the same places in both cases, but the the control surface will be easy to understand, and so I think it's a complementary suggestion.

H

But, but to the question of, have you considered having an api throw if it contains any main thread javascript because other you could otherwise, I fear you could end up with, because a readable stream is so flexible.

H

You can do teeing, you can do you can end up transferring these to workers, so you can have elements of the pipe that are both on main thread and worker threads multiple. So you have multi-threading issues made potentially right.

A

Now what we found with at the moment, the way you do and the way you do things like background blur and in.

A

In javascript is to paint paint a video stream onto onto a video tag that paints onto a canvas that you capture and then process and then generate the generators check from that which sounds horrible, but it's actually working fairly well, if you have a quite powerful machine and but someone had one of these features and tried to use it on the machines that are bought by the tom for using classrooms which go for the cheapest alternative that that its requirements and, of course well, we have to detect that case and turn off the feature.

A

So in some sense some cases it makes sense to say. Oh, this won't work, but there are so many other cases where it actually makes sense, low bandwidth, video, qvj pictures and audio.

A

So now I I wouldn't uh try to enforce use of workers in order to use this api.

H

Well, um I mean there are other concerns, though, like uh like, if you have a camera track, that has privacy indicators, that's necessarily tied to the current document, so you wouldn't be able to do this. I'm hoping we can't this won't be ever possible to send to a service worker, for instance, because then your camera access may outlast the page, and that would be bad. So this is limited.

I

H

General workers right.

I

So when you transfer a stream, you keep the source in the document, meaning that if the document the origin document goes away, then the stream will go away as well in the service worker, for instance, if you transfer a stream from a document to a worker and then to another worker, if the first worker goes away, the stream in the second worker will go away as well will be stopped.

I

That's the current design. People have tried to improve the design in the stream's. Api is pack, but it's it's actually a difficult issue. So that's where we are currently in terms of spec and implementations.

H

Okay, I just feel a lot of my concerns with main thread. Accidental main thread, processing slowing down the browser would be ameliorated if we maybe took.

C

H

Steps here and maybe uh started would say: let's only allow it in worker to begin with, see how that works out and then, if, if we see main thread needs and that main thread is performant once we have further along with implementations, because we don't even know if it's going to be performant in workers. Correct.

I

I completely share the the concerns and uh I think that there will be some presentations later on. That tries to do uh something along with outlines that you that you're suggesting for insertable streams- and I guess that the same model could be applied for these new streams as well.

B

I

B

We're gonna need to uh shut down this conversation and move on to the next presentation to keep on streaming before.

C

We do that. I never really got an answer. I didn't understand. The answer to my question: are people uh interested in the I mean? Is it reasonable to look at extending this to be able to work on the compressed media, packets or not? I didn't understand the answer.

A

um My answer is that I'm interested, but I think that it might not be strictly a mod on this interface. It might be a mod on the on the insertable streams right on the other side.

H

Okay, fair enough, can I also just mention that next week, there's going to be a breakout session on memory copies, because I think a lot of the what wd streams parts that we're um going to rely on here by streams and bring your own buffers are not well defined or implemented. Yet so.

K

Okay um yeah: can you copy the link to that memory? Copy thing on the chat? Please? Yes, I'll! Do that. Thank you. Yeah.

B

Thank you. Okay, so you went, you don't have the floor.

I

Thank you next slide. Okay, um so going back to instable stream, uh so arlo presented it in june and uh one use case among servers was entwine encryption and we know that it's a desired feature.

I

We know that native applications like google, do facetime zoom either have it or we'll have it soon and there's definitely an interest in enabling that support for web applications, as we discussed, I think it was in july inter meeting what about implementing that in asgs instable streams, and the consensus was that it's okay for prototyping, but there's a need for additional support, as per the instable streams explainer this document, so the instability in spec will define a mechanism from browsers to provide built-in transformers which do not require javascript access to the key or media, and these are good properties next slide.

I

So since we were excited by that, we decided to do an experiment. uh The experiment is mostly pen and paper right. Now, though, uh depending on our discussions, we might want to start building a prototype.

I

So with pen and paper we decided that uh yeah. It would be too long to redo the whole world, so we decided to build upon existing technologies. So there's s frame, which is the underlying format. Basically, as frame is an ietf uh it's, it will be in an etf spec. uh I think the itf is working on the via frame spec.

I

We also want to use cryptokey to convey key material uh and cryptokey is nice because uh you can either generate cryptokey from javascript or you can generate from native code. You can have under the hood uh properties like. Oh, this key is not exposed to gs, or this key is exposed to gs and in the future we could add a lot of properties to crypto keys without changing this proposal and of course we want to integrate with peer connection. So that's why instable streams is coming to play here.

I

So we we, um we finalized our an initial draft uh like a mock-up of an api which is available, there's a link in the spec in in the presentation. So you can look at that and specifically it's.

I

uh It has quite some some similarities with medus s frame implementation, which is uh some kind of s frame in javascript uh implementation, and it's done in a worker, so the worker has a kind of api that is exposing to the main thread to the pages, and it's uh quite close. So it's it's quite quite nice in a sense uh next slide.

I

So there is an example: you take a stream, you create a peer connection, you add a track and then you have a sender. So since you want to encrypt, you need a key. So there's a key management, the key management. You don't really know what it is. It may be implemented in javascript, because currently it's the only way, maybe in the future it could be mls implemented in user agents whatever at the end you get a crypto key and then in red.

I

You can see that you create a new sram sender stream, which is basically what would implement the s frame, transformation for senders and you set it as a transform to the sender.

I

So both the transform and the sram center stream are in red because it's uh not in the connection api and it's not in the install insertable stream spec yet, but we think that it's a it's a good model. If we were to remove the transform attribute there, what we would do would be a typical uh take very double stream pipe through this s frame center stream and then pipe two to the right of all stream. So it could be done as well and with the current instable streams api.

I

Of course, this model there. The setup is done in main file, but the actual processing can be done wherever is more appropriate, so it could be done in a dedicated crypto thread. It could be done in the media pipeline, it's really up to the user agent and that's nice, because it's allowing flexibility and possibilities to optimize things, while keeping the web developers life very easy. It's just a few lines of code, nothing to care except to do the setup properly.

I

So next slide so conclusion from this experiment: uh we think that it's feasible to add a native end-to-end uh transform and we also think it's a timely work item for a webrtc working group, because ietf is working on its part. There are a lot of customers that want it.

I

So one question I have to the working group is: should the working group start working on a native end-to-end transform, and one proposal I would make- is that yes, we should start working on it and we should start adding some requirements or guidelines such as no reliance on javascript access to either key or media content.

I

We should support webrtc, mandatory codecs, so audio and video codecs, and it would be nice to use s frame as the base format as well as cryptokey as well comments.

I

I have oversized as well, but I guess we could like discuss a little bit that first and then uh move on to other parts of it.

K

I'm just going to give a status update. It looks like the ietf. He is going to make a working group for his friend and that is going to be open to all different kind of key management. So there may be something interesting to be done in collaboration with our working group and richard here can tell you a little bit more about that. If you want.

E

Yeah, so we should have an s frame working group here, real soon now or a couple weeks.

E

This seems like a very sensible step toward leveraging that up into a you know the long-range vision of uh having stuff protected from js, um obviously not yet a complete vision. I think we need to get you know. Key management that was uh out of the hands of js as well, but like this seems, like seems like we can do it step wise, like get this frame locked in, get this thing to do the s frame transformation, and then we can add another thing that encapsulates the key management.

I

E

I

Seems like totally.

E

Sensible to me and happy to collaborate with uh to help facilitate the collaboration with the ietf relevant idea, for it.

I

Yeah, that's that's really. The design is really that to think that, uh like there's a key management system that will come in the future and uh but that's okay, it will take some time, uh but we think with that with good confidence, we could already have sram apis.

I

That would be used initially with javascript provided keys and would be able to migrate without uh deep changes to to apis to uh protected.

A

L

I think I am but uh um yeah, but I was also happy to let richard and you know, um since they were saying things I agree with um yeah, and this seems like a reasonable um next step along along the way um I like having.

L

um I like the fact that um that, uh as you can said, um this hot conceal is whether which thread it's in which, like makes life a lot easier because, like obviously it sucks to have feminine thread and the fact that this conceals it is really nice um and and and like you know, we want we rather people use this than use javascript, and so this encourages them to do the right thing rather than the wrong thing.

L

um uh One thing that I guess um I was thinking about and I apologize guys I was I'm trying to look this up in insertable streams before um uh um uh yeah. Sorry before, while I was waiting in the queue, um what happens if you get an integrity error in um uh in stream processing? That's not that's not specific to this. uh This. This would happen as well with the javascript stream implementation, but um how does one handle integrity? Errors is that is that?

L

Does this interval stream done to do that, or does that need to be added.

I

So this um you can provide a sign-in key and based on the signing key. uh If the sign the signature is wrong, then there's something that should be done. um I guess that, depending on the options we could provide as part of the creation of the s sender stream, we could provide some flexibility there. My initial guess that the default implementation would be if you're not providing signing keys you you do not validate them so you're, okay with and if you're, providing a signing key.

I

You probably want to drop uh the frames that that are wrong.

L

Right um that was an answer. That was an answer to a question I was about to ask so great. um um uh My question is dumber. My question is: what happens if the verification fails? Does insertable streams have an api for saying that there's a there's, a failure, or do you just like like how do you? Actually? How do you actually, the practical matter, um uh deal with the fact that a frame can't be verified.

I

um Yeah, it's it's up for uh it's not present in the api. So I guess that uh if we were to do that, there would be webrtc stats that would allow you to see that frames are uh dropped and the reason being as frame for instance, so we could provide that. uh We could also provide like, for instance, there's the issue where, when there's key rolling, oh, I receive a new frame and I do not have yet the key. For instance,.

C

I

So I was thinking yeah. Maybe we should add a call back there, but I'm not sure we actually want to add that complexity, and I think that yeah you, you don't have a key for the next frame, so you will drop it and you will get the next one and you will need an iframe or we will prefer a little bit and if we cannot buffer more then we'll say sorry, we drop on the floor and once you get the key, we'll ask for a new uh a new key keyframe, for instance,.

A

Great. Thank you.

A

M

A

This is stuff with that should actually be possible to prototype with insertable streams. Insertable streams doesn't care and when insertable streams you either pass the frame so frame or drop. It.

I

Yep yeah a different query and.

H

Okay, so as it's going to say that from an api side having sender.transform attribute, I like that better because uh it's it's more contained because with insertable streams, I often wonder well what happens if I don't listen to the incoming stream at all and just generate my own data from say web transport or something what happens then you know: can you totally.

I

H

A media stream track that way.

I

Yeah there's uh there are some slides about about that uh shortly.

C

E

So we could discuss that.

I

um So it seems that there's consensus that we should go in that direction. Right am I hearing well.

A

I think I think it sounds like a like a consent, consensus that wish that we should definitely spec this out and see if we can can land it. Yes,.

D

I

Looks good next slide, so I looked further and started to think. Oh, we have installable streams, so instable stream, when you're sending, is receiving and coding valid and coded media content, but it might produce whatever content it wants, so it could be valid and coding media content or it could be invalid media content.

I

uh Let's take vs frame example, for instance, vs frame will produce invalid media content in the sense that the media content will be encrypted and even worse it will prepend the media content with some keys, key ids and so on. So what happens there?

I

Actually, the u.s agent pipeline might expect to get valid media content for packetization and depocketization. So if you have a h364 with various nalu and uh you will not be able to pack a device correctly because all this is completely crappy compared to what you expect and on the decoding side, it's also difficult sfus.

I

They might expect as well to get valid content in the sense that they might want to parse media to identify useful information like keyframes and profile.

I

So that's that's. The problem next slide.

I

So before I stream, particularly, I believe that in the future we might have a generic future packetizer, but might work with codecs like everyone, for instance, and that might be totally fine there. We don't need to do anything whenever there's this packet sizer, but we don't have it yet. So what can we do there?

I

One possibility, if we really want to have it in x264 or vp8, would be to add an optional adaptation transform, which is basically you take whatever the instable stream is pro, is uh producing, and then you wrap it into a valid instable stream output.

I

So let's say that you have a keyframe, it's it's being encrypted, then you take the blob and you create a big nelly based on that and and you send it and then the sfu will be fine. um The user agents will be fine as well without any change, so one possibility would be to define this adaptation for x264 and vp8, which are mandatory. Video codecs, either within vs frame, transform or standalone.

I

So since we have don't have a lot of time, I will skip that, but I would really appreciate feedback on that. If that's okay.

J

You have uh dr alex in the queue.

I

Oh okay, go ahead.

K

um Actually, I think that the job has already been done, or I put it that way. There is a solution for that one one of the problem with just insertable frame and when you you change the the packet right is that you kind of already announced the which encoder you would use, and you already done the the um stp offer answer.

K

So you cannot change it and you have to deal with the fact that the payload heater will be encrypted and it's going to be different depending on each codec, and so what we did with the mod a few years ago was actually to check for vp8 h264 vp9 in case of svc, not svc and so on, and define a way to deal with that.

K

That has two parts which is on an rtp payload part and, let's say, frame marking on steroid, which is called dependency descriptor, which is indeed codec independent and that allowed to deal with all of that elegantly.

K

The dependency descriptor has been fully described in the annex of the av1rtp payload codec and sergio finished an open source implementation this week. So I can, I think I can liaise you with the people that have done that work and that will be part of the work in the itf uh x-frame working group anyway, and I know there is a draft document for exactly this problem, so yeah.

I

K

Maybe I can help you think with what what the other people have done. Maybe there's something interesting. It might not be perfect, but uh some some work has been done. There.

I

Yeah, but that would be nice. I I still think that um we need something in general, uh even without this frame, for instance, if insertable stream is modifying the stream, you need to send it and you need you don't want to change a lot as if use user agents for for for that. So, but I, I would be very fine with initially an strength, specific solution that would be great as well.

A

I

This was actually.

A

One of the design worries I had with the whole insertable streams is that if you change the payload format, you really should change the asdp too in a compatible way to mark to say what you're doing, but there's no hook for that at the moment.

E

Yeah, I think that that's also one of the things that was making me a little sad about us frame is that there was this risk. We were throwing away useful information as we're making the transformation so yeah. I think we'd be a difference to talking about ways to do that, both at the api level and at the protocol level.

I

Okay, good uh next slide, which is coming back to uh what universe. What's suggesting the transformer attribute versus exposing readable and way to go so instable streams are transformed by nature, meaning, if you look at uh so I took herald uh nice block uh images uh and just put ts in the middle, which are instable streams there, and if you look at that, you clearly see that this is transform right in super streams. They are not producing content, they are transforming contents and streams.

I

Api has support for transform and this model is actually validated and being used today with compression streams and encoding api. These are apis that are implemented in chrome and encoding. Api is also implemented in safari, so with writable stream, transform streams and readable stream. It's available in software tech review, and it's working fine there. So it seems that using the transform model for insertable streams will might have some benefits next slide.

I

So I took various examples there uh on the top left. You have the native transform, which is exactly the same as before, as I presented earlier, there's a current api example where you have uh the pipe through type 2 thing, which is in the main thread, and that's fine uh and I thought hey. Why can't we add like a new rtc transform that would take a worker, for instance, so you create it in the you create it in the main thread: it's there.

I

Maybe it has post messages under the hood that is already defined so that you provide some uh nice benefits to the web developer and it's just one object. It seems fine as well. uh What about a worklet? That's another approach uh there. You can see that there's some more api, like you, add a worklet to a connection and you, you add, add module, but the add module is already defined in for worklet in general. So you're, not you!

I

You don't have to redefine it, so it all seems that that's feasible for with the transfer model there, so that could provide us uh a nice uh choice in the future when it when we will be ready to actually uh decide based on experiments or on additional thoughts and studies. Next slide.

I

So some observations there, api with transform, is ergonomic, there's an intuitive use of transform and also it allows mixing native and gs transform uh very nicely, just as like people are doing. Like usually you text, you take the text, decoder stream, api, it's the transform and you pipe it, the fetch api response body and then you take that out to do something else. People are used to that and it's working fine. So why not piggyback on this development style here?

I

That makes a lot of sense to me: there's a potential for a good threading model, because we could provide uh some read limited api to be government fed by default and then there would be no need for transferring strength. In most cases, I hear that transfer transferring streams is possible. uh I implemented readable stream, I implemented writable stream, I implemented transfer stream uh text decoder streams and a lot of things.

I

I did the integration of the fetch api and I know that transferring streams is in the very easy cases can be done optimally, almost optimally, no memory copy and not blocking on the main thread, but there are edge cases where you start doing things and bang the optimal way is no longer possible, and that makes me think that uh we should allow transferring strings, but we should not require to use it in most cases. It should be uh by default of the main thread and in some specific cases, yeah. Let's transfer it next slide.

I

Yeah, so worker worklet- um I don't know uh most- will probably fulfill cases. Worker or worklet worklet has some advantages. It might be more efficient, like you could do. Synchronous processing, uh reuse of existing native threads uh on the media pipeline worklet might be less error prone noise to chair, for instance, uh but I feel like uh we need more studies there. Maybe we need more feedback from the web audio working group. Maybe we need uh additional feedback from a native transform stream implementation experiments.

I

So to me we should keep that in mind, keeping its scope of the discussion, uh but on my side, I'm not ready yet to provide like uh any guideline whether we should pick one or the other and that's the end of the presentation puts.

H

So, jan over here, I I like the direction and that the and I like the intent, which seems to be to get this off main thread, and I agree it's also early to decide exactly on the api shape. But um I think what we should do immediately is even if we use the existing post message solution where you create the the transform stream in the worker and then post messages to main thread or the other way around.

H

uh We should uh take steps, I think, to disallow main thread use in the existing insertable streams api early, because otherwise we might end up with web compat and have to break people later.

A

H

That make sense.

A

E

Just make sure just to.

I

A

That I disagree with you.

I

Okay, we could also simply only expose a transform- and in that case you don't no longer expose a readable and writeable stream on the main thread, at least the native ones. You would need to expose them elsewhere, which would be a worker or worklet, and in that case you no longer allow the potential food gun of a web developer, being lazy and trying to do the easy thing and it's working in in a big laptop and it's not working in a mobile device.

H

Yeah, I agree with that. I was curious. Are there any use cases today that require a main thread.

I

I don't know any you, you will need to uh require some synchronization, some mainframe synchronization data, so that and type synchronization. But when you're in the main thread uh you might be blocked so uh yeah. I don't know any use case here.

H

Just just for context, a problem with running a lot of event-based things or anything on main thread is that uh you're, I'm assuming this is going to be running off the main event loop and uh performance wise for real-time things. It's often very bad to rely on the main event loop, because so many other things may be queued.

H

So the right you want the event loop that you're on to be empty, mostly for real time, and that's hardly the case on main thread. So you can end up with a lot of uh jack.

I

Yeah, even even for audio processing, which might not be a lot of data, uh it's very sensitive to latency and that's why there's an audio workload, for instance, where people are using it to actually process captured data when they're encoding, then they're sending it to food data channel? We we know who are doing that and it's sort of working.

A

So one of the things that we one of the reasons why I end up with the shape that the currently has is.

A

That you know why unix doesn't have a move, cisco.

A

We have a link and we have an unlink, but we have no move.

A

That's because karen and richie thought that trying to combine into one operation, things that sometimes might.

A

Fail separately or be impossible for different reasons might just possibly be a bad idea, so it's possible that defining the transform is easier for users, but uh I have this nasty feeling that something is going to come and bite me for the cases that, where we absolutely don't want javascript to see the see the data like native transforms, then absolutely. This is a much much better api for that.

A

A

I'd like to share this on top of of the current api, it seems trivial and say people can use this and then let's experiment and find the use cases where it doesn't quite work.

I

Yeah, I I agree with that. I think that um the the medus s-frame transform, so the implementation uh done by sergio is roughly similar to that in a sense, um except it does not have the transform getter a transformator, but uh it could be a basic brick for doing that. Shin.

H

Yeah, I think my main concern is that sites would start using this on main thread and depend on it and they should have used workers, but they just didn't because it was fast enough on their fast machines and then later. If we try to decide. Oh, that was a bad idea and turn it off and throw a failure. We will break a lot of sights and maybe it'd be better to encourage them to move to workers early.

I

For the sake of.

H

uh Storytelling.

I

C

Should we should.

D

I

A lot of work too.

D

Okay, thank you. More discussion is needed.

B

Yeah, thank you very much uh u.n, okay, uh so we'll try to go quickly through webrtc svc. So a little bit of an update. We have four open issues, of which two are recent reviews, one from the tag and one from ping, and then there are two uh extensions: it has been implemented in chrome behind an experimental web platform features flag, so you can try it out.

B

um So here's some of the issues we'd like to chat about today, um I'll give you an update on uh the scalability mode table and the diagrams, how we're planning to maintain those and then we'll talk a little bit about the tag and the ping review comments.

B

Okay, so uh at last tpack we talked about how to maintain the scalability mode table and diagrams last year. The idea we had is hey make this someone else's problem, someone like ale media, and so we originally were thinking of referencing. The ab1 bitstream specification and just say: hey here are the mode names, and here are the diagrams.

B

um However, since then the disadvantages of this have become apparent. First of all, whoever do you see. Svc applies to multiple codecs, not just av1. uh There were new codecs coming down the pike, so just referencing an av-1 spec may not make sense. Another problem is that it was pointed out by implementers of ab1 that there are some limitations with respect to svc and av1 that result in very unnatural names and diagrams for some modes and also the diagrams in the av-1 spec weren't, very good.

B

The spatial id wasn't clear, so people felt hey, we can do better diagrams. So since then, we've made some updates in the editor's draft, where we actually have renamed some of the key and keyship modes. uh We do provide a link to the av1 names.

B

Just so people don't get confused, but I was felt there were better names than the ones that were in the a81 spec. Also. We now have our own scalability mode diagrams in section nine that uh people seem to think are a lot more clear than the ones in the av-1 spec, um and now we updated section 6.1, which talks about adding new modes where we require submission of dependency diagrams. You have to submit the diagram along with your mode name, to get into the table.

B

So I just wanted to just provide an update to the working group. Anybody have any comments on this kind of new approach to maintaining maintenance.

J

Not specifically on svc, but just to mention that there are very active discussions for the process document next year to have specific handling for registries.

B

J

Would uh be a great fit here.

B

Yeah, that's that's a that's a great point, dom yeah, because basically the spec itself is pretty simple, uh but almost all the changes would be to this to this registry of the table. Yeah. Okay. So um the tag review came up with a comment and they asked. Why is the scalability mode a dom string rather than an enum, and they pointed out that enums do offer automatic error, checking, for example, of a scalability mode value?

B

Isn't legal you'd find out right away, and then you can do little little things like you can check if the mode is within the list of scalability amounts to check validity.

B

So I would point out that section 5.1 we do specify some non-automatic checks, for example, in.

B

Set parameters: you can't set a scalability mode that isn't advertised as one of the support, so you have to put it in rtc rtp code of capability scalability modes in the list before you can set it so there's some non-automatic checks that are going on, but not automatic ones, um and so some of this question discussion of this revolved around what we just talked about, which is that uh the table has changed quite frequently and is expected to continue to do so.

B

Just uh since last tpack, for example, we had originally taken out the s modes and then they added them back. We've just renamed some of the key and key shift modes, and we suspect that as new codecs come about we'll we'll add new modes. So this table is constantly changing, um and that raises the question about whether automatic error checking actually adds any real value or just adds confusion.

B

For example, uh we already have, as I mentioned, the non-automatic checks, so you can't really set a mode that isn't supported and you'll get an error um and the problem. If you have the automatic checks it, it could result in some different behavior between browsers, for example, one you get a new mode or a rename, or something, uh for example, in the uh in the current implementation in chrome. uh That was done before these these this mode renaming.

B

So, um for example, the l3 t3 key shift mode with is not well, it's not supported in any browser, but uh in an older implementation it would treat could treat that as an illegal value, while in some other browser which had been updated, it would become legal and because of the potential for this table, to constantly update you, you could have these different behavior in different browsers, whereas the current mechanism would give you the same behavior.

B

That is you get an error when you try to set something: that's not supported, so the proposal is to leave it as a dom string, but we wanted to open this up to comment from the working group to see if there's any reason, people think that changing to an enum would make sense.

B

Let me put it this way: is there anyone that feels that it makes sense to change it to an enum.

B

A

That's the other issue.

N

B

And and, like I said, having different behavior in different browsers doesn't make sense to me. uh If both browsers would, if you can't set a mode, you don't support in either browser. Why would you know what what how does it help you to have automatic checking when the table's changing all the time? That may not make any sense.

O

Okay and- and you probably need to check any way if the mode is supported by the codec so right, you need to check the values in the first place and that's where that would be double checking.

D

B

um So now, let's get to the ping review comments. uh They posted something about get capabilities, leaking hardware capabilities without a permission, and they file this in the svc repo, not in the webrtc pc repo.

B

So basically here's the situation. The only thing that weber to see svc does doesn't change any of the methods or how they behave just adds uh to the capabilities dictionary by including a list of supported, scalability modes. So what's the additional fingerprinting surface from this? Well, uh whoever csvc is currently only supported on chromium-based browsers, but there are a couple of those. So you do get generic info on the user agent. Like you know, it's a chromium-based browser, but you don't necessarily know which one it is.

B

So you do get some information in that regard, but as far as the hardware capabilities today, I think it's true that svc encoding is only implemented in software, although I suppose that could change and the decoding capabilities are really only advertised for sfus, because at least for vp8, pp9 and 81, you can decode any any scalability mode, it's required as part of the codec, the decoder definition.

B

So at least currently we believe that svc, you aren't leaking any harder information from this list of scalability nodes.

B

The other thing is that we had an issue 49, which was filed in webrtc extensions, and basically that issue complains that the existing sync synchronous get capabilities method is not usable for exposing hardware capabilities, in other words, the person who posted that issue wanted to be able to describe whether the codec was supported in hardware software and found themselves unable to support hardware adequately, and they proposed an async method. Instead, so we simultaneously have the ping people saying it's leaking hardware and then issue 49 says no.

B

I want to actually leak harder, and I can't uh so both of those things probably can't be true at the same time. um So here's what we're proposing to do. We are going to update the security privacy section to include some of these things.

B

Some of it is already there, but we want to make a little bit more clear, um and then we have uh the generic get capabilities, issues um in issue 2460 and issue 49, and we can basically take it on there, but basically the position I would like the working group to take is that there's really nothing specific about webrtc svc with respect to leaking hardware capabilities.

I

Yeah, I agree with that. I think that, for instance, vp9 decoders may be done in hardware or in software, for instance, and it has nothing to do with svc there. It's just codec can be hardware.

B

It's the codec yeah or software.

I

I'm wondering whether the get capabilities being sync, whether we should just rely on the media capabilities for exposing hardware capabilities and never fix capabilities.

B

Yeah, I think that's that's an issue we should discuss, uh but I um kind of outside of this particular ping uh bug.

J

Yeah, I agree with your analysis. Bernard I wouldn't bother with not about user agents because that's describing a temporary situation and it's not adding fingerprinting surface anyway, but yeah, given what you described. I agree that there is really nothing exposed in svc. That's adding anything there.

B

Okay, so please uh put this down in uh assuming we have consensus. Please put this down in the notes, because I had closed this issue before and ping re-opened it. They were very, very adamant, so I want to make sure we can. We can close it with authority now. Well, the.

D

Way, I say to close it with an added note, or are you saying that we add an async method well,.

B

uh That I think the we should discuss adding an async method, but that's separate from, uh I would add a note to the issue in webrtc. Svc just say: hey, this is your generic issue and it really has nothing to svc and then, um in a minute, henrik we'll get to the generic slides, which yanivar was going to present to ping uh but never got to, um but but basically idea. Henrik is we can talk about an asic method, but it has nothing to do with with this particular ping book.

D

Sounds good. I was just gonna say that I think it's gonna come back to haunt us if we don't address it, but we can do that separately for now notice. Fine.

A

J

On the process bits, I don't know that the right approach is necessary to reclose the issue. I don't think going back and forth on. This will help, but I think what this gets us is clear consensus that the working group doesn't believe this exposed new fingerprint interface and then once we have a started that consensus, we can work with opinion figuring. The next steps right.

D

But if we want to take the concern seriously, we would have to update and add, like a note to get capabilities that this api must not expose hardware like going forward as well. I think.

H

Yeah sure the question dom uh it sounds. What I just learned is that uh from the chairs meetings, uh clarifying that horizontal review and wider review must show that we've addressed all comments, so would that include?

H

It would include.

B

H

J

Yeah, but for svc to be clear here, we're talking about an svc issue.

H

All right, yeah, thank you.

B

Okay, so um here is the generic slides that yaniva prepared for the ping folks about the generic issue of get capabilities. uh No, you didn't get to present it yeah. You want to.

N

H

Sure I mean uh I'm mostly referencing the documents that you put together bernard so, but it's basically saying that a site can learn about visitors underlying hardware without permission prompt that was the issue is filed. But then it's just basically saying. I think what you've said here is that the same information is available in the sdp but um which is inherently needed for peer-to-peer signaling and uh using use cases, for that is data channels, receiving media or sending media other than camera microphone and screen. For instance, camera element, capture stream and next slide.

H

So, um oh there, it is yes, so we already have uh fingerprint notifications in the spec about that get capabilities and create offer, and uh I think bernard you had a graphics hardware and fingerprinting document that linked to some of our thinking there that basically, these the same information can be inferred from other sources such as web gpu and webgl, and the performance api. So the proposed solution is to include a node relating to implementation issues of hardware, hardware capabilities and then I think, to ask for uh this to be dealt with outside of webrtc.

H

If someone wants to propose a permission, prompt for exposing uh graphics hardware information is that correct, bernard.

B

Yeah, I think that's what we're wanting to do.

J

B

J

Yeah, a couple of friends: first on the sdp creator, first thing here: I think when we have create offer for better for worlds but right now, if you I mean, I think the right approach right now is just to put create offer into your negotiation pipe and not actually ever look at the.

C

J

So I think saying it's actually needed for negotiation. Is we've made it needed, but whether it is still really needed? I think, is a fair question to ask.

J

I think the to me. The real point, though, is on the fact that a lot of this fingerprinting surface is indeed already exposed in other apis and that's getting a proper solution or mitigation to this requires coordination across these various apis. uh I've tried to get ping to take ownership of this, but I don't think.

J

I mean, I guess we need to figure that story out. My sense is that we cannot stop until this gets figured out because it's a really complicated issue, but I think we at least need to be very clear as to what our argument is, and you know I could be wrong on the negotiation negotiation thing, but if I'm not, then I think we need to be careful in how we present and uh how we present this and, as a result, what kind of mitigations would be possible or not.

H

Well, I don't think telling javascript don't look at the sdp is going to work, but maybe if we end up solving the s frame and uh browser-based keying down the road, we could at some point encrypt the sdp. That's a whole different, follow axe. Okay,.

B

Okay, anyway, I think that's probably it for now uh I'd like to turn it over to tim who's. Gonna talk about new webrtc, nb use cases, cool.

G

Thank you um so yeah, the theory theory here is it's a bunch. There are a bunch of people out there who are doing things with webrtc, which kind of mostly work but don't work as well as they should, and so I went and talked to a few of them and came back with these use cases that are like just on the edge of what webrtc one zero does, and I think that means that they're good candidates for improvement in in nv.

G

Basically, um you know these are things that we could fix for the the next version. um So yeah next slide, please so the the first one is p2p broadcast. um You know things like auctions live events. People want low latency, they want lower latency than hls and dash can do, um and you can do that with the web otc, but like there's a they the moment they do that they lose a bunch of features that they kind of love.

G

So it's a question of whether whether we can how many of those we can put back in and then can we also give them the thing. The other thing that we do with webrtc, which is to work behind that. So, if you think about an auction site, that's maybe um actually on a uh fiber to the premises link, so they've got the bandwidth to broadcast to their 500 bidders or 100 bidders or whatever, but actually they maybe don't have enough public ip addresses to do that so kind of bonus points.

G

If we can make this stuff work behind that so next slide, please I'm not proposing that these are like definitive requirements. The the the request, the ask is definitive. I think there are definitely things there that we need to do, but whether these are the right answers.

G

I I'm not really claiming that so kind of the idea here is to try and maximize the reuse of existing high latency streaming assets, so kind of create stats that allow people to see the things that they would have got from dash, um and also this is a big ask. I know, but clarity on the rules of autoplay, like a lot of these sites are are, are receive only essentially- and I also plea play on on on video and audio is just I get more bugs from that than I get from anything else.

G

So we just need to. We need to make that work somehow and it doesn't um and, and then yeah there's a final thing here is: maybe we could do some something trixie with allowing drm to work by kind of creating a our webrtc and then piping it up through the drm architecture, so that they can. I mean it's sort of like s frame only it would be nice if they could use their existing encryption um so yeah that that's the idea for that one. The next slide please.

G

uh Yeah, so so this is about cameras um in hospitals, factories whatever um and the use cases that quite often there's a nearby user. So there's a camera, maybe watching a baby or a patient or something and the user is maybe sitting at a nursing station or downstairs watching the baby monitor so they're, probably on the same network, um and then what happens when the the the router goes down like?

G

How do they maintain their linkage between those two devices that are probably on the same network, although not definitively, um and without needing to go back to the external internet as a making that a dependency? So it's kind of trying to raise the availability of these things next slide, please, and so like.

G

The assumption you can make is that these endpoints have already been connected and probably recently, um and what you want to be able to try and do is reconnect without recourse to to non-local servers and and that kind of a potential way of doing. That would be to make stun an offer answer in some way optional or predictable for reconnections, um and- and you know, maybe you do that with service workers. I don't know how you do it, but, like it's a it's a desirable feature, I think uh next slide. Please.

G

Next one is uh um decentralized internet. A lot of people using the dace channel for a bunch of interesting things um and matrix is one of the ones that I'm particularly interested in. So that's the entire peer-to-peer mesh internet messaging network, but what they really want is a home for their data channel um and and because the page life cycle just doesn't work for these encryption services, essentially or you know a bunch of other things that you want to try and do in that space.

G

There's a whole quite interesting presentation from matthew, hodgson about where they're going with this next slide. Please, but in essence, they want to have be able to kind of have data channels with a longer life cycle than just a page and and that would then allow you.

G

If you push them into a service worker or somewhere, then you would allow a page to issue a fetch which could then be resolved either with a cat from cash as it is at the moment or over https, as it is at the moment or by a service running in the service worker. That's doing some p2p over the data channel and it's that third option that we can't do at the moment next slide. Please and this one's about.

G

um If you see there's a bunch of people out there who are doing um communications apps on on mobile phones and smartphones, and you find that they're building them as native apps and they're, typically native apps lit wrapped around libweb otc, and you ask yourself: why aren't they aren't they progressive web apps? And the answer is that the robustness isn't there? You can lose the call way too easily.

G

um You know an incoming gsm call will just drop the uh we'll just drop the webrtc connection, and likewise, if the bro user accidentally browses off the page, then they lose the call and for the user point of view, it's kind of understandable. But the person on the other end has absolutely no idea. What just happened to that video call like there's no information whatsoever.

G

So next slide. Please.

G

The ambition here is to um is to try and at least allow the far end to have a sense of what happened like the fact that this call has gone away or may come back or whatever so some way of indicating that this is happening and to send um some sort of uh message to them in terms of audio or play out or something I mean. I don't know how this is done, but that's my suggestion and ideally you want an api to allow reconnection of a pre-existing call post to gsm interruption.

G

um Now, maybe if it's quick enough, that may just happen anyway with ice, but if you're over 40 seconds it tends not to.

G

In my experience and then the third one which is kind of a big ask, I realize, is to be able to park a track in the service worker so that you can move them between pages so that if you're co-browsing with a support user support, worker and you're co-browsing with somebody, you can keep the audio running whilst you're talking them through how to use their banking site or whatever you know, obviously huge security restrictions. One has to apply to that, but I think it's a desirable aim.

G

Next slide. Please.

G

So- and this is a again absolutely huge ask, but if you look at like the way that people are trying to deploy webrtc in their businesses, you find that actually, in a lot of cases, they have to kind of get a bunch of departments to get buy-in because, like you, have to spread webrtc out over a bunch of servers and a lot of that comes down to the way that um that the signaling is done in overweb sockets and I'm kind of put my hand up to having been responsible for at least part of that.

G

um But there are simple cases where we make it. Maybe could could minimize the signaling um such that you could bring up a channel with pretty much no signaling at all. I mean if you look at um uh whip, that's kind of that's a heading in that direction, um but but next slide please. I think we can go a little further than that.

G

Potentially- and I don't know if we want to talk about this- and maybe it's the wrong forum, but but in fact a url like that actually has everything you need to bring up a data channel like you. Could con you could compress the whole of of of the offer answer to um to something as simple as that and we've done a little a couple of little experiments with it and it and it's doable again.

G

There are some some interesting compromises you have to make like isolite, for example, and you have to assume who's going to be the server and the client. So there's like a bunch of the optionality in the normal offer answer goes out and you kind of define it by config, but it's doable anyway, so um yeah that I think that's pretty much all I wanted to say just that that you know those are the the options that I think we should be. Maybe looking at adding.

G

So if any of those kind of resonate with anybody.

K

I'm just going to make an additional sentence on week right. There's one use case on on streaming, which is streaming up to a social platform, being youtube live and so on and really that's the biggest case there.

K

People are used to rtmp, they want to put a url and be done with it and point it to a server and today with webrtc they cannot. So if they have a software solution, that's okay, but when they have a hardware encoder like a lot of people, have uh then it's not okay anymore, because they can implement the webrtc stack or take an existing library, but then for the signaling for every time they want to add a a new platform.

K

They have to have to have an implementation and when the hardware has shipped it's too late, and so whip was a solution to that as well. uh It makes the life easy for everybody that is developing hardware encoder. Do they do hardware encoder by default? Support youtube video motion? All of this! All of that, but support webrtc services is made almost impossible by lack of something simple. So the goal to weep is that use case with a simple url, and uh I agree there is a.

K

There is a lot more that can be done, so the url that team is proposing seems to have a lot of interesting stuff. It's just a draft right now. If people are interested in that use case, we're gonna speak about it at dispatch at the next idea.

A

There's an interesting trade-off in the url uh way of setting up connections, which is that, if you, if you exchange, if you send out the information in this form, you have no idea who you're talking to so you're, pretty much limited to saying that.

A

Okay, the server has to just just trust that you exist and it's okay. Talking to you and the client has to basically check the certificate, which has to be a legit search of the 1926741a5.

K

Right there is a little bit more about that security and authentication by by token, for example, in the in the world.

G

Well, so so this isn't to be clear: this isn't a whip uh you're right harold. There is a you're making some assumptions about the provenance of this. It does have the fingerprint in it, though so there's like there is a reverse check that you can.

A

G

That that's where the fingerprint turns yes, so you can't lose the fingerprint completely, um but yeah and and the assumption is that you know, maybe that maybe that is actually maybe you you make that a um checkable with https or something that you can check the the the uh host name with https and you get a guarantee that you're talking to the server you think service.

G

You think you're talking to um but yeah sure that you're making absolutely you're making a set of compromises about the use cases, as alex said that you're you're aiming at a particular. You know, you're, aiming at a particular service that has a particular set of capabilities and like that means that a lot of the optionality is at least potentially. You can get rid of it.

I

So the question would be: can it be done in javascript or what is missing natively for doing so and if, if it cannot be done natively and we need to do it- something, but that's fine if it can be done in javascript, I would guess that we could consider doing that. Natively, add native support as well. If we see that it's gaining a lot of uh usage, um because if there's a lot of usage, it means it's a good pattern and make and it's fine to actually ingrain it into the platform.

I

G

So that the thing you can't do um in in javascript, is you can't if you could set the icu fragment and you pass in javascript as opposed to having to use the ones you've given then this model would be possible, but because you can't set it in in javascript, then you can't do this for everything else. You could make it up. You could like just use templated uh stp and make it up make up what you thought you the answer would have been, but for the eu frag, you don't know that.

I

Okay sounds like uh a good issue, could be fined there. I guess uh the other thing. You were mentioning service worker a lot of uh in a lot of cases, uh and I was thinking about it like a long time ago. uh I still like it as I I think, shared workers might be a better place than service workers.

G

I think that the issue here that- and this needs really thrashing out with the when by looking at your use cases carefully, but it's to do with how you map the life cycle, like it's all about this life cycle, and I think I suspect that some of them won't work in in I don't know. Basically it needs needs pretty careful examination.

H

uh Just a high-level comment that you mentioned five use cases, but I think the last two are not actually use cases like call robustness and reduced complexity. Signaling sounds like betterments that you might even ask for within existing use cases right.

G

um I don't think that we originally thought that webrtc. May I don't know, but we you. I take your point, but I think they're sufficiently different from the way that one zero is currently experienced, that I think it's it's worth at least thinking of them as new use cases right I mean.

A

If, if web 1.0 satisfy the use cases, then something that webster once they cannot do is a new use case. Even if it's just yes do the same thing, but do it 10 times faster.

G

Yeah I mean these are things that people are attempting to do and then either failing or borderline succeeding. So I I guess that makes them a borderline use case, but, like you know, they still need doing.

A

They use cases you need to do this and we need to do them reliably or whatever. It says.

A

Yeah, so I I'd suggest so that there's a resolution on this point is: please write up the pull request for the me, because.

G

Yep will do and any of them kind of feel totally out of scope, we'll just put them all in that's molly, okay,.

H

Right just to clarify, there's no consensus at the moment correct.

A

It's a request.

H

Okay, thank you.

B

Okay, thank you jim. Thank you. uh Yanivar. You have the floor.

B

Yeah navarre,.

H

And unmute yes, so I originally had 15 minutes to do this. uh I reduced it to 10 and I added five minutes for a proposal if necessary. So this is more of a refresher. I was asked to do about the same origin policy wikipedia says it's an important concept in the web application security model- and I think most of you know this just uh having this set up, as uh I think it's going to be important and discussions coming later.

H

This prevents the same origin policy prevents malicious scripts in one page, from obtaining access to sensitive data on another web page through that page is dom which translates into the script. You can load a lot of things, but you can't necessarily see them or read them back and then what I highlighted in red, a strict separation between content provided by unrelated sites, must be maintained on the client side to prevent the loss of data, confidentiality or integrity, and there's it goes on to describe a site.

H

This is wikipedia entry, is fairly old, so describing a user visiting a banking site that doesn't log out if it weren't for same organ policy. Other malicious sites, the site the user goes to later- might be able to access user data by by sending requests to that banking site, which would be authenticated with the users session cookies for that banking site. So that's why we have the same origin policy. The next slide and wikipedia goes on to say applies only to scripts, and this is where we shouldn't believe everything we read on wikipedia.

H

What it means uh is what it's trying to say is that you might imply that to mean that yes, but you have access to cross-origin, media and other things that are not scripts, but that's actually not what it means. It means there's the barrier exists between the javascript and the content, which means that a lot of things may be presented to the user, but the javascript is not allowed authorized access to it. So that's true for images.

H

So if you try to draw an image into a canvas, you're allowed to do that cross origin, but the canvas becomes tainted, which means that if you try to get image data out of it, you will get a security error same for videos. You can play them. Fine, like ads play videos, but as soon as the javascript tries to access it by drawing it to a canvas and then getting the it could still see it. If you draw it to a canvas, so you can create a snapshot and display it, but you cannot.

H

The javascript cannot get access to it. This extends to peer connection as well, in that you can capture a video element into a stream and add it to a connection and send it and all the browsers today will send black instead of the content, and that's because, if they didn't it would be very easy to avoid the same origin policy by basically creating peer connection. One period connection two doing a local loop and send the data to yourself or.

K

H

Next slide, so here's a quiz: what are the obvious screen sharing risks, so the options I give are, you may accidentally share something unintended, the wrong tab or you may be scrolling too fast or other things on your desktop. Non-Visible parts may be included when sharing a window that aren't on screen right now or hidden behind all the windows sharing system audio may reveal other apps or sites that I'm using or d exposure to active attacks on my browsing sessions.

H

So uh the uh the last one is obviously not obvious. That's my point here so next slide uh and then do I understand as a regular user. Do I understand these risks, and can I navigate them?

H

uh It's tempting to say a yes on the first one- and maybe I understand the second and third one, but the last one there's no way even an web developer can protect themselves from so next slide and to explain that the slide I was asked to present is get display, media and same original policy and the tl, dr, is that's not a thing because get display media violates the same origin policy by design. So sharing a web surface under attacker control may expose the user to active attacks on cross origins.

H

The site can pop the malicious site can pop up iframes from other sites on the web, and it can embed media from other sites on the web and by displaying it, it has now an incentive to produce these pop-ups because it can now record them something you couldn't do before without this api, so browsers are supposed to require elevated permissions in the spec ie differentiate if the user picks a web surface to share versus a normal one. Firefox does this at the moment.

H

I don't believe other browsers do next slide, so why do we have screen sharing? Then it's an important use case for web to compete with native, so we've been, we've tried to mitigate this as best we can in the spec. It requires elevated permissions to pick a web surface over other choices.

H

The sites must not be able to direct users towards sharing these more scary surfaces. Full screen is considered a web surface since browsers may be visible on your full screen and there's a picker every time. There's no way to persist a grant permission, you can persist a block but not a grant.

H

So this has undergone tag, security, review and ping review and there's some links to the questionnaire and explainer which mentions this next slide.

H

And that's that's me for now, and I have a proposed five-minute proposal after this one.

M

Should I go now, I think so?

M

Okay, so, uh first of all, it could be that your proposal is going to invalidate everything I said, and if so, I'm sorry that I've taken all your time I'll try to be quick. uh So, as we know, the state of the art is that get display. Media allows you to display media, and there is a trade-off here that you've alluded to, which is, um on the one hand, uh by not allowing the application to constrain the selection of the user.

M

By saying you can only select the tab or you can only select the entire screen, then we don't let the application nudge you in a certain direction, and that makes a certain trade-off. On the one hand, the user is more likely to uh select the wrong thing, but on the other hand, the application cannot shove the user in a dangerous direction, um we'll get back to that in the next slide. But first I'll say what I'm proposing I'm proposing a similar, um a similar api.

M

That would be initially constrained to only selecting the current tab uh and it has some obvious benefits. uh It enables certain uh user journeys, so you can have an application that says hey. I would like to share myself to a meeting so, for example, if I'm right now in google, docs or google slides, uh then I could have a button. There saying hey share this one to the meeting.

M

I've got somewhere else, or maybe I've even got it in a pane next to me, or anything like that, and I'm sure that everybody can think of other use cases. So, for example, you can locally record a gaming session. You can record a reproduction of a bug. uh I know that somebody want we've got some partners who wanted to capture a screenshot, uh so there are all sorts of things that you could do, but of course we all knew about this before right.

M

So if we move to the next slide, uh we will try to die dive deeper uh into the security trade-off. So uh the.

C

M

Security trade off would be or privacy trade-off would be that well, the user cannot uh share their own thing. He is presented with a very simple, uh very simple, prompt uh that prompt can, by the way we can make it very scary. uh We can well, we being the user agent I can make it very scary, can make it repeating, uh can make it staggered by time.

M

So maybe the user is not allowed to click until it's been three seconds like any user agent can think of how they can make sure that the user just doesn't just click it away, uh but once this is done, the user shares exactly what he is expecting to share. He stays in the same tab and at least in the case of chrome, he's going to see the same old blue frame around the theme. So you would immediately understand that hey I'm sharing this.

M

On the other hand, um we you could claim that the application can now nudge the user exactly to the most under to the to share the thing, that's most under its control, and that is true um now.

M

Our argument would be that, except for the fact that we nudging there, uh this is actually uh more secure and the reason is it's more secure is because normally with well-behaved websites, um you eliminate the biggest risk, and that is, in my humble opinion, miss sharing, sharing the wrong thing and um just as uh just because I think I've got enough time. I can say that when you let the application know that I'm sure I'm capturing myself, then you enable all sorts of uh uh desirable behaviors like you could, for example, crop.

M

So if I now get the entire uh tab and they get the permission to do the I get the entire tab. Maybe I even say you know what I'm not even going to share all of this remotely I'm going to crop it and share just a part of it, and um that's it. That's all. I had to say.

B

So uh we'll give the floor back to you, yan avar,.

H

Okay, thank you and uh yeah, and thank you eli for uh bringing this to our attention. I think this is a good use case that is going to require uh us to to uh think of how to solve this. So I wanted to include uh and so based inspired by uh your proposal.

H

uh Last week uh we had a joint meeting with the media, entertainment and uh media and entertainment interest group, and this came up as potential interesting future features that the working group might be working on. So I'm going to present those slides here- apologies for the different layout, so just to recap what I told them is today web get display media.

H

How do you capture html, rendering is sort of the general concern?

H

The today would get display media web surfaces may be captured only if the user picks them carries significant risks yeah and that's why the api prevents sites from inference influencing choice um behind elevated permissions. So, ironically, it's it's ironic, then that sharing native apps is actually safer than web apps. So this is unfortunate since we'd like to promote web over native, and it's creates, as you mentioned, prohibited ux flow for use.

H

Cases such as presenting a google doc or recording in this meeting the next slide, so a better integration seems like it's something we should aim for. So what, if web pages stream themselves into a conference which was brought up by you by by elon and google? I think here and so the page could use existing tech peer connection to join an ongoing meeting and stream itself there if it could capture itself the good things here. I think my takeaway here is that the document only needs to capture itself and but to be secure.

H

The document must be origin, isolated as a matter of policy, and that would be across origin policies so that the document, because trying to ensure same origin uh with a dom object model otherwise, would be near impossible. I think because the document the javascript can make changes to the document at some point, unfortunately course only allows opt-in, which is not strong enough, since rendering a document from another origin is different from reading it.

H

You get a lot more information about the uh information than that in that uh other site, so we think a new policy would be needed. It might be something like uh just hand waving here, cross origin and better policy disallow uh next slide.

H

So if we can have that, which is a big if um this would be more secure, but it still would need permission because rendering itself may contain private info things like link purpling, that tells you your browser, history form. Autofill. Might you know browsers, might fill in your address credit card info that kind of stuff, and this will be captured in the rendering extensions like lastpass.

H

Obviously a bad idea there and file input elements sometimes contain private info, so active attacks is the fear here could harvest this information quickly and covertly by not putting this up initially, but then over time when you're not looking, it could paint this and maybe even use css coloring too, to do subtle, shading, whereas one pixel color difference that kind of stuff, and these are risks that are hard to explain to users in a prompt next slide, and then also this creates a new powerful paradigm, which is you can then take html, dom and create video out of it.

H

You can do remote browsing. You can also stream web apps lots of interesting ideas. I think, in addition to the use cases mentioned by a lot, it seems like a lower, lower level api to me than screen sharing both in use cases, behavior challenges and potential.

H

So suggestions I made for what a document shape might look like, would be document, capture stream or even canvas draw image document if we leave audio out because audio is not the hard part, I think- and we already have canvas capture stream if we want to go with a second approach.

H

Interestingly, that would put another scope for the webrtc working group to do something like that, which might not be a bad thing, because uh I think this would be something that dom security engineers might want to look at uh next slide and there's already another issue. 145.

H

That's asked for something that I think is related to this uh adjacent to this is to capture screenshots from dom a lot of browsers. Have this screenshot ability right now, but there's no exposure to javascript so depending on the use cases, since this would only be secure if you uh on pages that are using this new made up uh origin, isolated uh concept, uh you could in theory, then imagine you could draw image document to a canvas and the canvas becomes tainted. uh You can still display it.

H

But again, if you try to read data out of it from javascript, you would get a security error. Unless you have this policy, in which case you might be able to get your own image, data you'd still need permission, since it exposes the same origin, rendering with private user information, so we might have to since get image. Data does not return a promise we might have to but unfortunate, but fortunately create image.

H

Bitmap does so uh we might be able to do something there, and these I just put a fiddle in there as well, to show that these apis currently allow uh the allow these calls, but they produce security, error for cross-origin content, but not same origin content. I think that's my last slide.

I

Quick question: why are you focusing on document and not elements like full screen? Api is per, can be used for elements as well, so it seems like it would be valuable to go with elements as well.

H

uh Yes, I was more concerned in getting away from browsing context, because browsing context is the container within which multiple documents may live. So I think it would be concerning to. I think you we wouldn't want to capture the browsing context, because if you hit the back button, the four button you're suddenly the user might not expect the capturing to continue so uh document or lower in the document model might be, it might be doable yeah.

H

For instance, it should probably be document a body right.

G

There's a side case there about um like actually not really wanting to share the whole of your 4k screen. um So it'd be kind of nice to be able to say you know the left hand tab or something not tab. That's a bad answer, a left hand div.

H

Right, in fact, a lot of.

J

Screen cutting tools have.

H

A

We've investigated having a proposal for letting the letting javascript without it capturing itself, actually do cropping, but we have we're not as far as longer as long as that, we feel filtered to present this at the moment.

A

But yes, it's a good idea. People have thought of various uses for that.

H

We've also heard the request for cropping, especially with screen capture because of the large size, canvases bitmaps.

H

Of course our fear is that once we start allowing cropping we're into image, processing and people are going to ask for gaussian blur next, and I guess raw media access was supposed to uh solve this for us.

H

I guess you could do cropping and raw media access right.

A

Yes, I think so.

H

That's a bit odd, perhaps.

A

So the the huge difference between uh draw image or capture documents and browse the context, this.

A

What happens if you move off the page, because one would stop and the other one would display the next next page or next document.

A

So the question is: are they use cases where that's actually important they'll be able to move off the page.

I

You don't get disclaimer already allowing the case where the page is navigating and you continue capture of the browsing contact.

D

Yeah, it's it's possible in chrome to start sharing one tab and then temporarily, you you move and you show a different tab by clicking. You know share this tab. Instead, you know you show something and then maybe you go back to your slides and you say you know, share this document instead, I do think it's very useful to have it on the tab layer.

D

For that reason, but I also see a point in if you click the back button that that it could potentially go away, but isn't that already the case I mean if I close a tab where the track lives, it's gonna go away whether or not I'm capturing the current tab or a different tab. uh Probably.

H

D

Again, probably.

H

D

In my mind, this api- it it works this very closely to the existing stuff. It has the same more or less privacy concerns as the existing stuff, and so I mean to me whether it should put on document. as the api name or on the where get display media lives. Today, that seems more like bike shedding, rather than you know the core, the meat of the issue, and so I'm are you suggesting that this should be in a different working group because of the name.

H

Well, I'm not sure that the uh necessarily the task needs to be solved is within this working group, depending on how we define uh the task that needs to be solved, um I mean, if we, if we can all, we can already capture from a canvas. So if, if canvas had this ability, uh there would already be a way to do it, I guess, but my my main takeaway, I'm not necessarily married to an api surface here, it's more about.

H

uh I like the proposal from uh that a lot is bringing up. I like, I like the idea and the functionality that is brought up here, but that I I believe pretty strongly that we need to secure it and make it uh safe so that we disallow cross-origin content from being captured.

D

What what about the the primary use case of get display media today, which is I'm sharing, slides and then you know, while I'm sharing slides, I want to show you something so I move over to a tab. I show up the video and then I move back to the slides. I continue my presentation and those two web surfaces are the only surfaces I ever intend to share and I use get displayed media to do this and then I'm a stupid user and I share my entire screen and then I share something accidentally.

H

Right so I I would not necessarily put one privacy concern against the other. I think we should solve all of them and I I'm pointing out the difficulty in trying to move get display media in the direction of being more lenient towards sharing web services. When I believe at the moment, browsers are several browsers are violating the spec by not providing the elevated permissions.

H

So I'd like to not weaken that model, and I'm worried that going that avenue would mean we would have to re-get a security review from tag and ping, and that might be an uphill battle. So what I was.

H

What I think is encouraging, though, is that recognizing get display media right now, it's actually safer to capture native apps and that's pretty bad so having a more webby solution that better integrates web surfaces, I think, is the right approach and if we can do that securely, at least when it comes to cross origins, I would still need permission because rendering could expose same origin, user, local user information from the same origin.

I

So if we look at this meet google.com page that we are all looking at, does it contain cross-origin content, for instance, that that's something like? Is there a case where you could you could get away without any prompt and still get like 80 of the use cases that people are thinking of and leave the prom cases to get display media with? Maybe some small additions to or better prompts as part of get display, media.

A

I would say no we're now at minus two minutes.

H

Just briefly, uh I I say I think, no because rendering uh even without the cross-origin issue, rendering uh still exposes a google meet, is not gonna. Throw up a lot of links, but if a malicious site might throw up a lot of links and purpling of links would tell you what the user's browsing history is.

I

H

For malicious attacks.

I

It's very similar to passwords fields where, when you're capturing, you would not want to see the passwords letters even for a few milliseconds you, you would want the capture image to not show any letters so maybe for links. We could also do something very similar there, so the page is doing something and you render it in a special way that is safe, so links will not be decorated and maybe that's fine.

H

Right and extensions as well like lastpass, uh there's a big space here and uh worst case we're ending up with a different rendering target for html. So uh it's a tricky area.

M

A

M

If I could just.

A

M

A

Sensors- I don't sense the consensus on this on this point at this moment,.

J

So can I say that I hear I do hear consensus that the problem is interesting to solve, but that we don't really know the right way yet.

A

J

I think that's fair.

A

Push off the expensive, expensive pixels and the continued discussion of this until our next entry.

M

uh When is that,.

B

Oh, we still have to schedule it, but I assume it'll be next month.

A

Sometime in november,.

B

So, uh given that we only have one minute, I don't know if there's any wrap up or next steps to talk about harold.

A

So uh I the only only question I I forgot to ask at the end of ins of the breakout box presentation was: is this ready for working with adoption, or should we discuss some more.

A

This has a fair amount of interest.

I

D

For breakout fact.

I

Yeah, I think, there's interest. um I think we should like in terms of api shape. I think we should first like consolidate a little bit in suitable strings then apply something very similar to uh the breakout box thing so better other than that. Why not.

A

H

What you and said make sense to me too.

A

So that means we we adopt the principle of yes we're going to do something here, but we're we're not sure of the api shape yet, but that's a lot of commands. Let's see what goes okay with the chance we'll send out and call for consensus on that. Yep sounds good. We have a decision on something at least.

B

Okay, uh well, I think this concludes our second session at tpac. um We will be, I guess, posting there'll be a number of sessions which may be of interest next week as tpack month continues. But uh thank you.

B

A

F

F