Internet Engineering Task Force 108, 28 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IETF108-MOPS-20200728-1300

Description

MOPS meeting session at IETF108
2020/07/28 1300

https://datatracker.ietf.org/meeting/108/proceedings/

A

Much I will attempt to debug that at some point- probably not now, but possibly who can say.

B

um Whoops wrong one.

C

You know it's not.

B

Too early for the right.

C

Yeah, I think so.

C

This is the one I wanted there. We go.

C

D

Already had one crash of the the whole window frame, um I was in the note-taking thing, so I'm not going to run it in a separate outside window, but just fyi. If you run into the same thing, it just crashed, and I had to log back in, took a second.

C

Okay, all right! Well, uh it's nine o'clock! So uh since we're time constrained, I guess we'll get started. Leslie yeah.

A

It's nine o'clock somewhere all right! Thank you. uh This is the media, ops, working group. I assume everybody knows that since you elected to be here and it's not an anonymous room, please note well standard ietf policies apply I'll. Give you a moment to digest that.

C

No well I'll leave it there for a little bit longer.

A

um And well, while that's still there for another moment, I will say thank you, everybody for uh coming along and joining us today, it's nice to see a healthy turnout for this working group session.

A

um I think we've got a lot of interesting stuff on the agenda and um we'll just have to keep. If, if we keep pressing forward, it's we're not trying to be rude, we're just trying to make sure that we get through a lot of material that we have all right next slide, please! Yes, as I said, we have a fairly healthy agenda right here.

A

um My suspicion is that by the end of the session, we may be identifying material for an interim before the ietf 109 meeting, um and I'm going to say again thank you to glen dean for being our note taker, but please anybody else who wants to keep an eye on the kodi md window and participate in helping edit. The notes would also be appreciated and thank you to magnus westerlund for being our jabra scribe today.

A

All right, I think, that's it kyle anything else.

C

uh I don't have anything so I guess.

A

C

Just go to the first uh first agenda.

A

C

Yeah, uh where is it.

A

C

uh Jake, uh do you want me to run the slides for you.

E

Yeah, that would be great thanks. Okay,.

E

uh Okay, I'm jake holland, um I'm giving an update on our streaming operation. Operational considerations, draft um uh spencer, dawkins and ali have joined uh next slide. Please, uh so I think we discussed it at the interim. There is now a a session about um a section about kobit that spencer put in. uh We changed up the uh um the notes for contributors uh section and put it near the front where it's easier to find and read. um We added the template we discussed for how to submit the issues.

E

I have a slide on that at the end, so you can see it and the overall drop moved into its new home in the uh itf wg mops and thanks to kyle for help with that, uh and actually the gh pages problem got fixed. I think yesterday so.

C

Well, it's at least temporarily fixed.

E

uh Anyway, a few minor technical issues, possibly so just make sure to check the draft um for now uh yeah next slide. Please, uh the open issues list um is almost the same as last time, but I put in the links to the actual issue numbers um and uh we haven't closed a whole lot.

E

I think some of the authors have been a little busy um and the uh uh the new issues in the blue at the bottom here are uh just the new um uh references, some updates to the references for the new uh coded. It's actually named um extremely unpredictable traffic patterns uh to go with the unpredictable traffic patterns uh and the uh uh issue about the broken gh pages, uh where the um github uh website for this is is hosted uh next slide.

E

uh So there's also a discussion we had about latency. um Oh uh sorry, previous slide, there's one other point to state about um the ads thing: there's a pending pull request from matt stock. uh Thank you, uh some discussions on that and uh uh we'll be putting something in there. We think about the ads uh anyway. Next slide.

E

um So there was a latency discussion on the issues uh in the issues tracker. um It basically came up that uh there's some question about um different streaming technologies.

E

uh Rtp was mentioned, uh rtmp uh whether we should have a section that kind of outlines some some pros and cons and the issues regarding latency.

E

There were some scope questions about whether this is uh properly in scope for the working group. uh I think so. I added a link to the to the mail thread into the into the issues list, um but I think there are a few questions left hanging on that point. So if we get a chance, it might be worth.

E

I don't know uh taking it home on that. I think that um ollie had a uh uh some notes that there might be something useful to say about it without stepping really out of uh out of bounds on you know, providing guidance on use of rtp or anything. So um I don't know what that would look like, but I think that I would suggest that we go ahead and move things forward and then take a look at it and and review whether it looks properly in scope.

E

There were a few, but the main uh point of the of the open scoping questions is whether we should say anything at all about things like uh uh stadia. Where there's um you know streaming of video games that relies on the video stream and has tight latency constraints um and whether conferencing with the sort of real-time back and forth, should be brought up or mentioned as a as a point of.

E

Do we uh do we differentiate our content yeah? um So that's just a uh I. I think that that it's maybe a minor point, but something that could bear some further discussion and and making sure we're not stepping out of bounds on this uh so yeah next slide. I think I just have another yeah, please. uh Okay,.

F

Yeah from alexander the question about study I was do was to be differentiate on content.

F

Is there a difference between twitch streaming game screen or game content being streamed directly.

E

um So there's differences depending on the protocol that are that's in use, I think, is the way we're uh the way, we're framing it and sort of different use cases that have different requirements.

C

So I so twitch streaming is really just is really just consumption of video. It's like watching youtube, except you're, watching somebody game. It's not interactive in the same way that like stadia, would be.

E

There is some interactivity, I think, because you do have some back and forth discussion with the audience sometimes, um but.

C

It's not really the same.

E

Tight latency requirements.

C

Right, yeah, that's that's what I was getting.

E

At uh glenn, I think that was uh uh alex. I think that was alex yeah.

E

E

Okay, so outside of the latency discussion, uh I just wanted to remind everyone that um we would love to get a sort of list of issues that have been encountered uh as part of uh video streaming from people who who have encountered them uh tell us about them. This is the template that we have. uh There was one minor update to it. I think the um the trade-offs mentioned, because often it's it's trade-offs, but uh it's roughly what it looked like last time, but I don't think we've gotten anything in this template form.

E

Yet um so I will send spencer after you, if you don't send something in he'll, be super nice at you um and uh yeah. I guess uh that's, that's all I've got so do we want to have the latency discussion now or just a exhortation to please uh consider it and send something to the list.

A

I think that we need to. um I don't think we have the time to have it. Today I mean I recognize that it's an issue- that's been kicking around for a couple of months, um but what I think I would like to do um and kyle jump in if you think this is, uh if you'd like to handle it differently.

A

What I think I would like to do is say: let's frame up the discussion as an agenda item, either on an interim or um or on the next regular meeting, um and I think that your notion of trying to start writing some stuff up so that we actually have material to consider and aren't just some wrestling over our experiences with particular technologies is, is probably the best plan. Does that make sense to you, kyle.

C

Yeah, I I think definitely uh having something teed up for a discussion at the at the next interim would be a good idea. I think we should probably not wait four months to the next meeting, so if, uh if jake, if you can throw something together, you know we'll figure out some time to uh hold an interim um well before uh ietf 109.

A

And and it it's not necessarily throwing something together to compare and contrast what might or might not be in. But what I heard.

C

uh She probably can't hear us she might have like pulled her uh microphone slightly out of her out of the jack.

C

I'm just texting her side channel so.

C

Oh technology, how many engineers does it.

E

C

Okay, um yeah so uh leslie's gonna gonna figure that out, but in the in the meantime, um I can't remember what we were saying. What was the uh what was leslie saying.

E

Something not necessa. um She was agreeing that that we should write something up and uh not necessarily about having a thumb wrestling match over what's there, but um I think she was going to give some helpful advice there that I yeah right.

C

Yeah, actually, I think it makes for including.

E

C

Yeah, I think it makes sense to have some proposed content and then we can um and then we can refine it uh help refine it at the meeting rather than rather than trying to have a meeting where we, uh you know, like a composition meeting which might not be super.

E

Productive any um are you still on the uh uh don't check it in or can we check it and pull it out after it doesn't matter.

C

um I mean, I think so, if you, if you put it into oh, that's a good question.

E

C

E

There are people yeah, we'll figure.

C

E

Yep all right, uh I guess that's it, then all right.

C

Great okay, we're gonna move on now. Thank you, jake any! Oh any other questions. Actually, oh yeah.

C

Is there anybody in the queue.

C

Doesn't look like it, oh leslie's back. Did you want to add anything? Can you hear me.

E

C

Looking good, okay, all right, let's, uh let's move on since we're uh kind of time constrained. So thank you jake. uh uh How do we get out of this uh click? The camera icon to stop video yeah there you go all.

G

C

So next up on the agenda is uh chris culbachus um and he'll be talking about uh best practices for end-to-end workflow monitoring from the streaming video alliance. Oh, I have to stop my screen share there. We go.

C

And you're up for me, hello, I can, I can see your screen. Can you uh can you hear me? Okay,.

H

A

No apparently not.

C

Okay, I can hear you now leslie, oh.

A

Different microphone.

C

Yeah you're still a little quiet, but at least I can hear you.

A

Yeah, it's gonna be quiet because it's a different microphone, although I will note two things: one, exactly nothing changed, I'm sitting here with my headphones plugged into the computer and audio failed on meat echo, and I could see I can see the little nice vu bars bouncing up and down and even says, I'm you know filling the I'm I'm at risk of clipping for how loud I'm talking right now, but anyway, I'll file, a bug report afterwards,.

C

You sound good at the moment so anyway, uh so christopher uh uh you wanna continue. um Can you hear me hello?

C

I can hear you uh-oh. Hopefully he can hear us dope. Okay,.

H

I can hear you can you, can you hear me.

C

Well, actually, I don't even know if he's in the chat, but uh but yes we can hear you.

H

Okay, perfect, so, okay, um so I'll start, okay, so um hi everyone. My name is chris kobaccus. It is my pleasure and honor to be here and present at the ietf108, and thank you very much for having me here today. I will provide a quick status update and an overview of the best practices for end-to-end workflow monitoring project. uh This project is being worked on by the streaming video alliance as part of the measurement quality of experience work group. um Just a little bit of background about myself.

H

I've been a member of the sva since 2018, mainly focusing on measurement and quality of experience and its applicability to live streaming and advertising.

H

I've also spent the last 10 years in the video monitoring space, the first seven at rogers communications, a canadian operator as a technical, lead, building, end-to-end digital tv and ipv video monitoring systems, and most recently I spent three years at sim wave, a canadian perceptual video quality measurement startup in a technical product role.

H

So quick glance at the agenda, I would like to take the next 15 minutes to just uh go through the contributors to the project. The industry, motivation for the project uh goals of the project, the qe definition adopted by the project project scope, a quick overview of the proposed framework, followed by an example, application of the framework and finishing with an outage detection example using the framework approach and then finishing off with a five-minute q. A.

H

So the project was born out of the industry's need for systematic and comprehensive approach for real-time video streaming monitoring as such, this project's contributing companies, along with the rest of the sva membership comprising representative mix of video workflow stakeholders, including content owners, for example, disney service operators, for example, sky and now disney, as well, with disney plus, for instance, and vendors specializing in quality measurement for video streaming, including data zoom, opticom, simulator and thomas edwards, is the measurement here we work. Group chair and myself, along with brenton of touch streams, are the two project co-leads.

H

The document is currently undergoing an sva review process and the plan is to publish in q3 of this year.

H

So what we're seeing is that the industry continues to have a need for a comprehensive end-to-end, video monitoring framework for media operations and knock teams. This is especially important as they need to be able to proactively detect failures and degradations and efficiently carry out customer impact assessment, workflow, fault, isolation and root cause analysis in order to reduce mean time to detection. I mean time to resolution.

H

Similarly, engineering and technology groups need to be able to measure system performance and establish perceptual quality trends in order to characterize and optimize workflows.

H

However, to this day, these diagnostic and measurement activities can be extremely challenging for the media operations teams to illustrate here we're showing a prototypical end-to-end video streaming workflow view. We use this view in the best practices dock. It is a very high level view. It emits a lot of complexity. However, it does focus on the key video workflow functions and video transformations from left to right. We have source production which feeds into encoding.

H

Then we have transcoding packaging, origin content, delivery network and customer premises, equipment such as phones, tablets, browsers set the box connected devices, etc.

H

So, for instance, when when a customer is experiencing an issue such as a frozen screen, teams need to quickly qualify uh primarily where in the workflow has the failure occurred. For example, is it the customer environment, the delivery, the content, processing or even the source? And secondly, what is this scope of impact on the customer population? Is it a single device, a single household single region, multiple regions or even an entire subscriber base.

H

And in general it is not straightforward to establish the scope, nature and the root cause of video service impacts in many outage or degradation scenarios. It's not sufficient, for example, to only instrument the player or the cpe, or to only rely on cdn, probes or cdn logs for video quality information in the following example.

H

Suppose we want to know the quality that each of our subscribers is receiving, so we've instrumented the players so we're getting metrics such as video startup time, rebuffering ratio, average media bit rate, video, start failure and they're all showing okay they're, all showing green and we've also instrumented the cdn and the origin, and we're getting metrics such as availability response time bitrate for both manifests and chunks and again they're all green they're, all showing okay. Everything seems fine and metric wise from these three monitoring points.

H

However, then viewer is getting a frozen screen, so one potential scenario that makes this possible is as follows: the source can fail, the encoder loses input. However, the encoder does continue to generate a well-formed video stream output with non-changing pixels and the rest of the chain doesn't know any better and is happy to deliver the content along and play it out for the viewer and resulting in a frozen screen to the customer.

H

um So source degradation or failure is just one of many potential root causes that can result in a bad video being consumed by the end viewer. This next example shows how video quality degradation may not even occur on the network or transport layer or to get all together.

H

Instead, it can occur on the content layer itself so similar to the previous example. We're monitoring our players in our delivery at cdn and origin. These metrics show healthy delivery and play out metric measurements are within okay ranges in green, however, then viewer again is receiving poor perceptual video quality. For example. It's not not a sharp it's fuzzy blocky so in this example we're receiving a pristine source. However, the encoder is too aggressive it over compresses the video, but still produces a well-formed video stream.

H

Again again, the rest of the chain doesn't know any better and is happy to push the content along and play it out again. Then viewer ends up with poor video quality.

H

The previous example showed that network layer monitoring is not sufficient either. In other words, we also need content level monitoring. This third example shows another interesting property of video workflows. Video workflows can regenerate transport layer packets. This is another reason why video workflow issues are difficult to diagnose. In this example, again, player and delivery metrics show a healthy video stream. However, the video quality that the viewer is receiving is unwatchable such as pixelating, video and choppy audio, so in this example, we're receiving a pristine source and the encoder actually does a great job with compression.

H

However, we have packet loss in the network segment pulse 10 code output, but since the packet loss is not high enough, then transcoder still continues the transport process and produces a well-formed stream. This video stream, although well formed, is missing some video content information.

H

However, the rest of the chain does not know any better and results is bad. Video quality to the end viewer.

H

In general, many underlying factors can contribute to to contribute to poor customer viewing experience, not just the examples mentioned above different root causes at different parts in a workflow can introduce different impacts, different source issues such as poor source quality, different content, processing issues affecting encoding transcoding packaging, such as for ankle quality packet losses or jitter for transport quality. Evp idr are not aligning properly drm issues, etc.

H

Different publishing issues affecting the origin, such as ingestion, replication and hosting issues, um as well as different distribution issues affecting the origin, cdn, the internet, the last mile and different playout issues um such as the home environment and device issues, including wi-fi device, player, sdk and even usage.

H

All these factors can contribute to bad viewer quality, not to mention other aspects that can also contribute to bad user experience, including latency advertising compliance, loudness cloud issues. The list goes on and on.

H

Another complication is that the resulting end, your quality of experience in general, is a confluence for a mix of multiple unrelated issues, as these impacts occur along the workflow they propagate downstream and they mix together, making it difficult for media operations teams to isolate the different issues, delineate workflow areas at fault and identify what causes. Moreover, the impact is not necessarily the same for every viewer different viewers may be experiencing different quality impacts, depending on where they are geographically, what device they are using and what assets they are watching.

H

In this illustrative example, the resulting consumer impact is due to several root causes. Each root cause contributing its own quality impact picture macro blocking is introduced at source due to packet loss pictures losing sharpness of the encoder due to over compression ringing, noise is being introduced into the picture and content level at the transcode.

H

Top profile went into outage of the packager and just latency. Spikes at the origin are causing jitter and distribution, cd and edge. Cache failure is causing unavailability in a specific region, and a bug and latest firmware of the player is causing intermittent playback issues. For a particular type of.

H

Device so based on this motivation, the project goals are to create best practices on implementing a system to capture qoe and other measurement data at each monitoring point across the workflow identify streaming. Video metrics together at each monitoring, point establish guidelines on how to calculate or leverage existing metrics at each monitoring point and investigate potential for an end-to-end, metric or metrics across all monitoring points.

H

So the project adopts the itu-t definition of qoe, essentially, as we saw in the motivation section, any issue impacting the workflow at any point in that workflow has the potential to impact the degree of the light or annoyance of the user of an application or a service in the best practices document. We're not attempting to define a single qoe metric.

H

Instead, we're focusing on different monitoring methods and the different monitoring metrics that are available and how these can be used in concert to achieve an end-to-end monitoring framework.

H

So the scope of the project is iptv. Ott systems focuses on live workflows, the scope is end-to-end from source suggestion to device output and play out. uh The framework can be applied to both cloud-based and on-premise based workflows, including hybrid workflows, and the monitoring is geared towards operational use cases, for example, detection delineation of issues. um The monitoring instrumentation is also focused on probes, logs and player apis, so, for example, application performance. Monitoring solutions are not considered in this framework, uh for example, node, cpu load or memory utilization or temperature, etc.

H

So the best practices dock proposal proposes an end-to-end monitoring framework as follows: firstly, establish an end-to-end scope for monitoring of a service, for example from sources through to device output.

H

Secondly, decompose the end-to-end monitoring uh scope into modern points: uh for example, sources, input, encoder output, transcoder output, package output, origin output, cd and output player output, device output, then decompose each modern point into measurement layers content, for example, the video encode quality itself. The envelope, for example, the network and transport performance and availability, metrics auxiliary, for example, any other relevant modeling information such as drm subsystem, encryption key exchange status, authentication, subsystem, health and statistics and status of logins of subscribers and player out error and error messages, for instance.

H

Fourth, for each monitor point and for each measurement layer describe where is the monitoring taking place, for example? Is it at the source on premises? Is it pulse transcode in the cloud? What is being monitored? Exactly, for example, is the? Is it the video content itself, or is it the impact to transport stream packet headers, or is it the ip packet headers? For example?

H

How is the monitoring carried out, for example, whether using a physical splitter or a sniffer or a tap, or is it a network igmp join or is it http get or is it a player, sdk api or even a device, hdmi output, and why is the monitoring being done at the particular monitoring point, for example, is it for junk and junk outsource monitoring to establish a baseline, is it to quantify encoder, compression impact, etc and fifth, as a last step, aggregate individual monitoring point results with the end-to-end approach in mind.

H

First, on a per monitoring point basis, then, on a subsystem: scope, for example, source processing, publishing delivery payout and finally, across the entire end-to-end scope.

H

So the following is an example of a framework application for an ott service operator. In this case, the operator receives high bitrate compressed mezzanine source from a third-party content owner the operator transcodes and packages the content and then sends the packaged content downstream to a third-party cdn provider.

H

The third-party cdn hosts and distributes the content, and once the content is distributed, the viewers consume the content using the operators, devices and players.

H

So, with respect to monitoring, since the framework is modular in nature, we can choose which monitoring points to deploy, given the monitoring needs and constraints in this example, since the operator does not own the option. Encoder, therefore encoder output, aka transcoder input, is monitoring 0.1 here using igmp join to the output of the encoder.

H

The operator uses a probe for both the video content and the network and transport layers. Ipdp and p2ts, for example, are being monitored since the operator owns the transcoder. Hence the transcoder output is monitoring, point 2 and similar to model 0.1, also, measures, content and network delivery quality using probes. This is done for all multi-bitrate profiles, since the operator owns the packager. Hence, package output is monitoring 0.3.

H

Here the operator uses probes using http get in order to measure various iptp http and hls manifest and chunk network metrics. As with the transcoder, all adaptive bitrate profiles are monitored.

H

The content quality is not measured at the package or output, as the content is encrypted by the packager, since the operator also owns the device in the player. Hence the operator collects player data, for example, play out events and errors using the player api, making it modern point four.

H

It's also worth noting that at the player we do not monitor content nor the network directly. We are limited to what the player api can provide. We are also constrained by the compute resource of the consumer device.

H

So, in summary, the best practices document presents various options as to how one can define a high-level end-to-end monitoring approach. The different monitoring points, different monitoring layers and the different monitoring options, including purpose advantages, disadvantages methods and metrics.

H

As a last slide today, this next example shows an outage detection using an eight-point monitoring application of the framework.

H

In this case, let's suppose that we have a video marker blocking consumer impact, as was described in an example at the beginning of this presentation.

H

This is the case where the source was coming in clean and the macro blocking was introduced, post encoder due to packet loss, hence monitoring wise. Since the source is coming in fine, our content and network level metrics show a healthy source. However, at encore output we do detect impact on both content and network layers. For example, our video qoe algorithms show dipping scores and, for example, our network metrics indicate packet loss or signal loss working downstream content-wise.

H

Our qoe algorithm shows impact at transcoder output as well, however, again since the packager encrypts the content. Thus, we cannot measure content quality across the delivery, including at the player.

H

Nonetheless, even though the content is encrypted, we can still modern network delivery. In this example, transcoder output does not show any network level issues as the transcoder regenerates, the ipudp mp2ts stack. Similarly, the packager and the abr content delivery, for example, iptcp http hls stack metrics, do not show any issues.

H

The downstream player is also.

C

Just a uh just a a brief reminder: you've got about four and a half minutes left and that that includes time for questions.

H

Yeah, thank you. So the downstream player is also happy to play the content out with good startup time no rebuffering and the highest quality profile being played out all player event. Metrics, look good! No errors are coming in from players as well, however, pulse device. Once again, we do detect perceptual picture breakup with qoe algorithm on content level.

H

So that concludes the presentation, and um thank you very much, and uh this is the q a section. Thank you.

C

All right thanks chris, uh so uh anyone have any questions. uh Just uh I guess you click on the the hand, uh the microphone hand icon to get in the queue.

C

Any any questions, no.

C

A

Yeah, so maybe not a question so much as a comment. um One of the reasons why I thought this was really interesting.

A

Work to have here was because it really does show the spread of um all the things that need to go right in order for um for video to work well, and um I think that I, I think everyone understands that a at a certain intellectual level, but having it sort of spread out here and sort of compartmentalized into the different steps, I think, is really useful um and, as you said chris, this is all about um what the sva is working on in terms of how to measure stuff um and how to how to a framework for looking at how, to you know, rate um different, the quality of experience or detect issues with quality of experience and impacts downstream uh from problems upstream.

A

um So I think that there's a lot of interesting stuff here, that's more than one can get to in a 15-minute presentation. Wonderful though it was um when approximately, when do you think the sva document is actually going to be published and publicly consumable.

H

Yep, so this should happen um in um imminently uh in in q3, most likely um it's it's already been reviewed by the membership, and now it's being reviewed by the board um and afterwards uh we will make any remaining updates that are required and then the document will be published.

H

So q3 most likely.

C

All right, um glenn.

D

All right, can you hear me.

C

D

uh Oh, you know: one of these crystals strikes me and this sort of a broad question is that you know this uh presentation did do a great view of uh for application and encoding delivery, all those pieces that come together and then to make it work. uh You know one of the things that that you guys didn't think that I think may be of interest to the future here at the ipf.

D

Is the larry just below that stuff, uh which is never clear because a lot of these pieces, I know you know some of them- go over uh well managed network. That means, like you, know, the the video production team might be able to manage their network, but a whole bunch of other pieces of this go over unmanaged networks, uh the open internet- and- that's always, you know difficult to understand from an algorithmic perspective in the player, I'm wondering if there's potentially future work, where we look at.

D

How do you connect that higher level and give it awareness of what it is behaving on the open, internet uh and other networks? Of course, where I can better, you know, inform um people try to fix the problem, but also maybe algorithmically be able to respond to it more dynamically and I'm not sure I'm going to question my own question in the notes holding my back.

H

You know that's uh uh thank you glenda yeah, that's um so, for example, uh this this site here shows the the service operator example, and in this case I mean um you know this could be a managed network, but this could also be an unmanaged network and, as you can see here, the the service operator has third parties providing the sources and also has third parties providing the origin and cdn functionality.

H

So, and in this case this could be an ott streaming service operator, and you know, the third-party sources, for example, could be coming in to the operators transcoders over the open internet and also, similarly downstream. uh The service operator could be providing the packaged hls content to the third party cdn provider, again over the internet to the cloud. So um you know these types of deployment patterns here and you know, could speak to also unmanaged. um You know: end-to-end unmanaged, uh video service, video services, including you know, hybrid, hybrid, video services.

H

uh You know partly on premises partly on cloud um as well. As you know, composed of multiple stakeholders. um You know third parties providing uh different components and and content etc. So there are definitely you know also. You know the internet could be interspersed anywhere here um along these uh workflow stages and- and yes absolutely, uh you know in terms of the monitoring work, that's that that we're we're working on. um It also speaks to the to the network and transport layers, not not just the application layer, not just the content layer.

H

um All of these layers, what you know, what we're? What we're trying to say is that all of these layers need to be systematically monitored all the way from content application through to transport and network.

C

All right, I think, we're uh we're just about out of time. uh Thank you, chris. um uh Is there an? Is there anything that uh if people want to follow up with you, how should they get in contact with you.

H

Absolutely uh what I can do is uh just provide my uh email at the end, um it's christopher.coolbachus.gmail.com um and I can provide my contact information afterwards.

C

All right, thank you. Thank you.

C

And up next is alexander gillard and it looks like so. Can you hear me alexander.

I

C

G

Have your, I don't see your.

C

Screen quite yet yeah there we go, although you go yes, can you hear me perfect? Yes, can hear you just fine all right. um So hang.

A

On these slides are not posted in the meeting materials. Is that correct.

G

um I share them with you specifically, um as you asked yesterday,.

A

Okay, all right: oh okay,.

H

G

Looks like your name is not there, so I might have missed that, but I I thought I had shared them with you yesterday. I will share them right away. If you're, okay, I mean.

C

Yes, we'll we'll add them to the we'll add them to the meeting materials right after the meeting. Apologies for everyone who uh was following along at home.

A

If, if I have in my inbox, I can be uploading them during the meeting for anybody who is.

G

Not right I mean so, I wrote those slides to be a basis for discussion, so it's uh it's a good material for reading and discussing later on the mailing list a little bit less to present. So I can maybe make it a little bit faster when I present first is a disclaimer. I didn't spend a lot of time on it, not enough to to make it a real conclusion or anything.

G

So anything here is just a list of question and opinion and uh to check my understanding and to check that what we want to achieve through the the test of different protocol as it was proposed.

G

um There are a few questions uh when it comes to be able to be fair and to be um informative right, we need to be able to have a good basis for comparison and reproducible tests. So people do not trust me and do not trust anyone, and and do the thing by themselves again and are able to reproduce the results. We're gonna we're gonna reach uh and sometimes fairness is. Is it doesn't matter?

G

Even if you know a comparison is not exactly fair between hls and rtmp, for example, which is not one of the things we're going to do, but I take as an example. You still are interested in certain condition about the result and the data right.

G

So some of the the tests we're going to do are not going to be fair but might be interesting for for the community, and I like the direction from the chair and from the group about which question are of interest eventually, if we go there, so a small little problem to have a strong base for comparison is to know what we're testing right to make sure that we're not testing a moving target and that the day after we finish testing, I just updated, and there is a new feature and you need to do everything again.

G

So there is a little slight concern about the the versioning of srt, but this is not a big concern and that can be worked with a with the team there.

G

My biggest concern really to make a fair comparison is what to compare srt or rtmp and other with uh it. Did a fair comparison if I take hls, for example, right which is stack, you have hls, https and http then go down to tcp. um Would it be fair to compare fcls with https?

G

um I don't think that's the case, but they don't exactly do the same thing. So is srt a transport or a media transport and the example I'm giving here. The definition is um a transport is something that is not specific to media. It can transport media, but it can also transport other stuff and it doesn't have anything that is media specific.

G

Just like you can put media in https, but you're, not reaching hls right hls will have all these information about the the number of tracks. The type of the tracks which codec was used uh is their subtitle and so on and so forth. That comes on top of https and so comparing them directly like this is difficult. So we see a lot of non-media transport being used nowadays to transport media. uh I could cite web socket, for example, and so the question is: is it? Is it a fair comparison? Number one and number two?

G

Is it a comparison? We want to do anyway right because people use it that way, so people are interested in knowing what is the most appropriate when it comes to to to transporting media and what are the the trade-off in between those different protocols, whether they are a dedicated media protocol like rtp or are not really like quick by default right? Is it still interesting to use quick directly?

G

um So there is a lot of different angle to that question that I bet I read and discussed on the mailing list after um there's, actually three slides of them to try to be as thorough as possible. Then the question is: which features do we want to test? I think I saw a lot of documents that were describing srt advantage over rtmp on a different feature, so I think those should be included if only to show the contrast between srt and rtmp reliability.

G

Is it full reliability, no reliability, partial reliability and, depending on the layer again right, is the reliability coming from the transport? Are you using tcp or is the partial reliability coming from the media protocol, like in the case of of rtp, the resilience.

F

G

To network condition the security, however hard it is to test and firewall. Not troublesome, are four points that I think should be tested to a certain extent. I might be missing other points that are important, and here I'm welcoming feedback from the group to tell me what they think would be of interest.

G

I think it's also important here again to show um where the reliability, resilient, security or or firewall travel salt capacity comes from in the in the aussie layer decomposition, because you could then make a have a better understanding of the choice between two stacks that share the same underlying layer.

G

um Finally, I think there's something we don't speak about too much, but that's the global use case. I think the previous uh presentation did a very good job to show that the the delivery of the media in streaming from the source to the viewer as a lot of steps- and I don't think there is a single protocol that is used for every single one of them right. So my understanding from outside is that srt or tmp today, and sometimes with rtc, are used on the first mile from uh the capture to the ingest node.

G

Then there is a platform that does the chunking and the encoding, the re-encoding, the transcoding, the abr, all that things into hls and at which point hls is used for the final for the delivery side of thing right. um If that's the case, it would be very interesting to narrow the scope down to that first mile, because we're not in the p2p mode anymore, but we're on a client to server use case and that's a difference that is being increasingly made uh in the past two years.

G

I think in the scope of the web transport discussion in in the scope of h3 discussion for the delivery, because there is a lot of mechanism with respect to net travel so and and firewall travel, so that can be simplified or completely removed from the from the description, if you're in a c2x use case, my my advice, my feeling is that it's better to put it in in that scope to have a fair comparison again, if anybody think that's not the case or as an argument against or is interested in testing it in another use case.

G

Let me know I'll put it in the list there, where last week, some quick transport hackathon ongoing. I don't have the result yet, because everybody that was at the hackathon is attending ietf.

G

There were some new announcement with respect to web transport as well with uh will law the chief architect of akamai becoming the chair of the w3c group of web transport, where transport will have a w3c and ietf group, and so there is a lot of things that show that cdn, web and internet interest are starting to get together and speaking together on this topic, um it would be maybe interesting to wait a little bit to see um the result of that hackathon and the discussion that happened this week at ietf 108 before we decide exactly what what we're going to do with these tests.

G

So my gut feeling it looks like srt will not be a full media transport um to be discussed. Given that the definition I gave, even though it can transport media, it has some advantage over rtmp, but those advantage might overlap with existing functionality in rtp or quick for that matter. I think that was the the result.

G

That magnus comment last time is that the overlap with quick is is big, and we need to take a look at that as well, uh but it doesn't mean it's not an interesting discussion and it looks like there is a real usage in the in the first mile capture to ingest and if, even only in that use case, it would be interesting to spend time to get some data and compare with the other uh protocol used and apparently rtmp is, is really the the good comparison, because that's what being used for ingesting most of the live uh platform.

G

As far as I can tell today um slight question, anybody knows if one 109 will be in bangkok or online.

C

uh We don't know yet the decision.

A

Has not been made and we're at time for this agenda item um the agenda item that that I thought we were going to have was a discussion of we were going to following on discussion a month or so ago on the mailing list about comparing different transport protocols and how crappy the comparisons are um that we were actually going to do um our own effort where different people could participate instead of just building um instead of having arguments over personal perspectives and or gut feelings about things. So um I think that we need to.

A

You can send the slides to the list. We will post them to the meeting materials. The discussion ongoing is still. What are we going to do in a hackathon in order to actually do some of this work, because I think I do think it is actually worth doing the work of figuring out how to talk about um different different protocols for transport of video and media pursuant to the agenda item we just had a moment ago.

A

um So, yes, I think that um please send your slides to the list, because there's a lot of material to sort of dive into that, um and and let's focus as a whole on how do we? um How do we move forward to do some actual testing? Does that make sense.

G

um I I agree it makes sense, but before we start testing I mean we need to define the scope and what we're trying to achieve. So all those slides were about. Can we narrow the scope to a few protocols that are really comparable and for use case we actually interested in, and I left everything open there, because I, I really don't have the answer, but what you want.

A

Then it'll be a great list. Discussion.

G

A

All right thanks and now we are past time. I I have to shut this down because we are past time for the um for the meeting and uh we do have an aov who did send in some slides. So I would really like to at least get that aired slightly so um moving on to the next presenter.

A

Thank you, kyle. Do you have the slides.

C

um No, I didn't see them in the meeting materials uh you.

A

Should do they were there.

C

uh Maybe I just missed it.

A

Yes, it's the first one on the list. Media operations use cases.

C

Got it, I will have it up in one second and.

B

B

C

Okay, there we go.

A

Good and we have a presenter renun.

A

Yeah, seeing him enlist.

C

A

Send audio there, you go.

C

Oh okay, it's at the top.

A

There you go and we are and- and I will preface this by saying- we are going to lose our connection in something less than five minutes. So.

A

That's odd, I keep saying accepting sen, I'm gonna stop touching buttons kyle. Could you please say yes to renin.

C

Yup brendan can you uh he keeps appearing and disappearing.

F

This has been a common I've seen this before, and it seems to be commission problem, uh usually on this underside right.

C

On on their side, yeah.

F

They allowed the media to be standing and.

C

Oh yeah, they might maybe they didn't. Do the accessibility, the change in the accessibility system, preps, okay, we we won't have enough time for him to resolve that before.

A

Before we get the hook.

C

Before we get kicked off yep so um so uh we should uh consider this as a topic for the for the for an upcoming interim.

A

Yes, and with apologies for the technical glitches, um all right interim hoe, uh any other comments before we um before we close or are closed.

C

Anyone get into the queue, no sounds like that's a wrap, then.

A

Yeah, I will say thank you to all of our uh presenters who put time and effort into bringing their topics to us today um and thank you to everyone for showing up and for our next challenge. uh I hope we can get to a point where we can have a discussion, because this seemed very, very quiet um all right thanks. Everybody.

C

All right thanks.

C