OpenJS Foundation Open Visualization Collaboration Space Community Meetings, 24 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: UCF TAC Meeting 2021 06 24

Description

UCF TAC Meeting 2021 06 24

To learn more about critical open source JavaScript projects like Appium, Dojo, jQuery, Node.js, and webpack, and 27 more checkout The OpenJS Foundation: https://openjsf.org/

A

B

B

B

Hi everyone excuse me, this is Caleb from Linux Foundation, just a small request, um as you are logging in. um Can you please update your name and the participants list to include your first and last name and your organization? It just helped me with capturing attendance in the minutes or the notes um you can do that by going to the participants list clicking on your name and then clicking on rename.

A

A

B

B

As everyone's dialing in just a gentle reminder to, please add your full name and organization to your name in The participants list. Thank you.

B

A

C

A

D

Let's just wait for one one more minute for people to join, so we can start.

B

Thanks John I'm, just a heads up for everyone, normally I uh manually record these after everyone's here I did change the settings to automatically record I missed the last meeting.

B

um I was excuse me um out of the office and so I realized. That was a gap since I wasn't here to manually click record. So now it's just set up to automatically record our sessions.

D

Great thanks, thank you.

C

Yes, I was going to ask that so we don't have the recording from the last one right in the presentation.

B

Yeah I wasn't here to click record, so I was like. Oh, that was a gap. I should I set this up to automatic I'm, usually at all the meetings. um So now that was that's the only recording we don't have.

D

Okay, uh thank you. Everyone for attending June, 20 June's at UCF pack meeting. uh This is Shan you're, a tech chair uh this week. We have a very exciting uh agenda. We have invited um sorry for butcher. Your name uh Abner is that correct, yeah after he is a. uh He is working with the New York time R D team, and he has done so I'm very excited project using loadersgl for photogrammetry, um a 3D immersive web experience to record the physical world.

D

um So it was further Ado. uh Let's welcome Abner, to give a presentation on his work um thanks and feel free to start sharing your slides, I'm gonna stop my.

E

D

E

Yeah uh yeah thank you for inviting me to speak in here. This is my first time and it's nice to see everybody here. So um we are. What I want to talk about. Is the work I've been doing uh for the New York Times R D um regarding photogrammetry and how I use logosgl to produce uh the story about the Battle, uh Gospel Choir, so yeah without further ado?

E

um A little bit about me. So I joined the New York Times Rd about one year ago as a consultant that they needed somebody to research, work for streamlining, uh like pgis data, and actually it was more about Point clouds um when I joined and I do have another role: I'm, a doctoral researcher at Alto University and the alto media that in Finland and I'm working on tele robotics for conflict resolution, I'm based in Finland, originally I'm from Israel.

E

um For me, both of my goals, kind of fit to the saying in a way to the same pattern and I call this kind of a transition with from reporting to teleporting. So for the New York Times, you might say I'm teleporting remote places to people or to people's phones and to people's computers, and then, in my other role, I'm teleporting, the people using robots in remote places, but it's both is about like teleporting and a physical environment.

E

So um when I, when I, when I joined the R D um the photogrammetry and then my work uh with the point, Cloud was rather separate. So they already had a working workflow, a proven workflow with photogrammetry uh that was using the Ombre 3D, also actually finish uh company and then running with 3js. And if maybe some of you know there were some very really beautiful works that they that they did before before I joined I.

E

This picture is from diversity, Plaza that they did and uh there's a link here to the r d website, where you can see more demos and then um I was doing the research uh how to do.

E

Point clouds and I looked at several libraries, like uh poetry and others, and mainly how to integrate with 3js, which was really like a prevalent at the end times, r d and then ultimately, I ended up suggesting to use Lotus GL with ccmi and 3js so using ccmi and for tiling the point clouds and then uh Lotus GL for loading them all the way to 3js.

E

So we did some very nice experiments, for example, in this image like uh in the wildfires in California, how you could recognize that the Damage Done of the Wildfire, with the landslides in Puerto Rico, and none of this was released as a story as as of yet. But this was the research done and I hope. We will continue with this.

E

But then at the same time as something happened with the photogrammetry, which is uh that company Umbra 3D was basically purchased by Amazon and then abruptly shut down Elder services, so uh New York Times unable to use them anymore, and they were looking now for solution for photogrammetry and basically that's where I came in because already doing this work with lotus GL and with the point clouds. But knowing that actually, they can work also from meshes for glts and for photogrammetry, so started this endeavor to try to adjust um work.

E

I did for uh for photogrammetry. So now let's go get some more the technical of how? How exactly uh who did this? So, as you may know, there is a kind of overlap between 3js and Lotus GL a bit in this part of loading uh gltf. There's a draco compression, there's the loading geograph in itself. There's the texture, processing and then um I had to do this work kind of figuring out which which does what the which Lotus GL would do and which 3js would do and how to synchronize um between them.

E

um And, firstly, then what I did was having Lotus GL process, the Draco compression and 3js process, the textures now I'm, already more towards uh also doing the Draco in 3js.

E

But then there was also a lot of adjustment needed just to get started, because the logos TL was rather adjusted to this vstgl framework that GL and others and and I, had to kind of um make up this context of the French State and also do some changes in synchronization about when the tile can be displayed. So I had to do all this synchronizing work, but eventually got something basic working um and then it was more about optimization.

E

So at first, uh when I tried to show the photogrammetry models which I've we first tiled and cesium ion and the performance was not so good. There was a lot of tiles and there was a lot of uh like overhead happening and maybe some memory release. Maybe some problems with the prioritization. So I did some uh had fixes into Lotus GL uh for that, and mostly about the priority prioritizing the tiles, uh that's able to load thousands of tiles and then prioritize.

E

The network requests um based on really which tiles are the most important and then other things I had to do was to fix some missing stuff in the Traverse the mechanism. So the logusgl code is kind of borrowed from the CCM uh JS code, Seasons library, for showing through tiles, but it was kind of incomplete. There was some missing thing: it's still incomplete, but there were some things were very necessary like handling in these nested tile sets and so on. So I did that as well and then doing some performance testing.

E

uh How like how many workers to use uh how much memory to you so ended up, at least for photogrammetry I can say that using one worker has the best performance overall in Mobile and desktop another thing I started to do, which is still incomplete, is research to my. We can call it's like the Holy Grail of the 3D title conversion, which is called the skip traversal, which is used.

E

um It was research in season as well, but even there it's not complete yet, but it's a kind of uh hierarchical traversal mechanism, which is much more efficient, but it's also much harder to implement, requires lots of use of either shaders or the depth depth buffer.

E

So that's still in research, Feel Free by the way to interrupt me with questions.

E

um But anyway, once uh we finish with the optimizing of the of the loading itself and showing the tiles, and then we went into the tiling uh process. So initially we used CCM ion. But then, when we start started to put more complex and larger models, we uh ended up with all kinds of issues, both visual artifacts and both uh performance crashes and all and all sorts of problems with using cesium tilers.

E

So we ended up actually using reality capture reality captures export and so reality capture was the software to use the r d for for the grammatory and and they actually have their own LOD export and feature which goes directly to 3D tiles, and we did have better visual quality, but actually also, this one had lots of problems with the how it's structures, the tiles but and still kind of- was easier to handle the problems with that one. And we did a lot of experimentation about the different parameters that you would choose.

E

When you're tiling I wrote here, I can say that we ended up with choosing 10K like 10 000 triangles per node, which is a tile, and this is not the default number like they have a much higher number initially. But when you're handling very large, uh very large models, then you would like actually to decrease this, because you end up with tiles that are just way way too big.

E

um Also worth mentioning it's easier to self-post the tiles when you use reality captures 3D export because um you basically get the task immediately and it's got the ccmline cloud for anybody who doesn't know. 3D tiles is an open specification um which is created by by this company cesium, but it open specifications. So anybody can implement the the tiling mechanism and the loading mechanisms.

E

And so even this wasn't enough, because the tiles themselves had still had issues after they came from reality captures where something's missing, for example, they were not Draco compressed. They I think this is going to be solved in the future versions, but it wasn't at that time and then also the images were compressed initially to actually to webp, which requires all kinds of uh additions.

E

If you wanted to work on iOS, for example, so what I ended up doing was working on uh like basically based basing on the seasons, 3D title and glts pipeline, which is also open. Source I wrote some scripts to Draco, compress everything and also the textures to the basis Universal um format, the compressed the image texture, which actually worked very well sorry, and then there are still issues because the tiles themselves and the mesh the meshes were not so optimized coming out from reality. Caption, especially the UVS and the texture.

E

Baking had like lots of issues so I had it. I had to put another optimization step for the for the gltf, and this time I tried. The blender I wrote some script, which runs blender on on the on on the tiles and optimizes the meshes. That was still not enough. So we ended up actually using install on a third-party tool. Probably we're not going to continue uh using this one, but look for other options, but basically we had to write scripts to also optimized into the individually each tile and in the process.

E

But after a lot of trial and error and optimization, and basically we wound up with something we can call satisfactory. And the final result was this piece that once you I think you'll get the presentation. Then you can see the link here. uh It was a piece about, as I said, like um The Gospel Choir in Bethel and um John near our project, leader, managing the recording and there was a 3D 3D recording of the of the Gospel Choir.

E

And you have this experience when you're flying around the church and giving the gospel I think worked out very well.

E

E

Yeah, now that all the work is done, we are planning to to open source. Basically everything uh that we did and I'm really happy that uh the New York Times r d. They are putting up the resource uh for this for this kind of thing. So for, for example, recently I have been spending most of my time, uh working for the New York Times, but actually uh working with lotus GL.

E

So I'm really happy that I was able to uh to have to do these contributions, and we, you know um already most of the changes are posted to Lotus GL, and we are also going to release the 3js if you were as a standalone and then we're also going to release more components, for example, uh control components uh where you can do like this storytelling using scrolling and the camera uh using the mouse scroll to to run the cameras and we're going to release that as well as as part of and bigger open sourcing effort, and so yeah I'm very happy.

E

We we are able to to do this.

E

Yeah, that's all I had to say thanks very much.

D

Yeah, thank you so much for presenting it's always great to see our open source World being used for you know, reporting and journalism. Did you get any feedback of the uh the coral projects you did on the.

E

Well, all of the feedback I saw like from comments on the website and from uh and other people uh in the New York Times, uh then I was really good and um the experience was was very smooth, uh didn't have and and too many of course, we we worked a lot that it doesn't crash, like you know, optimizing the tiny bit of detail that doesn't crash on an iPhone 12 or an iPhone 10, and we really had to do some um very specific uh numbers that to make sure that doesn't crash on any of those, especially with the iPhones, because they are very restrictive allowed with the amount of.

F

E

Allow you to use so we have some detection uh that which device you are using and then setting up the parameters uh accordingly. So after a lot of testing in different devices, I think we managed to reach rather stable uh level of experience so worked out very well.

D

I was actually thinking. Have you put it on like a 40 4K display making like 3 000 it's a much larger on the big screen.

E

You mean like how it worked on the like higher end devices or well.

D

Yeah yeah, have you tried it and like a much bigger screen for this.

E

Yeah well I mean, of course, it worked also in a bigger screen, but I have to remember. We are here aiming kind of at the lowest uh denominator. There's many viewers, many readers of the New York Times- that all are all they are using, is their phone.

E

So we could use the parameters to logosgl to optimize um to change a bit the quality level based if you are on the desktop or on the mobile, um but still I would say we are not maybe at the maturity level of the technology that we can easily show like a very high def experience to desktop, while still maintaining the the mobile working with the same model. Of course, if we are using different models, it's easier, but right now there is still a lot of restrictions.

E

um We have to put on the model itself to support uh the lowest denominator right.

C

Hey Abner, this is this is great. Congratulations. It's a pretty awesome project. I was just actually playing with it, as you were going on the on the website. uh One question so uh so on the improvements that you get you've done on the loaders, do you see that actually it will I mean it's to you or two others like?

C

Will you have any impact on on Deck GL and the way that it loads, also like MB tiles or other car like uh I, know that they all use like the loaders under the hood is any other things that you've done. That will improve the performance of loading other type of things, not just the 3D.

E

I think yes, I also saw when submitting the PRS, for example, that there's even in the i3s traversal the different type of tiles in the loader gel it's same. It's the same code and I saw uh uh one that mentioned that it actually solved one issue with viewing, for example, in buildings in New York using i3s, so that uh improves that as well also with Point Cloud. So the well I think you can comment on that. Also, uh but uh I saw some improvement as well.

E

um I think mostly it's oriented towards this type of photogrammetry experience uh like we had, but it also has potential to improve other.

C

F

Could make a comment so um so we do have a um you know, a certain set of code to do uh traversal of traditional quad trees for 2D tiling, and then we have this separate traversal mechanism, which obviously is more sophisticated for uh for the oaked trees that are used in 3D tiles, and uh we have uh I'm sure you and I have been discussing.

F

um You know if there are ways to kind of unify this into one traversal system so that we can benefit from this. But.

A

F

Of an ongoing discussion and not concluded I think we're right now in the face of kind of expanding to make sure we support, you know a breadth of different tile sets and then once we have the understanding how that works. I think you know we can clear picture emerges of how we can do this. You know in a kind of unified, good way, I.

C

Would say: well one idea that I was having, as you were, sewing it. That means like it goes towards. You know how we can promote better, uh the the whole BCR car like ecosystem. So for me this is a great example of how you're, like an ecosystem like this, you know with a project that is very specific, for something can have impact on the on a wider collapse. So so I was wondering if you will be up for uh for a blog post that we can publish uh on maybe on the visdl.

C

You know explaining about the you're like what you've done and so on, and then you know we can do a corollary of how what other things you know will be uh benefiting from from your work.

E

Would have to be, of course, coordinated with the with the New York Times theme with the r d, but uh if, if uh conditions I would, of course be be happy to um there's one other thing and now I remember I wanted to mention about what you talked about kind of specific requirements and just kind of interesting. Maybe another difference in the requirements that, uh when the New York Times that that we had was that because there's more emphasis on like the photojournalism.

A

E

We had much more emphasis on high quality of textures, rather than showing high quality of meshes, so we are showing like usually rooms or streets. So we had we could. We could run a lot of compromises um in the like how we show, like the complexity of them, of mesh, the 3D 3D models themselves, but then we couldn't compromise too much on the textures. So this is not so relevant to the Lotus GL code, but it is relevant to um how we like the workflow, that we use for processing the tile.

E

So just came to my mind now, like another thing, interesting fact,.

E

B

Hi this is Gail. Are you able to share the slide deck that you presented to USA.

D

B

D

I have I have that slide and I will put it.

E

Right and I sent it to you, yeah.

D

What's uh what's the next project to I I know that photo photogrammetry is a series of products that here's time plan to release and what's the next thing for this to come out.

E

uh I I think the next thing now there's the effort they're actually around the open sourcing, and uh there will be some like demos and simple guides on how to add on like open, searching the process of photogrammetry at the New York Times uh I. Guess if John is here like our uh project leader, he can uh expand on this mod, but I. Think more uh more uh is about this right now and then um I'm I'm not sure what will be the next uh for the parametry project.

E

But I do know that there's going to be more down the line.

E

G

Afternoon, could you comment on uh like for making these pictures for this particular gospel story? What kind of device do you need to.

A

Collect this game.

E

uh You mean like what camera was used.

G

E

I I wasn't involved in that process, um but it's John here, but.

C

H

Everybody yeah: this is John Canary yeah. Yes,.

H

A

H

Exercising and listening to this during my lunch break, so apologies, um but but no we use um Cannon I'm. Sorry.

A

H

87S, uh the version four and basically, we set all the parameters to be somewhere.

F

H

Reduce the amount of artifacts we get for you know across ISO and lens distortion. uh We also use um to improve the mesh quality, so we do I think we did maybe 30 lidar scans of the space as well as flew a little bit of drones shots.

H

um We have a brief article that I can share later that kind of talks a little bit about our workflow. It's tend to focus more on the audio front, but tell us a little bit about it in the um in the uh on the photogrammetry capture. The other thing I will just sort of reiterate. Is that um what Abner mentioned? Is that we're going to be releasing a set of instruction sets really kind.

D

H

Over our process and what we're trying to do is um with with iPhone 12 Pro and with some of these newer mobile phones just try to reduce the difference in quality that you can get with a Sony A7 together with a mobile phone, so that we can get similar level of detail. It's not the same, but um we we're excited about the capability of the mobile phone. In that summary, we'll dive into in Greater detail as well as I'm, providing an explainer about how to use floater's, GL and and how we process it with RC.

H

A

That'll be coming out.

H

Three or four weeks.

E

I posted a link now I think John. That was you what you were referring to about that Insider already protest more about the audio, but it's a bit more about the process.

G

Thank you thanks.

D

uh All right do we uh do you have more questions from the audience.

A

C

This is a comment: it's amazing how much fun you guys have at the New York Times. This is amazing. Well done guys very interesting.

A

Yeah congrats I I've, just look I had to sign up on my other screen here. It's it's really I've, never seen anything like this before it's so cool.

D

It's always amazing how many, how much you know technical research and Innovation that New York Times can put into to make this happen, I'm sure, even for the photo capture part of things, you know, you think your own send different cameras, stitch them together, generating cities paid that that by itself is already a thesis, a master thesis that I actually uh used to used to do when I was at architecture school. So I know how much uh to how many, how much things has to happen just to make this made this website.

D

um I think that then that's it, we were concluding uh concluding today's meeting. Thank you, Abner and Zhang to come, presenting you are presenting to the developer, who wrote the code that you know for you to build this so I'm sure all of them are very excited to see their work being used. uh You know in such an Innovative way and by the New York Times. So thank you so much and you know, building up this direct channel to the developer will help will also, you know, help you.

D

They will also be very happy to come and help you whenever you have problem using those libraries, so yeah.

E

I actually wanted to say that that the help from uh Eve and the team like in Lotus jail has been really really great, uh like answering so quickly to do all the the pull requests and then helping to solve the issues so um yeah. We really think it will. It's been real pleasure to work with the team.

D

All right, um I think that will be it for this month's meeting and um uh I. Have a I got a question from John Johannes to be able to share the meeting, invite to a broader audience right now, I think we only have that um on the UCF list, email list and if there's a way we can better share the calendar, invite so you know whenever you guys want to invite more people. You understand the link out. uh I, don't know Gail. Is there a way to better get this to to other.

B

um We have it published to GitHub and to the website and but I think really I think the those are kind of if people come to them, they find them. But if you have people in your network, you can add it to, uh anyone can add themselves to it and if they have trouble figuring it out I'm happy to add people individually as well, but it's a public calendar public invite. So anyone who sees it can just add it to their calendar if they want to.

B

But if you have people who need some help with that I'm happy to help. um Excuse me sorry, analogies today, um but really I mean we have it published, I think it's just a matter of helping get it out there a little bit more um Your Word of Mouth, probably yeah kind of sharing it around um I'll.

D

Make sure that everybody who are on the calendar has has access to add more people, I think right now, I have been humanely, adding them so.

B

Yeah, um it should be completely open to the public, so it's a public calendar um I'll copy the link right now, just pop it in chat, so anyone should be able to to view it there. um People can add it to their personal calendar. We don't have it so that everyone can edit it, um but it is publicly viewable and you can add it to your own calendar. If you want- and if you want someone added I can add it, um you can Sean, we can add other people to help admin it as well yeah.

B

It's lots of flexibility in How, We, Do, It,.

D

Cool yeah I mean I, think whatever just uh every time when I send an email or just share the link in the calendar link as well so yeah, you know, for that link will make sure, for my email will have be able to have the recipient accesses yeah. Absolutely all right! Thank you guys for today and thanks again, Emir and John. We'll uh see you guys next month.

A

D

A

You thanks for the presentation.