Node.js Node.js Interactive North America 2016, 15 Dec 2016

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Multimodal Interactions & JS: The What, The Why and The How by Diego Paez, Despegar

Description

Multimodal Interactions & JS: The What, The Why and The How - Diego Paez, Despegar

The term 'multimodal interactions' (MMI) on the HCI field refers to the situation where a system offers many ways of interaction.

This talk combines an academic subject with the daily JS we already know. It is an effort to bring together two, at first, different worlds looking for a win-win situation. On the one hand, academia could enjoy getting a novel approach to a particular problem thanks to our awesome JS. On the other, the industry can get new ways of interaction which can be applied on a variety of context and/or products. In short, the talk contains: - A quick introduction to MMI (Multimodal Interactions) - A particular novel approach to add support for MMI on Web Apps. - Short list of related open problems or possible research lines to pursuit.

About Diego Paez
Diego is a JS developer with a Computer Science degree and a passion for HCI. He was born in the *southernmost* place in America, **Tierra del Fuego**. He moved to La Plata where he got his degree at UNLP. Diego has co-founded LaPlataJS a local JS community and [GEUT](http://geutstudio.com/) a *mysterious* side collective project. He is currently working for Despegar, the biggest web travel agency in Latin America.

A

Hello, everybody I'm so glad to be here. Sharing some of what I learned was I was researching about this subject. You know this is the cell shape of my thesis.

A

Well, so hope you can enjoy this well after the talk you can reach me and ask me if you have any doubt about this or anything related, it's okay, so, okay, who's, the speaker, I'm Diego, like Julian, told us, and my friends call me daca. So you can call me that way. It's cool I'm, a sharp escritt developer, with a patient for a HDI or human-computer interaction. Well, I'm, currently working for this PR, which is the greatest revelations in Latin America, and it's a cool place to work.

A

You know you have a lot of technological freedom there and you can propose your own stack and a lot of sass and well. Of course they love nodejs.

A

Well, let's move on this is the agenda for today, I like to introduce first some common terms and then move to the foundations. He will see the standard architecture for multi moral apps, and then we will compare with a different approach. You know more suited for web application and node applications. So then I will like to mention some things that maybe people here come can help me to improve this project.

A

Okay. So what is a multi-modal interaction? Now? How can we define this, since this is a compound concept, we're going to take it by parts if you will through the layers in order in order to understand what am Ultimo interaction? Is we shall be talking about this?

A

The human-computer interaction, the whole thing and the concept of feedback you know so, on the one side, the human-computer interaction is the area, research that focuses on the interfaces between people and the computer. You know, on the other side, this is idea of feedback and when we interact with the world we affected. Somehow you know- and we can perceive this.

A

This is the feedback you know, at least in in this concert, so.

A

At the beginning, it's only you on the real world outside it when I say the world I, meaning I'm, referring to anything, it can be nature and can be a system, as we known so well, as I was saying, it's only you in the world and you can start interacting. You know, and this interactions usually have an output that can be consequence of a previous input and so on and so forth in a change fashion. You know so decision making is based on feedback and the way we are knowledge.

A

This new input is perceived through our senses, so we are taking these things by layers. Unit of all senses. Senses are a kind of a highway for our interactions where each lane can be seen as a single sense using single sense. We can perceive the world.

A

As you may know, there are different type of lanes. You know it can be a paper world or a great one. So the type of the lane is the mode.

A

The feedback we get from this mode defines an interaction. So it's like an exchange and when we focus on two or more of these mouths we are dealing with us. I am Ultimo interaction, no it's taking it as a whole, for example, touching something to indicate maybe a gesture and then receiving a confirmation sound. This is a multi, more interaction, and this means some trivial at the beginning, but you can get complex easily. You know so well why.

A

Why are multimodal interactions? Important I think this is important to me when I receive any new kind of concept to have like a foundation to have somehow why I'm learning this? Why I am receiving this information? So this has some aspect to consider.

A

We can say multi, monitor actions are an important tool to make the human-computer interaction more accessible. By allowing more natural interactions following this line, accessibility can be improved. You know the same way. We are adding more interactions modes into a system. We are like adding new hardware devices and it's like you are adding more doors to your application to your system. It's it's like. There are more ways to enter to your system for people you know and then, of course, commercial speaking. The use of multi more interactions can be. Maybe your next killer feature.

A

You know, users can not only get a better user experience, but over more. The ultimate goal is to get the user. Maybe a customer. Why not addicted to our system because of this new way to interact with the system you know, and last but not least, is the IOT. You know: I Otis have grant a green field for multi more interactions, because it's all about it. You know we can benefit from the use of sensors actuators and we can create easily prototypes with a low cost.

A

You know projects like twenty five makes things even easier because they they make things super easy to connect with with the node world. You don't have to call Arduino, or something like that is. You are always a with triangle: the same code base you know so well now that we have some concept in mind. You know some new thing in mind: let's see some common multi model interaction, architectures.

A

Well, like I mentioned before this was like the deceptions of my thesis, so it's something that you can see in a lot of papers, different things well in the HCI research field. The whole concept of multiple interactions is not a new thing. You know, in fact the term was coined by Richard ball in the 80s in his famous war, known as put that there you know you can find it on YouTube. It's it's really impressive for the time you know, but here we are seeing something different. It's like a little bit more more.

A

This is from the Bruno Dumas at all another important research researcher on the field, and this is the suggested architecture for building multi model applications. This graphic is important because we can see some key aspects that any multi model library or the framework must contain- or at least known about so technically speaking in section one I think. Yes, there is section one you have the input modalities. You know this is like the way we get data into the system.

A

They are in charge of sense in the world and generating the system input. This raw input goes into a thing called recognizers, and this recognizers their shop, is to normalize this data to something that we can use later, and this is section 2 in section 2, we have some data and we can sterilize in it. We can start using it, and this is a part of our multi, more interaction system.

A

Church of these, this part is usually known as the fusion engine, and this company combines the data into something that we are going to call interpretations.

A

Then, in section 3 we have the synthesizers and the synthesizers used the interpretations generated previously to generate a physical reaction that that we can perceive you know such as a beep or a vibration, some something like that. So these are like the important part.

A

Well now that we have this, we can move to this particular approach, so we are introducing something called plus ultra & GS. You know this is a way that we can using these models. We can consume the multi mode interactions inside our node application, so these are big names. I was like I into rib meat. All she at the time when I thought of a name for this oof, that's the name anyway, plus ultra the first one works as a message gateway.

A

You know it's like a query for interactions, and then we have the GS model, which is a more versatile one which can be used a standalone you know independently or to to create device drivers, or maybe inside the web application or a node application, because I'm mention about the world web application. But this applies I, don't know if you, if we make something like a Dexter app with using electron, we can use the same concept here. So it's the same so well. The chisel is a way to the developer to help him consume these interpretations.

A

A

Okay, so we have another graphic here, and this is like a general overview of the previous architecture from Dumas, but adapted to know the applications. So in section one we are taking the concept of the input-output modalities introduced by dmoz and we have changed it to distributed architecture. You know where every client can have one or more recognizers and one or more synthesizers. Why not? Here? We can use the GIS model to introduce these new recognizers into our system.

A

Remember a recognizer is any device that can sense the world and the synthetizer, on the other hand, is any device that can produce a reaction. You know a physical thing into. We are seeing blue soul train action this this message a with this help us to separate concerns between the developer application. You know the developer code and the code of the modalities ountry.

A

We are seeing the application itself. You know that that's another layer here you can use the GS model to in order to consume and deal with interpretations. You know here you can create interpretations, it's like a top layer. We can see that way. These interpretations can can be seen as a set of interactions.

A

So let's say we are interested we're interested in tactile interaction. You know touching something and then also we are interested in a specific voice common. You know.

A

If we combine this to inter interaction story, we can create a unique interpretation, so this interpretation can be seen can be composed using these two interactions. So what an interpretation is this is the combination of different interactions.

A

Here is, you know, a little graphic, a different one, with a different arrangement, with the focus on the communication. You know the modalities are add into the system using a driver like I mentioned before.

A

Well, you can call your own drivers, and this is an important part, because it depends on the device that you are using to sense the world. There are a lot of devices, for example, I have to use the leap motion, I, don't know if you know this this device, but is it's used to to get hand gestures? You know so well. I have to call a particular device driver to get the lip data. You know into this system and well.

A

There is also a well here in order to create this device driver. We are using to the GS model. You know we're going to see an example later so well, using this small or morality device can communicate with a plus ultra message gateway. You know, so it's like. We can start sending messages into the into our system. So the message gate. We then distribute this system these messages and then, on the other side, we can see the web application of the node application itself. You know this.

A

This part can be seen as the consumers for the modalities for the data generated previously and as I mentioned before. On this side we consume interpretations and thanks to the fusion engine that I mentioned before, when an interpretation is detected, the web application will notice them, so the web developer can add any hook. You know to do something about this and well. Of course, we are using the J's model to the sales models is using some it's using browserify. So that's why works standalone and inside the account of web application?

A

So what an athlete theory? Let's see some code. Well, if you don't load this, it's not going to work just magically you're going to maybe it's better if you use docker or something like that, something like that, because plus ultra and it's radius instance. So if you just make this docker compose up, everything will happen. Otherwise, you have to install manually.

A

You already seen stencil setup that well, if you do the first part you just can get the message gate with running you know, and then the other part is to install the J's model inside of inside a particular folder. You know, in this case we are trying to create a new device driver, a multi mall device driver. So we are creating an NPM project inside of this.

A

So this is the things that we have to install to start playing with this. Well then, the other important part is to create a multi model device driver. So there we are seeing the the important part I think.

A

Here we are going to mix some of the GS code with the particular code related to the device that we want to integrate. You know, so the important part is to require the GS molality driver, which helped us to connect the device with the with the plus ultra platform, and then we are going to do something simple. That is an inner it. It's like. We are like composing, somehow our code with the code of the of the tool. Then there are some important methods that we can call.

A

There are more than that, but it's just to get an idea. You know the fission one or the new event. If the code they put inside the fission will recall when efficient efficient, a happens, the fishing's are related with the synthesizers. So when efficient happens, we we have the chance to synthesize something and create a physical response.

A

The most common example from for a physical response is to create a vibration. You know that's a really simple example: whoa that was the part to use it to create a device driver. You know a multimode device driver, but then there is another part which is the application side. You know so well here we are using this inside of our node application, so we have like a first. We have to create an instance of the GS. The the plus altar platform was think to be like path.

A

You know how a platform as a service, so that's why it needs or requires an app key, but never mind and forget that the thing is that we have to create an instance at the beginning and authenticate. Then we create the molality. You know this. This first part is like an abstraction. You know it's like metadata, something like that. The Ender is the then. When you have a modality, then you can.

A

You can instance molality driver, you know, and then you can connect your mobility with the driver, so one modality can use one or many drivers. You know different drivers, and maybe you can change it in runtime, so we are connected with a specific driver that can do something: physical, that can sense the world somehow or that can create an interaction. Somehow.

A

Finally, we are going to add this modality into the system you know, so this is just a simple as at the morality when the modality is connected to the system. The thing that happens is that every message that the modality sends is going to be distributed between all the clients.

A

So well well we're getting closer to the end of the talk. So if you are interested in some of the concept that I have introduced, you can make your contribution. You know to how this project to become a better one.

A

Here are some things that that maybe we can work on. You know I found this treat to topics like the most important ones to be tackled in order to get a better project. You know, so one of the things that we can do is to Spearmint of it using WebRTC to enable peer-to-peer interactions.

A

This way we can remove the plus alter the message key with a requirement you know or maybe distribute that that thing and also thinking about peer-to-peer interactions as a concept feels natural. You know well. Another aspect for improvement is to try to think to define interpretations, maybe as components. There is a w3c spec that mentioned this, and maybe we can based on that on this. The final thing, last but not least, is the developer experience. You know the Korean API is really bad I'm. Sorry for that, at the moment it wasn't a priority.

A

You know I was trying to create a proof of concept, but if the project moves on, we must to deal with it somehow and I think we can explore more like a reactive or a functional way. You know to improve the API, so, like I said, if you have experience with any of these concepts or technologies, and you are interested in the project, maybe your help is welcome here there are some Russia cripples links.

A

Well, that's all. Thank you very much.