e-NABLE e-NABLEcon 2019, 13 Oct 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: e-NABLEcon 2019 - Susannah Engdahl and Dr. Biswarup Mukherjee

Description

Susannah Engdahl and Dr. Biswarup Mukherjee share their work using sonomyographic control of upper limb prostheses.

A detailed write-up can be found here: https://drive.google.com/open?id=1ZBzx0Y_mm700YYkTy2DZ4nHfT65yctzy

More information and discussion about EnableCon 2019 here: https://hub.e-nable.org/s/e-nablecon-2019/

More information about e-NABLE here: https://enablingthefuture.org

A

All right welcome back, so we had prepared some slides. A lot of this is repetition. A lot of this is repetition. From the morning.

A

So we're just going to start off and reiterate a couple of points, so we wanted to make about ways that we feel some of my graffiti is an advantageous way to control the prosthesis and then we'll move on to some more technical discussion about our work towards miniaturizing the ultrasound systems and is.

B

There anyone here who was not here for the morning presentation, okay, good.

A

So the first point we would like to make is that existing my electric technology, for the most part, is controlled using surface lecture biography, so we're reporting muscle activity electrical activity from the muscles through this in surface, and this causes some problems in terms of control, which we can understand by considering the sensory motor system as a whole. So if you think about someone with an anatomical arm, they have had an amputation.

A

They have motor commands that are sent down from the brain through the spinal cord out through the nerves and ultimately arrive the muscles where they cause a contraction and initiate a movement. Now when those muscles contract they trigger what are called proprioceptive receptors that are within the structures in the arm, and these pro receptor receptors give us a sense of where our body is in this space, and we do have obviously other sensory receptors as well and following a mutation many of those disappear, but the motor pathways and the pervious active pathways are still intact.

A

The problem with a lot of existing my larger control technologies is that it becomes difficult for people to access these proprioceptive pathways because the way in which they're asked to contract their muscles in order to operate the prosthesis doesn't necessarily match, with the way that the prosthesis moves so they're sending commands out to their muscles that doesn't exactly match what the hand is actually doing and there's this mismatch and they can make it difficult there and how to control it to control the prosthesis.

A

It's not an intuitive control strategy, so this is one of the primary areas that we see our semi ography system being beneficial to reader. At this point, ultrasound again is measuring muscle defamation. So, as we take that cross sectional view through the arm, we can see how those muscles size and shape change over time and because perception, the sense of perception. The sense of where your body is in space is related directly to muscle deformation.

A

There's this direct congruence, then between the ultrasounds images that we're recording and the sense of perception that the subjects should be able to access.

B

And the previous slide, the color segmentation matching each muscle with in hand.

B

A

That's a great question: I should clarify. Did that image with the colors is just real, illustrative purposes? An actual ultrasound image is not quite so pretty as no.

B

A

That's a great question is already very yeah.

C

A

Great, if it looks that perfect, our jobs would be a lot easier, but it's just real estate purposes. Thank you all right.

A

The next point that I want to reiterate is that our training protocol is very quick and very intuitive, as I mentioned this morning, the.

A

There's a sound plan: I hope you guys can hear that.

D

Yeah, it's definitely the video.

A

So essentially, what we do is we ask the subject to perform a series of five different contractions. Some of this video were that we're asking them to do a key grasp, so they perform this five times interspersed with five rest phases and once again we can extract relevant if.

E

You overlaid on that same color.

E

The waveform fingers.

A

They will relate to some extent, but so.

E

An attentional Elektra so was that displacement we're seeing on the graph yeah.

A

The x-axis is time the y-axis is the correlation between an ultrasound image recorded when the Sun is at rest and the current frame so.

E

Different actions, don't necessarily discriminate right.

A

This will follow this same training procedure or more different grasps it doesn't. It doesn't affect what grass you're performing in terms of the training but towards your point, I will say that we are able to very accurately discriminate between those different grasps after training. So what we're looking at here is a confusion matrix for anyone who's not familiar with looking at these. Essentially, what we're plotting here is the percentage of the time that someone is intending to perform a specific graph Aron grasp, excuse me and that we actually send that compress command to the prosthesis.

A

So, in an ideal world we would have it line at 100 percents, going all the way across. We see point one percent, but on average across five different grasps and the five different amputees we tested, we had an average accuracy of nearly 97 percent. If.

E

The response is similar depending on how does the training doesn't distinguish? I, don't understand if I go like this, you get a certain waveform and I. Go like this I get the similar waveforms I. Do it five times?

E

How is it distinguished in the software alright.

A

Maybe let's take a step back to.

C

A

C

A

We have our training. Databases has a series of images that correspond to or.

E

Should yes, it's based on the actual image based.

A

On the actual image, okay,.

B

A

I'll talk through this anyway, so we have this database of reference images every time that we record a new image. That image is compared to everything, that's in the database and we calculate correlations between the incoming image and everything in the database. So what comes out is this colorful graph you seen here? So these are the actual Sun high traffic signals that we're interested in I know it's a little bit fuzzy to see, but these big Peaks that stand out is basically the system identifying that someone is performing in grass.

A

So there's very strong correlation between the incoming ultrasound image, something that's in the database. Yeah.

B

So I want to see if I understand this visit. There are five grasps. Yes, you have you take. You have a series of photographs of this grasp and.

C

B

And you got a series of photographs for all of the other graphs grasps at any given moment. You look at the live image and you say which grasps, but does this image correlate with that's.

C

What you're looking.

B

E

B

We're more closely resemble okay,.

E

That's wonderful right.

A

E

It's pattern analysis, but.

B

The correlation here is actually what pixel to pixel correlation.

D

A

So these correlation signals are then passed to a classifier.

A

C

A

That's the last step: once we have these correlations signals, then the last step is basically, you know if we have a peak to which of the five arrests was that gorgeous at that point, it's a very straightforward, send a command to the prosthesis in it. Initiate that grasp.

E

So there is a minor dwell time for the image processing and everything that your video actually shows. Second,.

D

Because the processing that is actually the do they the hand, is actually actually yes.

E

It's the coolest.

D

The processing wheel is probably less than it would be, putting.

F

It was, it was very uh term Iowa Trek in my budget is wagging. It's like. There was no.

B

E

So it's really processing time, but.

D

Keep in mind that this is.

B

C

B

D, the underlying processing I think I now understand. I, don't understand how that gives you what you showed before, which is all the intermediate postures all.

A

Right, so that's a great tie-in to the next point. Thank you all right, so we know that we can very reliably identify res States and motion completion States. But the question is what about everything in between great if we can identify those.

A

So it's the same idea here that if we reconsider this training protocol, we do collect ultrasound images that correspond to rest and motion creation, but we can record all the way through that movement as well, and so these transition states are detectable. We can follow the same so.

E

You interpolate.

A

So we can identify every step between zero and a hundred percent and again we're just to drive on this point further, because we are there's this direct congruence between the users intended motion and we can map that all the way across the percentage of completion. There's that correlation with muscular. This is a proprioception and the prosthetic hand function putting this all together. We do believe that it has enables a much more intuitive controls. Humans.

B

A

Yes, I goes back to the idea that, in order to essentially trigger those four receptive receptors its directly depending on muscle information, so as the muscle contracts, we will be gain that sense of proprioception. So if we're able to track that contraction across the entire movement cycle, then we should have access to that perceptive cue across entire movement cycle.

B

My the way, my muscle contraction feels to me, is proportional to the muscle, contraction and.

B

A

B

Closely tracking my feelings than with other techniques, yeah.

A

Concise way to put it yeah, thank you just to give a more visual example of this positional control. We designed an experiment where people were asked to control the position of a cursor on a screen, so the blue dot that you see on that upper right graph is the cursor. They can move it up and down on the screen and it moves in direct proportion to the level of muscle contraction. So if it's all the way up at the top of the screen, that said, you know maximum contraction.

A

So the task was to try to match the position of that cursor with targets which is represented by the red line here. I own, this place short video and that's a little bit hard to see the trace at the bottom, but you can see that he's visually. At least it looks like he's able to kind of match that target position pretty well.

A

And in fact, if you plot this out over a lot of different target positions, over looking at here is the blue line corresponds to the position of the cursor that the subject was controlling those little diamond sheets. If you can see them are the target positions, and so you can see there's pretty close, that a lot of discrepancy between the target position of the actual position is. Somebody was able to achieve all right, so I'm sure these videos earlier, but again putting all these pieces together.

A

We're really optimistic about the opportunities of this control methodology provides for achieving intuitive control and improve functional outcomes. This is an area of ongoing work in the lab, but for now we'll move on to the session of the miniaturized technology.

D

Hello, it's really nice to be here. Thank you for the opportunity, so all the work that Susannah highlighted we.

D

So most of the work that Susannah highlighted, so we work with individuals with upper limb amputation for over a couple of years, and we did this study with a lot of some systems. The natural question would be for most of you who have seen an alarm sound system. It looks kinda like a small mini fridge right. So how would you get this inside a space? Maybe this big or even smaller, sometimes right so to understand why Poisson systems have not been miniaturized the extent that we need it to be.

D

We need to understand how it works right. So, let's dive into what goes on inside and system, so primarily a non-person system will have two parts very simplistically right. So.

C

You have an ultrasonic transducer.

D

As the part that you see collision holding in his hands and actually navigating on your body to see those images and then there's a back-end which either looks like a tablet in some cases, sometimes like a mini-fridge which is doing all the hard work right, so it's doing all the processing. Now, if you break open one of these transducers, what you're going to see is there are individual piezoelectric elements, so piezo electricity is essentially the phenomena driving a material with voltage and it generates sound.

D

So in ultrasound we drive each of these elements with a very high voltage and what happens is these small elements they are? They are arranged in an array. Usually you get an array of 64 128, sometimes even higher counts of these elements. So you get upwards of 64 elements, so it takes up a bit of space right. So you have an array that has 64 elements taking up space. So that's one concern so in order to reduce this form, factor of the question is: do we actually need these many sensors?

D

So that's one question: now: each of these sensors, as I said, gets excited by a voltage right. So when you excited by a very high voltage, he sent out a small amount of acoustic energy into the into your body. Now this.

A

Acoustic energy.

D

Is reflected back and you receive some of that back now. The way under sound works is that it's looking for echoes right. So it's almost like a batch hearing out for echoes and trying to locate where an object is and that recognition of how far the object is comes from the measurement of time.

D

So you measure how long it took for sound to come back in order to figure out how far the object is so measurement of time when it's in the body, it's in the order of centimeters and the speed that sound travels in, but around 1540 with m/s. These time periods are in the order of microseconds, so you're measuring very precise amounts of time. It's very difficult.

D

You need very high speed digitizers, which increases the cost complexity, power, consumption and the form factor of clinical or some systems. So these challenges make it really difficult to actually miniaturized for the Sun systems, but for our application, the questions are like I said.

D

Is there an optimal number of transducers that we can use and get away by if you're doing a five motion, classification right, you know, all we want to do is a gesture recognition task or a proportional control task. Do we really need 64 sensors? So that's one critical question, second: is: can we even with lower voltage, can we send out low voltage pulses as a result?

D

Merely you get a weaker signal, but is that degradation in signal quality acceptable if the task is just to classify images, if you are not bothered about a nice interpretive image, can you get away with these constraints? And finally, can we measure depth without measuring time? So if you don't have to measure time, then you don't need to make very precision measurements with using digitizer. So these are the challenges that we are trying to solve.

D

So in order to understand the first problem, can we pan this down to maybe a few sensors instead of 64 sensors right? So let's look at what a typical order. Sound image looks like so. You have depth on the y-axis, so you're, looking down through the layers of tissue down into your body and on the y-axis.

D

What you have is the individual scan lines or the individual elements that have basically passed on for some signals into the body, and you stack them together and you get this nice beautiful image now as you're scanning through the body in time. You get these stacks of images and we call these stacks of cinema as you go as he stacked most of these images together, as you start mourning these images together, you get a nice cinnamon loop. That shows you know your muscles moving now.

D

If we select all of the 64 or 128 channels, maybe three or four channels right. What we get is what is called an M mode in which M here stands for motion mode, but with these images, the depth and scanner images are called beam or images or brightness mode images, as opposed to these.

A

D

On the right, which are which are called EV mode images motion mode. So what are we seeing here right? So, if I select, let's say the blue slice right, so it's a slice that corresponds to one line or not being on image, and it goes across fine, so you're tracking one transducers image the image that's coming out of one transducer over time, so along that line, whatever movement you've seen over time, you're going to be able to plot that. So now this axis is time.

D

So the y-axis and the second row of images is time so we're looking across the time axis and the reason why this is called motion.

A

D

Is because you are now looking at one one transducer and you can really nicely see these movement patterns if.

C

There's an interface that's moving.

D

Up and down, you can nicely track it. So now we select, so we have a blue scan line. We have a red scan line and we have a green scan line, so this can correspond to three different sensors placed equidistant from each other, and so we get three different images: three different M mode images. Now, if you move our might, if I move my index finger and my god, what did we get right? So if he superimpose this images- and we see what sort of patterns emerge out of it right.

A

So when I move my index finger, what do you see is.

D

Most of the movement is actually mostly restricted to M mantras, 2 &, 3, so sensors, 2, & 3 pick up that most of that motion and even and are they're actually distinct right. So if you, if you're just looking at n 1 2 & 3, you can see that the amount as distinct patterns. Whereas if you look at the ring finger movement, you see that it's located in all three sensors. So you can find information about this movement in all three senses. So even with just three sensors we can see.

D

Distinct patterns emerge out of three different motions that we are looking at you. So that gives us a sense of the fact that if we sparsely sample this nice beautiful to some image- or if you don't have as many sensors, you can still still do really robust classification.

D

So these patterns will emerge out of these senses and we have done some formal, formal work where we have systematically shown that if you reduce it down to as low as four sensors, you can still retain classification accuracies higher than 90%, and that's really was the key finding that led us to believe that. Yes, we can pare it down to just four senses to get it to work so from there on what we did was we developed wearable art, asan's, transducer patches, so, on the left hand corner.

D

What you see is if I take one or two sound transducer, so not personal transducer will be that white dot in between. So each of these elements can be excited electrically and they will emit sound into the body. So we have four such elements. I have only shown three here, but there are four elements over there and this is placed on a flexible silicon based patch and the electrical signals, and the back end show you later, and this can be placed directly on the skin.

D

Then you can tie it with the velcro strap or something like what Chad was showing John Marty. You know with a like yeah with a within a socket and that would be secure on your skin and it would be able to image through and the.

A

Images that you would see.

D

From them would look very much like these m1 images that you see and we can use this to classify emotions. So, yes,.

B

So is just piezoelectric: transducer is putting out the signal. Is it also detecting on the echo? Yes.

D

Yes, yes, yeah! So that's a great point, so these are not just images. These are also receiving the signal and processing it, and so the way we process the signal, that's actually a good segue. So now we have transmitted the signal we have received.

F

D

The cost was less than 4%, for so it's the reason is the beta is so rich and and there's a lot of redundancy in the muscle Wow. It would be probably so. The largest muscle group in the forearm is probably around one square, centimeter plus cross-section right, so you don't really need that much information. Sure.

F

D

By sparse sampling, if you can still get away by doing this so yep, so now we have attached these some transducers. What remains to be seen is you know the other part of this system where the back end, but in order to reduce the bulk of the back end, we really need to move traditional ways of processing the signal. We cannot be exciting. These transducers, with high voltage pulses high voltage pulse generation requires large circuits. We cannot do that so we actually adopted a new technology called time. Delay spectrometry this.

A

Was actually developed for radar applications.

D

Back in 60s by this genius, individual called Richard Heiser, and this was used for a very long time to make acoustic measurements. You know like how do you soundproof a room? What.

A

Are the sound characteristics.

D

Over room stevie time, delay, spectrometry was used extensively for this application and we adopted those applications adopted this technology into our facade imaging and we developed our own system, which uses a low voltage pulse and solves many of the problems with the back end, electronics, and so this is one of the ports that basically excites and receives signals from one sensor or one channel. This is still not the form-factor ideal form factor that we would like. So this form factor is about five centimeters by 3.5.

D

The new generation of this board is going to be around 1.5 centimeters square.

C

D

C

A

D

It's going to be the size of an EMG electrode, essentially, so each of these units basically sends our signal receives it processes it, and you have a stack of four such sensors and you can come. You can basically control and receive signals from that awesome band direction, so it's receiving from this and the ultimate image that you get from it would be these end. More images.

F

D

These so I'm simplifying.

B

D

Again but you know so.

C

D

A traditional office on system: what are you doing? It's almost like hitting the transducer with a hammer right, so it's a high voltage short trying pulse. What we are doing is we're spreading the energy out over time, so the total energy is the same, but it's it's routable time. So the small voltage that we use our about 2.5 volts peak to be compared to at least forty two hundred volts peak-to-peak. So those are much easier to generate. Yes,.

B

Certain remote control toys.

B

D

Those sensors VRS on sensors, but they are operating at a much lower frequency, so typically aeroacoustics. So if you're transmitting into the air, you need frequencies of around 100 to 200 kilos, sometimes 400 kilohertz, because otherwise it would attenuate a lot. The higher the frequency higher that generation in inside body. We actually medical artists on imaging. It happens above one one megahertz.

D

It will work for very shallow depths, so it will probably be able to see probably be able to see like maybe half a centimeter inside your body.

B

D

Be four centimeters plus.

D

For RC cars, so yeah.

A

D

This image, what we're basically trying to show is, you know, as you do different motions. These patterns emerge out of these motions and we can do this. Real-Time imaging at about 25 words, which is sufficient for most applications and beyond. Personally control actually, and we get really nice robust, snr, signal-to-noise ratio of about 30, DB and we're talking about this.

D

In the morning we have shown that these sensors are safe for long-term use and well below clinical regulate regulatory standards set by the fda as far as far as spot reduction is concerned, is still a little bit high and we would like to bring it down, but it's still within the range where we can operate it off, of a prosthetic for study battery people, $4,000 ampere our battery would, he would probably take about 10% of the battery life, so you shortening the hand would actually be severe.

D

So finally, does this work so do we you get the same kind of classification accuracies. The answer is yes, so we tried classifying with these M mode images, nine different motions, and then we see them, and this is with able-bodied individuals. We see that the classification accuracy is above 90 94%, almost 95% classification accuracies. Yes,.

D

What do you mean? Oh so, I, actually don't remember what motion one is so so usually what we see is there's a large confusion between them and Barbara's women. They look very similar, said mean I mean key grip. That's a.

F

F

D

So that's actually a very good point. Yeah.

D

So you've seen this video before, but just to emphasize so now what you're seeing here is the variable band on healthy individuals forearm. This is imaging the cross-section of their forearm, with.

A

Four sensors.

D

And this has been that has been classified and will MATLAB is script and we can.

E

How sensitive is positioning of the sensor? It's.

D

Not very much so we don't actually carefully place this well, we have done some analysis that shows that if it's in the upper 40 percent of your forum, you do not really need to matter.

E

Oh just kind of centered, just kind.

D

Of centered, so we are only imaging the extensor muscle group, not the flexor muscle group I'm. Sorry, we are only imaging the flexor muscle group, not the extensor muscle. It would.

C

Actually, help to.

D

Do both there are advantages to doing both, but in general it really doesn't matter. If you do see a degradation in signal quality, maybe it makes sense to move it to a higher muscle power. We haven't done tests with individuals with limb loss where positioning might be critical. So you know in order to get nice views of the residual musculature, you may have to move it around and play around I.

E

Don't know if this is true, but it would seem to be that the closer to the wrists shall we shall we must go to rises to the skin and therefore you might get a stronger signal of you. Correct.

D

Yes, you're right yeah, so most individuals with limb loss I mean at least with fans radial amputation, they're not likely to see anything more than so. We want to be as proximal as possible. I'm.

A

D

That's proximal as possible, so.

C

D

This video is not playing so I'm not going to try it. So the next step is our random research goal would be to miniaturize these, to an extent that we can put it inside. The prosthetic socket socket do I keep getting very close to it, so the version 2 of our system isn't works. Hopefully we have a grant to get this done in the next year. So, but we see applications of this beyond prosthetics research, where you want to see muscle movement, movement intent. This can be a complementary technique to EMG.

D

We are also trying to develop very small sensors and very low-cost sensors that can be paired with that can be used as a direct control. Some a terrific direct control system to control.

F

D

Yes, so we have done some studies where you know we that's what we do is we present the user with a target, a virtual target right and we ask them to achieve it, and then we remove this visual feedback and we ask them to hold on to that project and we look at the stability of that signal. So how stable are you? How stable is a grip when you're actually not looking at your hand, you're not looking at that signal, so we see that that signal is pretty stable.

D

You are very good at actually maintaining that level, but if you, if I, ask you to leave that object and reacquire that object, we are not very good at estimating the position of our fingers or a grasp without actually looking at it. We needs either tactile feedback, haptic feedback or we need visual information, and we have shown that you know there are ways to get around it with even very rudimentary tactile feedback, yep.

E

Good question the original system process.

B

E

So the onboard processor but smaller obviously is it- was a proprietary coterie. So the.

D

Court for now is proprietary, but so this is NSF funded. This is funded through taxed at office right, so we don't see any reason for holding this back. That being said, under the PA on the graph.

E

You see, I have my promise. Your turtle team wrote your own code. Yes,.

D

Yeah I mean we may have used open source and you know where we have to attribute but other than that yeah, it's all right. Yes, so yeah. We are interested in exploring normal haptic technologies to go with this, and we feel there's an acute need for intuitive hatrocks to go with this intuitive way of controlling devices. So I'm not I'll skip these pack, all my lab members and collaborators, and we are looking for active departments to help individuals with limb loss get to this technology. Learn about this technology.

D

We are also looking for subjects and volunteers to participate in the study. So any help in that regard and technological collaborations are most welcome. Thank you. So much.

D