Numenta 2015 Spring NuPIC Hackathon, 5 Jun 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OpenBCI Primer (2015 Spring NuPIC Hackathon)

Description

Conor Russomano, cofounder and CEO of OpenBCI, describes how his device works for NuPIC hackers.

A

This is connor Bruce Amato is the CEO and co-founder of open b/ci who manufactures high-tech jewelry. As you can see, the.

B

Potter demonstrated.

C

D

E

C

Thanks guys for all taking the time and watching the debug, the system line, so this is open bc is, are really cool logo, so open bc. I was started by me and my business partner, Joel Barbie, so Joel was my physical computing teacher at Parsons, and so I learned how to use in Arduino and I learned introductory electronics and a lot of the code that I've learned was working with him. He had started his stint in to open source hardware with the pulse sensor, which you can see on the upper right.

C

It's an optical heart rate, monitor for arduino, and then he was contacted by somebody on an SBIR garba grant and the DARPA grant was to create low-cost EG for non-traditional users, and then he knew that I had spent my whole year. My whole time in my master's program, doing EEG related art projects, so he reached out to me and he was like you want to work with me on this night. Absolutely so then, after about three months, we took our first prototype to maker faire and everyone at maker faire I was like.

F

This is unbelievable.

G

We have to have.

C

This, and so we decided to kick-start so so Joe and I- that's the two of us with 3d printed headsets.

C

So we you know what we set out to do is essentially build this, this piece of electronics, this prototyping tool that could be used by a variety from different people, so artists, researchers, game developers to extract bio signals from the body and allow you to do anything that you want with those signals and a lot of our at least a lot of my inspiration came from Lego as a who.

D

G

C

With Lego at some point, they're like them, who.

D

Here actually thinks it.

C

Shaped the way that they think quite substantially so Lego has this amazing model where they give you, you know an instructional guide and how to build the castle or how to build the you know the pirate ship and they give you all these pieces and then, when you're done you're like oh on board of my pirate ship, you can do whatever you want with it, and so what we decided is we're going to apply the same approach to to turning people into cyborgs.

C

So, like essentially figure out what pieces you want and then stick them together, and so Arduino did a really amazing up who hears worked with Arduino before so. For those of you that don't know, Arduino is a electronics prototyping tool. Essentially, so it's really great for beginners, because you can look.

B

G

C

In a push button- and you can write a few lines of code to push the button to make the LED turn on and off, but then a lot of experience electrical engineers actually use it as their initial prototyping tools, so they'll build shields for the Arduino that do very powerful things. For instance our first open bc.

C

I board was an arduino shield and then we took the components that we're working and the ones that we wanted and we manufacture our own pc, be based on the shield combined with the arduino and the great thing about our do. We know is that it's programmable, so you can apply code to the microcontroller and have everything to be everything operating on hardware and never actually doing anything on the computer. So we wanted to combine all of these aspects with biosensing, as you can see so.

D

We ran a Kickstarter in.

C

Late December of 2013 or December of 2013, and we were- we were successfully funded, we more than doubled our goal, which was awesome. We weren't sure if we were going to hit it, but it turns out there are a lot of people that are interested in doing the same thing that we were doing, and this is the work that you see.

C

In my hand, this is a diagram of the various components, so there are 8 x, 0 potential inputs, so there are eight channels that you can listen to EEG, ECG or EMG, so any electrical potential coming out of your body, there's a high-powered analog to digital converter. So this is 24 bit resolution very sampling rates, so the theoretical sampling rate is up to 16 kilohertz, but for eg you don't really need more than 250 muscle.

C

You generally want a little bit higher, so maybe a thousand kilohertz or 500 kilohertz or 500 Hertz rather or one kilohertz. There's an accelerometer there's the the same, microprocessor that the Arduino hands, which is the atmega328p, and then we have a different variation of the board, which is a 32-bit, pic32 microprocessor and there's wireless communication and a local SD card. So.

H

One of the.

C

Big one of the big issues about sending EEG data over the air is that you run into a bottleneck, a data bottleneck in your wireless communication most of the time, and so we put the local SD card there so that you could write high-resolution data at high sample rates, the SD card and end down sample to get it over the air. In that way, you can visualize your data live, and you can also have all of that data for post-processing at a much higher sample rate.

C

D

Yeah, this is kind of a quick summary of EEG.

C

The brain and your body gives off a lot of analog electricity all the time and what we do is we take that analog signal and we convert it into a digital signal and I'm sure many of you are familiar with this, but the higher resolution, your data packet, the more The.

D

Closer, your digital.

C

Signal ends up looking to your analog signal, and so we have a really high resolution: data data, packets, 24-bit resolution.

C

So here's a little demo, so your brain is like an ocean of electricity and when you're working with eg you're standing on the beach, essentially in your washing waves crash on the beach and you're trying to make inferences about what's going on inside of the ocean and so obviously the more the more beaches that you can stand on. At the same time, the better inferences you can make, but ultimately you're watching waves crash on the shore and see what you have to do is you have to look for outliers.

C

You have to look for things that that stand out, maybe a succession of high amplitude waves, and then you try and attribute those to some other contextual information. So.

D

If you look at a standard battery over.

C

Time you'll see a steady voltage as the battery starts to lose life. The voltage will gradually drop with EEG your it's very similar to touching the multimeter to both ends of a battery except you're, touching into on your head and then the minus side, your reference and that's usually a place. That's electrically neutral, like your earlobe or your pass code, which is the bone behind your ear and then what you'll see is a variable voltage over time. So you'll see this this fluctuation. Is this waveform? That's a raw EEG waiting on the bottom.

C

So this is an example of way of addition. You see a plus B equals C with eg. What we're trying to do is we're trying to look at the raw wave and we're trying to extract frequencies from that raw way. So when you hear the term brainwave, I'm sure, we've all heard the term brainwave, but what it's literally referring to is reverently prevalent frequencies or predetermined bins of frequencies that we're extracting from a raw signal.

C

So when I close my eyes and all done with this later, my visual cortex will produce an alpha frequency and you'll, see that and you'll see that waveform in the raw signal very pronounced, and this is actually what it looks like. So that's a one of our early versions of our GUI.

D

C

Then, if you look at the time domain here, if you count the little waves per second you'll notice, that there's about 14 kind of high amplitude waves that are dominating the signal and then here's the FFT graph and then at about 14 Hertz, you see a spike and the.

A

D

C

The back of the head, you see that the high opacity saddam know it's there and it's because your visual cortex introduces very strong alpha when you close your eyes. So here's a better insulation. This is our newer, a newer version. We're doing so here you see, channel 2 is connected to my forehead and I'm, going to demo this in a second actually, whatever I'll just devil this in a second.

C

So this is these are our products. We have an 8-bit board, a 32 bit more than a 16-channel, our indie kid, and then what we've been doing.

C

You know we put a lot of effort into the hardware and making the hardware extremely modular what we started: building SDKs and drivers for openframeworks processing and a lot of existing signal processing tools for biosensing, so bio, Explorer, ECI to the house in Great, Bay and there's a huge there's, a plethora of ease out there, and so a lot of our community members have actually already started building these bridges.

C

B

D

C

Of different projects that people have started implementing so on, the upper left is a motor cortex classification. So a PhD student in France is trying to classify motor cortex signals and then them to just a simple game where he basically has a binary input. He can have a character jump up over like like almost like Mario. He just has to avoid the gaps and his character constantly runs word. He has to jump over them here. We have five different people plugged into the same open bc.

C

I Borg controlling a floating shark flowing through a room, and this is a diagram of how it works. So what we do is we have kind of a commander who is dictating to the rest of his team, which action the shark needs to perform and by closing your eyes introduce an alpha. So each person's alpha is mapped to a different trigger of the sharp, and so the commander will say forward and then the forward I will close his eyes.

C

His brain wave will produce the alpha and move forward and then they rotate through their respective jobs. Well, there's other ways of doing this. With a single user, so one technique is called SSB EP, which is a steady-state visual evoked potential. So if you look at flashing frequencies of light, your visual cortex will actually start mimicking those frequencies, and so you can have a screen or blinking LEDs at different frequencies and you can have each frequency math to do a different controller output. So if you look at you know, it hurts that's map to left.

C

If you look at 12 Hertz, that's I have to write. You close your eyes and your doubt produce produce alpha and moves the robot forward, and so we've done that as well, and then.

D

We've also worked on.

C

Controlling 3d printed prosthetics with EMG signals from the hand and then 3d, printing electrodes, so there's a lot of new materials out now: conductive materials, conductive spla, so we're looking at 3d printing, the entire TG headset. So these are different prototypes that we've been working on, and here I'll turn this on.

C

This is the ultra cortex, which is you know, one step beyond the neocortex.

C

So the ultra cortex has a twitter account by the way you guys well, follow it and we're probably going to launch a Kickstarter campaign later this summer to print and distribute ug headsets that work with the open bc. I board we're also looking at 3d printing everything every component of the headset, so working with printing companies like blocks away, who are looking at metallic ink, in addition to new types of filaments to print electrodes and components, and then just have the actual EG hardware, the amplifier just plug into the back here.

C

So why do we do it? For people like you, and actually you know we're big fans of what you meant is doing, because everything that we do is open source and it's very community driven. So since we're an open source company, we basically live or die depending on how well our community supports what we do and at the same time we believe that, as we move forward as a species as humanity, we need to make sure that the technology that we build that's going to be most instrumental in the future of who we are.

C

What we do is is transparent and and understood by everyone, and so that's a lot of what we stand for behind open b/ci. So that's that thank.

D

C

I'll actually mo the technology, so this is, we have an open source processing duty. So processing is a creative coding framework based on java, it's not the fastest framework, but it works, especially with there's a lot of really cool graphics, libraries that are unique to the system.

C

C

If you want to look at this code, it's all online on.

A

C

Yes, so here you can select from a very series of syrah.

D

You can select your board essentially.

C

And name a data file, so anytime, you're logging, anytime, you're, recording data live from the board. It's storing it in a CSV that you can then access for post-processing. You can have eight channel order, 16 channel board.

C

Here you can playback and previously recorded files. If you want to.

G

Generate synthetic data.

D

C

If this is all connected directly in the powers on cool, that's what you want to see so now, I'm going in.

C

This is not the entire screen, because my screen resized so.

D

G

Doesn't make any sense right.

C

Now, because I'm actually connected.

F

C

I've connected my head, I'm, going to demo all three bio signals at the same time, so I've got EEG, which is my brain activity about EMG, which is my muscle activity and ECG, which is my heart activity.

C

Alright, so what I have to do- and this is a really cool feature of OBA DCI- is that while the board is actually streaming, data, I can reprogram the various channels to optimize them for different things. So channel 4 is listening to a bio potential measurement from my right, forearm.

D

C

My left wrist and therefore since sample across my chest, you see if we AMG spike, not sure why this is kind of early cuz I'm touching.

C

Okay, refrain from touching or pins. Hopefully this fixes you should.

F

C

So I'm going to do is I'm going to remove channel 4 from the reference that is being used for like EEG channels.

C

I'm also going to remove it from the bias, so the bias is a ground establishing a common ground between the board in my body and in doing so the bias is kind of a souped-up ground. So what it does is it senses, common-mode noise on all the channels and then takes that common mode, noise, waveform inverts it and then send it back into your body to create destructive interference for that signal.

B

C

I'm not going to I'm not going to force you guys to send me just to do that again, but essentially what I'll do is go back to the demonstration here. I wanted it end with us live, but if you guys want to stick around and watch it afterwards, I think there's just a lot of moving pieces right now. So this is what I was going to demo to you, which is the exact setup here, is heartbeat in channel 4 boom boom.

C

Here you see in channel to this huge EMG spike, that's a result of meat closing my eyes, and so, when you close your eyes, your visual cortex will produce this alpha wave form, and here you can see in the right channel around 12 Hertz 12 Hertz is the frequency, or you know, most people's alpha fall somewhere between nine and 12 Hertz, and so the various regions of your neocortex, like your visual cortex, is a very of your brain.

C

Your occipital lobe is very big, and so what happens is as you close your eyes, and your visual cortex is not processing, tons of patterns and decoding and looking at and observing things in the environment. As a result, the like, when your eyes, are open. Your visual cortex basically looks like static. It looks like noise because everything is doing it's doing its responsibilities, but then, when you open or when you close your eyes, your neurons, don't really they don't know what to do. They're kind of just like.

C

Oh we're chilling now we're in chill mode and as a result, they kind of start. They don't stop firing, they still need to fire, and so they end up finding each other they define requiring in synchronization, and it starts producing this. This alpha frequency and so different regions or other regions of your brain behaves. Similarly, so if you relax every muscle in your body, your motor cortex will begin to produce a new frequency and so one motor court.

C

One technique of applying EEG for interaction is to have someone sit very still and then tell them to think about moving some limb or piece of their body. So if they engage their right hand, they don't even actually the move it, they can think about.

C

Moving the right hand, the entire motor cortex will be in immune frequency and then that region, that's responsible for activating the right hand, will start looking like static, and so you can build classifiers around that, where you can look at right hand versus left hand versus feet, and so that's a common technique for building interactive tools using EEG, but interaction I would argue is like not the best thing to do with the EG.

C

I think that there are many better ways of building interactive tools, for instance EMG, so EMG signals are very easy to produce and even quadriplegics can blink their eyes right. Their teeth, and these EMG signals we've trained ourselves over over our lives to have very strong control over.

C

So a lot of the novelty of BCI and EEG- you realize it wears off very quickly but like in media, doesn't really bad job for the industry about saying mind control this mind, control that when, when really like, that's not the most practical, practical application of EEG I think the EEG is the weave were so far away from hitting the plateau of non-invasive, BCI and understanding what it's capable of but I, think those those practical applications are not going to be interaction.

C

I think it's going to be understanding, brain activity in the context of environmental stimuli and looking at diagnosis for medical medical billet for diseases, and things like that. So a combining you GE, with neurostimulation like transcranial direct current stimulation for closed-loop systems, well yeah, so I'm bummed that I couldn't get the device working on this whole thing, but.

G

If you guys want to stick around I'll set it up on, my computer and I can show.

C

You, but if you have questions you have any in sanaa.

E

D

You close your eyes, you.

E

Know you actually see a different type of signal, but if you start thinking about seeing the same thing, you saw when your eyes open.

D

E

The brain start firing the exact same way when you are actually watching somebody run eyes open versus when you're close. Yes,.

C

To a certain degree, so it won't be as strong of a signal but I. Imagine like imagined movements will activate the same regions of the motor cortex is actual movements and I. Think that's I'm not I'm not a neuroscientist, but I think that that is true that you will get those regions of the brain will be will be active right.

E

So like when you initially close your eye, that's just a temporary like temporary signal that you're getting just because you're closing your eye and resetting. But you start thinking about the same thing that you were seeing and you may go back to what it used to be before you close your eye.

C

Hypothetically hypothetically but, like you know a construct like that, something that's so specific is very difficult to detect with EEG. So, for instance, like person, riding bike would be almost impossible to pick up with EEG, whereas eyes open versus eyes, closed is possible or alertness versus mind, wandering states so kind of more I, guess you could say, emotions or higher level mental states, as opposed to very granular patterns like that are very difficult to kind of like going back to the ocean analogy.

C

You know it's difficult to look at that tectonic plate at the bottom of the ocean, but you can see big wind patterns influencing waves.

G

But what about athletes that use mental you know see themselves running over and over again, and then they see their times increasing. Would that have some kind of effect that I'd be able to assist athletes, for example, hope so.

C

G

C

Are a lot of theirs, so red bull is actually doing a really interesting, really interesting project right now and it's called Project endurance and so they've actually been rigging athletes up to EG, EMG, ECG and all of these sensors and then trying to map optimize athlete's performance, both physically and psychologically, using like neurofeedback techniques. So I would look into that project. It's really cool budget yep.

H

Do you have a sensor somewhere near motor cortex? Can you pick up the new van yeah.

C

I mean so I, just stuck these electrodes wherever wherever I wanted to, but you could array them all over your head or you could target the motor cortex specifically. So that's one advantage to our system is that you can essentially build.

H

D

C

Or build your own head where to target very specific locations.

H

To the previous comment, then,.

D

H

Closing is only going to affect an alpha. It's.

E

H

The visual cortex and the motor cortex is separate game, but I had another question. Witches.

H

Do you imagine that this be able to detect and emotional response in.

G

Other words, if you ask somebody a.

H

Question on which they're likely to have a strong opinion we could do you think you could infer from the signal whether they have a strong positive response was a strong negative response. There.

C

Are people claiming.

D

C

Be doing this already with EEG, so there are companies like software companies that are developing proprietary software for emotion, emotional detection, and they were, you, know, I think it was a company in Israel. I was out there for brain tech. That was doing this and running people through ad campaigns for the election, like the recent election in Israel, and they were getting. They were classifying emotional states and emotional responses to different political ads, so it is happening and I do think it's possible. It's not easy, though. Even.

H

Even with even with his level others as opposed to classical EEG, with whatever the 17 sensors yeah.

C

So I mean the EEG zeg you know like if you have a high sample rate and you're in a good environment with low noise. If you can build a Faraday cage to prevent alternating current and light from influencing signor you'll get a good signal. If you have a high sample rate and good luck and good connection and I think like open b/ci is overpowered for a lot of the applications that it's being used for.

F

D

F

Somewhere that one of the issues is some kind of a helmet that maybe incorporate some of this stuff with the pilot where's. You know I think about that. Yeah.

C

Actually, it was a DARPA solicitation or DoD solicitation, I think recently about Air Force, wanting to admit DG and maybe even neurostimulation, like tDCS techniques and have closed-loop systems inside of a fighter pilot helmet, I, don't know much more than that, though, but I think it is being researched so.

B

Yeah, just to follow allowed me a lot of the research that their funding it's based on the ability to sort of detect sort of one pilots or one soldiers fatigue or don't fatigue. The.

G

B

Of sort of at what point they really want to sort of push their soldiers to the limit. At what point can we say? Okay, these soldiers can make, or these pilots can make really good decisions without, like their sleep factor, fatigue, effectiveness, I, think the bulk of why they funded where it's going into yeah.

C

Yeah I know that, like with tDCS, that's a big one is uh I think it was a study done with the drone operators and they tested. You know because tDCS who's familiar with tDCS transcranial direct current stimulation, so there's a whole field of T blank s which is transcranial this or that stimulation.

C

So they're trying out direct current alternating current magnetic random noise and they're, basically just sending electricity into the head and trying to influence brain activity and see what happens and it's pretty widespread thing now, there's a whole DIY, tDCS subreddit and it's a very active community or people that are sticking 9-volt batteries, their head and trying different things out.

C

But one study that was done and apparently found statistically valid results, was tDCS used for like improved or enhance performance through, like essentially closed-loop systems to optimize alertness, and so they tested tDCS against ritalin nuvigil, caffeine and a control group. But they found tDCS to work more consistently and for longer durations of time than any of the drugs which was pretty crazy.

C

So on yeah, any more questions.

F

F

E

C

We don't do any tDCS and I haven't done much research, but I'm kind of I pay attention cuz, it's interesting and terrifying at the same time, but I think this standard like acceptable range is 2 to 20, milliamps I. Think, like the you know, most people are not sending more than 20 milliamps and low voltage signals.

D

Through the right.

C

B

You're, so your input on open, DC I can handle that kind.

F

C

Don't do any of this I mean I, don't know.

F

D

Saying like if some were joined at once, he asks each other yeah.

C

I, don't think it wouldn't be wrong, yeah, so I, don't think so. Otherwise, you'll have to watch G boards.

C

A

Thank you, Connor really.

A