Numenta HTM Visualizations, 23 Sep 2013

Previous Meeting

⏯

youtube image

►

From YouTube: Cerebro for NuPIC

Description

A CLA visualization tool for the Numenta Platform for Intelligent Computing.

A

All right, hey.

B

Guys, thanks for joining me here on this presentation, so basically it's not gonna be a presentation. It's gonna be a demo of this tool that we use for debugging the cla uh called cerebro.

B

C

B

And I'm recording.

D

C

I'm eventu eventually it'll go up on the web, so don't talk about any company confidential stuff.

B

So this is all going to be open source. This is all survivor, non-confidential, basically a tool I built um a while back and it's gone through multiple iterations, but it's been a great way of diving into the guts of the cla and it's a good way of understanding. What's going on so.

A

What is cerebro.

B

Well, here are some options: it's either a dvr.

A

B

Cla algo a visual debugging tool for the cla, a.

A

Data set generator a local web app.

B

Another client of the opf, which is our core framework or a tool for finding news. It's actually all of these things. It is all of the above, basically cerebro. What it does is it allows you to create data, sets and and run cla models over them, allowing you to edit the different parameters, all the little itty-bitty parameters that we swarm over or even ones.

B

We don't you can hand edit them run models over them and at every record it will actually store the internal state of the cla algorithm, all the permanences, all the connections, and then you can kind of go backwards and replay uh the data set and actually look at how the state evolved over time.

B

As the data came in and the movie x-men it's used for finding means, and so real quick just right now, before pre-open source, it's located in project cerebro to run it super simple: you do python cerebrum, you specify a port number and it is a local web application. It's not built to actually be hosted yet there'd be some trickery to have that happen, but you can just download it right locally and then go to your browser and go to localhost. Convert.

B

And again, I'm going.

A

To demo all this, but I just want to get.

B

Your heads prepared so you're ready to absorb it. So.

A

The first key, primitive.

B

In cerebrum are data sets so there's two different types of data sets. There are file based which, right now it doesn't just take in a.

A

B

Straight, what it actually takes is our grok description.pi files, which are a fully qualified experiment that describe a model with all its parameters, as well as a data file. To point to so it specifies the data.

A

As well as the model.

B

Parameters um and actually you can load two- you can load a base description file and you can load uh what we call the sub description files that kind of override some parameters, which is usually what we get pre and post swarm the base description files. What we get pre-swarm just specifies everything and the the sub description files usually get after this form. That tells you what specific parameters this form found.

B

uh Alternatively, you can write a function to generate this head, and actually, I think this is one of the really cool features of it that to generate artificial data sets, you can do it really easily by writing python code uh and I put in a bunch of useful utilities that I will take great pleasure in demoing to you guys to show you how like just some nice tricks and utilities for creating realistic or you know, a wide variety of data sets and.

A

You can create those and same with csv, so you don't even have to use cerebro.

B

To run models, if you just want to use it to create data sets all day, you can do that as well.

B

B

D

A

A

B

A

A

B

D

B

B

In the super folder right now,.

B

Either description files or a function, data set first.

D

I'll show you.

B

B

All right, so I'm going to load up a description file here. This is our anomaly detection, uh the hot gem data that we have actually this one's for our anomaly detection.

B

So let's check that out so below the description file as soon as you, let's show.

A

You notice on the left here.

B

A whole bunch of text showed up. Unfortunately, the indentation is not great, but really what that is is this is the dictionary.

A

C

B

Parameters so at the top we have the classifier parameters alpha the steps that it's trying to predict. uh You can see here it's a temporal anomaly model and then all the encoder parameters are here so for consumption. You have your n and your w. uh It's an adaptive scale! Encoder! You can look at the impacts of that.

A

B

A

A non-adaptive scanner encoder.

B

um And then we have all our sp parameters over here. We see that it's set to a random spatial puller, so.

D

A

Can mess around with these and actually what you can do.

B

We can edit these later uh and re-run experiments, but just.

A

Start off with these.

B

Are kind of what the parameters were in the description file as it was loaded we just clicked on the experiment, tab and just hit run.

B

And you can see over here, the blue is our predictions and the green is the actual and the.

A

Bottom here, because it's an anomaly model.

B

uh It shows the anomaly score, but it wasn't an only model like that graph of the job you can see it's checking along.

B

And from the interest of time, I'm just going to pause it right here and.

A

We're using digraphs graphs here so you.

B

Know you can zoom into the middle and see you know what kind of predictions are we.

A

Getting here what's going on.

B

I don't know where the space.

B

Oh, it's there's no special pool enabled so.

A

B

A

B

You'll notice here is that this is a non-spatial cooler model. um Let me just for the sake of clarity. Do us patient pull them off? First.

C

That's a feature.

B

huh That's a feature.

C

That is a feature.

B

A

B

Build up to that so, for the sake of those in tv land, we're gonna do.

C

It normally so.

B

C

It does it just start over yeah.

B

So you get running again start over also note the enthusiasm on the buttons, because that's how you should be feeling welcome with this. um So.

A

It is basically.

B

A little playground for the cla, so you're going to write, I'm going to write a little bit more just to show what's going on and then I'll talk about what all these outputs are. What all these displays are, um hopefully.

A

Every I think, if you look at the video, the colors might.

C

B

Great but I'll talk through them.

C

So now it's it's really done is. Is that graph at the bottom still anomaly? It's still.

B

The anonymous course so the anomaly score for.

A

The beginning with this change is starting out pretty high yeah, so let's stop it for a second.

B

And take a look what's going on so, let's zoom in a little section right here again, remember so.

A

Now, basically, this is why I say me when I say it's a dvr for.

B

A

The right here on.

D

The left here we have the predictions which is.

B

The actual on the right we have the spatial puller. The spatial cooler is 2048 columns right now with 4d on um the two-dimensional layout of this vision. Puller is irrelevant, it's just for keeping it all. On the page.

A

But it could have very well just been the.

B

One-Dimensional thing, the the hollow grayed out circles are inactive columns.

A

The yellow columns.

B

Are active and if a column is predicted.

C

So cla terms, those are represent: columns of cells, four.

D

B

I don't know why the predictive ones aren't showing.

C

Up, oh, those were should be blue circled if they are.

B

Predicted um and the anonymous course says they are predicted. I don't know why they're not showing up there.

D

B

Just use a special data set here just so.

C

Stitching is due yeah I'll, make sure in mask. Ask whatever I need to mask yeah.

A

I'll run it by super tight before.

C

B

It here's just another one. I know it does better.

B

Again, this has been my side project, so bear with me.

C

B

B

um So we'll run this unnamed customer mom.

C

Okay, great there's.

A

C

B

Yeah, so basically, if a point, if the columns are on active columns, are yellow that's what shit's up there if they were columns that.

D

B

Predicted have the blue outlines on them, so you can.

D

See on these little blue outlines.

B

Around here, if.

D

They match you, gotta get a.

A

B

Over a yellow circle- and it looks like that so.

D

Here the anomaly score.

A

B

0.8 because most of them are unpredicted, some of them work and.

A

Then you can use the arrow keys here just to step through.

B

uh Record by record surprise, so you can see you know the space builder saying the same. We didn't predict this one, but.

A

D

We predicted it all and just step by step.

B

You can cut through um what's going on on the bottom here.

A

Or actually so, you can use the arrow keys to step through.

B

Say what I found really useful is like you say you want to compare two points. Sometimes it's useful just to jump back and forth between them to see what the two representations are so say. I want to jump between this point over here where there's a spike.

B

What's going on over here versus, like that versus like this spike over here well,.

A

I can notice that you know.

B

This is at 68 and this is a 145. So that's a difference of.

B

80 73 77, so you basically the.

A

B

Here, if you want to jump uh across multiple records, you put how far you want to jump across in that box and then you hold down shift in the arrow keys and you can jump, and I actually did that math right for me, um you can jump from point to point and that's really useful, just for comparing two faraway points and seeing what the representations are is.

C

That in the documentation anywhere.

B

It's in the video documentation.

C

B

So this is the spatial puller. What comes before the spatial puller is the encoders, so.

A

B

Bottom here the encoders also have their.

A

B

And I split them up by field, so you can, in this data, set there's two fields, uh a demand which is an injury and the day of the week. So.

A

B

Actually, as we go through, you can kind of see that the day of the week, the same concept with the the the energy uses. The demand is jumping around it's hard to see because it's on the bottom right of the screen there and then, as we jump down, you can see that the energy usage jumps down and.

A

I'll show you some more interesting examples.

B

Of one that's useful in the beginning, uh a little bit later and.

A

Then eventually so.

B

These are five minute data points, but eventually we'll see the time of day change as well or the day of week. What.

D

Is the days of week wrong again on the road.

B

It's the encoding apple for the day of the week, so.

A

Right now, these are five minute intervals, so.

B

The day of the week is.

A

Staying relatively.

B

Constant but yeah.

A

So you can see around here. This is where.

B

Midnight occurs, so the representation is changed from the day of the week and.

A

Right now it's not forming accordingly, so it's kind of running off.

B

The end of the page, but the top you can.

A

Also see here how the day of the week has a very low resolution, encoding.

B

It's only using, like you know, 28 bits to encode that.

A

The demand is using around 500.

B

It's using a very high number of bits, so once you buy, the circuits are all very small, so you can see the amount of resolution you get in those two encodings.

C

So there's both actually rows of circles. One's just a bit huge.

B

Yeah one's just huge and one's really small, okay, so.

D

The circles are proportional to the number of boots.

B

Yeah, of course, the idea was to scale them, so they all fit within.

C

B

C

They could be also tiled, so they.

B

Use multiple rows, but um at.

A

The bottom now you can also see that within.

B

uh We have some debugging output too, that is just textual, so you can see the data at time. So just it's just reiterating what the record number is here, but then.

A

You have what the raw input record.

D

B

So you can just confirm what your suspicions were about. What data is going in and right now we just have the inferences of what's coming out. So there's no anomaly label. We can see what the anomaly score is should match. What's in that graph, uh you can see what the multi-step predictions are. So each time set. What were the predictions.

A

And a little bit later, I'll show you some verbose output as well.

B

That'll be real real fun.

A

Any questions uh right now just about.

B

Running a data set and the file data sets in particular the function. Datasets will be a lot more fun, trust me, and none of them will belong to customers.

B

A

So function data sets let.

B

Me just reload the page, so this is where you can procedurally generate a function.

A

D

B

One of the easiest data sets.

D

B

Generate is a solid. You know. Just clients accounts one, two, three, four up to n and back uh so.

D

One thing uh the easiest way to do that is.

B

Basically, the way this.

A

D

B

It's a python function called get record that def is already given to you and it takes in the time t, which is basically the record index and also a helper buffer of history. So.

A

If you want to construct a saw wave.

B

It's a repeating function. What function, uh what mathematical operation do you think you.

D

B

So you can say that you know if you wanted to be a five point or like a five step. Solid, we just say uh t mod five and so t is gonna count from zero one. Two three four five.

A

B

You'll get that wave, and so you can say if you want.

A

B

Field to be called foo, say fields, foo is t mod 5 and you say I want to run that for 200 iterations, you say create it, and now it's created that model. You see, you can see all the parameters on the left side. Here it's created a multi-step prediction model which is with some basic uh encoder parameters and everything. This is just the default.

A

If we wanted to, we could save the csv.

B

And if we were to open that up, it comes.

A

Out in a nice drop format, foo.

B

And then at the type, and if there was a flag, it would be there too, and we can also go back and run it.

B

uh Unfortunately, for the running code here, like the scrolling code, isn't working, but it's running all of them in here we have basically 200 records of the green, the solid and then the blue are predictions. So you think it's a while for us to catch up there, but um and then the same as before. You can step through and see what all the different references are. You can see foo at the bottom here you can see the different encoding outputs, it's getting as we.

A

Step through and eventually you're gonna see.

B

A

Starting to predict these, if there was an anomaly, if I specified a normal score.

B

It would have been showing up down there as well. I can actually do that by just editing the parameter in here and the inference type I'll change it from temporal multi-step to temporal anomaly.

B

And run it again.

B

And you see the anomaly score going on right here uh and how do you use this to debug? Well, one.

A

Thing, you can say is like oh I'm using an adaptive encoder. That means it doesn't really know what the min and max is.

B

A

B

First, actually, so, if you look into the code, our default min and max for adaptive scale,.

A

Encoders is zero.

B

A

A zero here should get the.

B

Min value which you can see with the encoders, because all the bits are all the way to the left. The next.

D

One is the one so that actually.

A

B

The max value goes.

A

All the way to the right.

B

What happens after that? What's going to happen when we get to two well.

A

The logic for the adaptive encoders right now is that.

D

B

We just set the max to whatever the new math value is so.

A

Two is the new.

B

Max so it should be encoded as the max value again. So it should also have all the bits to the right and that's what happens and the same thing for three and four and.

A

You get back to zero, it should.

B

Be the min value again because that's now the min and max are zero and four. But.

A

B

Is not the maximum it's somewhere in between so the second time it comes somewhere in between and the third, so you can see the evolution of that encoder, all the time and the corresponding spatial polar representations.

B

You can see that one two three all get that same spatial, polar representation as well and then, but now after the second pass, they all get different. So you can see how the learning is affected by that.

A

What, if I want some more details about, what's going on in the spatial.

B

Pool of that, like I can't really visualize here, uh what you can do is there's a parameter here called the sp.

A

B

Which is how what level of debugging output you want to put, and usually these would come to standard out, so I can.

A

Set it from zero and set it to two.

B

And I'll run it again.

A

D

Again, this is artificial data set right based on the formula that you used.

B

One that you can see it's going all slow enough, because it's got to capture all that output, so 200 is going to take a long time, but you can see at every point also now I get all this data output so at every.

A

B

I can go through and it tells me like you know, what's connected how many things are firing. What's the density yeah inside the individual radius so- and I get that now at every point- it's a little slow when I try to scroll through, but you can see like this is at 0.56 like if you just check me a little hard here. Give me a.

A

B

A

One thing about using siri.

B

A

That revolves output can.

B

Get um very big very fast: you can.

A

Actually see, I got a.

B

Max document size, error out of.

B

But you know at times have seven I get that and types of eight these are.

B

I think these are just different column connectivities.

B

It's actually been more.

A

Useful in my experience.

B

To use the temporal floor output, so we should do that in a.

B

B

So here you can see which cell like at each point which cells were connecting to which other cells, if backtracking was happening, it had to go, it lost the sequence I had to go back. It'll tell you that they can tell you which columns are bursting or which new segments are created, and you get again you get that as a play-by-play for every record. So you can see why it's learning why it's not learning?

B

uh What's taking so long, so that's kind of all the debugging.

D

B

That you get from siri rose right here and again. Every parameter that you ever need will be right here on the left.

B

I don't think if there's any good.

A

B

To demonstrate, I don't think so robust I mean we can.

A

Play around with encoders.

B

And uh see the different resolutions, but.

A

Last thing I'll show: you is.

B

You can actually create some pretty interesting, uh interesting data sets here. So, for instance, if you want one that kind of depends on the history, you can.

A

And has some randomness.

B

You can say that say we're going to have a hundred.

A

B

B

B

B

It's a little bit noise.

B

So, like you know, ten percent of this will be.

B

D

This is uh this version is committed already, or is it.

B

A

This is going to be a saw wave just a little bit of noise.

B

In there, when I run that.

B

You can see that noisy.

B

B

So that's like a good way to simulate the noise from because, in all of our customer data, you never have super clean data like there's always a little bit of variability. So you can simulate that here and see how it affects the learning and you can see it takes a little bit longer to learn because of all the different sequences. Now.

A

B

You can do is, like I said before you can say that foo is uh basa is something that you know one two, two three or four, and you can also have conditions in here. So you can you.

A

Can say that foo randomly sample it? You can also say.

B

B

If the field of food, if it ended up being.

A

B

Then I want bar to be 40 or 10.

B

If it's 2 I'll make it 20.

B

You can do kind of weird conditionals like that and then run it.

B

And right now only foo will be shown because I've set the predicted field to foo.

A

B

D

A

I could also set it to bar, and then we could.

B

uh See that, but uh so foo is just going to be some random number in one two three four, but you can see that if you look at the raw end plate here whenever food is four like four would be a hundred. That's one.

C

Yeah, it's always going to be 100. If it's not one, because your else is uh you.

A

C

D

Bro, can you feed the real data in there.

B

uh Via the description files right now, you.

A

Have to create a description.

B

File for it, of course, using experiment generator, but.

A

B

See foo and bar at the bottom here so they're both being created and you can see how yeah it's straightforward.

A

Get 100 and the first.

B

D

B

D

But in order to to run you need, it only use this artificial.

C

Data you can't fit like customer data into it. Can you.

B

If you want to create a description file or for it, you can.

D

B

Yeah, that's why I was showing.

D

C

A

D

You can use the t to load from the right from a list on your function. Key the.

B

Oh yeah yeah, you loaded all the data into a giant array. That's literally, you could do that too.

B

D

A

B

This by now go to sandbox, you can actually open files or anything from here you know only exposes certain uh libraries.

D

uh The last thing you can do is like you could have something like this.

B

And you could say yeah on the first one come on.

B

B

B

Else, uh you can say have some temporary variable equal to.

B

Actually, let me say so for the first two times bar is going to be zero. Otherwise we're going to reach back two time, steps in history and get what uh bar was and plus one so.

A

You can actually make something that's dependent on itself or.

B

Depending on who, from two time, steps to go, I can create that what is temp.

C

B

D

B

A

B

Get the actually it's a little hard to visualize, um but trust me. It works.

D

B

But actually what another thing you can do is simulate a random walk, so say.

B

A

Want to say it feels if it's zero.

B

Is one um else.

D

B

So you reach back one and.

D

A

B

Plus random so we'll just add the.

B

B

A

What this should do.

B

Is it'll start off with food equals one and then every time it will take the previous.

A

B

And add a plus or minus 0.5 to it, because random. gives you 0-1 randomly, and so you can see how well drop. Oh, no.

B

A

Guys actually want to see that.

B

Nice, try! No I'm happy dude. I just don't want to bore.

B

D

D

A

B

Sometimes I do.

C

Maybe it's rounded up random dumb.

B

You see the air reporting isn't great.

A

A

D

Why does it go up so much? I don't know. I think I have to look at that.

A

C

D

At oh well, it should be.

C

Plus or minus 0.5 right well, if the initial value is positive, it might set a trend.

D

C

Actually, if you do it again, it might go down.

B

C

I don't know if the seat might.

B

Be set, though, let's see here's what I said um but yeah. I think this is one of those unstable equilibria where, if there's.

B

um Know you make a lot of fun data sets like that. Basically, with all these tools, I had one example. Let me see if I can find it that was like. I had a cleaner version of our hot gym data.

D

Set one question.

B

D

um So if you can go back to that.

B

D

B

It show you the prediction.

D

Made at that given time step or does it project it like into the future? That's.

B

A good question because at one point it had them lined up. So it was the prediction for this time set and the actual for this time step.

A

B

There's a bunch of code changes where that got lost and I.

D

B

It got me implemented, but I'm not a hundred.

D

Percent sure, because when you show the soft tube, even though the anomaly score was zero and it was perfect overlapping, it was like.

B

Yeah, so it may have not. That may not have been reimplemented on this chart. That's questionable.

B

I used to have one where I took our hot gem, one and kind of recreated it in.

A

A function like this just so I could.

B

Add varying levels of noise, so I won't actually do it all, but I had like a day of week.

B

Variable that was t mode.

A

B

Was thinking that t was each hour so the hour of the day was that.

D

B

Be every every t is an hour divided by 24 hours of the day index four a month. Seven is the day of the week.

B

Top day was t 24 and you could do things like you know. If day of week, you know if it's five days.

A

D

A

B

B

And then you can add varying levels of noise. That's a good way to defend everything, uh yeah any questions in general about cerebral, that's, basically it that's how you run it. Those are all the different features in there. What libraries are available from from this uh from.

D

B

They are random, numpy string and math, but it's a really easy code change to add more questions.

A

C

Sandbox will still, let me write an infinite loop right. Oh.

A

C

B

It's not a great sandbox.

A

B

C

Just thinking about I'd love to put a version of this online, but there's a lot of security issues.

B

Well, the function, data something can be optional if you were to have a hosted version. That's.

C

The coolest thing it is kind of crazy, but um so we might be able to harden it. Maybe yeah, um and also the verbose output, probably wouldn't be able to run like a hundred thousand steps, because that would be true.

B

Although I will say that one of the benefits I've had with working with this is, uh I don't know if it'd be suitable for a hosted version, because what I would do is I'd, modify the cla code and then run this.

C

Oh right so yeah, I just want an example to show people.

A

C

This introduces a lot of concepts to people.

A

C

The white paper, but I mean, if you've, read the white paper, you understand the theory. This gives you a much better understanding of how it's implemented. I think.

B

A

B

Yeah, I think that's a great.

A

Idea so yeah, it says on the.

B

Generator plus some version of the.

A

Yeah and then for girls athlete you can.

D

Meet like for just an.

B

Example, maybe you don't need.

D

It yeah just wanted to ask what uh was it implemented? How was it.

B

How's it implemented, so I have a web dot, pi backhand um and again this was my foray into learning about how these things work. So it's probably not written in the way. It's meant to be right here. um So it's got that and the running experiments actually run happens either.

D

B

A separate thread or a separate green light to try to keep it responsive. So as it's crunching the data, you can click on stuff and then just uh jquery and boot trap on the front end for all these things,.

A

D

B

For the these little dot, charts and.

A

B

For the storage, although it doesn't have to be that was just. I also want to see those deals with okay.

D

B

D

For security purposes,.

A

D

uh Re-Implement this function in javascript on the user.

A

C

B

Yeah just send records.

A

B

A

If they freeze.

B

Anything it'll just freeze the browser.

A

B

Yeah, that's totally, and uh the library I'm using for, like is ledger called code, mirror to do the highlighting and stuff it already.

A

B

A javascript theme too, so it's.

C

B

C

Other questions about it.

D

C

B

You guys use it more. That's the first time I've understood adaptive encoding, just seeing an action like that. So.

C

That's awesome.

A

B

And then so that's actually the reasons why you know it's. Sometimes it's issues is because that first run or whatever, at least in these examples, it was just the first one, but that spike can happen anywhere. Yeah.

D

And you can even like see I'll fly.

B

huh And that's: why we'll break this break that adaptive encoder, but you can actually even say.

B

A

And you can just make it a non-adaptable skull.

B

You can say it's a scalar encoder, you can say the min valve say you want it like for some reason clip it. So two.

D

B

No, it doesn't have to match.

A

B

Min and max of the data so.

A

What it does is it eclipse if you.

B

Go beyond so what should happen.

C

Here, if you have to.

B

Did I not do that yeah? I did not you guys.

D

B

Thing better than I do. Yes, I create it and then.

D

The problem is right now it.

A

B

Like it, it regenerates.

A

That so, you kind of have to.

B

A

To make your changes again,.

B

And then you run it pop.

B

It and- and you can see here that if you look at the encoders uh first of all the first one if we look at it, should not have that issue anymore of like uh like zero than max max max. But zero.

A

B

My family is having that issue.

D

Yeah, usually you don't have.

B

To usually after you bounce out, you should just do it.

B

Oh sorry, this for some reason there's this new thing underscore classifier input. I don't understand. What's going on with that, so there's like this field f, you can see kind of appears twice in this classifier thing and then, as the f over here.

A

B

Have to apparently enter this one.

B

So update recreate it.

B

Let's see what goes on with this, so first one should be zero yeah, so you can see now one doesn't go to max again: zero. One two are all min and threes in the middle already have four five. Six seven eight will be max and.

A

B

See the clipping happening and how the beginning isn't affected by all that stuff. That looks like hot gym.

D

The prediction's deal.

A

D

A

Can kind of understand.

B

That. Because all everything after four looks the same to.

A

The cla, so it's going to predict the same things so yeah.

B

It's a fun little player to mess around with it.

B

Thank you guys for hearing me out. Thank you very.

C

Much yeah that was cool thank.

B

B

Okay, like I said it's all checked in including that little powerpoint.

C

Soon, to be open source.