Numenta HTM Community Meetup 14 Mar 2016, 14 Mar 2016

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Advanced HTM Processing with Apache Flink - Eron Wright

Description

HTM Community Meetup - March 14, 2016
https://github.com/nupic-community/flink-htm

A

I'm going to talk about integration project that I'm working on was recently published to the new community site. This.

B

A

Effort to integrate one of the implementations of HTM with a popular emerging streaming framework from a Chico epic funny I'm, going to tell you words about a Pecha point because I mean on further, but it is similar to apache spark, but it is a intended for distributed. Real-Time stream, processing of information like from Casca or ETFs.

A

Can you do it on top of yarn clusters or a scallion clusters, and well I'm going to walk there. Today is a project that allows you to use HTM to do an ollie detection of information as it flows through these distributed streams.

A

But when you're working with HTM, it's a library and you're likely going to integrate it into maybe a big data solution. These are the kinds of problems that you might face when doing that, you might need to I need.

B

To a great my data before.

A

Feeding into an engineer, network or I need.

C

A

Many independent models in a distributed environment in a high scale, maybe I want to connect to a cafe data source or drive outputs to HDFS.

B

A

Maybe your data is coming in out of order and you'd like to reorder the information as it enters into a scam since.

B

A

Very important when you're doing temporal learning and finally, maybe you need reliable stream processing role. We measure this induction care about checkpointing and reliability mechanisms, and so with that link becomes very interesting homeland HD, because these are problems that flings designed us all.

B

A

Say he works hopefully in itself now the project is a top-level project from Apache it just reached 1 point 0. It was a research project that was first developed by consortium of German recipes funded by the German Research Foundation, and there is a start that commercializes point artisans input.

A

B

Are some of those features that.

A

People talk about the flip. Flip provides a programming framework where you write jobs or programs similar to action, sparkle, Oh or do MapReduce jobs. Only. You use the language of plank out of a gadget point, and you do this with Java Scala the pipeline.

A

Fling provides features like a distributed data flow graph. Anything you describe the problem you're trying to solve and then a place.

C

Of optimizer deploys.

A

An execution graph on some custard to execute this distributed processing. It works with unbounded strings, meaning the applets. It has paired powerful reliability and state fulness primitives.

B

Minute so, for example,.

A

Htm network is for sustainable entity. It's cumulating state as it.

B

A

That's that's integrated into place, check, pointing and recovering mechanisms. So if a worker and over a crash we can actually recover the model is being trained on machine.

A

There are extremely powerful data manipulation primitives, with inflate flame, provides when doing support. So you might say, show me but think this stream of records and every 10 minutes like to look at it, maybe do some aggregation or find.

B

A

Value this, so what's the art of that it supports partitioning and that's very important when you've got a large-scale stream, it may be partition. For example, kappa has a as emotional partitioning and with flame you can actually training norris HTM models in parallel across those different partitions of the strength it supports joint. So we can actually bring the spirit, streams of information together and using time lingos we can join them in outer or inner joins, and those can become aggregate records that we then pushed into HTML and finally, link provides iteration support.

A

So it's the graph is up as a bag that can have loops in it. So the open from something can become the input to itself.

A

Now people ask me: I'ma, show you some pictures first to some of the things you just described. This is uh this is plain website and they claim we win. Here's.

B

A depiction of.

A

This ordering problem where information that's arriving into your system might not be temporarily work because a dried in creation from different sites and some of them have a length so flakes.

B

Actually, work.

A

With those supplements, as they come into your system,.

A

This is a depiction of the beautiful wrath and some.

B

Of the checkpoint is.

A

Embedded supports I just.

B

Want to give you a visual sensitive, this grass.

A

Well, the new support millions is between blink and spark stream. Is that great deals entirely in? What's called the enzyme, which means when we construct windows Satan, say, give us the last every 10 minutes worth of data. This idea of 10 minutes is not eating clapping or so it means look at the time stamps inside the messages for records and your input here, Justin and organize the information in that way. So I want every 10 minutes. In the event time domain.

B

A

Flip is really amazing: I was invest in class streaming processing system and, while spark is really great, too I love start quite a bit and done a few projects in spark and M shimmering together. I think that HTM in Flint, the best friends because of HTM, is intended as a screaming temporal learning and that just SAS military connection with the massacres.

A

B

I'm going to do a couple demonstrations.

A

For you, I show you to get apps that are included in source tree very negative. One of them is hot gym, we'll see how to the myth- and we will see New York city traffic I'm, going to walk through actually I need the race motorcycles. All I'll show.

C

You a little bit how to.

A

Htm traffic stops near see traffic in life blog boy, sir.

C

C

That's community supporters. Do hey thanks a lot.

C

So, let's, let's first compr minute of a Love Machine. Maybe you are the other side move.

A

The mice day, ok, so I'm. This group is again a little app that shows the use of salt river view, which is a cloud source of temporal data and expertise and drive the state it through a flight data flow. Well, this picture.

B

A

That removing the service actually.

B

A

Partitions Oracle streams of data and we're going to do is read those streams in parallel using a flame data source of the river swords, there's a paralyzed sore second scale, and if, as pulls in these strengths, it can drive them each of the set of HTM network instances. There will be one instance / straight so we'll learn an anomaly score for each dependent streams, and this will happen throughout our fleet cluster, where each a shadow process finally will aggravate intimately scores into a result set using a projection.

A

Okay, so first thing I'm going to show you my track model I'm going to play applications it using your own data types that come from your domain is very important. It's a strong type system and as Varitek you can work with solid, a slug guns which I pojos tools, you're all typed. In my case, I, don't work.

B

A

Paste box, I've defined all the traffic report, the.

B

Meda traffic report.

A

First, wants to some stream some time.

B

A

Speed some couples.

A

I'm just going to show you have team yet, and this is the data source for the drawing on. This is the NYC traffic resource and it's.

B

A

You see provided by dimension and it defines the ski and trouble and if I show one of these strings, really quick.

A

You'll see that you get from the five.

B

So first thing my dad's I find this.

A

Case class from all the second thing, I'm doing is writing network definition, and this defines that that hmm past information through this is a very simple single layer, civil region and because I'm not an expert in the use of a sham this summer girl.

A

But if adverse the purpose of editors and everything isn't others when you put a sham is earlier, you must take data that comes from your main.

B

A

Transforming into msgr the encoders are the way english that's published, and so you can see here. I use a date encoder for the eighth time and the scalar encoder.

B

A

With some routers.

A

This here is the shackle for a typical pleasure.

A

What will happen is we'll create an environment object, which is the context. Object is plenty and we're going to write a series of data transformations.

A

We are constructing the graph and when we finally has to keep the wrath, the jobs actually deployed and run at some scale.

B

A

He said we're going to work with we're going to create a versus using a track record.

A

And the purpose of four see here: I.

A

Wish I let it select a strengthened less, but if you need to see my tea party it all the sense now.

A

So I'm going to give this.

A

B

Just going to register this source midplane.

A

I'm ready this is Allah is solid, is wonderful, no offense to the closure post resonance, which.

C

Are using these two elbow, it's Devon.

A

Yes, citrus saw and out longer hold on, but when you're seeing here in some of the mission.

B

Mars is a solid ESL for.

A

A ship out allow me to write and working conditions.

A

Solo, this output so far we're going strong. Is it totally order series of traffic report elements a strong word, and you know what I forgot? Oh here here have a super one and you just see it that the strings now.

A

So they see, I was really bored with the type information. So all the times we work in the native types of language and it's just a really well secretaries.

A

Now this issue like oh well, that she's learning now you find that and I want to do- is select.

A

What's going to happen, is one or more mechanical instances will be created and how many partitions of toys.

A

And a series of inferences I mean we need to know this project project, the output, power, Lisa secrets- and this is when the dude screws- two bolts just pair of information and assistance- is only for together with original it works, and this comes to Polly. Mitch of this part.

A

B

Going to be my.

A

Now any chef here, what the injury is the greatest ring. So if you look at the title information of this and has to go enemy ship, you can see that this is the data stream contain two bowls of your chakra fourth plus in office.

A

We can change these together as like that, but let's just now running this.

A

You can see you know, my processors are all right, because electric trains newer, songs fairly and you can see from an output, will quit the second. You can see the circle thread. Gator some far left you can see. There are numerous all being trained device if I stop this and you're, seeing not only stores all right, you're, seeing Fred left and with your eyes. Well,.

A

Actually is trained in a fresh.

A

A

Competitions like it could be some other property industry. My traditional training hall.

A

For I skin I did you expect not.

B

A

Forbidden those.

C

Cheese and bread.

B

A

When you jump ahead to Elmo's okay, well number one I'm going to show such marvelous form.

A

Okay, so this is the final form you will see in the source tree right now. This is something that replicates the end.

B

A

The tutorial video that's Matt put together what you see is obtaining the traffic data from riverview, creating with all investigation interval on august third through four, and we want to find anomalous rights.

A

So we assemble those and only scored, but I really only want to learn up until the point when I get to the end of the investigation.

A

So I'm just filtering the reports up until the investigation ends and then first then take those stores and specifically call out the ones that are in the investigation. Time window and I want to pick up once it happens, and so when we were kissing.

A

Okay, so now the switch gears will be run like a hot Jam. Hot Jam is very similar and won't miss much to the focus time get a good look previous reminders from work at the moment. What part is STM, what what? What? What is important, what purposely well the way that these this is like a fluent syntax is typical, just plain, and you put operators into this syntax to.

B

Make transformations.

A

Over these data streams, as you go and it's a whole bag of transform, so that's not a single flow can be made a direct.

A

It's a graph of transformation technique who, in and out even has a statement back on themselves, so.

B

The very rich language.

A

For building these stream flows, but.

C

To answer your.

A

Question at the what we put this learn element. um That's the point in which we did certied HTM operator in to show.

C

The network creation, where.

B

You're, putting together the HTM.

C

Components that built the network I think that's what he's asking that Network instance.

B

You're sending into the learn function. That's.

C

Where it just happens, the big coder creating the region's and moving them together, yeah.

A

So Gilles wish to hot Jam, for example, here's the classic network definition of hot jam and again it has a time stamp as a consumption.

B

A

Then at the point at which the point of actually actually has get to.

B

A

Have supplied them as input and and many.

C

Instances any instances of this network.

A

Would be created.

A

Oh yeah, if you think you can see from differences only underline under the hood, where little grander line that's intelligence, weight telling you that this is what's called an implicit conversion. What oh I think the.

C

Scholar has a.

A

Capability, similar to c-sharp, where you can extend in classes that you don't write, you don't own with new functionality and the compilot person to do so.

B

A

Is actually extending the data strange support, op anomaly? What are you having to find better graphic model? Three? No. What happened was at the top of the top of the.

B

A

We import arse ecology itself for from nominee attention to play, and then what happens is this is this is some solid business, but this is all I should mention that this is all accessible to java programs as well. The dsl elements not stand so if you saw a java code and you want to use HTM inflamed, that is perfectly fine, and that makes you use a static as a static, Reggie pointer to page system.

A

So I'm going to run this now online cluster. Let's just run our option.

A

Here, I'm going to take just the online text here, going to run what is going to do. Is it's going to read the power, the CSV, this class projects and we're going to write out some knowledge.

B

A

This has some nice x couple, but let's actually switch into flint see why? What we'll see here now? Is it Flint the show? But my cluster has four tasks. Waffles means it's April over and four things in parallel, the sections of the local cluster just running on my map um fling this type of oh no scripts for the various public clouds as well, really scale sup.

A

What we see is that I have a completed under there. Just ran for 25 seconds, and it's kind.

B

Of a nice charts at this show, this is a.

A

Little graph, you see running from the concept recall source through HTM network through to some sandwiches ready.

A

B

Is some metrics in there as well? So we can see.

A

That our HTM stuff processed 4300 records with a total millisecond time, twenty-three thousand milliseconds. What does that mean 45, and it was only one.

B

A

Then, in the end expect.

A

So here we needed it to the class of concept evolved, containing the timestamp actual predicted.

A

Words over this farce will go from here.

A

So we walk through a line cook like John government there's.

B

A

More things to do next to make this really raw number one.

B

A

Check pointing stuff of life is so powerful, but it depends on ability to serialize your state. Well guess what HTM town doesn't support serialization today, but.

C

Should and will to make this really work. Just people get serializable like your dog right well,.

A

Didn't I chat about how to see on that he knows best. What to do there will we.

C

B

Done tonight we.

A

Can we can have a totally integrated sharpening system when.

B

You bring more.

C

About those memory, it's.

A

Gonna blow your mind, it late, let's say you're, drawing on a half the screen and you blew up both lenses system recovers. The offset into the resource Africa is actually retained. Edition plug data so.

B

A

Back to where left off in the half the string, it can hook in chi exactly once, processing of sharing information from pathway through HDFS and we get serializing. Htm models will be there as well all right. The river.

B

Source actually.

A

Quick to expand the string, what I'd really like to do is sleep on the interval is defined in the sources that just write next thing.

B

A

Is a pretty great basis, I think to do swarming on HTML I would like to demonstrate the viability of that understood little more. What happened is hard after.

B

All this is a distributed.

A

System and can easily stand up p.m. on licenses for us and I do believe looted. We could use it as the basis for Sauk Valley I. Don't have support memories.

A

C

Use the question: swarming module.

B

A

Essentially, you hand code your model today, but there is. There is august wilson, interesting anything else. You swarming on top of the java library. This.

B

A

We're supposed to the mini site, I just got started with it: there's a little bit eat it show you I run these animals and bill is why I like to hit some resolutions and see where this goes thanks again,.

A