►
Description
How to use ML.NET in an Event Hub consumer to get real-time data predictions.
C# Consumer code - https://github.com/jwood803/MLNetExamples/blob/master/MLNetExamples/EventHubPredict/Program.cs
Python notebook producer code - https://github.com/jwood803/MLNetExamples/blob/master/MLNetExamples/Notebooks/ML.NET%20Event%20Hub%20Producer.ipynb
ML.NET Playlist - https://www.youtube.com/watch?v=8gVhJKszzzI&list=PLl_upHIj19Zy3o09oICOutbNfXj332czx
Contact:
Twitter: https://twitter.com/JWood/
Blog: https://jonwood.co/
Gear used (affiliate links):
Mic - https://amzn.to/2YEXtxI
Mouse - https://amzn.to/2ZtASoQ
A
Hey
everyone,
so,
as
your
event
hub
here
is
real-time
data,
ingestion
service
and
so
I
figured
since
you
can
do
real-time
data
ingestion.
Why
not
be
able
to
do
real-time
data
predictions
with
Annette?
So
that's
what
we're
gonna
do
in
this
video
we're
gonna,
use
event
hubs
and
then
use
in
Medinah
to
make
predictions
on
streaming
data
from
event
hub.
So
the
first
thing
we
need
to
do
is
and
then
there's
your
portal
here
and
we
need
to
create
a
new
event
hub.
A
So
I'm
gonna
create
a
new
resource
now
I'll
just
type
in
an
event
hub
here,
so
give
it
a
name.
I'll
do
see
in
monnet
event
hub
is
that
available
pricing
tier
I'll
just
do
basic
put
my
subscription
put
it
into
a
resource
group
and
give
it
a
location
and
I'll
hit
create.
So
that
might
take
a
few
minutes
to
create
and
so
well
let
that
do
its
job
and
deploy
all
right
so
that
deploy
it
successfully.
So,
let's
go
to
it.
A
One
thing
to
note
when
you
create
an
event
hub
here,
is
that
you
actually
create
an
event
held
name
space,
not
exactly
an
event
hub
itself,
so
to
do
that.
We'll
click
on
this
event
hub
a
head
button
here
and
we'll
give
it
an
its
own
name:
good
ml
net
age,
all
right
and
we'll
just
create
that
and
that
got
created
we
can
go
down
here.
A
All
your
event
helps
will
be
down
at
the
bottom
here
and
we'll
click
on
that,
and
now
this
is
where
our
event
hub
instance
is
so
we're
gonna
leave
this
up,
because
we're
gonna
need
this
to
get
some
connection
strings
and
all
that
let's
go
to
visual
studio
and
I
already
have
a
dinette
core
console
project
loaded
and
so
in
here.
I
have
a
model
folder
and
inside
that
I
had
the
housing
model
file
already
loaded
and
we'll
be
using
that
for
our
predictions
and
that's
asking
NuGet
packages
here.
A
A
This
is
so
we
can
serialize
and
deserialize
Jason
and
T
string
and
all
that
feel
free
to
use
Newton's
salt,
if
you're
more
comfortable
with
that
I'm
just
gonna
use
Microsoft's
implementation
of
it.
Alright,
so
remember
when
I
said,
we
need
to
get
a
couple
of
things
from
our
as
your
portal
here.
Let's
go
ahead
and
create
some
constant
strings
for
them
or
do
the
event
hub
connection
string.
A
And
it
would
do
an
event
name
so
go
back
to
our
portal
here.
Our
event,
held
name,
is
what
we
created
up
here
in
the
net,
the
age
for
event
hub
for
the
connection
string,
I'm
gonna,
go
to
shared
access
policies
and
then
add
a
new
one
and
I'm.
Just
gonna
do
manage
here
and
I'll
just
give
the
name
of
manage
policy
and
create
it
all
right.
A
So,
let's
created
we'll
click
on
it,
and
then
here
is
where
we
can
get
connection
sharing
for
the
event
hub
and
we'll
just
paste
it
in
here
and
our
main
thread
here.
I'm
gonna
get
a
consumer
group
and
that's
gonna
be
from
the
event
hub
consumer
client
from
the
package
that
we
just
downloaded
a
default
consumer
group
name
and
also
need
to
get
a
storage
client
for
using
the
new
blob
container
client
instance,
and
we
get
this
as
well.
A
This
comes
in
from
that
event
hub
NuGet
package,
so
we
don't
need
to
do
that
separately
and
then,
here
we
need
to
give
it
a
blob,
storage
connection
string
and
a
blob
container
name
already
had
these
defined
up
here
in
this
block
connection
region
uses
to
pass
in
into
our
event.
I've
been
hooked,
client,
and
this
is
mainly
used
to
kind
of
keep
track
of
all
the
events
it
had
been
read.
So
it
doesn't
reread
any
of
those
events.
A
So
next
I'm
going
to
create
a
processor
with
new
event,
processor
clients,
control
dot
to
bring
in
the
namespace,
like
the
others
and
I
pass
in
the
storage
clients,
the
consumer
group
they
recreated
above
there
and
then
the
event
hub
connection
string
and
then
the
event
hub
name
that
we
created
just
up
here.
That
gives
us
a
new
event
hook,
processor
and
then,
with
that
I
can
do
processor
that
process
event
async
and
here
I,
give
it
a
method.
A
I
said:
I'm
subscribing
to
this
method
here,
subscribing
to
this
event,
pretty
much
and
now
create
another
method
here
called
event
a
process
event
handler
and
then
also
in
the
processor.
We
can
call
process
err
async,
so
anytime
the
errors
may
happen.
We
can
listen
to
those
and
process
on
the
quarterly
and
I'll,
create
a
process
error
handler
nothin.
So,
let's
create
these
clear,
do
private
static
task
and
bring
in
that
namespace.
A
The
first
thing
I
do
I
was
create
the
process,
error,
handler
and
I.
Think
here
and
that's
gonna,
give
us
a
process
err
event:
args
Rahmani
in
here
I'm
just
going
to
write
out
a
couple
of
things:
the
partition
ID
an
area
has
occurred
on
it
and
then
I'll
write
out
to
the
console
the
exception
message
and
since
it
returns
a
task.
I
do
return
task
back
completed
task
and
it
should
be
process
area
event
handler
alright
and
do
something
similar
to
the
process.
A
Event
handler
nothin
here
takes
process
event
arns
as
a
parameter
and
here's
where
we're
going
to
do
all
of
our
work
and
for
event
hubs.
You
need
kind
of
two
two
sets
of
code
pretty
much
this
one
we're
doing
here
in
c-sharp.
This
is
what
we're
gonna
be
using
the
process.
New
events
that
get
into
our
event
hub
and
a
little
bit
of
this
will
do
another
one
where
we
actually
produce
events
that
goes
into
our
event
hub.
A
A
So
we
need
to
make
that
into
a
string
of
those
bytes
and
to
do
that,
we
do
encoding
that
utf-8
and
let's
bring
in
this
namespace
of
the
system
that
text
namespace
and
from
there
you
can
call
get
string
that
takes
in
an
array
of
bytes,
but
because
it's
read
only
memory
of
bytes
I
need
to
call
to
array
on
it.
I
said
I.
A
Guess
me:
the
string
representation
that
bytes
data
that
comes
in
from
our
event,
that
is
a
that's
gonna,
be
a
string
and
we
need
it
into
our
prediction:
output
class
that
we
usually
do
with
our
in
Linette
models
and
to
do
that.
I'm
gonna
use
that
system
that
text
ID
Jason
nougat
package
that
we
used
in
my
call
Jason
serializer.
A
A
Okay.
So
it's
just
like
usual.
Let
me
paste
in
what
I
had
before
so
I.
Don't
you
know
see
me
type
in
all
this
there
good.
This
is
the
pretty
much
the
same
thing
that
we
see
any
other
videos
what
we
do
with
the
housing
data.
We
left
a
low
column
and
call
name
attributes,
and
you
probably
need
those
but
I
just
left
a
man
just
penis,
alright.
So
now
that
we
have
our
data,
let's
first
create
a
new
in
context.
A
We
bring
in
this
namespace
now
we're
gonna
get
the
model
and
to
do
that,
we'll
do
context
that
model
dot
load
and
this
takes
in
a
string
of
where
the
model
is
located
and
that's
in
that
model.
Housing
model
that
zip
location
that
I
showed
earlier
and
it
returns
out
a
data
view
schema
of
the
input
schema.
Okay.
So
and
now,
since
we
have
our
model,
we
can
create
a
prediction:
engine
using
context
model
create
prediction:
engine
use
a
housing
data
only
to
create
a
housing
prediction
class
as
well
and
pass.
A
All
right
so
with
the
prediction
engine
created
I
can
now
create
a
prediction
using
that
prediction
engine
and
call
predict
on
it,
passing
the
data
and
then
now
all
we
have
to
do
is
we
can
write
out
to
the
console
and
we
just
do
prediction
is
prediction
predicted
house
value
and
then
I'm
gonna
return
task
completed
tasks
summer?
What
we
did
up
here
in
all
right?
A
So
that's
our
consumer
app
right
here
now:
let's
I'm
gonna
switch
over
to
a
Jupiter
notebook
and
show
how
to
use
Python
to
talk
to
our
event
hub,
and
we
can
use
that
to
create
a
producer
script
to
give
us
our
data,
to
put
it
Dalton
right.
So
I
have
Jupiter
notebook
created
here
to
be
able
to
talk
to
my
vin
hub
and
Python.
I
need
a
pip,
install
Azure
you
vent
hub
and
using
the
exclamation
point
here.
I
can
run
shell
scripts
and
I
already
have
this
installed.
A
So
that's
cool
and
a
couple
of
imports
here
from
your
event
hub
that
a
import
event,
producer
client
and,
from
your
dad
event
hub
and
port
event,
data
and
I'm,
going
to
import
the
JSON
in
space
as
well.
Next
I'm
going
to
create
a
variable
to
hold
that
connection.
String
and
I
can
copy
this
once
again.
A
So
next
we
do
another
producer
with
the
event
producer,
client
and
then
I
can
call
from
prediction
from
connection
string
and
pass
in
a
connection
string
from
our
variable.
There
and
and
I
can
also
pass
in
the
event
hub
name
which
I'll
give
from
here.
Since
we
already
used
it-
and
we
have
the
producer
now
so
now-
I
can
create
some
event,
data
that
can
wait
on
it
in
Python
and
call
producer
that
kraid
batch.
A
It's
it's
just
a
batch
in
our
our
data
and
send
it
all
at
once.
So
we
don't
send
anyone
that
one
at
a
time
and
then
I'm
gonna
create
a
data
object,
and
this
is
just
going
to
be
the
housing
data
object
and
I'll
paste
that
in
here
it's
just
some.
Some
random
data
that
I
used
will
call
event.
Data,
add
it'll,
be
a
new
event
data
and
in
here,
I'm
gonna
use
JSON
that
dumps
in
a
pass
in
the
data.
A
So
what
this
does
is
that
it
takes
this
dictionary
data
and
then
it
dumps
it
into
a
string
into
a
JSON
string.
When
you
send
in
event
hub
data,
it
does
have
to
be
in
a
string,
but
you
can
pretty
much
give
it
whatever
you
want
in
JSON
format,
so
that
adds
to
the
event.
Data
batch
and
next
I
can
call
a
weight
producer
to
sin
batch
and
then
sending
that
you've
been
datum.
Let's
go
ahead
and
run
this
and
send
it
to
our
event.
A
Okay,
all
right,
so
that
ran
so
we
should
have
a
message
in
our
event
hub
all
right
so
back
to
our
consumer
c-sharp
code
here
and
we
actually
have
to
finish
our
processing
steps.
So
we
set
up
our
processing
events
methods,
but
we
have
to
tell
the
processor
to
start
processing
and
we
used
to
start
processing,
async
and
then
I'm
also
going
to
do
a
task
delay
and
to
wait
a
few
seconds.
We'll
do
just
do
two
seconds
here,
we'll
wait
two
seconds
before
we
tell
the
processor
to
stop
processing.
So
we
start
processing.
A
We
wait
two
seconds
for
the
process
and
then
we
stop
the
processing
of
our
event
hub
data
here.
So
let's
run
this
and
we
should
pick
up
something
here.
There
go
so
picked
up
our
item
here,
so
we
have
our
payload
there.
It
is
and
we'll
deserialize
it
look.
If
you
look
at
our
data
here,
we
didn't
get
that
data
deserialize
correctly
from
what
our
payload
shows
so
see.
A
A
So
how
do
we
fix
that
then
use
an
attribute
called
Jason
property
name
and
give
it
the
name
that
we
expect
that
comes
in
in
our
JSON
data,
so
we
use
on
the
tude
in
this
way
it'll
match
on
this
JSON
property
name
instead
of
trying
to
match
on
the
actual
c-sharp
property
name.
So
we
do
that
throughout
the
rest
of
these
properties.
Real,
quick,
all
right
so
have
all
the
rest
of
the
properties
on
here.
A
Note
that
the
median
house
value
doesn't
have
that
property,
because
this
our
label,
so
we
don't
expect
that
to
come
in
and
our
JSON
data.
So
we
don't
need
to
map
that
out
all
right.
So
let's
run
this
again
and
see
how
this
works
all
right.
So
now,
I
got
it.
I
get
the
payload,
so
we
got
some
items
and
a
deserialize
and
let's
make
sure
okay,
so
I'd
realized
correctly
this
time
and
I'll
continue
on
here.
A
Oh
and
I
forgot
to
add
consoled
wreath
on
here,
but
now
that
we
rented
this
already,
let's
send
this
data
again
as
another
batch
since
it's
already
processed
that
first
one
and
that's
run
this
again.
You
should
pick
up
again
so
picked
it
up,
we'll
continue
on
here
and
there
good.
So
now
we
have
a
prediction
that
data
predicted.
That
house
would
be
worth
two
hundred
and
sixty-six
thousand
dollars
and
we
can
reformat
this
to
currency
if
we
wanted
to,
but
the
I'll
end
things
there.