►
Description
Presented by: Avinash Sooriyarachchi
Streaming data processing has moved from niche to mainstream, and deploying machine learning models in such data streams opens up a multitude of possibilities. However, most data teams shy away from this approach because of its complexity. In this talk, through retail industry-specific applications, Avinash will explain how Databricks Delta Live Tables, AutoML, and Github Actions can be used together to make production-grade streaming machine learning simple and feasible.
As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub
Thanks!
Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github
About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com
A
A
And
machine
learning
used
together
is
incredibly
complex.
Using
the
right
tool
set
can
simplify
it,
so
you
can
get
to
Value
quickly,
I'm
Avi,
sirachi
and
I
work
as
a
Solutions
architect
at
databricks.
I
love
my
work
because
I
get
to
work
with
machine
learning,
engineers
and
data
scientists
and
organizations
across
the
United
States
to
solve
business
problems
with
machine
learning.
Speaking
of
native
bricks,
we
are
the
inventor
and
Pioneer
of
the
data
lake
house,
a
topic
which
I'll
come
back
to.
A
We
have
strong
routes
in
open
source
with
Apache
spark
ml
flow
and
Delta
Lake
all
coming
from
databricks.
Currently,
we
are
well
over
4000
employees
strong
and
are
growing
up
rapidly.
Now,
during
the
course
of
this
talk,
I'm
going
to
mention
the
term
lake
house
a
lot,
it's
actually
Central
to
the
topic
and
not
just
the
buzzword.
A
So
it's
important
to
describe
what
it
is
before
I
dive
into
streaming
and
machine
learning,
so
the
databricks
Lakehouse
platform
is
unique
in
three
ways:
one:
it's
simple
data
only
needs
to
exist
once
to
support
all
of
your
data
workloads
on
a
single
common
platform.
Two,
it's
open!
It's
based
on
open
source
and
Open
Standards
to
make
it
easy
to
work
with
existing
tools
and
avoid
proprietary
formats.
Three
is
collaborative
data.
Engineers
analysts,
data
scientists
and
machine
learning
Engineers
are
able
to
work
together
much
more
easily
without
any
artificial
silos.
A
A
It
was
fairly
simple,
as
you
can
see
here,
a
simple
UI
and
a
service
in
the
back
end,
integrating
user
input
with
natural
language
processing
and
returning
the
analysis
to
the
end
user,
but
one
of
the
problems
I
had
was
how
do
I
distinguish
between
users
and
Bots
based
on
clicks?
These
clicks
would
be
continuously
generated
and
the
process
of
distinguishing
between
Bots
and
real
uses
would
need
to
be
done
using
an
automated
process
in
a
cost-effective
manner.
A
Now,
for
a
solo,
startup
founder
in
an
incubator
in
Iowa
calling
away,
this
was
too
much
of
a
challenge
to
handle
so
I
put
it
on
the
back
burner
and
focused
on
more
important
things
to
keep
the
business
afloat.
No
surprises
I
never
got
back
to
it.
This
problem
is
not
unique
to
my
past
self.
Now
imagine
you're
a
developer,
an
e-commerce
company.
You
have
the
same
problem
instead
of
Sellers
and
bias.
You'd
have
some
bot
generated
clicks
too,
and
you
want
to
detect
the
Bots.
A
You
need
a
way
to
automatically
detect
bots
in
your
continuously
generated
click
stream
data.
So
a
solution
that
uses
machine
learning
to
detect
these
anomalous
users
who
outliers
in
the
Stream
of
data,
would
be
ideal
for
this
to
come
up
with
a
solution
for
this,
we
need
to
take
a
look
at
a
couple
things.
First
of
those
is
streaming.
The
definition
of
streaming
data
is
data
that
is
continuous
and
unbounded.
A
There
are
a
large
number
of
streaming
data
sources,
change
data
feeds
from
databases,
click
stream
data
from
websites
like
the
one
we're
talking
about
machine
and
application,
logs
application
events
and
mobile
and
iot
data.
As
a
matter
of
fact,
most
of
the
world's
data
is
streaming
data,
and
your
organization
probably
has
lots
of
it.
Now,
let's
go
a
little
deeper
stream
processing
is
different
from
the
traditional
definition
of
processing,
traditional
processing
loads,
the
data
into
a
storage
location
to
create
an
artificial
boundary
and
then
processes
the
bounded
data
in
a
one-off
manner.
A
So
this
is
a
two-step
process.
On
the
other
hand,
stream
processing
is
continuous
and
unbounded
just
to
be
clear,
I
practice
processed
a
whole
lot
more
than
just
data
processing,
but
processing
data
in
batch
or
in
stream
is
a
core
part
of
why
it
does
really
well.
Another
key
aspect
of
Apache
spark
is
that
it
has
a
consistent
API
for
both
batch
and
stream
processing,
but
they're
using
SQL
or
python.
A
lot
of
key
constructs
in
traditional
batch
processing
are
carried
over
to
Apache
spark
structured
streaming.
A
This
is
of
processing
data
in
a
single
batch
data
is
processed
in
a
series
of
micro
batches.
As
you
can
see
in
the
diagram
you
could
think
of
the
data
stream.
As
an
unbounded
table
view,
data
in
the
data
stream
is
equal
to
new
rows
appended
to
this
unbounded
table.
Many
people
associate
streaming
with
low
latency
processing,
but
this
doesn't
necessarily
have
to
be
the
case
all
the
time.
For
example,
you
can
use
structured
streaming
to
perform
incremental
processing
periodically,
for
example,
daily
or
hourly.
A
We
can
go
as
far
as
stating
that
batch
is
just
a
special
case
of
streaming.
If
you
look
at
a
long
enough
time,
Horizon
write
the
Syntax
for
the
transmissions
in
a
streaming
pipeline
is
one
thing:
putting
it
in
production
in
a
reliable
and
scalable
manner
is
a
whole
new
ball
game.
There's
a
lot
of
complexity
in
production
scenarios
to
the
point
that
data
Engineers
spend
a
lot
of
their
time
on
tooling.
Instead
of
getting
value
from
data.
One
framework
that
simplifies
this
process
significantly
is
Delta
live
tables
or
DLT.
A
Dlt
uses
a
simple
declarative
approach
to
building
reliable
data
pipelines
streaming
or
batch.
Whether
using
python
or
SQL,
the
online
DLT
engine
figures
out
dependencies
live
monitors,
data
quality
and
automate
away,
literally
all
the
operational
complexity
Additionally,
you
can
treat
your
data
as
code
and
apply
modern
software
engineering,
best
practices
like
testing,
error,
handling,
monitoring
and
documentation
to
deploy
these
pipelines
at
scale.
Now,
let's
talk
a
little
bit
about
machine
learning.
In
a
nutshell,
machine
learning
involves
extracting
patterns
from
data
and
using
those
patterns
to
make
predictions
on
new
data.
A
Now
there
are
a
number
of
caveats
and
nuances
to
this,
but
at
a
high
level
this
is
accurate.
It's
also
important
to
dig
a
little
deeper
and
understand
the
machine
learning
life
cycle.
Imagine
you're
a
data
scientist
at
this
e-commerce
company.
You
start
with
the
data
you
receive
data
from
an
upstream
process
likely
a
data
pipeline.
Then
you
do
some
exploratory
data
analysis
and
perform
feature
engineering
to
mold
this
data
into
form
that
a
machine
learning
algorithm
can
be
trained
on
and
also
Infuse
domain
knowledge
into
these
inputs.
Then
you
perform
model
selection.
A
Finally,
you
put
this
trained
machine
learning
model
in
production
in
stream
real
time
or
batch
after
a
while,
the
model
performance
degrades
and
you
have
to
rerun
the
whole
process
end
to
end
all
over
again,
and
this
process
is
usually
automated
with
some
intervention
from
humans
at
Key
stages.
Again,
there
are
a
number
of
caveats
and
nuances
to
this
by
the
high
level.
This
is
a
common
listing
pattern
for
most
use
cases
in
production.
A
Now
there
are
many
platforms
to
do:
machine
learning,
I
know,
data
breaks
pretty
well
I
work
here
and
I
can
think
of
a
number
of
reasons
to
go
with
data
breaks
for
your
machine
learning
workloads.
Let's
start
with
the
data
machine
learning
is
a
data
Centric
process
garbage
and
garbage
out,
so
ensuring
the
quality
of
your
data
that
you
feed
into
machine
learning
training
process
is
imperative.
A
Storing
data
in
Delta
is
key
to
ensuring
this
Delta
tables
are
basically
data
store
in
parquet,
with
an
open
source
format
co-located
with
a
Json
transaction
log
that
sits
in
cheap
object,
storage
in
the
cloud.
These
two
things
together
allow
acid
transactions
and
ensure
data
quality,
with
the
cost
benefits
of
using
object
storage.
In
addition
to
this
on
databricks,
you
have
a
feature
store
that
leverages
Delta
under
the
hood
to
ensure
discoverability
and
reuse
of
features.
The
lake
house
caters
to
both
professional
data
scientists
and
citizen
data.
Scientists.
A
The
process
of
logging,
the
model
with
mlflow
automatically
creates
a
user-defined
function
or
a
UDF
which
encapsulates
the
inference
logic,
whether
you
want
to
deploy
the
model
as
a
service
in
stream
or
for
batch
inference,
and
this
reduces
the
developer
overhead
significantly.
Now,
let's
take
a
closer
look
at
the
whole
idea
of
automating
machine
learning.
Automl
has
played
a
significant
role
in
making
machine
learning
accessible
to
more
people
than
it
used
to
before.
A
Given
some
training
data,
you
could
automatically
generate
a
Dependable
Baseline
model
that
could
be
modified
by
a
data
scientist
and
put
in
production,
but
not
all
automated
machine
learning
Frameworks
provide
the
end
user
with
human
readable,
annotated
code
that
can
be
modified.
One
automl
tool
that
does
so
is
database
rml
database
automl
provides
a
glossbox
solution
to
rml.
It
provides
the
benefits
of
automation
while
giving
the
end
user
full
ownership
of
code
and
models.
You
can
run
automl
from
the
UI
or
the
API
automl
will
test
different
models,
tune,
HEPA
parameters
and
return.
A
The
list
of
models
produced.
There
are
three
important
aspects
to
consider.
First
is
tightly
integrated
with
ML
flow.
All
the
models,
trained
and
corresponding
metadata
are
neatly
organized
into
an
experiment.
A
data
Express
notebook
is
returned
and
finally,
a
represent
trial.
Notebooks
can
be
edited
as
desired
to
customize
the
model,
with
all
models
automatically
logged
to
ml
flow.
It's
easily
deployed
an
rml
model
for
batch
streaming
or
online
inference
via
the
model
registry.
Now,
why
would
you
want
to
combine
both
streaming
and
machine
learning
together?
A
Also,
if
Done
Right
using
the
right
tool
set
streaming,
machine
learning
inference
is
likely
going
to
be
cheaper
and
more
performant
than
deploying
the
models
as
points
for
a
number
of
use
cases.
Now,
let's
put
everything
we
discussed
together,
we'll
walk
through
the
steps
of
building
out
the
streaming
machine
learning
interest
pipeline
for
anomaly
detection
in
quick
stream
data
originating
from
our
friends
Ecommerce
website.
A
In
the
database
lake
house,
we
have
an
incremental
ingestion
mechanism
called
the
autoloader
as
click
stream
records
land
in
object,
storage,
the
autoloader
processes,
this
data
and
writes
it
to
a
Delta
table.
Incrementally
we
call
this
table
a
bronze
table
which
is
essentially
a
table,
withdraw
data
to
which
a
schema
is
applied.
A
Then
it
performs
some
Transformations
on
this
data
to
get
to
a
silver
Delta
table.
When
you
perform
further
aggregations,
you
land
this
Delta
Data
in
a
gold
Delta
table.
The
series
of
Delta
tables
is
what
we
call
The
Medallion
architecture
in
the
lake
house.
Let's
assume
that
our
developer
has
access
to
some
label
training
data,
then
you
can
use
databricks
automl
to
obtain
a
classification
model
that
can
predict
which
is
which
Boss
versus
real
users.
Since
the
model
is
logged
with
mfo
automatically
you
get
the
user
defined
function
or
the
UDF
inference
code.
A
As
a
result,
you
can
register
this
UDF
and
use
it
in
the
pipeline
just
like
using
any
other
pyth
or
SQL
function.
This
entire
pipeline
of
Transformations,
written
in
SQL
or
python
can
be
deployed
as
a
Delta
live
tables
pipeline.
Selecting,
The,
Continuous
mode
for
running
this
pipeline
ensures
that
the
data,
transformation
and
model
inference
is
run
as
a
streaming
process
and,
of
course,
infrastructure
management,
data,
quality
monitoring
and
all
the
other
operational
details
are
abstracted
from
the
end
user
and
automatically
handled
by
Delta
live
tables.
A
As
we
discussed,
you
can
also
use
databrick
SQL
a
tool
within
databricks,
primarily
catering
to
SQL
analysts
to
query
the
gold
table
with
inferred
records
and
build
dashboards
that
constantly
update
as
the
click
stream
is
processed.
You
can
see
the
DLT
pipeline
created
on
the
top
left
here.
The
dag
is
infrared
from
the
transformation
logic
and
automatically
created
by
the
DLT
engine,
the
continuously
updated
dashboard.
What
monitoring
anomalies
built
with
databrick
SQL
is
on
the
bottom
right
when
you
work
on
a
project
like
this,
it's
highly
unlikely
that
you
would
do
it
alone.
A
So
when
you
are
working
as
a
group
adapting
best
practices
for
data
science
and
engineering
code,
development
is
critical.
To
help
with
this
repos
in
databricks
is
a
feature
that
allows
you
to
have
repository
level
get
integration
with
Git
providers
like
GitHub
or
GitHub,
Enterprise
server
and
others.
You
can
develop
code
in
a
databricks
notebook
sync
it
with
a
remote
git
repository
and
use
git
commands
for
updates
and
Source
control.
A
Here
we
clone
a
remote
repository
with
the
code
for
the
DLT
pipelines
in
a
couple
of
clicks,
so
you
can
edit
the
notebooks
and
sync
it
with
the
remote
git
repo.
We
can
use
these
notebooks
in
repos
to
create
a
DLT
pipeline
that
could
be
tested
and
put
in
production.
Now.
What?
If
you
have
to
change
some
aspect
of
this
DLT
pipeline
that
you
put
in
production?
A
Let's
say
you
want
to
deploy
a
different,
more
optimized
model
in
the
Stream,
for
example,
an
isolation
for
us
model
has
been
trained
on
a
large
volume
of
data
that
performs
better
than
our
automl
model.
Now.
We
move
into
the
realm
of
cicd
I,
don't
know
about
you,
but
when
I
think
of
cicd
and
GitHub
I
think
of
GitHub
actions,
if
you're
unfamiliar
with
GitHub
actions,
you
could
think
of
them
as
tasks
or
units
of
automation
that
are
triggered
in
response
to
certain
actions
that
occur
in
our
GitHub
repository.
A
There's
a
whole
Marketplace
of
these
actions
and
there
are
a
couple
from
databricks
themselves.
We
use
one
of
them.
For
example,
GitHub
actions
uses
yaml
files
to
Define
what
steps
to
take
in
the
GIF
workflow.
The
first
couple
of
lines
here
describe
the
trigger
for
these
GitHub
actions.
So,
according
to
this,
any
action
we
describe
in
this
file
will
only
trigger
on
the
event
of
a
pull
request.
A
In
short,
here
we
check
out
the
development
branch
and
run
a
set
of
unit
tests
in
this
test,
notebook
with
the
help
of
the
databricks
Run
notebook
GitHub
action.
This
is
run
in
the
databricks
compute
cluster
we've
specified
in
the
ml
file.
As
you
can
see
here.
Let's
say
you
have
this
trained
isolation
force
model
and
want
to
introduce
that
to
the
pipeline.
You
have
to
change
the
corresponding
cell
in
the
notebook.
They
didn't
make
a
pull
request.
This
pull
request
triggers
the
GitHub
action
to
run
all
the
units
tests.
A
As
you
can
see
here,
once
the
tests
are
run
and
all
the
checks
pass
and
everything
is
good
to
go.
You
can
merge
the
full
request
and
delete
the
branch.
Now,
let's
zoom
out,
we've
tried
different
model
approaches,
Auto
ml
models
or
isolation,
Force
models.
Then
we
go
through
the
process
of
model
selection,
as
we
talked
about
before.
A
Since
all
of
these
models
are
logged,
you
can
basically
copy
and
paste
the
UDF
code
from
the
UI
in
the
notebook
where
we
register
the
UDF.
This
notebook
will
be
the
one
that
causes
the
DLT
pipeline.
Once
we
create
a
pull
request.
This
regards
a
get
workflow
where
it
checks
out
a
branch
and
a
job
where
a
notebook
with
unit
testified
as
the
model
you
can
merge
and
have
the
DLT
notebook
pick
up
the
code
changes
with
new
ml
inference
logic.
A
If
the
tests
fail,
you
can
look
at
the
job
execution
details
in
database
to
see
what
went
wrong,
because
you
have
full
visibility
into
that.
This
is
not
limited
to
this
anomal
detection
problem
or
click
stream
data.
You
can
also
perform
sentiment.
Analysis
on
social
media
feeds
using
the
same
architecture.
The
data
passing
through
the
pipeline
would
be
text
which
Delta
supports
and
the
model
will
be
a
sentiment
classification
model.
You
could
use
a
sentiment
classification
pipeline
from
the
hogging
phase,
Transformers
library
and
log
it
with
the
ml
flow
to
obtain
the
UDF.
A
You
can
even
go
further
with
this
and
do
brand
specific
polarity
analysis
on
the
social
media
feed.
Here
you
would
provide
the
desired
classes
to
classify
on
to
the
zero
shot
classification
pipeline
from
hugging
phase
and
around
the
streaming
Pipeline
with
DLT.
In
either
case
you
can
use
the
model
pipeline
out
of
the
box
with
zero
training.
I
encourage
you
to
apply
this
to
a
challenge
that
exists
within
your
organization
that
could
be
solved
using
stream
processing
and
in-stream
machine
learning
inference.