►
Description
Presented by:
Hamel Husain, Staff Machine Learning Engineer at GitHub
Jeremy Lewi, Software Engineer at Google
From automating mundane tasks to reducing inefficiencies in developers’ workflows, machine learning has the potential to scale your team’s results like never before. However, the practice of deploying machine learning for enterprises is relatively new. In this talk, Hamel and Jeremy will demonstrate how GitHub Actions and Kubernetes can be used to orchestrate machine learning workflows in new ways that increase transparency and reliability of these applications. By borrowing best practices and technologies from DevOps, they’ll help you learn how to deploy machine learning solutions with confidence.
About GitHub Universe:
GitHub Universe is a two-day conference dedicated to the creativity and curiosity of the largest software community in the world. Sessions cover topics from team culture to open source software across industries and technologies.
For more information on GitHub Universe, check the website:
https://githubuniverse.com
A
A
Let's
get
started
so
when
most
people
hear
about
AI
what
they
think
about
is
a
lot
of
very
high
profile
in
very
exciting
success
stories
in
terms
of
AI.
So
you
have
self-driving
cars
like
way
mo.
You
have
alphago
in
which
a
computer
program
beat
the
world's
best
go
player.
You
have
everyday
applications
like
google
translate,
but
when,
when
ml
engineers
and
data
scientists
for
in
the
field,
think
about
AI,
they
have
a
very
different
experience
and
life
story.
A
So
this
Twitter
quote
has
been
receiving
a
lot
of
attention
in
the
ML
infra
and
data
science
community,
because
it
really
sums
up
pretty
nicely
the
sentiment
and
feeling
of
a
data
scientist
on
a
day
to
day
basis,
and
the
idea
is
basically
data.
Scientists
are
able
to
rapidly
and
quickly
develop
a
model
and
prove
that
this
model
is
able
to
improve
some
important
business
metric
and
then
they
would
try
to
get
that
into
production
or
into
some
application
and
that's
where
they
still
run
into
roadblocks.
A
So
they
end
up
spending
months,
trying
to
get
this
into
production
and
oftentimes
they
fail.
And
what
we've
learned
at
Google
is
that
one
of
the
reasons
that
these
systems
is
so
difficult
to
get
ml
into
production
is
that
when
you're
building
an
ml
application,
you're
actually
building
a
complex
distributed
system.
A
So
this
is
a
famous
diagram
from
a
paper
that
Google
published
called
the
hidden
technical
debt
in
ml
systems,
and
the
key
idea
is
that
most
people
tend
to
think
about
ml
as
being
about
complex
machine
learning,
algorithms
and
difficult
math,
but
in
reality
this
ends
up
being
only
a
small
portion
of
your
ml
application
or
system.
In
reality,
what
you
have
is
a
lot
of
systems
and
infrastructure
that
surrounds
that
it
is
necessary
to
support
that
ml
algorithm.
So,
for
example,
you
need
infrastructure
to
collect
data
and
pre-process,
it
extract
features.
A
You
need
to
clean
the
data
and
make
sure
that
it's
valid
and
you
can
use
it
in
production.
You
have
to
manage
all
the
resources
like
to
compute
in
order
to
do
both
the
training
and
the
survey.
Once
you
get
your
model
into
production,
you
have
all
of
the
usual
ops
challenges,
monitoring,
logging,
auditing,
etc,
and
so
in
reality,
most
of
your
time
and
energy
goes
into
to
all
these
surrounding
systems
and
infrastructure.
And
that's
what
makes
getting
an
imbalance
of
production
so
difficult
and
so
challenging.
A
So
one
of
the
things
that
we've
learned
at
Google,
dealing
with
complex
distributed
systems,
is
that,
if
you
deploy
more
often,
this
often
reduces
the
likelihood
that
problems
will
occur
and
there
also
reduces
the
severity
of
the
problems
when
they
do
happen.
So
this
has
been
backed
up
by
research.
So
there's
a
DevOps
research
report
called
the
Dora
report
that
comes
out
every
year
and
in
2019
the
data
basically
confirmed
this
conclusion
that
if
you
deploy
more
often
you
will
all
have
less
problems,
and
this
is
a
key
principle
of
Google
I
sorry's.
A
A
So
for
traditional
applications,
continuous
integration
and
continuous
deployment
has
a
very
rich
ecosystem.
So
we
have
this
cycle
where
we
want
to
do
development,
you
know
create
make
a
change
to
our
application,
like
our
web
application.
Push
that
to
our
source
repo
then
build
it
test
it
and
deploy
it,
and
so
we
have
a
very
rich
tool
chain
for
supporting
that.
A
We
then
have
integrations
for
gluing
the
steps
together,
so
you
have
plug-ins
4
vs
code
for
things
like
git
and
finally,
you
have
off-the-shelf
platforms
like
Jenkins
that
can
put
this
all
together
for
ML
C
ICD
is
similar,
but
it's
difference
and
these
differences
create
challenges
which
are
preventing
us
from
having
having
rich
toolchain,
as
of
yet
to
help
us
to
see
ICD
from
machine
learning.
So
as
an
example
with
with
with
with
ml,
we
have
new
tools
as
an
example.
A
A
lot
of
data
scientists
and
ml
engineers
are
using
Jupiter,
which
is
a
note
booking
environment,
to
develop
their
models
because
it
provides
really
rich
visualizations
and
is
really
great
for
interactive
data
analysis.
But
this
is
these
toolchain
right
look.
Jupiter
lacks
some
of
the
integrations
that
we
expect
from
our
existing
tool
train.
So
as
an
example,
we
don't
really
have
great
tools
with
Jupiter
to
automatically
build
artifacts
like
containers
that
we
would
typically
want
to
put
into
our
CI
CD
pipeline
to
automate
the
deployment
and
getting
them
into
production.
A
The
first
thing
you
have
to
do
is
you
have
to
train
the
model,
and
that
can
take
a
lot
of
time
and
it
can
also
be
computationally
expensive.
The
other
problem
is
that
often
times
when
you
have
a
model
like
ranked
a
ranking
model
or
a
recommendation
model,
it
can
be
difficult
to
evaluate
the
performance
of
that
model
offline.
You
may
have
to
deploy
that
model
into
production,
test
it
on
a
subset
of
traffic
and
see
whether
it's
actually
driving
more
traffic
to
see
whether
it's
improving
quality.
B
Thanks
Jeremy,
so
I
just
want
to
show
you
an
example,
a
concrete
example
of
what
happens
because
of
this
lack
of
DevOps
for
machine
learning
or
more
concretely,
CI
C
D
for
machine
learning.
This
is
a
real
PR
from
the
coup
flow
project,
and
this
is
really
indicative
of
the
PR
that
you
will
see
on
any
machine
learning
project.
B
So
this
is
really
problematic
for
a
traditional
software
kind
of
development
lifecycle.
We
don't
want
to
guess
whether
a
code
should
be
merged
or
not.
So
you
might
typically
see
on
github
when
using
a
CI
CD.
You
have
integration
test
smoke
tests
a
unit
test,
whatever
they
might
be,
and
you
have
these
robust
systems
that
help.
You
monitor
that.
So
you
don't
have
to
guess.
B
Like
I
just
showed
you
in
that
PR,
and
so
we
want
to
bring
that
same
paradigm
to
machine
learning
so
that
you
can
take
the
guesswork
out
and
then
there's
a
lot
of
other
things.
So
you
want
to
be
able
to
block
PRS
from
being
merged
in
production,
for
example,
if
the
test
didn't
pass
or
if
the
test
is
not
even
run,
which
we
also
want
to
bring
into
machine
learning.
B
So
what
I
have
here
is
a
PR
example
and
in
this
PR
I'm
changing
some
parameters
of
a
model.
So
if
you
look
at
this
file,
I'm
changing
I'm
increasing
the
size
of
an
embedding
later
from
50
to
85.
The
details
are
not
important,
but
the
idea
is
I'm
changing
something
about
this
model
and,
for
example,
one
thing
that
you
may
want
to
do
differently
from
traditional
CI
CD
is,
you
might
not
want
to
trigger
a
full
run
of
your
model
on
every
push,
because
machine
learning
can
be
very
compute
intensive
and
time
intensive.
B
You
might
want
a
human
to
give.
You
give
the
final
word
on
when
they're
ready
to
run
the
full
test,
so
using
github
actions
we
can
orchestrate
something
like
a
chat,
ops,
so
I'm
gonna
issue,
a
command
here
run
full
test,
slash,
run
full
test,
make
a
comment
and
that
will
trigger
Gift
of
actions
to
start
this
process.
B
So
in
this
case,
we're
building
the
containers
they're
being
pushed
to
the
docker
registry
that
we
have
and
then
we're
building
the
kubernetes
spec
that
will
then
deploy
emitting
this
machine
learning
pipeline
to
Kufa
low
to
get
run,
and
so,
if
you
notice
the
the
context
for
this
is
dropped
into
the
PR.
So
actions
build
a
container
and
it
submitted
the
job
to
coop
flow.
To
run
this
machine
learning
pipeline
I'm
not
going
to
go
into
a
lot
of
detail
right
now
about
how
that
works.
Jeremy
is
gonna.
A
B
Learning
workflow
system
that
runs
on
coop
flow
again
this
is
available
from
the
PR
and,
as
you
notice,
we
created
a
checker
on
so
this
yellow
dot
here
is
going
to
stay
open
until
the
end
of
the
machine
learning
job.
So
eventually,
this
job
will
keep
running
the
the
training
will
complete
and
it'll
clear
that
check
run
now,
keep
in
mind.
This
is
running
on
the
infrastructure
of
your
choice.
This
was
in
this
specific
example.
B
So
while
the
model
was
training,
we
logged
a
lot
of
statistics
to
an
experiment
tracking
system,
in
this
case
we're
using
weights
and
biases,
and
that
experiment
tracking
system
captures
lots
of
information
about
model
performance
and
other
system
metrics
that
you
might
want
to
see
what
happens
is
when
the
model
is
done.
It
queries
that
and
brings
into
the
PR.
So
this
is
what
weights
and
biases
looks
like
you
can
use
any
experiment
tracking
system
you
want,
but
just
to
give
you
an
idea.
B
This
is
a
system
where
you
can
log
very
detailed
metrics
about
your
machine
learning
models,
training
run
things
like
a
lot
of
things
that
are
locked
to
standard
out,
and
you
see
here
various
model
artifacts.
So
again,
this
is
dropped
into
the
PR.
So
all
runs
that
correspond
to
this
Shaw
that
this
PR
is
run
on
is
dropped
into
this
PR
and
it's
dropped
in
as
markdown,
and
you
can
customize
this.
B
B
So
let
me
go
back
a
little
bit,
so
this
shows
that
model
being
deployed
and
then,
when
it's
done
being
deployed,
it
will
report
back
success
or
failure
of
deployment
into
the
PR
so
that
you
can
go
and
see
if
this
code
was
deployed
and
when
it
was
deployed-
and
you
can
have
a
record
of
it
all
within
the
PR-
and
this
is
just
the
dashboard
of
the
this
particular
model
being
deployed
to
Google
Cloud
functions.
So
this
is
really
exciting.
B
We
showed
here
in
this
example,
kind
of
an
end-to-end
workflow,
where
you
have
machine
learning.
Ci
CD
model
is
running
on
infrastructure
of
your
choice,
in
this
case
we're
using
GPUs
on
the
machine,
learning
infrastructure
of
your
choice,
and
you
can
compose
various
github
actions
to
achieve
that,
and
that's
really
new
and
exciting.
That
is
gonna,
allow
more
transparency
for
people
working
on
machine
learning
and
that
are
using
github.
B
A
You
Hemel,
so
now
we're
gonna
talk
about
how
we
go
about
building
that
CI
CD
workflow,
that
it
Hamel
just
demoed
by
using
coop
flow
and
github
actions.
So,
let's
just
let's
start
off
by
talking
about
what
we
need
to
do:
ml
for
CI
CD
for
ML,
there's
really
four
things
we
need.
We
need
tools
and
applications
that
allow
us
to
develop
our
models.
That
includes
editing
the
code
as
well
as
training
our
models.
A
We
also
need
applications
that
we
can
use
to
deploy
our
models
and
create
you
know
serving
endpoints
that
we
can
hit
with
HTTP
or
G
RPC
requests.
The
second
thing
that
we
need
is
we
need
to.
The
third
thing
is
tools
for
automation
right,
so
we
need
that
PR
workflow,
where
we
can
automatically
trigger
those
training
and
deployment
steps
like
camel
showed
in
response
to
actions
by
the
user,
and
so
for
that
we're
going
to
be
using
github
actions
and
then,
finally,
what
they'll
need
is.
A
We
often
want
to
have
a
manual
review
process
where
humans
decide
whether
a
model
is
actually
improving
performance
and
therefore
decide
to
push
it
into
production,
and
for
that
we're
going
to
use
github
and
the
standard
review
and
approval
process.
So,
let's
start
with
coop
flows.
So
what
we
started
to
see
about
two
years
ago,
with
an
ml
space,
was
that
companies
that
were
really
successful
with
machine
learning
and
we're
really
on
the
cutting
edge.
A
We
started
talking
a
lot
about
the
platforms
they
were
building
in
particular
to
solve
this
problem
about
how
to
continuously
integrate
and
deploy
models
into
production.
So
Google
was
one
of
the
first
and
we
started
talking
about
tf-x,
which
was
our
internal
machine
learning
system
which
we've
started
to
open-source,
and
one
of
the
key
problems
that
tf-x
was
designed
to
solve
was
continually.
We
reach
your
models
as
new
data
arrived
in,
pushing
those
models
into
production.
A
There
are
other
companies
also
started
talking
a
lot
about
the
platform's
they
were
building
lift
was
one
of
the
big
ones.
They
talked
about
a
platform
called
lift
learn
which
enables
their
data
scientists
to
rapidly
develop
models
using
jupiter
notebooks
and
then
push
those
into
production.
So
a
lot
of
other
companies
like
Bloomberg
stripe,
it
Airbnb
have
been
talking
about
the
platform's
they've
been
building
and
how
they've
been
building
these
platforms
on
top
of
open
source
applications
like
10,
0x
tree
boost
and
onyx,
and
then
also
on
top
of
kubernetes.
A
So
we
decided
to
start
building
coop
flow
as
an
open
source,
kubernetes
native
platform
permission
learning
so
that
all
the
companies
that
wanted
to
use
and
needed
an
ml
platform
but
didn't
have
the
resources
to
build
their
own
platform,
could
also
get
the
advantages
and
begin
to
use
ml
successfully.
Just
like
these
bigger
companies.
So
our
mission
with
coop
flow
is
to
make
it
easy
for
everyone
to
develop,
deploy
and
manage
portable
distributed
machine
learning
on
kubernetes.
A
We
are
an
entirely
open
source
project.
We
have
an
incredible
amount
of
momentum,
so
we
have
over
30
companies
that
are
contributing
and
building
on
top
of
coop
flow,
and
we
have
an
amazing
number
of
PRS
and
unique
authors
every
month.
So
I
think
we're
up
to
over
80
unique
PR
contributors
every
every
28
days
or
so
so
getting
back
to
the
CI
CD
workflow.
The
first
step
is
developing.
Your
models
with
coop
flow
is
developing
your
models
and
we're
going
to
show
you
how
cupola
makes
that
easy.
A
So,
as
we
said
before,
jupiter
is
becoming
one
of
the
standard
tools
in
the
data
scientist
toolbox.
We
had
a
panel
a
couple
years
ago
if
the
cupola
Summit
and
we
asked
a
bunch
of
data
scientists.
What
is
the
tool
you
most
can't
live
without
and
they
all
said
notebooks
in
Jupiter,
so
with
coop
flow
we've
made
it
super
easy
to
spin
up
notebooks
on
on
kubernetes.
A
We
provide
you
this
very
simple
web
UI,
where
you
can
launch
a
new
notebook
and
then
begin
to
use
that
to
develop
your
model,
but,
as
we
showed
before,
one
of
the
big
challenges
with
Jupiter
notebooks
is:
how
do
you
go
about
converting
those
notebooks
into
artifacts
like
dr.
containers
that
you
can
begin
to
in
degree
into
a
CI,
CD,
workflow,
and
so
for
that
income
flow
we've
developed,
fairing
and
fairing
as
an
SDK?
That
makes
it
easy
to
convert
notebooks
into
docker
containers,
and
so
the
idea
is
that
fairing
is
divided
into
three
steps.
A
You
have
preprocessors
and
what
the
preprocessor
does
is.
It
basically
takes
your
jupiter
notebook,
which
is
really
just
a
json
file
and
converts
that
to
a
python
file,
and
it
basically
does
that
just
by
sort
of
stripping
out
the
markdown
and
then
allow
you
to
tell
it
using
various
filters
and
other
kinds
of
syntax.
What
sells
to
include
and
what
sells
to
remove.
So
you
can
remove
your
visualizations
or
some
of
the
other
cells
that
you
don't
need
in
your
production,
artifacts
and
once
you've
produced
that
Python
file.
A
Farren
provides
a
builder
which
makes
it
easy
to
build
the
docker
container
from
that
Python
file,
and
so
it
can
do
things
like
generate
a
pitfall
of
requirements
file
to
install
your
Python
dependencies.
Then.
Finally,
once
you
have
that
container,
we
actually
need
to
deploy
that
on
kubernetes
and
so
for
that
Ferenc
provides
deployers
which
allow
you
to
easily
create
kubernetes
resources
using
that
docker
container
that
you've
built,
and
so
the
the
idea
here
is
that
by
building
this
sdk,
we've
provided
the
integration
for
you.
A
A
So
the
first
one
is,
you
typically
want
to
train
your
model,
and
so
for
that
kubernetes
provides
a
bunch
of
different
applications
called
custom
resources
that
make
it
really
easy
to
train
your
model
using
different
frameworks
like
tensor
flow
or
pi
torch
or
extra
boost,
and
the
idea
here
is
that
we
give
you
a
very
simple
declarative
spec.
So
you
can
give
us
a
very
simple
yeah,
mph
specification
that
focuses
on
the
high-level
information
that
you,
as
a
data
scientist
want
to
provide.
A
So
in
particular,
you
just
have
to
specify
the
image
which
is
the
docker
container,
that
contains
your
code,
and
then
you
would
have
to
specify
the
number
of
workers
that
you
want
to
use.
If
you
want
to
run
your
kit,
your
job
just
so,
you
can
take
advantage
of
training
large
models
in
parallel,
and
so
this
simplifies
the
management
process,
because
the
data
scientists
no
longer
has
to
manage
individual
resources
or
multiple
containers.
They
just
have
to
spend
create
this
simple
kubernetes
specification.
A
Another
big
challenge
in
machine
learning
is
hyper
parameter
tuning
a
lot
of
times.
Machine
learning
models
or
deep
learning
models
have
hyper
parameters
that
the
only
way
to
train
them
is
to
try
models
with
a
bunch
of
different
values
and
see
which
one
performs
best.
So
a
simple
example
with
deep
learning
is
you
often
have
the
number
of
different
layers
to
use
in
your
model
and
the
only
way
to
see
which
what
the
desire
best
value
is
is
to
try
a
bunch
of
different
values.
A
Finally,
you
actually
have
to
deploy
the
model,
and
so
for
that
we
provide
k-f
serving
and
so
km
serving
as
an
application.
Where
again,
one
can
provide
a
simple
declarative
spec,
where
all
one
has
to
provide
is
the
URI
of
your
model.
So
this
could
be
a
protocol
buffer,
outputted
by
Tenzer
flow
and
then
under
the
hood
KF
serving
takes
care
of
all
of
the
complex
DevOps
tasks
or
ml
ops
tasks
that
you
would
need
to
actually
deploy
this
model
into
production.
A
So
as
an
example
of
some
of
the
complex
DevOps
challenges
that
KF
serving
handles
for,
you
is
KF
serving
will
automatically
auto
scale
the
number
of
instances
your
model
has
deployed.
So
if
you
have
a
lot
of
requests
coming
in
and
you
need
to
spin
up
more
capacity,
it'll
automatically
add
more
containers
and,
as
the
number
of
requests
decrease,
it'll
reduce
the
number
of
containers
you
have
deployed
so
you're,
not
wasting
resources
or
money
on
compute.
You
don't
need.
A
It
can
also
handle
complex
DevOps
challenges,
like
you
know,
rolling
out
changes
in
doing
things
like
Canaries
and
rollouts.
On
the
machine
learning
side.
Kf
surgeon
provides
a
lot
of
advanced
functionality.
As
an
example,
we
provide
model
servers
that
can
easily
take
a
Python
function
that
does
inference
and
then
wraps
that
with
an
RPC
arrest,
server
that
integrates
with
things
like
sto
to
provide
monitoring
and
metrics
that
you
need,
if
you're
actually
going
to
deploy
this
into
production.
It
can
also
handle
things
like
inference.
A
B
Thanks
Jeremy,
so
just
to
kind
of
circle
back
and
just
in
case
anybody
doesn't
know
what
github
actions
is.
Yet
probably
everybody
does
at
this
point
in
the
conference
you
can
github
actions
makes
a
lot
of
this
stuff
very
easily.
So
in
just
in
case,
you
haven't
seen
get
up
actions
you
specify
with
the
llamó
file
and
the
dot
github
slash
workflow
of
directory
of
your
repo.
B
You
can
trigger
it
on
any
github
event
and
then
you
can
also
use
other
predefined
workflows
from
other
people
without
rewriting
all
the
code,
so
it
it's
very
modular
and
composable
in
that
sense,
and
that
makes
it
really
good
for
machine
learning
ops
because
you
can
inherit
other
people's
parts
of
other
people's
workflow
without
kind
of
redoing
it
rebuilding
it
and
there's
a
large
library
of
existing
actions,
including
some
actions
for
machine
learning
ops.
So
you
can,
you
can
find
those
on
the
marketplace.
B
B
B
So
again
we
talked
about
how
we
started
from
a
pull
request,
and
then
we
build
the
environment
from
the
code
in
that
pull,
request,
specifically
docker
containers
and
then
showed
a
way
how
we
can
orchestrate
training
of
those
models
with
github
actions
and
reporting
the
status
back
to
the
PR
and
also,
at
the
same
time,
collecting
metrics
about
your
model
and
reporting
that
back
to
the
PR
and
then
ultimately
deploying
the
model
and
again
up
actions
is
a
great
way
to
orchestrate.
All
of
these
things.