►
From YouTube: Conquering a Kubeflow Kubernetes Cluster with ksonnet, Ark, & Sonobuoy - Kris Nova & David Aronchick
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Conquering a Kubeflow Kubernetes Cluster with ksonnet, Ark, and Sonobuoy - Kris Nova, Heptio & David Aronchick, Google (Intermediate Skill Level)
Kubernetes provides a unique environment for running distributed applications. In this demonstration we learn how Ark, Sonobuoy, and ksonnet are helping to craft a kubernetes configuration build for rock solid machine learning. In this talk we explore a kubernetes cluster configured for infrastructure optimization as well as data experimentation, model porting, and cloud specific configurations for advanced machine learning. We learn about the constructs behind the kubeflow project, and take ownership of them and our infrastructure with Ark and ksonnet. We validate the health of the cluster with Sonobuoy, and ensure that as machine learning engineers we are operating the healthiest cluster we can.
About David
David Aronchick was the Senior Product Manager for the Google Container Engine and led product management on behalf of Google for Kubernetes. David has been helping to ship software for nearly 20 years, founding and being part of the management team for three different startups, as well as squeezing in time at Microsoft, Amazon, Chef, and now Google. David is co-founder of the Kubeflow project, an effort to help developers and enterprises deploy and use ML cloud-natively everywhere.
About Kris
Kris Nova is a Senior Developer Advocate for Heptio. She is a Kubernetes maintainer and contributor. She wrote the book on cloud native infrastructure. She spends her free time climbing mountains.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Go
cool
hi,
everyone
I
am
Chris
wow.
This
is
really
loud.
I
am
talk:
quieter,
Chris,
Nova,
I,
work
at
hefty
Oh
I'm,
a
developer
advocate
I,
I,
co-authored
a
book
called
cloud
native
infrastructure
and
I
spend
a
lot
of
my
time
in
open
source
kubernetes
working
on
the
oh,
so
important
infrastructure
layer
underneath
all
of
this
cloud
magic
that
we
run
on
top
and
today
we're
gonna
be
talking
about
conquering
cube
flow
with
a
handful
of
hep
geo
tools
that
are
open
source
that
we're
gonna
be
demoing
for
you
guys
later
on
today,
yep.
B
I'm
Dave
our
uncheck,
that
is
all
that
is
also
very
loud
and
I'm.
Very
loud
I'm,
a
product
manager
at
Google
I'm
been
around
the
kubernetes
community
for
a
while
and
I'm
co-founder
of
the
cube
flow
project.
So
I
would
like
to
start
with
a
simple
question:
how
many
of
you
are
running
as
multi
cloud
right
now,
one
or
more
clouds?
Okay,
the
rest
of
you
are
lying
and
I
can
prove
it
right
now.
How
many
of
you
have
laptops.
B
Congratulations,
your
multi
cloud.
You
have
a
second
deployment
that
you
use
for
testing,
dev
experimentation
exploration,
doesn't
matter,
you
need
to
keep
that
in
sync
with
your
cloud,
and
that
is
the
problem
that
customers
are
facing
today.
The
majority
of
customers
are
already
thinking
about
multi
cloud
and
despite
those
19%
lying
about
being
multi-cloud,
multi
cloud
is
the
reality.
Thank
you
and
realistically
it's
not
just
one
cloud.
It
is
multiple
clouds,
Amazon
Azure,
aww
I.
Excuse
me
Google,
digitalocean,
IBM,
so
on
and
so
forth.
Sure
those
are
all
hyper
scale.
B
They
are
ready
to
go,
but
there
also
is
on
Prem.
There's
your
training
rig.
There's
your
cluster
underneath
your
buddy's
desk.
There's
your
laptop
these
all
environments
that
you're
gonna
spend
time
coordinating
getting
lined
up
and
keeping
in
sync.
So
the
problem
is,
as
you
get
more
clouds
you
get
mo
problems.
B
Setting
up
a
single
cloud
is
hard.
Setting
up
a
multiple
cloud
is
hard
to
the
power
part.
It's
keeping
a
lot
of
things
in
sync:
there's
almost
always
something
that's
going
to
change,
and
you
end
up
running
into
more
and
more
problems,
and
then
you
get
to
not
just
the
set
up,
but
also
the
a
pro
lot,
and
then,
on
top
of
that,
you
got
to
sync
everything,
so
it's
just
over
and
over
and
over
again,
you're
gonna
run
into
more
problems.
B
Yes,
it
is
impossible.
Excellent,
so
kubernetes
helps,
as
you
saw
today.
Kubernetes
provides
a
great
abstraction
layer
under
the
hood.
It
gives
you
a
set
of
cloud
native
api's
that
work
anywhere
as
long
as
you
are
using
a
kubernetes,
conforming
cluster,
most
clouds
have
great
hosted
solutions
or
have
good
communities
and
documentation
for
setting
it
up.
They're,
also
a
great
set
of
third-party
tools.
Cuba,
corn,
cops
so
on
cube,
ADM
to
help
installation
as
well,
and
especially
with
the
new
cluster
api
stuff.
B
Again,
a
lot
of
that
stuff
becomes
abstracted
away
from
you
and
the
best
part
is
everything
is
declarative
right.
So
you
can
read
your
llamo
and
understand
exactly
what's
going
on
and
if
you've
done
your
job
right,
they
should
just
use
an
underlying
kubernetes.
Abstractions
and
kubernetes
takes
care
of
the
rest.
So
that's
great,
except
of
course
you
want
more.
So
you
need
to
solve
day
2
and
with
that
day,.
A
Kubernetes
anywhere
is
a
great
one
and,
like
David
said,
there's
managed
providers
as
well,
but
we
run
into
all
these
problems
that
we
only
see
on
day
two
one
of
the
big
homes
we
we
talk
about
is:
how
do
I
backup
my
cluster
and
and
to
me
I,
feel
like
ok,
that
can
mean
a
lot
of
things.
What
does
that
mean
to
you
backing
up
your
cluster
in
the
sense
of
like
okay?
If
it
fails,
how
do
I
get
it
back
or
backing
out?
A
My
cluster,
in
the
sense
of
like
I,
want
a
record
of
what
we're
running
like.
Even
if
it's
not
a
failure,
recovery
mode
yeah,
you
still
need
to
have
a
list
or
some
sort
of
complete
document
that
says
here's
what
you're
running,
so
how
do
I
ensure
multiple
clusters
at
the
same
time,
this
is
another
problem
show
of
hands
who
here
runs
more
than
one
kubernetes
cluster.
A
Who
here
only
runs
one
cluster
one
hand
went
up:
okay,
I'm,
like
what
have
more
questions
now
I
wish
I
could
ask
you
guys,
but
like
wait
for
later,
most
people
are
running
multiple
kubernetes
clusters.
There's
various
reasons
for
doing
this.
Some
people
do
it
because
they
want
a
separate
environment
for
dev
or
for
staging,
or
they
want
a
separate
environment
to
do
some
load,
testing
or
there's
different
use
cases.
So
how
do
you
ensure
conformance
across
both
of
those?
How
do
I
know
cluster
a
is
gonna
behave
the
same
as
cluster
B.
A
This
is
an
age-old
problem
in
tech
that
we've
seen
time
and
time
again,
and
it's
just
resurfacing
itself
in
kubernetes.
Furthermore,
how
do
I
encapsulate
my
application?
I
have
an
application.
That
is
this
idea
that
is
important
to
me,
and
my
company
and
I
have
these
kubernetes
resources
that
are
very
ambiguous.
How
do
I
connect
those
two
together
and
group
these
resources
together
to
create
an
application?
How
do
I
manage
complexity?
This
is
software.
This
is
why
we're
all
software
engineers.
A
A
Yep
do
has
some
of
the
answers
show
of
hands
here.
Who's
heard
of
hefty
Oh
before
Wow
almost
everybody's
hand,
went
up
so
yeah
we're
a
startup
we're
out
of
Seattle.
The
co-founder
of
the
core
founders,
Joe
and
Craig
came
out
of
Google
Joe
spent
some
time
at
Microsoft
for
a
while.
They
helped
bring
some
of
the
tools,
David
works
on
to
life
and
they
started
a
company
called
how
do
I
now
work
for
them,
some
of
our
core
values,
our
FTOs
committed
to
open-source.
A
We
believe
that
it
is
important
to
share
and
that
learning
from
each
other.
In
a
safe
scientific
environment,
where
it's
ok
to
fail-
and
it's
ok
to
try
things
and
it's
okay
to
experiment
is
something
that
we
value
as
a
company.
Furthermore,
recupera
ninety's
experts,
we
have
a
ton
of
people
out
the
company
I
routinely
see
on
Cooper
Denny
sig
calls
I'll
be
like
sitting
on
my
desk
and
will
be
somebody
down
the
hall
and
I'll
kind
of
look
over
and
I'll
realize.
A
Oh
we're
both
on
the
same
open-source
call
and
our
video
is
recursively
going
into
each
other.
Furthermore,
we
have
a
passion
for
honest
technology.
When
we
write
technology
we
want
it
to
be
honest,
we
want
it
to
be
open.
We
wanted
to
work
well
and
if
it
doesn't
work
well,
then,
let's
talk
about
it.
Let's
figure
out
why
it's
not
working
well,
let's
make
it
work
better.
A
A
That's
like
it's
a
big
thing:
it's
a
conformant
tester,
so
you
can
run
this
on
your
kubernetes
cluster
and
it's
going
to
run
the
exact
same
set
of
conformance
tests
and
we're
gonna
get
an
output
from
that
in
that
output.
If
you
use
a
tool
I'm
going
to
show
later
called
the
FDA
scanner
can
tell
you
things
visually
or
there's.
Actually
some
output,
you
can
get
locally
at
your
command
line
and
you
can
go
through
and
you
can
say,
is
this
cluster
conformant
to
kubernetes
standards?
A
The
reason
this
is
important
is
because
it's
it's
creating
uniformity
across
kubernetes
clusters.
The
cluster
I'm
testing,
my
application
on
is
conformance
to
your
cluster
you're
testing,
your
application
on.
So
you
know
more
of
those
it
works
on
my
computer,
not
your
computer,
Oh
docker
is
gonna
solve
that
problem,
and
now
we're
liking
that
problem
all
over
again
with
kubernetes
clusters.
So
that's
what
have
to
guess
unemployed
does
case
on
it.
Who
here
has
used
case
on
it
before
show
of
hands?
Who
would
like
to
use
case
on
it?
Awesome
so
case
on.
A
It
is
a
configuration
management
tool.
It
encapsulates
your
application.
It
encapsulate
serious
other
higher-level
paradigms
that
are
important
to
software
engineers
and
it
creates
templates
and
kubernetes
resources
for
you.
So
you,
as
a
user,
are
kind
of
pulled
away
from
a
lot
of
the
noise
of
the
the
kubernetes
resources
and
you
kind
of
get
a
simpler
interface
to
work
with
where
you
can
define
things
in
your
app.
What
port
am
I
listening
on?
A
What
docker
image
am
I
going
to
be
pulling
this
thing
from,
and
it
keeps
it
simple
and
will
actually
go
and
create
the
resources
for
you
to
deploy
those
to
kubernetes
help
do
arc
this
one's
my
favorite
show
of
hands.
Who
here
is
use?
Hep
do
arc
and
I
guess
I.
Probably
shouldn't
have
mitad
that
I
favorite.I
they're,
all
my
favorite
I
love
them
all.
Equally,
who
would
like
to
use
happy
org,
okay,
Arcas
dope?
It
goes
through
and
it'll
actually
read
all
of
your
kubernetes
resources.
A
Yes,
it
will
even
read
yours
here
at
ease
for
you.
It
uses
an
API
endpoint
that
will
describe
all
of
the
resources
you
have
defined
in
your
cluster.
It
will
iterate
over
those
and
will
back
them
all
up
and
stick
it
in
a
tar
ball
from
that
tar
ball.
You
can
undo
it
and
you
can
go
the
other
way.
You
can
take
all
these.
A
B
Right
so
enough
of
that
vendor
talk,
let's
get
into
some
real-world
examples:
I,
don't
know
how
many
of
you
were
at
the
keynote
today
we
introduced
cube
flow
0.1,
I'm,
just
gonna.
Give
you
a
quick
run-through
for
anyone
who
wasn't
there
I'll
tell
you
what
cube
flow
is
and
why
we
decided
to
do
it.
Everyone
who
was
there
can
take
a
quick
nap,
so
the
idea
is
cube.
Flow
is
cloud
native
ml.
B
It
is
an
ml
stack
designed
with
all
of
the
core
concepts
that
kubernetes
has
empowered
people
with
to
make
everyone
so
productive
around
their
applications,
and
those
concepts
are
basically
loosely
coupled
micro,
service-oriented
dynamic
scale,
true
distributed
computing
via
an
API,
specifically
relative
to
those
micro
services.
Ml
stacks
are
composed
of
a
lot
of
micro
services.
Now
you
know
here:
I
have
13
micro
services
theoretically
listed
here,
but
it
may
even
be
more
than
that.
Each
one
of
these
may
have
their
own
micro
services
and
I'll.
B
These
are
all
really
common
situations
and
again,
this
is
where
you
do
start
to
get
into
the
reality
of
multi
cloud
where
you
may
already
have
a
entirely
data-driven
pipeline
and
back-end
built
as
a
spark
cluster
or
Hadoop
cluster,
or
something
like
that.
Then
you
know
your
data
scientist
may
want
to
explore
on
their
laptop
and
then
you
bring
it
to
an
accelerated
training
rig
to
do
your
model
building
and
then
event,
obviously
eventually
rollout
and
it's
not
just
composable
from
micro
service
to
micro
service.
B
Using
the
native
tools,
like
case
Anette,
to
parameterize,
based
on
the
environments
you're
rolling
out
and
not
changing
your
core
deployment
at
all.
So,
like
I,
said
cue
flow
comes
to
some
of
the
rescue.
It
does
give
you
code
based
deployments
for
all
it
uses
common
OSS,
tooling.
We
don't
write
a
brand-new
ml
stack.
We
don't
write
brand
new
data
processing,
we
don't
you,
you
know
you're
able
to
use
jupiter
notebooks,
just
like
everyone
else
does,
and
it's
all
transparent
and
parameterised,
and
that's
key.
You
can
look
at
it.
B
It's
not
this
opaque
blob,
it's
written
in
everyday.
You
know
code
that
you
can
expect
you
can
write
tests
around
and
all
that
kind
of
good
stuff,
and
you
get
that
reproducibility.
You
can
pick
up
that
cube
flow
and
you
can
hand
it
to
other
people.
I
mentioned
it
already
already.
It
uses
criminais,
T's
native
abstraction
and
it
wires
everything
together
in
a
cloud
native
way,
specifically
around
something
like
tensorflow.
So
while
cube
flow
has
flow
in
the
name,
we
mean
machine
learning
flow,
not
tensor
flow.
B
B
Does
who
knows
what
a
customer
research
definition
is
a
lot
of
people
it's
more
people,
but
for
those
that
don't
you're
able
to
wire
together
multiple
services
in
a
single
object
and
have
them
wired
together,
but
both
deployable
and
wired
together.
So,
for
example,
with
tensorflow
you'll
have
a
master
server,
parameter
server
and
worker
nodes
and
you'll
want
to
wire
them
together
using
service
endpoints
that
all
comes
in
a
single
cube
flow.
B
So
cute
blowed
day,
two
is
all
about
production,
as
Chris
alluded
to
earlier.
There
are
so
many
things
that
you,
you
know
it's
very
easy
to
roll
out
using
kubernetes
now
anyway
used
to
not
be
so
easy,
but
now
it's
very
easy,
but
once
you
install
that
complex
application,
you're
going
to
want
to
have
multiple
jobs,
you're
gonna
want
to
have
multiple
namespaces.
You
need
to
figure
out
ingress
how
you
know,
users
and
other
services
actually
get
into
your
platform.
The
jobs
need
to
communicate
in
an
intelligent
way.
B
You
want
to
use
multiple
environments
and
and
keep
them
in
conformity,
because
things
tend
to
drift
over
time.
You
want
to
backup
that
data
and
you
certainly
want
to
migrate
that
data.
So
again,
these
are
all
day
two
problems.
We
with
cube
flow,
provide
the
elements
of
it
and
we
all
do
this
via
native
kubernetes
AP.
Is
we
don't
solve
any
of
these
problems
natively,
but
to
show
how
we
do
solve
them,
using
fto
tools,
cool
so.
A
I
I
just
moved
to
Seattle,
and
my
hobbies
include
going
to
the
Google
office
for
lunch
to
get
kale
I
remember
going
over
there,
nice,
hey
David,
you
want
to
do
in
cube
con
talk
and
we
start
talking
and
I
realized.
Machine
learning
is
probably
one
of
the
hardest
things
to
run
in
kubernetes.
It's
got
state.
There's
a
ton
of
complexity,
going
on
there's
a
bunch
of
different
pods,
doing
different
things.
They
all
need
to
communicate
together
and,
frankly,
I've
never
even
worked
around
machine
learning
before
so.
A
Naturally,
I
was
obsessed
with
it
and
I
accepted.
The
challenge
like
challenge
accepted
and
I
was
like.
Let's
go
run
this
thing
in
kubernetes
and
let's
do
it
with
the
head:
do
tools
that
we're
going
to
show
you
today
to
give
you
a
really
good
great
toolbox
as
a
cluster
operator,
it
has
somebody
trying
to
manage
this
complexity
of
machine
learning.
A
Cool
thanks,
I,
don't
want
like
off
into
the
microphone
for
all
you
guys,
so
we're
gonna
be
looking
at
the
three
tools
and
we're
going
to
be
doing
this
with
could
when
I
use
cube
flow.
Who
here
has
actually
looked
up
the
cube
flow
repo
before
okay,
who's
planning
on
doing
that
like
sometime
in
the
near
future?
Because
of
this
talk
and
other
ones:
okay,
yeah,
okay,
so
this
is
the
repo
it's
really
great.
A
A
Let's
break
that
one
out
and
let's
create
a
new
tab,
and
you
all
say:
when
is
that
good?
No
big
enough,
a
little
bit
more
say
when
I'm
gonna
keep
going
until
somebody?
Okay?
Should
I
go
back
just
kidding?
Okay,
so
we're
gonna
use
a
tool
called
cube
context.
It
allows
you
to
switch
between
different
various
kubernetes
clusters.
I
have
four
clusters
I'm
interacting
with
right.
Now
we
have
a
couple
of
gke
TPU
clusters
and
we
have
one
in
Amazon
and
we're
gonna
be
doing
a
cube
flow
migration
back
both
between
amazon
and
google
kubernetes.
A
Just
so
to
start
off,
we're
gonna
show
you
what
is
up
and
running
in
our
amazon
kubernetes
cluster.
Right
now
I
have
k
alias
to
key
Bechdel.
So
every
time
you
see
me
type
k,
it's
just
short
for
Quebec
doll,
so
I
can
get
namespaces
and
we
can
just
see
what
your
namespaces
are
up
and
running
right
now,
conference
Wi-Fi,
so
you
guys
can
just
like
speed
this
up.
So
we
have
cube
system
and
we
have
a
couple
of
other
things.
A
The
way
it
works
is
you
can
come
in
here
and
see
the
CML
and
it's
going
to
deploy
this
automatically.
So
you
can
copy
this
command
and
we'll
generate
this
endpoint
for
you.
So
this
is
Q,
Bechdel
friendly
and
then
it'll
actually
come
and
install
these
resources.
Can
you
guys
I'll
read
this
out
there
zoom
in
so
we
get
a
namespace.
We
get
a
service
account.
A
We
do
some
R
Becky
things
and
we
get
a
deployment
or
I'm
sorry
a
stateful
set,
and
we
need
to
deploy
some
pods
pretty
straightforward,
and
it's
going
to
pull
all
of
this
and
run
it
automatically
on
your
cluster.
So
we
can
copy
this
command
run
it
on
our
cluster
I
feel
like
this
is
like
the
cooking
show
saying
where
it's
like,
and
then
we
open
the
oven
and
it's
like
ready
to
go.
A
So
you
can
see
here
it's
creating
the
kubernetes
resources
for
you
and
if
we
come
back
to
the
web
page
here,
it
should
automatically
detect
that
we
have
deployed
this.
It
I
should
switch
over
to
the
your
cluster
is
now
scanning,
so
we'll
come
back
and
check
this
page
out
in
a
second
and
we're
gonna,
look
at
a
failed
run
from
earlier,
because
I
feel,
like
failures,
are
what's
important
here.
A
Let
me
copy
this
guy,
so
this
can
take
anywhere
from
30
to
sit.
There
goes
30
to
60
minutes
to
run.
It's
gonna
run
a
very
similar
to
the
kubernetes
end-to-end
tests
and
when
it
gets
done,
you're
gonna
get
an
output.
That's
gonna
say
here
the
conformance
tests
running
on
your
cluster
and
it's
gonna
tell
you
which
ones
are
okay
and
which
ones
failed.
A
So
to
look
at
a
failure
we
can
come
in
and
we
can
say.
Oh,
we
have
a
lot
of
conformance
issues.
So
these
are
the
things
that
are
going
to.
Let
you
know
that,
according
to
the
spec
for
kubernetes
at
the
CN
CF
is
set
out.
Your
cluster
is
breaking
some
of
those
rules,
so
applications
and
services
might
behave
differently
on
your
cluster
than
they
would
behave
in
other
clusters.
A
So
usually
you
can
come
through
and
you
can
fix
these
one
at
a
time
who
here
has
ever
looked
at
a
kubernetes
end-to-end
test
before
okay,
so
you
guys
kind
of
know
what
we're
looking
at
we're.
Gonna
look
at
one
of
these
failures.
Really
quick.
The
name
is
kind
of
what's
important.
You
have
the
sig,
that's
responsible
for
it.
A
So
the
very
first
thing
I
do:
whenever
I
have
a
kubernetes
cluster
is
I
just
run
this
thing
and
then
I'll
kind
of
run
it
every
so
often
maybe
a
week,
maybe
every
other
week,
just
to
keep
an
eye
on
my
cluster
to
make
sure
nothing
is
sort
of
like
seeping
out
or
there's
no
conformance
issues
that
are
coming
up
in
the
background.
So
as
that's
running,
we
can
now
look
at
hefty
Oh
arc.
A
Actually,
let's
do
case
on
it
first,
because
we
want
to
install
the
app
first
we're
gonna
go
through
the
cube
flow
installation
steps.
Here
again
somebody
say
when
it's
like
a
well
and
I'll
walk
you
through,
what's
going
on
like
kinda
behind
the
scenes
in
case
on
it
as
we
go
and
we
actually
set
up
a
cube
flow
cluster.
So
the
first
thing
we
want
to
do
is
we
want
to
export
a
namespace
variable
called
cube
flow
and
then
we're
going
to
create
a
namespace
for
that.
So
all
capital
namespace
equals
cube
flow
I'll.
A
Call
it
cube
flow
I,
don't
want
to
call
it
Q
flow
live
somebody
pick
a
word:
I
can
use
I'm
gonna
be
Mountain.
Sorry
I
mean
not
sorry.
Okay,
we're
gonna,
call
it
Q
flat,
Mountain,
okay,
and
that
way
we
can
just
look
for
a
mountain
as
we
go
through
the
whole
demo
here.
So
now
we
want
to
create
the
namespace,
so
we
can
copypasta
this
in
RAM
KaBlam.
We
have
our
namespace
we're
gonna
export.
The
version
of
cube
flow
we
want
to
use
version,
is
equal
to
0.12.
B
Right
now,
until
zero
point
one,
we
had
released
just
as
things
came
streaming
in,
we
will
have
a
zero
point
two
later
this
year
and
we
expect
to
hit
1.0
this
year.
No
promises,
of
course,
but
right
now
that
that's
basically
the
cadence
we
do
over
time
expect
to
get
to
a
cadence
similar
to
what
kubernetes
is.
Where
we're:
releasing
we're,
bundling
up
everything
up
and
releasing
once
a
quarter.
A
Awesome
so
the
next
thing
we
want
to
do
is
we
would
wanted
to
find
our
application.
Remember
I
brought
these
up
earlier.
These
are
ideas
that
are
important
to
humans.
Those
things
that
cause
all
those
problems
and
we're
gonna
call
this
one.
We're
gonna
call
it
Q
flow
Mountain
as
well.
I
like
helps
keep
Blue
Mountain
like
this
is
actually
kind
of
fun
now.
So
the
first
thing
we
want
to
do
is
we
want
to
initialize
our
app
with
case
on
it.
A
You
can
see
which
version
of
case
on
it
you're
running
in
by
doing
KS
version,
so
I
have
the
latest
version.
It's
zero
point.
10.1
and
I'm
gonna
do
a
chaos
in
it
and
it's
gonna
init,
the
app
name,
couplet
mountain
so
what's
happening
here.
Is
it's
actually
creating
some
local
files
for
us,
some
of
which
are
JSON
it,
which
is
where
the
case
on
it
name
comes
from,
so
we
can
look
in
this.
Let's
actually
go
here.
A
Let's
go
to
this
repo
I
want
to
create
a
new
directory
and
I
want
to
be
able
to
make
sure
I
can
push
this
up
when
I
get
done.
Github.Com
slash
David's
last
name:
/q
flow
demo,
slash
we'll
make
a
new
directory
we'll
call
it
Q
flow
Mountain.
Sorry
this
may
be
a
directory
called
cube
floating
in
your
repo.
Well.
A
So
in
this
case
on
it
an
it
here-
okay
cool-
this
will
be
a
little
bit
easier
for
us
to
explore
if
it's
the
only
thing
in
the
directory.
Okay.
So
if
we
list
this
out,
we
can
seem
you
have
a
directory
called
queue
flow
Mountain,
which
is
the
name
of
our
app
and
we
go
in
there
and
we
can
see.
We've
now
got
don't
get
ignore
a
dot
case
on
a
file
and
not
yeah
mol
components.
Environment
slid
in
vendor,
but
what's
really
important
here
is
the
app
that
you
animal.
A
So
you
come
in
here
and
you
look
and
it's
like
whoa
I
have
my
terminal
zoomed
in
that
much
and
that's
my
whole
application.
So
this
is
taking
the
complexity
of
a
kubernetes
app
and
slimming
it
down
and
making
it
as
small
as
possible
and
as
simple
as
possible.
Now
it
is
configurable,
you
can't
extend
it,
so
you
can
get
the
dynamic
configurations
that
you
could
get
with
the
rest
of
kubernetes.
A
But
the
point
is:
is
that
there
is
same
default,
so
you
can
sort
of
assume
a
lot
of
that
along
the
way
so
sort
of.
If
you
don't
need
it,
don't
don't
define
it
so
anyway,
we
have
our
server
here.
This
is
our
kubernetes
api
server.
Again,
it's
run
and
amazon
we're
running
version.
One
point
eight
point
four,
and
we
can
actually
tree
out
the
components
directory
here.
There's
our
live
sonnet
file
and
we
can
tree
out
the
Lib
directory,
which
these
are
more
abstractions
as
well,
ultimately
going
to
kubernetes
resources.
A
So
here
we
can
come
in,
we've
already
changed
directory
into
our
app
name
and
we're
gonna
set
environment,
and
this
is
what
I
feel
like
makes
case
on
it.
Unique
is
that
I
can
have
different
environments
with
the
same
application.
So
this
is
where,
like
David
had
mentioned,
it's
gonna
be
important
if
we're
running
in
Amazon
versus
Google
or
if
we're
running
in
a
stage
versus
production
or
dev
versus
production,
or,
like
my
super-awesome,
intense
google
teepee
use.
What's
what's
my
one
liner
here,
TP
use
our
tensor
flow
processing.
A
Units
that
are
designed
to
to
run
tensor
flow
in
Google,
so
I
can
set
my
environment
here,
which
is
equal
to
my
namespace
cube
flow
man
and
we're
gonna
add
this
registry
notice.
This
is
just
part
of
the
github
repo,
so
we
store
all
of
this
up
and
yet
we
add
our
registry
and
we
can
install
these
three
packages
now.
These
are
the
things
that
actually
describe
your
application
or
in
the
case
of
cube
flow
applications
or
components
of
an
application.
A
Okay,
its
generate
BAM,
and
we
can
come
in
here
and
we're
gonna
skip
this
part
because
we
don't
we're
not
concerned
about
anonymous
usage,
metrics
and
we're
gonna
go
straight
to
the
good
stuff.
We
actually
want
to
take
action
against
our
kubernetes
cluster,
so
this
is
going
to
be
the
thing
that
actually
goes
and
deploys
all
of
our
different
resources
to
kubernetes.
So
chaos
apply.
The
namespace
is
going
to
be
default
and
it's
going
to
be
the
cube
flow
core
directory,
so
we're
gonna.
A
Let
this
thing
run
it's
going
to
reason
about
a
few
things,
make
sense
of
it
and
then
actually
go
and
take
action
against
our
cluster
again,
using
whatever
cube
config
I
have
defined
in
this
case
AWS.
So
now,
let's
look
and
see
what
namespaces
are
ready.
Take
it
namespaces
cool.
So
now
we
have
a
cube
floatin
down
here
at
the
bottom.
A
It's
been
here
for
five
minutes
because
we
created
it
a
couple
minutes
ago,
but
we
can
actually
get
some
pods
there
and
see
what's
going
on
in
couplet
mountain
and
actually
see
what
a
cute
flow
is
running
under
the
hood.
That
looks
good
and
we
can
see
that
the
the
containers
are
creating
we're
bringing
cube
flow
up
to
life.
That's
it
all.
The
complexity
of
machine
learning
is
done
it's
out
of
your
way
and
as
soon
as
these
containers
come
up,
we
can
jump
in
a
Jupiter
notebook
and
start
training.
A
Our
model,
which
is
pretty
simple
and
I,
think
it's
a
pretty
decent
user
experience
so
to
grab
the
jupiter
notebook.
We're
gonna
get
the
TF
hub,
the
tensorflow
hub,
which
is
how
you're
gonna
connect
to
the
rest
of
your
machine.
Learning
stack
and,
let's
see,
if
I
can
remember
this
command,
I
might
have
to
look
it
up,
but
it's
port
forward.
A
A
A
So
we
want
to
go
here
to
localhost
8080
or
actually
you
only
one
do
8,000
right.
So
there's
a
Jupiter
notebook
up
and
running
so
I
can
sign
in
the
default
username
and
password
is
literally
anything
you
want
and
I'm
not
gonna
do
it
now,
because
they
would
totally
burn
my
laptop
up.
But
this
is
where
I?
Could
you
start
my
server
and
actually
start
training
models
and
that's
where
the
software
engineering
folks
would
come
in
and
actually
start
managing
their
app?
A
So
using
hep
do
tools
like
case
cyant,
where
you're
able
to
take
an
arbitrary
app
cube
flow,
encapsulate
it
and
deploy
it
to
kubernetes,
and
now
we
can
go
through
and
we
can
configure
multiple
environments
to
deploy
the
same
application
and
parameterize
all
those
environments.
So
it
changes
accordingly,
pretty
cool
all
right.
So
now
it's
like
hi,
I'm
in
machine
learning
software
engineer
and
they
come
to
me.
The
I'm
playing
the
of
the
cluster
operator
and
they're
like
alright
Nova
Google
has
these
new
TPS.
A
We
want
to
move
over
we're
in
amazon
now,
but
we
want
to
move
over
to
google
and
when
you
sold
us
kubernetes,
you
told
us
the
whole
point
was
to
be
cloud
agnostic.
So
that's
why
this
tool
arc
came
to
be
so
arc
does
backups
and
recovery,
but
it
also
allows
you
to
encapsulate
your
kubernetes
resources
and
we're
going
to
do
a
cross
cloud
arc
deployment.
So
in
order
to
install
arc,
you
would
go
to
the
arc
repository
here,
and
can
you
guys
see
this
ok
great?
A
You
can
come
to
your
cloud
provider
and
you
can
go
through
to
run
Arkin
AWS
and
the
installation
looks
like
you
need
to
create
an
s3
bucket
as
three
bucket
is
storage
external
to
your
kubernetes
configuration
so
that
you
are
backing
up
in
a
place.
That's
isolated
away
from
the
rest
of
your
kubernetes
cluster.
After
you
create
your
bucket.
We
want
to
create
a
user
that
we're
going
to
use
for
arc,
so
here
we
would
create
an
I
am
user.
A
A
Here
we
have.
How
are
we
doing
on
time?
Oh,
we
have
one
minute
left.
Oh
man,
okay,
I'm
gonna
go
really
fast.
We
can
cut
out
the
AWS
examples
and
we
can
see
our
config
okay,
so
I'm
gonna
do
the
world's
fastest
backup
and
restore
okay,
so
arc
backup.
This
is
a
cool
cuz
it'll,
be
timed.
Our
backup
create
we're
gonna,
call
it
Q
plumb,
Outen
and
second,
let
me
do
it
BAM.
So
we
created
that
and
now
we
can
do
a
que
delete
namespace
cube
flow
Mountain.
So
this
is
a
disaster.
A
Wait
for
it
wait
for
it's
deleting
everything
so
just
clobbering
the
whole
namespace.
We
can
do
an
arc,
restore,
create
we're
gonna
call
it
cube,
fill
mountain
and
we're
gonna
do
from
backup
cube
flow,
Mountain,
BAM,
and
so
that's
restoring
now.
So,
if
I
do
cake
in
namespaces,
it's
restoring
in
Amazon
split-screen
magic
because
we're
gonna
fly
through
this.
A
We
now
want
a
KTX
over
to
our
Google
teepee
use,
which
has
already
configured
with
arc.
In
the
same
way
we
configured
it
with
Amazon,
except
we
pointed
it
at
our
s3
bucket,
and
here
I
can
do
an
arc
backup
get
you
can
see.
We
have
all
these
various
backups,
the
important
one
here
being
cube,
slow
mountain
and
if
I
can
get
nodes
over
here,
you
can
see
that
we're
actually
running
no
longer
than
Amazon,
but
we're
in
gke
come
on
in
Internet
and
we
can
do
an
arc
I'm,
just
gonna
type.