►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Machine Learning the Kubernetes Way - Ania Musial & Keith Laban, Bloomberg
The Bloomberg Professional Service (aka "the Terminal") provides data, analytics, news, information and communication to professionals in business, finance, government, and philanthropy. Through our internal machine learning platform, Bloomberg is applying advanced AI and GPU-accelerated compute to dozens of domains such as NLP, computer vision, time-series analysis and personalization. In this talk we'll discuss how we adapted Kubernetes to support a diverse set of machine learning workloads; how an ecosystem of data connectors enables rich collaboration; how we address multi-tenancy, data security and privacy; and finally, how we support heterogenous compute requirements and hardware acceleration.
To learn more: https://sched.co/GrSJ
B
Just
a
quick
note
from
their
dream
team
here
so
keep
an
eye,
are
both
the
senior
software
engineers
at
bloomberg,
with
the
combined
15
years
or
so
of
tenure
and
on
a
daily
basis.
We
work
on
the
same
internal
data,
science
or
machine
learning
platform,
but
we
do
sit
in
two
different
parts
of
engineering,
which
at
least
most
days
encourages
a
healthy
balance
of
between
the
minds
of
distributed
computing
and
the
minds
of
AI
researchers.
B
If
one
of
my
jokes
are
funny
today,
that's
because
he
wrote
them
and
so
just
a
quick
shameless
plug
at
the
bottom.
If
you
want
to
check
out
a
little
bit
more
about
Bloomberg
data
science
and
kind
of
what
we
do
papers,
we
publish
I,
invite
you
to
look
at
our
blog.
So
our
outline
for
today
looks
a
little
bit
like
this
funny.
B
You
don't
count
well,
it
kind
of
looks
like
this,
or
at
least
it
did
way
back
way
back
when
the
company
was
founded
in
the
1980s
there's
an
actual
physical
terminal.
Hence
the
name
which
stuck
along
with
the
beautiful
colorful
hand-assembled
keyboard
that
you
see
here
and
the
Nasdaq
was
electronic
in
the
1970s
and
by
the
80s.
Electronic
trading
was
really
becoming
the
norm.
So
with
this,
the
terminal
was
really
truly
an
innovation
at
the
time
because
it
provided
unprecedented
command
line
driven
access
to
market
data.
B
So
you
should
really
feel
an
affinity
for
our
users
already,
because
what
else
do
you
need
in
life?
Besides,
you
know
a
colorful,
a
mechanical
keyboard,
some
key
caps.
You
know
nice,
shell
and
some
command
line
driven.
My
aliases
yeah
flash
forward
to
now
looks
a
bit
more
like
this.
It's
technologically
evolved
quite
a
bit
since
then,
but
it
still
comes
with
a
fancy
keyboard
I.
Nevertheless,
the
purpose
is
still
largely
the
same,
and
it's
to
provide
real
time
market
data
news,
pricing
quotes
and
analytics
as
a
service.
B
B
They
might
graph
its
closing
price
over
the
past
30
days,
maybe
even
search
for
it
being
mention
in
the
news,
and
in
order
to
do
this,
they
load
up
what
we
call
a
function,
those
of
you
with
an
acute
sense
of
detail,
or
if
you
saw
Kelsey
Hightower's
talk
this
morning,
you
might
realize
that
it's
called
a
function
because
that
used
to
actually
invoke
a
Fortran
subroutine,
but
today
it's
a
synonym
for,
like
a
nice
specialized,
GUI
or
application.
Now
the
temp
the
terminal
has
tens
of
thousands
of
these
and
they're
all
very
tightly
interconnected.
B
So
here
we
have
one
such
function.
News
trends
and
clients
really
use
this
to
find
out
which
companies
or
topics
are
most
discussed.
On
Twitter-
and
they
do
this
by
ranking
them
by
the
highest
highest
volume
of
related
tweets,
you
can
sort
these
based
on
the
largest
spike
and
also
the
increase
or
largest
total
volume.
So
I
mentioned
the
all.
These
functions
are
interconnected
and
in
this
case
to
provide
context
for
the
ranking.
B
There
are
charts,
there's
some
real-time
pricing
data,
there's
a
representative
tweet
on
the
right-hand
side,
and
each
of
these
cells
are
kind
of
linked
aside
to
other
types
of
functions
on
the
terminal.
So
you
can
really
go
down
the
rabbit,
hole
and
all
of
our
users
kind
of
have
their
a
secret
special
sauce
that
they
care
about
our
signals
that
they
care
about
when
they're
making
decisions.
B
So
just
what
them?
That's
one
tab
of
one
function.
You
can
kind
of
already
see
the
potential
right,
so
within
a
tweet
we
might
have
a
named
entity
detection
and
linking
pipeline
which
could
extract
this
Monsanto
point
text
and
then
kind
of
blink.
It
buy
it
to
the
Monsanto
US
equity.
That's
on
a
terminal
to
figure
out
that
this
is
about
the
topic:
agricultural
chemicals.
We
have
to
have
some
sort
of
you
know,
topic
classification
system
and
to
detect
publication
spikes.
B
B
Well,
if
we
make
the
assumption
that
events
move
financial
instruments,
so
they
move
securities,
then
for
our
users,
the
question
really
becomes
how
do
I
figure
out
which
events
move,
which
instruments
and
how
do
I
figure
out
the
occurrence
of
events
of
interest?
Does
anyone
remember
the
mortgage
crisis?
Maybe
okay?
B
So
from
an
engineering
standpoint,
the
question
here
really
becomes:
what
kind
of
tools
do
we
have
at
our
disposal
in
order
to
kind
of
automate
those
to
make
it
provide
this
as
quickly
as
possible
and
a
slightly
more
modern
take?
This
is
the
impact
of
social
media,
our
favorite
unaccessible
of
rumors
and
unvetted
events.
That
has
the
potential
for
getting
a
serious
advantage,
and
this
really
happened.
B
You
know
a
mistake:
tweet
on
the
AP
wire
about
an
explosion
that
caused
Barack
Obama
to
be
injured,
caused
the
market
to
react
really
sharply
and
quickly,
and
the
point
here
is
really
just
to
get
across
how
much
potential
market,
in
fact
there
is
or
how
much
money
there
is
at
stake.
If
you
have,
you
know,
above
in
your
software,
so
I've,
given
you
a
few
examples
or
applications
that
we
have,
many
of
them
are
in
text
processing,
but
it
doesn't
really
stop
there.
B
The
possibilities
are
infinite
and
it
really
stems
from
the
types
of
data
that
we
have
available.
So
we
have
plenty
of
image
processing
problems,
ranging
from
extracting
tables
from
press
releases
to
geospatial
data
with
live
shipping
routes.
You
know
crossed
with
oil
rigs
crossed
with
weather
patterns
that
might
hit
them
anomaly
detection
we
saw,
but
it
also
comes
in
other
flavors,
such
as
monitoring.
Oh,
the
back
insistence
that
we
have
that
actually
power
the
terminal.
Now,
there's
an
entire
world
of
risk,
modeling
or
forecasting.
B
We
get
hundreds
of
billions
of
ticks
a
day
as
well
as
other
types
of
events
that
we
saw
audio
wise.
We
have
many
transcriptions
we
can
make
from.
Let's
say:
earnings
calls
and
there's
a
wealth
of
information.
Retrieval
problems
such
as
those
within
the
search.
You
know
relevancy
and
ranking
domains
in
between
the
two
pictures
of
those
terminals.
B
I
saw
you
at
the
very
beginning
the
time
the
price
in
information
has
really
gotten
much
shorter
due
to
automation
and
markets
have
gotten
a
lot
more
efficient
and
their
users
and
their
expectations
are
really
getting
much
more
sophisticated.
So
the
interleaving
of
any
of
these
types
of
data
sets
really
becomes
a
powerful
necessity.
B
So
these
are
the
kinds
of
questions
they
should
spend
the
majority
of
their
time.
You
know
really
thinking
about
so
you
know
one.
What
is
the
data
that's
relevant
to
my
business
problem?
Do
I
need
two
years
of
tweets.
Do
I
need,
you
know,
takes
from
the
SMP
now.
What
is
my
data?
Look
like
what
formats
are
the
timestamps
across
my
fingers
for
ISO?
You
know
eighty
six,
it
was
what
features?
Can
I
glean
from
this
data
and
can
I
count
on
them
being
consistent
production?
What
metrics
should
I
use
to
measure
success?
B
What
am
I
mixing
maximizing
or
what
I
might
minimizing
what
types
of
algorithms
are
most
suitable
for
this
problem
and
how
should
I
count
tune
or
tweak
them
and
once
it's
deployed
when?
Is
it
appropriate
to
update
my
mom'll
such
that
you
know
the
live
data
isn't
drifting
too
much
from
my
original
training
set.
So
you
know,
you've
probably
noticed
here
that
there's
an
implicit
assumption
that
whatever
structure
infrastructure,
you
need
to
kind
of
answer.
These
questions
is
readily
available
and
these
are
really
kind
of
what
questions
right.
B
But
ultimately
the
efficiency
of
your
work
and
the
accuracy
of
your
model
really
depends
on
what
tools
you
have
available
so
dealing
with
this.
How
kind
of
takes
up
too
much
of
the
time
so
in
an
ideal
world
to
tighten
up
a
sort
of
model
development
lifecycle
that
we've
outlined
here?
These
are
our
requirements.
B
I've
broken
these
down
into
a
couple.
Different
categories
and
I'll
fill
the
bill
points
in
a
minute,
but
compute
wise.
You
know,
experiments
are
very
power,
hungry,
they're,
very
burst
or
don't
bursty
and
sees
a
little
nature.
They
require
our
on-demand
specialized
hardware.
You
know
very
resource
hungry
in
terms
of
data
liquidity,
the
notion
of
data
provenance
kind
of
where
it
came
from
how
it
moves
through
the
fabric
of
time
is
really
crucial
for
debugging
and
those
sorts
of
model
forensics.
B
This
typically
will
come
off
as
I
need
access
to
everything
in
a
perfect
format,
with
no
errors
at
snapshotted
for
all
the
time
and
then
a
separate
copy
for
each
of
my
experiments
would
be
perfect
and
on
the
right
here,
a
very
broad
category,
then
terming,
a
machine
learning
workflow
are
kind
of
ease
of
use.
It's
as
much
as
a
about
the
developer
experience
as
it
is
about
scientific
reproducibility.
If
you
want
to
actually
ship
any
of
your
models
to
production.
So
let's
pull
into
a
bullet
points
and
we
have
toolkits.
B
We
have
experiment
management,
we
have
zero
network
overhead,
etc,
and
it
actually
looks
quite
demanding
and
verbose
in
there
too.
Many
took
that
things
in
here,
so
we
can
just
think
look
at
the
slide
later,
but
I'll
point
out
one
thing
that
I
have
listed
that's
under
there
somewhere,
you
may
have
seen
it
in
some
other
case
studies.
B
It's
the
notion
of
hyper
parameter
tuning,
so
these
are
parameters
that
are
not
necessarily
I
learned
or
produced
as
the
result
of
training
your
model,
but
rather
something
that
you
provide
as
an
input
or
a
configuration
to
your
training
process.
So,
for
example,
what's
the
maximum
possible
height
of
my
decision
tree
or
if
I
wanted
to
run
the
same
set
of
experiments,
give
me
all
of
them
with
a
neural
net,
two
layers
of
the
post
2/3.
B
As
long
as
you
structure
your
code
properly,
which
we
don't
always
do,
but
the
heavier
your
computation
and
the
more
clever
you
have
to
be
about
how
you
schedule
these
sort
of
workloads,
because
the
great
search
is
not
always
feasible.
So
clearly,
this
method,
an
ideal
and,
with
some
sort
of
you
know,
infinite
amount
of
time
and
resources.
Maybe
it
would
be
possible,
but
life
is
short
and
so
is
headcount,
so
to
make
an
honest
attempt
at
achieving
Nirvana
to
require
it
will
require
the
collaboration
of
just
more
than
just
beyond
the
ml
practitioners.
B
If
so,
in
particular,
these
are
kind
of
what
I'm
sees
our
stakeholders
we're
trying
to
keep
all
of
them
at
least
happy
enough,
and
they
all
have
kind
of
different
priorities
that
sometimes
come
in
conflict
with
each
other.
So
the
ml
folks
that
we
just
saw
with
their
their
wonderful
demands,
there's
DevOps
Mike
James,
whose
ideal
world
is
the
complete
separation
of
application
and
hardware.
There's
an
infrastructure
engineers
like
Keith.
You
want
to
provide
a
robust
generic
solution
and
like
not
happening
compromises,
there's
product
managers
who
are
like
how
much
does
this
cost?
B
Ok,
and
if
you
deal
with
any
sort
of
sensitive
data
which
in
our
domain
we
definitely
do,
there
are
security
officers
that
have
to
be
involved
in
kind
of
every
step
of
the
process.
When
we
want
to,
let's
say,
integrate
new
policies
into
a
mature
network
at
Bloomberg
figure
out
an
appropriate
data,
permissioning
model
of
our
standards,
as
well
as
be
part
of
reviews
for
situations
such
as
am
I
even
allowed
to
export
this
model
into
production.
B
And
of
course,
we
can't
forget
the
wonderful
people
who
actually
pay
for
what
we're
doing
for
whom
privacy
you
know.
24/7
uptime
is
a
serious
priority,
so
we
all
have
to
come
together
to
some
sort
of
managed
solution,
a
grand
social
experiment,
if
you
will,
and
so
with
that-
and
what
was
everyone's
emotional
state
before
began.
Working
on
this
thing,
I'm
not
going
to
call
this
failures,
but
rather
kind
of
prior
work
or
market
research
or
kind
of
legacy
code,
or
things
that
still
making
money.
B
So
when
I
first
started
at
Bloomberg
eons
ago,
there
were
a
handful
of
people
that
were
just
sitting
in
my
row:
kind
of
experimenting
with
machine
learning
out
within
the
cord
news
infrastructure
team,
and
that
has
since
grown
into
an
entire
organization.
So
company
wise.
We
have
teams
that
are
dedicated
to
building
a
very
sophisticated
and
opt-out
form.
B
As
you
kind
of
saw,
as
well
as
information
retrieval
teams
that
specialize
in
building
models
into
certain
systems
a
shout
out
if
you
follow
learning
to
rank
and
sold,
learn
if
you
follow
that
community
and
I
should
note
that
they
don't
just
write
up
notebooks
and
toss
models
over
the
wall.
That's
not
what
we
do,
but
we
kind
of
take
pride
in
ownership
of
the
entire
software
development
lifecycle,
which
is
a
little
bit
different
than
other
places.
B
So
back
then,
and
when
I
started,
and
these
rules
based
systems
really
kind
of
turning
into
machine
learning
based
systems,
the
MLB's
systems
really
grown
out
of
extensions
of
existing
applications.
It's
not
really
new.
It's
a
how
the
formation
extraction
has
been
done
everywhere,
but
you
know
you
have
a
giant
rule
set
or
grammar
that
you
just
deploy
with
your
monolithic
service
so
enter
our
first
status
report.
First,
three
columns
should
look
like
failure
fairly
familiar,
but
the
fourth
one
kind
of
represents.
B
You
know
the
support
teams
or
the
stakeholders,
or
even
Mike
back
there
up
in
some
cases.
So
our
data
was
about
much
smaller.
Compute
was
really
limited
to
our
service-oriented
architecture
and
the
routing
that
we
had
a
hyper
parameter
tuning
at
that
I
mentioned
earlier,
wasn't
really
being
done
by
too
many
teams
due
to
limitations
and
how
their
systems
were
set
up.
B
B
So
the
teams
were
really
their
own
support
fast
forward,
a
little
bit
at
some
point,
like
many
other
companies,
big
data
era
rolled
around-
and
this
wasn't
really
a
replacement
right,
but
rather
kind
of
grew
alongside
the
other
Bowl,
because
you
have
to
keep
me
systems,
so
he
built
up
a
giant
data
Lake
in
HDFS,
but
given
the
Sin
City
of
the
data
there,
you
know
it
was
very
highly
secured.
So
it's
useful
if
your
applications
kind
of
contained,
but
sometimes
you
have
to
you-
know
graph
or
some
sort
of
other
parts
for
your
model.
B
B
If
you
were
lucky
enough
to
get
them
installed
there,
then
you
had
to
kind
of
figure
out
how
to
reliably
get
them
working
with
various
security
and
authentication
protocols,
which
is
always
a
wonderful
joy,
especially
for
long-running
jobs
like
furnishing
linen
workflows,
and
these
typically
kind
of
ran
in
our
yarn
as
support
wise
having
a
Hadoop
platform
team
was
really
nice,
though
not
every
use
case
on
that
compute
platform
was
really
ml
specific
there.
So
their
expertise
was
really
well-suited
for
classical
distributed
systems.
Problems
and
less
so
I
can't
tell
her.
B
My
topic
model
is
spitting
out
completely
outrageous
values.
Is
it
because
of
how
spark
is
set
up,
or
is
my
machine
code
kind
of
full
of
bugs
so
at
some
point?
Neural
networks,
kind
of
became
the
top
performing
class
of
models
and
many
domains
of
machine
learning,
and
we
wanted
to
support
various
specialist
groups
at
Bloomberg
to
explore
their
use,
and
this
happened
very
organically.
The
same
way,
then,
on
every
business
problem
is
suitable
for
machine
learning
approach.
B
Not
every
machine
learning
problem
should
be
solved
by
using
neural
networks,
so
we
gave
an
access
to
an
interactive
GPU
farm
to
establish
data
experts
within
we
bring
engineering
to
kind
of
assess
their
practicality
for
the
types
of
problems
that
we
are
working
on.
So
having
built
up
many
wonderful
datasets
in
the
past
couple
of
eras,
this
became
yet
another
exercise
and
data
Olympics
right.
We
had
access
to
blazing
hardware.
In
some
ways
this
was
kind
of
a
minor
step
back
to
our
rules-based
era,
where
folks
would
really
step
on
each
other's
toes.
B
But
only
at
this
time
it
was
some
much
more
expensive
hardware
and
we
got
some
isolation
by
running
in
containers
we
began
to
which
were
also
really
useful
for
kind
of
installing
the
latest
neural
network
packages,
but
going
back
to
practicality
of
scheduling
what
about
ETL?
Now
we
have
kind
of
folks
that
are
including
together
pipelines
between
you
know,
vastly
different
environments,
because
your
post-processing
really
happens
on
your
own
servers
or
maybe
within
hadoop,
so
support
wise.
All
sorts
of
questions
come
up.
You
know.
Why
did
this
machine
blow
up?
B
Why
is
this
essary
Kelly
might
notebooks?
Can
someone
get
the
latest
tensor
flow
drivers
installed
and
so
on
so
looks
like
leading
up
to
this
we've
identified
a
few
different
things:
one.
There
are
some
fairly
peculiar
types
of
workloads
that
machine
learning
our
world
requires
and
based
on
the
limitations
of
the
previous
systems.
I
just
talked
about
this
needs
to
be
needs
to
have
some
serious
thought
put
up
front
in
regards
to
data
liquidity,
proper
multi-tenancy
and
really
stability,
and
up
until
now
there
have
really
been
kind
of
ad
hoc
solutions
for
each
of
these.
A
B
A
A
A
So
when
we
chose
kubernetes
is
because
it
offered
all
the
abstractions
we
needed
to
build
the
platform
that
we
wanted
to
build
and
deploy
and
run
these
processes
in
a
way
that
we
can
utilize
compute
much
easier.
So
there
is
jobs
and
pause
and
deployment
and
so
on,
and
these
are
all
like
declarative
ways
to
specify
your
application,
but
in
a
kind
of
abstract
way,
we've
all
kind
of
found
these
abstractions
to
be
a
little
too
abstract.
A
It's
called
our
tensorflow
job.
So
from
early
on,
if
we're
looking
at
the
the
spec,
we
wanted
to
own
this
concept
called
frameworks.
So
instead
of
ml
practitioners
having
to
learn
docker
and
losing
focus
on
the
the
job
at
hand,
they
can
just
require
one
of
our
managed
frameworks
and
we
built
a
bunch
for
the
most
common
languages
and
toolkits
for
data
scientists.
So
you
require
a
baseline
environment.
A
Like
this
tensorflow
1.7
python
3,
and
then
we
can
own
the
base
operating
system
set
any
sort
of
internal
proxies
configurations
that
we
need
and
make
sure
all
the
drivers
are
set
up
right
and
so
ml.
Folks,
don't
have
to
think
about
that
and
now,
because
people
don't
create
their
own
docker
images,
they
need
some
way
to
get
their
runtime
dependencies
into
the
base
framework,
and
so
they
can
do
that
by
specifying
their
packages
and
the
module
they
want
to
become
their
entry
point.
A
A
A
A
A
Maybe
the
amount
of
data
they
were
trying
to
crunch
and
choose
a
small,
medium
large
sort
of
t-shirt,
size
and
specify
here
with
also
the
CPU
GPU
prefix
to
into
the
schedule
or
what
they
needed
for
hardware,
and
if
this
doesn't
work,
you
need
to
go
deeper.
We
do
have
custom
sizing
for
a
specific
requires,
request
limits,
and
this
last
bit
here
is
just
for
the
arguments
that
get
passed
to
the
application.
A
Here,
you
can
see
they're
going
to
use
the
Hadoop
identity,
which
was
specified
earlier,
so
we've
taken
the
step
further
and
extended
it
to
multiple
different
types
of
CR
DS.
In
order
to
satisfy
all
of
the
themes
from
earlier,
we
have
Python
and
JVM
jobs
to
provide
an
appropriate
price
base
framework
for
many
of
the
other
common
data
science
choices.
A
This
job
takes
as
a
template
all
of
our
other
types
of
jobs
and
basically
schedules
a
black
box
hyper
parameter
tuning
cycle
over
your
parameter
space,
so
setting
all
this
up
has
really
given
us
a
good
groundwork
framework
for
onboarding
individuals
and
to
scale
for
more
people
and
sort
of
allow
our
ml
folks
to
think
about
ml
without
having
to
think
about
compute.
Specifically.
A
So
at
this
point
we
sort
of
have
parody
in
this
data
column
with
our
previous
iterations
HDFS,
s3
and
various
other
data
sources.
We've
also
combined
almost
everything
from
the
big
data
world
and
the
GPU
world
in
terms
of
compute
well,
at
the
same
time,
making
it
more
ml,
specific
and
declarative.
By
introducing
native
kubernetes
CR
diaz
and
with
using
CR
DS,
we
can
sort
of
now
split
out
what
ml
practitioners
care
about
and
operational
teams
care
about,
because
we
have
kubernetes
scheduling
it.
A
A
So
some
of
these
folks
were
before
having
exclusive
access
to
maybe
their
own
team
server
VM,
which
had
like
a
quick
feedback:
loop,
toolkits
they're,
familiar
with
like
native
UNIX
eli's,
and
now
we
want
them
to
move
to
the
cloud
and
use
tools.
They've
never
used
before
to
debug,
which
may
be
feels
like
they're
debugging
in
a
black
box.
A
Additionally,
straight
porting
things
like
things
that
run
into
JVM
or
use
Python
multiprocessing.
If
those
weren't
running
in
containers
before
you
might
notice,
there's
some
different,
interesting
behaviors,
you
try
to
move
into
a
container
environment
and
from
an
infrastructure
perspective.
When
we
started,
we
really
wanted
to
build
something
that
could
be
injected
into
this
MD
LC
model
development
lifecycle
and
the
way
that
software
development
lifecycle
works.
We
want
to
enforce
our
best
software
development
practices
on
that,
but
that
doesn't
always
necessarily
lead
to
the
best
ease
of
use.
A
So,
just
to
recap,
when
you
had
this
server
or
VM
or
development
environment
setup,
you
have
a
repple
or
maybe
even
a
local
Jupiter
notebook.
So
now
we
want
to
bring
that
interactivity
to
the
cloud,
and
so
the
first
thing
we
did
was
expose
a
Jupiter
job
which
I'm
sure
many
people
are
familiar
with,
and
it's
also
serve
a
hot
topic
in
the
community
about
how
it
should
be
presented
to
people.
A
A
And
this
coupled
with
a
lot
of
education,
about
how
the
sort
of
cloudy
paradigm
works
has
made
people
a
lot
more
comfortable
with
switching
over.
So
things
are
starting
to
look
up
a
little
bit
for
ml
practitioners,
but
with
new
every
new
bit
of
power
given
to
users,
we
have
to
worry
about
how
that
power
is
going
to
be
abused
from
the
infrastructure
side.
So
in
this
case
all
those
greedy
interactive
users
are
probably
going
to
be
clocking
up
our
resources
and
locking
GPUs
and
CPUs
right.
A
So
now
we
sort
of
get
to
some
better
problems.
In
the
past,
we
just
wanted
to
get
people
on
the
platform,
but
we
had
no
people,
so
we
don't
have
to
worry
about
scale
and
we
just
kind
of
were
able
to
build
something,
but
now
everyone's
using
it,
people
are
using
interactive.
We
need
to
figure
out
how
to
make
it
fair
for
everybody.
A
So
the
question
now
is:
how
do
you
make
users
care
about
sharing
this
kubernetes
cluster
and
while
we're
doing
that
by
forcing
it
with
infrastructure?
So
first
thing
you
want
to
create
a
namespace,
and
this
is
something
we
did
from
the
start.
I'm
every
user
gets
their
own
namespace
and
namespaces
obviously
can
have
a
resource
and
security
benefits,
and
we
knew
there
was
going
to
be
a
lot
of
things
that
we
were
gonna
add
into
the
namespace.
A
So
we
created
this
higher
level
concept
called
a
workspace
and
a
workspace
is
basically
just
another
C
or
D,
which
defines
all
these
things.
You
see
here
all
right,
so
it
was
sort
of
interesting
because
in
the
beginning
we
wanted
to
get.
It
was
all
about
like
user
adoption
right.
So
we
sort
of
had
this
small
cluster
that
we
can
put
the
bill
for,
and
other
teams
didn't
really
have
to
worry
about
getting
budget
to
join
into
our
platform.
A
But
that
didn't
really
allow
us
to
give
production
level.
Sla
is
because
we
can
never
guarantee
we'd
have
capacity
to
schedule
right,
but
now
people
are
getting
comfortable
that
they
want
to
join.
They
want
to
use
the
platform
we've
built,
so
we
need
to
figure
out
how
we
can
sell
it,
sell
it
to
them,
so
we
split
it
out
into
sort
of
two
different
tiers.
A
We
have
this
user
tier,
which
is
more
for
developing
and
testing
and
has
lower
priorities
for
scheduling
and
then
a
shared
workspace
which
you
pay
for
with
a
budget,
and
then
you
sort
of
get
a
resource
quota
limit,
but
a
much
higher
priority
during
scheduled
time.
So
now
all
the
sputters
we
were
worried
about
can
sort
of
still
squat,
but
at
least
they're
paying
for
it
and
another
small
addition.
We've
added
was
deadlines
to
our
CR
DS
in
the
way
that
jobs
and
pods
already
support
deadlines.
A
A
But
if
you
want
to
support
both
a
production
level,
SLA
and
easy
onboarding
with
this
sort
of
low
priority
and
might
schedule
at
some
point
tier,
you
need
to
sort
of
introduce
this
idea
of
preemption
now
and
some
applications
like
tensorflow
spark
a
little
bit
Jupiter
a
little
bit
have
visibility
for
checkpointing
built
into
it,
but
the
more
naive
toolkits,
maybe
don't
have
that,
and
it
needs
to
be
added
or
adopted,
which
is
going
to
be
now
another
challenge
that
ml
practitioners
are
gonna
have
to
face.
But
on
the
other
hand,
we
can
guarantee
better.
A
Sla
is
because
we
no
longer
have
to
worry
about
specific
resource
questions
and
squatters
and
so
on,
because
serv
built
into
the
system
now
so
now
that
we've
made
the
cloud
a
little
bit
easier
for
users
and
we've
given
them
access
to
compute.
We
need
to
remember
that
we're
still
building
a
full-featured
ml
platform
and
that's
only
part
of
the
story
right.
So
let's
just
talk
about
some
other
features.
We've
been
thinking
about
and
adding
into
the
platform.
A
The
first
one
is
a
history
store,
so
you
know:
coop
control
is
great.
Kubernetes
is
great
for
getting
the
jobs
which
you've
submitted,
but
if
you
have
multiple
clusters
or
many
many
jobs,
SUT
might
not
scale
the
way
you
want
it
to
so.
We've
taken
that
and
put
it
into
a
separate
database
which
allows
us
to
tag
and
annotate
jobs,
so
you
can
sort
of
look
at
multiple
trial.
A
Multiple
runs
of
a
trial
together
and
aggregate
them
after
the
fact
of
running
them
in
kubernetes,
we've
also
been
adding
features
to
collect
metrics
and
performance
information
from
the
models
which
you've
built,
and
so
we've
really
been
able
to
roll
this
into
a
full
model.
Development
lifecycle
workflow.
A
So
if
we
look
at
the
status
report
again
now
we're
basically
just
adding
features
to
our
ml
workflow,
so
continuing
on
that
trend,
we
need
a
couple
of
more
features
that
a
full
platform
should
have
so
up
until
now.
We've
talked
a
lot
about
training,
a
model
and
other
types
of
data
science
and
manipulation
that
might
actually
not
necessarily
end
up
with
a
model
as
the
output.
A
But
if
you
are
training
a
model,
you
probably
want
to
serve
some
predictions
with
that
model
right,
so
at
Bloomberg
we
actually
have
a
bunch
of
different
ways
to
deploy
applications
which
in
turn
can
be
used
to
deploy
model
prediction
service,
but
you
might
be
doing
inference
in
some
other
ways
as
well.
So
I
knew
before
mentioned.
A
Solar
is
learning
to
rank,
which
is
basically
a
prediction
serving
service
that
lives
inside
of
the
solar
search
engine
itself,
and
so,
if
you're,
training
on
our
platform,
you're
gonna
have
to
go
and
move
to
another
ecosystem
to
do
that
anyway.
But
a
lot
of
people
want
the
seamless
way
to
go
from
training
to
serving
and
don't
need
anything
too
specialized.
So
it's
a
great
candidate
for
a
full-featured
platform,
and
so
right
now
we're
working
on
kein
ADA
to
build
model
to
serve
lambda
pipelines
and
then,
once
that
exists,
we
can
serve
integrate.
A
Some
of
our
folks
are
working
on
bringing
that
to
kubernetes.
So
just
to
tie
this
all
together.
Bloomberg
has
6000
engineers,
so
really
it's
an
engineering
company
at
heart,
with
a
huge
wealth
of
data
available
and
so
to
allow
our
engineers
to
build
products
utilizing
this
information
quickly
and
efficiently.
We
really
need
to
simplify
all
of
the
roles
involved
and
so
with
kubernetes
and
the
data
science
platform,
which
we've
been
building.
We've
been
able
to
split
those
roles
up
between
infrastructure
and
ml
specific.
A
So
thank
you
all
for
coming
out
to
the
talk
we
had
a
couple
of
other
presentations
from
Bloomberg
I.
Think
they've
all
happened
already
and
if
you
haven't
seen
them,
you
can
see
them
online
and
if
you
want
to
be
my
teammate,
you
could
click
on
that
link
and
I.
Don't
think
we
have
time
for
questions
so
I
will
meet
anyone
that
wants
to
talk
on
the
side
over
here.
Thank
you.