►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building a Medical AI with Kubernetes and Kubeflow - Jeremie Vallee, Babylon Health
Engineering AI systems at scale can be difficult, especially in highly regulated environments like healthcare. Many challenges arise, such as ensuring reproducibility, controlling data access policies, and running highly secure infrastructure. But with some planning and meticulous engineering, this can be achieved. At Babylon Health, we've leveraged Kubernetes, Kubeflow, Argo, Istio, OPA, and many other Cloud Native technologies to provide a secure research platform for building and scaling medical AI models across the world. In this talk, we will share our experience so far, give an overview of how these components fit together, and explain our vision for the future of our platform. We will demonstrate how using open-source CNCF technologies can help you achieve your goal of experimenting, training and serving your AI models at scale, while operating in a regulated environment.
https://sched.co/UaWf
A
Ok,
everybody
we're
gonna
get
going
here
soon.
I
just
want
to
adjust
myself
I'm
Matt
Baldwin
I'm
gonna
be
your
track
host
today.
I.
Thank
you
for
coming
out
to
building
a
medical
AI
with
kubernetes
and
ku
flow
by
Jimmy
Valli
who's,
an
infrastructure
lead
at
Babylon
Health
prior
after
the
session.
I
want
to
encourage
you
to
go
to
the
schedule
app,
please
rate
the
session.
If
there's
any
questions,
I
asked,
please
hold
those
to
the
end.
A
B
It's
yeah
hi
everyone.
Thank
you
for
coming
to
this
session
today,
I'm
going
to
talk
about
a
transformation
from
running
a
set
of
training,
jobs
and
new
beauty
on
netbooks
and
all
other
AI
workloads
on
bare
metal
servers
to
a
running
them
in
a
cloud
platform
on
top
of
communities
and
the
multi
region
way,
and
actually
a
multi
cloud
way.
B
But
it's
it's
happening
in
the
context
of
health
care
and
it
comes
with
different
other
requirements.
So
my
name
is
Jeremy
I
work
for
a
company
called
Babylon
health,
and
our
mission
is
to
provide
an
affordable
and
accessible
health
service
in
the
hands
of
everyone
on
earth,
which
is
a
big
mission,
and
it
comes
down
to
two
main
challenges.
The
first
one
is
accessibility
and
the
second
one
is
affordability.
B
So
for
tackling
the
accessibility
problem,
we
have
to
look
at.
What
can
we
do
to
bring
people
closer
to
healthcare
professionals
and
what
we
did
in
2014?
Is
we
launched
a
project
that
would
allow
you
to
get
a
digital
consultation
with
a
doctor
from
your
mobile
phone,
and
so
you
could
have
a
video
or
audio
consultation,
and
it's
been
five
years
now
and
we
reach
over
two
million
digital
conversations.
So
the
service
has
gotten
a
lot
of
traction
and
is
working
well
but
then
comes
the
problem
of
availability.
B
Accessing
healthcare
professionals
is
expensive
and
actually,
according
to
the
World
Health
Organization,
we
actually
have
a
shot
shortage
of
about
5
million
healthcare
professionals
in
the
world.
So
it
becomes
a
very
hard
problem
to
scale
accessing
healthcare
professionals.
But
there
is
something
that
we
all
know
is
easy
to
scale
in
getting
cheaper
and
cheaper
every
year,
and
there
is
a
computer
do
to
work
on
the
affordability
issue.
B
We
started
thinking
what,
if
we
could
use
computing
technology
to
bring
some
of
the
services
that
would
you
you
would
usually
get
from
a
healthcare
professional
and
also
serving
it
from
a
device
that
most
people
have
which
in
this
case
is
a
mobile
phone,
and
so
we
created
different
sets
of
products
that
kind
of
help
you
get
get
started
with.
Managing
your
health.
We
have,
for
example,
a
symptom
checker
when
you
can
tell
the
the
app
if
you
have
a
headache,
for
example,
and
then
yeah
we'll
try
to
understand.
What's
going
on.
B
All
of
this
requires
different
type
of
technologies
and
we're
working
on
things
like
NLP
computer
vision.
We
also
do
Bayesian
networks
and
all
sorts
of
other
other
products,
and
that
means
a
lot
of
research
and
a
lot
of
computes.
So
I'm
going
to
tell
a
bit
about
how
this
all
started
in
our
company
about
15
months
ago,
I
had
the
discussion
with
researchers
at
Babylon
and
they
had
built
a
graph
based
neural
network
and
they
were
running
hyper
parameter
tuning
on
our
on-premise
servers
and
they
wanted
to
improve
the
efficiency
of
our
models.
B
But
it
became
quite
clear
that
we
not
only
didn't
have
enough
compute
to
run
this
hyper
parameter
tuning,
and
we
also
didn't
have
the
orchestration
to
make
that
happen.
It's
it's
actually
very
complex
to
orchestrates
hyper
parameter
search,
especially
when,
when
on
Prem
like
this,
like
we
were,
and
at
the
same
time
our
company
just
had
migrated
all
of
our
workloads
to
kubernetes.
So
we
thought
all
right.
B
B
We
run
about
fifty
nodes
for
week
and
that
resulted
in
1606
over
three
terabytes
of
RAM,
so
much
more
compute
than
we
than
we
had
locally
and-
and
it's
actually
quite
a
it-
was
a
really
good
success.
For
us.
We
managed
to
get
good
results
and
the
the
team
published
a
paper
that
used
this
experiments
as
the
experimental
set
up
of
their
theory
and,
at
the
same
time
we
were,
our
company
was
growing.
B
We
were
embodying
more
and
more
teams
doing
AI
models
and
we
we
wanted
to
find
a
way
to
make
these
teams
more
efficient.
So
we
started
looking
at.
How
can
we
do
to
maybe
centralize
all
of
the
things
that
arrived
that
that
gravitates
our
on
machine
learning?
So
how
do
we
manage
infrastructure?
How
do
we
manage
our
models?
And
so
we
we
thought
of
this
idea
of
having
a
platform
where
you
could
take
from
92
production
very
easily.
B
They
are
experimenting,
training
tuning,
your
model,
validating
your
model,
I'll
talk
about
it
a
bit
later
in
details
and
then
deliver
them
to
production,
but
we
have
to
think
about
being
in
the
context
of
healthcare
and
trying
to
understand
what
does
it
mean
to
do
this
in
the
context
of
healthcare?
So
the
first
thing
that
comes
to
mind,
obviously
security,
but
also
we
wanted
a
good
auditing
solution
so
a
way
to
find
what
happened
on
the
platform
and
for
what
reason,
and
also
another
important
thing
was
safety.
B
We
wanted
all
of
our
models
that
get
produced
by
this
platform
and
produced
by
Babel
in
general
to
be
safe
once
once
they
reach
production.
Another
thing
that
we
wanted
to
keep
in
mind
is
that
working
in
healthcare
and
deploying
your
stack
in
different
countries
you
operate.
You
have
to
think
about
where
the
data
is
and
most
of
the
times
the
data
will
need
to
stay
in
the
country.
B
B
I'll
talk
about
a
bit
the
the
use
cases
that
we
have
and
what
we
are
planning
for
the
future-
and
my
hope
here
today
is
that
I
can
show
you
that
using
open
source
technologies
and
technologies
that
you
can
find
the
cloud
native
world
can
help
you
also
and
build
your
own
platform.
If
you
need
to
ok,
so,
let's
start
with
the
base,
it
all
starts
with
the
infrastructure
itself.
There
are
a
few
things
to
keep
in
mind
where
you're
building
a
machine
learning
platform.
The
first
thing
is,
you
actually
have
to
manage.
B
You'll
have
to
prepare
for
4
different
type
of
workloads.
Some
teams
are
going
to
need
GPU
nodes.
Some
other
teams
will
needs
CPU
intensive
nodes.
You
have
to
make
sure
that
you
are
prepared
for
all
of
this,
and
so,
where
you're
running
in
a
cloud
environment
you
can
use
Auto
scanning
groups,
so
make
sure
that
you
prepare
different
type
of
nodes
for
that.
We
currently
use
six
different
type
of
nodes
at
Babylon.
B
Now
another
thing
to
keep
in
mind
is
you
want
to
make
everything
that
you
have
in
that
cluster
private's
some
clusters?
By
default?
We
have
a
public
humanities.
Api
I
really
recommend
you
to
make
it
private
and
same
thing
goes
for
the
nodes
we
run
all
of
our
nodes
in
private
subnets
and
the
access
is
very
restricted.
We
also
encrypt
all
of
our
nodes
and,
depending
on
the
cloud
provider
that
you're
going
to
use,
you'll
see
that
sometimes
it
comes
for
free
and
you
don't
have
to
do
anything
if
you're
using
AWS.
B
One
thing
that
we
found
out
is
you,
so
you
must
encrypt
your
EBS
volumes
if
you
want
encryption
at
rest,
but
also
make
sure
that
if
you
want
your
route
volumes
to
be
encrypted,
you
need
to
encrypt
the
a.m.
eyes
before
deploying
them,
and
for
this
we
use
Packer.
So
we
prepare
all
of
our
notes
with
using
packer.
We
have
down
the
nodes
and
then
we
encrypt
them,
and
then
these
a.m.
is
get
encrypted.
B
B
We
also
want
to
use
OPA
as
an
admission
controller,
and
that
allows
you
to
kind
of
like
filter
some,
some
errors
that
could
be
sent
by
users
such
as
forgetting
to
put
a
label,
though
so
OPA
could
actually
block
the
request
to
create
a
pod
and
say
no.
This
label
must
be
here
Oh,
for
example,
looking
different
CPU
requests,
and
you
want
to
enable
a
humanities
audit
logs,
which
is
very
important
to
know
exactly
what
happened.
B
You
can
find
all
the
events
that
happen
during
the
during
your
workloads
after
you
have
the
QB
on
this
cluster,
and
you
want
to
bootstrap
it,
and
so
for
this
we
used
get-ups
methodology.
We
deploy
everything
on
that
cluster
using
get-ups.
The
tool
that
we
chose
for
this
is
flux
and
flux
is
a
tool
that
will
listen
to
changes
on
the
github
repo
and
once
the
change
happens,
it
will
pull
the
latest
information
and
apply
this
in
your
cluster.
There's
a
few
examples
of
things
that
we
deploy
right
here,
what's
really
good
with.
B
That
is
that
we
can
use
github
as
the
source
of
truth
of
everything
that
happens
in
the
cluster.
It's
very
good
for
auditing
as
well,
and
you
can
see
who
decided
to,
for
example,
a
great
issue
at
2:00
a.m.
in
the
morning.
For
some
reason
it
really
reduces
human
error,
so
we
don't
have
to
operate
the
cluster
ourselves
anymore.
We
let
flux,
apply
all
of
the
changes.
B
All
right,
another
thing
that
you
want
to
configure
and
secure
is
your
networking
layer
on
kubernetes.
We
wanted
a
way
to
provide
a
zero
trust,
which
means
that
we
didn't
want
service
to
be
able
to
reach
another
service
without
having
us
configuring
that
specifically,
we
also
wanted
a
way
to
encrypt
communication
between
services
and
perform
util
Charis.
B
I
put
an
example
of
a
configuration
that
would
allow
service
a
to
talk
to
service
see,
so
you
create
a
service
role
that
allows
a
call
to
service
c2
to
go,
and
then
you
bind
it
to
the
service
account
of
service
a
and
you
can
do
all
kind
of
different
configuration
on
this.
You
can,
for
example,
if
you
need
to
restrict
access
to
the
Internet,
and
you
can
use
it
to
to
help
you
with
that.
There's
other
configuration
to
keep
in
mind.
B
Oh
okay,
let's
talk
about
cluster
management,
so
at
first
we
we
had
to
think
about.
How
would
we
structure
everything
that
runs
on
a
equipment
is
cluster,
so
we
decided
to
come
with
the
name
of
a
project,
everything
that
will
run
everything
that
our
users
will
run
on
the
platform
will
be
a
project
and
that
project
will
be
isolated
from
other
projects
so
that
you
wouldn't
actually
be
sharing
resources
directly.
B
For
this
we
arrived
with
a
NFS
like
a
solution
where
you
can
actually
attach
as
many
pods
as
you
want
to
the
same
volume.
So
if
you're
running
a
distributed
workloads,
they
could
all
be.
Writing
to
the
same
folder,
for
example,
which
is
quite
convenient,
so
we
created
a
project,
custom
resource
on
communities
and
an
operator.
So
every
time
a
user
would
a
project
we
would
set
up
an
m-space.
Some
are
back
rules
and
the
persistent
volume
partition
a
bit
later.
B
B
Ok,
so
let's
talk
a
bit
about
coop
flow
in
flow.
Is
the
machine
learning
toolkit
for
kubernetes
and
it
comes
with
many
different
components.
You
can
see
the
amount
on
the
screen
right
now
and
what
we
really
have
about
queue
flow.
Is
that
it's
very
modular,
so
you
can
actually
decide
what
you
want
to
install
and
then
based
on
your
needs,
then
you
can
deploy
deploy
them
so
here,
I
give
some
examples
of
what
we
use
with
queue
flow
I
talked
about
khatib
earlier
and
we're
also
big
fan
of
notebooks.
B
So
we
use
the
jupiter
notebook
controller.
We
have
tensor
flow
training,
jobs
and
pie,
table
jobs
as
well,
and
we
started
using
our
go
workflows
for
some
of
the
use
case.
Actually
one
use
case
I'll
talk
about
later,
but
all
of
this
is
deployed
again
with
the
same
get-ups
metallurgy
about
humans
ago.
Cubes
removed
all
of
its
deployment
tools
to
customize,
so
you
can
easily
play
with
customized
generates
a
big
yamo
configuration
put
that
to
your
github
branch
and
have
a
tool
like
flux.
Deploy
that
automatically.
B
You
can
also
use
other
tools
for
cg
like
I'll.
Go
I'll
go
CD,
which
is
a
good
one,
and
since
0.6,
which
came
out
this
summer,
cube
flow
is
multi-user,
which
means
that
you
can
access
and
control
the
access
to
some
of
the
UI
sub-q
flow,
like
repeat
on
that
books
for
several
people,
and
this
is
an
example
of
how
it
works
under
the
hood.
There's
been
a
talk
already
at
cube.
Come
about
this,
so
I
won't
go
into
too
many
details,
but
the
idea
is
when
a
request
goes
through
a
load
balancer.
B
We
have
an
invoice
filter
that
will
redirect
the
traffic
to
the
queue
pro-earth
service,
and
then
you
can
integrate
it
with
your
IDP
I
put
Dex.
Here,
it's
an
open
source,
one
we
use
alpha
31
and
it
works
right,
and
so
once
you
authenticated,
the
queue
of
service
will
inject
a
header
into
the
into
the
call
that
will
then
get
to
the
queue
flow
gateway
and
will
redirect
it
to
the
trippity
notebook
that
you're
trying
to
to
access.
B
Another
part
that
we
had
to
think
about
is
how
do
we
access
data
and
how
to
configure
secrets
on
the
platform
at
Babylon
we
have
a
specific
team,
that's
dedicated
to
providing
access
to
data
sets
and
they
built
a
an
access
layer
and
we
integrate
with
that
layer
from
our
platform.
But
the
idea-
and
what
you
want
to
keep
in
mind,
is
that
you
really
want
to
audit
who
gets
access
to
the
data
and
that's
really
what
we
wanted
to
do.
B
Okay,
this
is
actually
pretty
important.
We
when
we
created
the
the
platform
very
early
on,
we
had
users
sending
us.
Okay,
I,
want
to
see
exactly
what's
going
on,
I
want
to
see
how
my
training
job
is
doing,
so
we
need
to
find
a
way
to
collect
all
of
the
metrics
that
were
running
on
the
cluster
and
we
also
wanted
for
for
people
like
me,
live
on
the
Emer
upsides.
B
We
wanted
to
know
how
the
cluster
was
doing,
and
for
this
we
went
with
a
very
obvious
solution
which
is
Prometheus
to
collect
the
metrics
of
our
clusters
and
graph
a
net
to
visualize
them.
What's
really
good
with
graph
a
nice
that
it's,
you
can
automate
your
dashboards
quite
easily.
So
this
is
an
example
of
a
dashboard
that
we
provide
to
our
users
and
every
single
project
that
gets
created
gets
its
own
dashboard
on
graph
Anna.
B
So
people
can
follow
and
we
add
more
and
more
monitoring
and
metrics
after
that,
and
we
also
want
people
to
expose
metrics
during
their
jobs
so
that
if
they
want
to
build
custom
dashboards,
they
can
fuel
instant.
Listen,
sorry
that
we
learned
our.
If
you,
if
you
want
to
use
graph
Anna
and
you
want
to
deploy
it
on
a
Chianti's
cluster,
there's
a
helm
shot
for
this
and
it
is
quite
convenient,
every
single
dashboard
can
be
be
applied
as
a
config
map.
B
B
So
you
might
have
seen
this
in
the
first
slide
about
monitoring,
but
one
thing
that
came
out
also
very
quickly.
When
people
started
using
the
platform,
they
wanted
to
understand
how
much
they
were
going
to
spend
so
again
keep
in
mind.
This
was
running
on
the
cloud
and
everyone
was
actually
asking
us.
Ok,
but
like
is
it
gonna
be
expensive,
like
I
need
to
know,
so
we
try
to
find
a
way
to
expose
like,
like
you
can
see
here.
B
I
expose
the
cost
of
every
research
project
and
we
try
to
look
around
for
open
source
tools
and
there's
actually
not
a
lot
of
them.
We
we
found
one
which
is
a
third-party
tool,
but
osburgh
open
source,
their
cost
model
called
cube
cost
and
you
can
check
it
out.
It
actually
exposes
the
metrics
to
Prometheus.
B
Okay,
so
taking
a
break
here,
having
a
look
at
what
we
have
so
based
on
all
of
this,
we
we
had,
we
actually
managed
to
create
a
cluster
where
people
could
only
men
start
training,
jobs
and
notebooks
in
equivalencies
cluster.
We
introduced
monitoring
and
we
had
our
back
controls
for
our
networking,
multiple
TLS,
that
was
quite
cool.
At
that
time,
the
users
were
actually
talking
to
the
communities
API
directly,
and
so
one
thing
that
we
found
out
is
it's:
it's
actually
a
bit
complex
for
some
users.
B
Some
users
don't
really
want
to
know
exactly
how
to
build
a
Chianti's
manifest
and
I
would
argue
that
that
that's
where
another
problem
that
we
had
is
that
we
still
were
running
on
only
one
single
cluster
and
wants
to
find
a
way
to
kind
of
manage
several
clusters
at
a
time
and
be
able
to
spin
up
resources
in
different
regions
of
the
world.
And
lastly,
we
started
by
telling
users
to
just
create
projects,
get
up
styles
so
again
create
a
llamó
file,
put
it
on
github.
B
B
B
Again.
If
we,
if
we
wanted
to
actually
control
some
of
an
abs
on
business
logic
about
permissions
from
those
API
calls,
we
can
use
OPA
and
one
example
would
be
if
you
want
your
project
by
default,
to
have
GPUs
disabled
because
you'd,
rather
maybe
contact
the
administrators
to
say.
Ok,
this
project
is
now
allowed
to
use.
Gpus
GPUs
are
expensive
on
the
cloud,
so
you
might
want
to
control
that
you
can
actually
bring
your
own
business
logic
into
oppa
and
and
filter.
Those
calls,
which
is
very
convenient.
I
mentioned
about
complex,
complex
objects
before
so.
B
This
is
an
example
on
the
right
of
a
it's
actually,
a
very
simple
pods,
that's
running
one
scripts,
and
that
pods
is
asking
for
a
GPU,
k80
GPU
in
that
use
case.
It's
also
trying
to
mount
the
filesystem
and
mentioned
about
it's
using
a
an
actual
pool
secret
for
a
private
repo,
and
it's
asking
for
16
gigs
of
ram
and
8
CPUs.
This
is
quite
actually
quite
big.
2
2,
2
2
right
and
you
can
actually
have
a
lot
of
mistakes
in
this.
B
If
you
miss
the
indentation
of
your
limits
or
resources,
or
if
you
put
them
outside
of
your
container
spec
communities,
will
not
necessarily
give
you
an
error,
but
will
not
take
that
into
account,
and
you
have
these
issues
where
people
were
actually
wondering.
Why
I
doubt
if
you
would
never
come
up,
but
that's
because
they
put
it
in
the
wrong
place.
What
we
did
with
this
API
is
that
we
try
to
also
simplify
how
we
spin
at
this
and
on
on
the
left
side.
B
You
can
see
exactly
what
a
user
was
technically
trying
to
do,
which,
in
this
case
just
just
run
a
script.
Here's
my
image,
here's
my
my
script
and
I
want
to
run
it
on
this
type
of
node.
Ok!
So
now
that
we
build
this,
we
actually
had
the
platform
where
we
could
spin
up
resources
in
different
parts
of
the
world,
and
we
had
a
fast
like
a
much
better
on
boarding
process
and
people
were
able
to
create
projects
and
start
a
notebook
in
less
than
20
seconds,
and
it
made
them
very
happy.
B
This
is
a
small
example.
Actually,
you
have
a
use
case
where
yeah
you
can
see
creative
project
select
the
region
and
give
it
a
name
and,
for
example,
you
want
to
spin
up
a
cutie
on
a
book
and
you
can
just
use
the
UI
T
to
this.
Do
you
yeah?
You
can
see
some
of
the
features
that
we
have.
You
can
choose
the
size
as
well
and
you
can
speed
it
up.
Oh,
you
see
a
small
error.
This
is
a
screenshot
from
the
dev
Craster,
so
sometimes
not
as
stable
as
we
want
it
to
be.
B
Okay,
so
let's
talk
about
a
bit
some
use
cases
and
actually
there's
one
which
is
very
interesting,
the
clinical
validation
of
our
symptom
checker.
That
was
one
of
the
biggest
part
of
the
work
we
did
for
the
platform
does
I
mention
before.
If
you
used
the
app
and
you
tell
them
that
you
have
a
headache,
for
example,
we
have
our
symptom
checker
with
which
is
a
Bayesian
network,
and
it
actually
has
two
jobs.
B
B
We
we
had
our
doctors
creating
thousands
of
different
use
cases
and
we're
running
this,
evaluating
both
the
questions
as
by
the
by
the
symptom,
checker
and
also
the
outcome,
and
we
wanted
everything
every
single
use
case
to
be
treated
properly
and
also
to
be
safe
in
the
outcome
that
we
would
give.
This
was
massive
computing
work
and
it
would
actually
used
to
take
ten
hours
running
on
our
own
prime
servers,
and
that
was
very,
very
annoying
for
our
developers
and
researchers.
They
basically
had
to
spin
up
the
job
and
then
come
back
the
next
day.
B
You
know
see
if
it
failed
or
not.
They
made
the
whole
feedback
loop,
very,
very
slow.
So
what
we
did
is
we
try
to
encapsulate
all
of
this
and
we
moved
it
to
running
on
top
of
kubernetes
and
we
use
algo
for
this.
What
we
did
is
we
optimized
with
the
clinical
validation
code
and
we
so
parallelized
the
api
and
model
parts.
We
were
actually
using
our
go
to
spin
up
as
a
demon
job
about
sixteen
or
twenty
different,
well
replicas
of
the
the
model
with
the
api
and
what
was
really
cool.
B
So
with
this
is
that,
since
all
of
our
production
platforms
run
already
on
docker,
actually
deploying
this
in
the
docker
container,
the
containers
that
we
had
here
were
exactly
the
same
as
we
would
potentially
be
deploying
in
production.
So
we
were
sure
that
this
was
the
right
version,
so
the
artifact
was
exactly
the
same,
and
by
parallelizing
we
were
actually
going
at
first
for
50%
reduction
to
five
hours.
B
But
after
a
few
weeks
of
work,
we
actually
managed
to
take
it
down
to
20
minutes
so
really
big
parallelization
and
now
the
team
is
working
on
actually
putting
this
on
every
pull
request.
So
every
time
you
change
anything
to
the
model
or
to
the
code
of
the
app
we
get
to
feedback
in
in
actually
now
less
than
20
minutes,
which
was
a
massive
success
for
the
team
and
one
one
good
lesson
that
we
learned.
B
Okay,
so
now,
what
are
we
working
on
for
the
next
steps?
So
the
first
thing
that
we
really
want
to
have
is
offering
a
common
common
serving
solutions
to
our
users.
Most
of
our
developers
are
still
figuring
out
how
to
serve
their
models
so
mostly
using
tensorflow
serving,
but
some
some
others
like
the
bayesian
network,
I
talked
about,
they
use
their
own
kind
of
API
wrapper
around
their
models,
so
one
kind
of
provides
a
common
framework
for
that
and
we're
looking
at
KF
serving,
which
is
a
very
interesting
tool
on
top
of
K
native.
B
B
We
also
want
to
improve
the
user
experience.
Everything
on
the
platform
runs
on
docker
and
it's
it
still
sometimes
a
bit
annoying
for
users
to
actually
have
to
build
a
docker
image,
maybe
run
through
some
CI
CD
and
then
get
it
deployed,
and
then
you
know
have
it
failed
and
then
having
to
reiterate.
So
we
want
to
take
that
step
away
from
them
and
build
the
image
directly
on
the
platform
to
improve
the
experience.
B
And,
finally,
we
we
are
adding
more
tools
actually
pretty
much
this
year
around
how
to
track
all
of
our
metadata
centrally
and
again.
That's
something
where
some
teams
are
investigating
different
solutions.
We
are
trying
to
find
the
best
one
to
track
all
the
models
that
we
have
in
the
company
and
we're
mostly
looking
at
ml
flow
and
flow
metadata,
which
are
very
promising.
B
Ok,
so
I'm,
gonna,
wrap
up
now.
One
thing
I
want
to
say
is:
if
you're,
if
you're,
looking
to
actually
build
a
research
platform
on
top
of
Q
on
tees
I
would
I
would
actually
argue
that
you
found
a
really
good
platform
for
that.
I.
Think
Chianti's
fits
a
lot
of
our
use
cases
and
it
was
a
pleasure
to
work
with
and
finally
yeah
well
cube
flow
is
the
best
machine
learning
tool
for
Chianti's
that
we've
seen
out
there
it
get.
It
has
a
really
great
modularity
and
was
very
easy
to
get
started.
B
You
can
just
use
k
fctl
to
deploy
it
and
then,
once
you
get
more
comfortable
with
it,
then
you
can
play
with
customized
and
maybe
only
deploy
the
components
that
you're
interested
in
and
for
for
security
and
compliance,
especially
now
domain
on
healthcare
or
even
in
banking
or
finance.
There's
actually
many
open
source
out
there,
the
open
source
tools
that
you
can
use
and
contribute
to,
and
that's
something
that
we
we
found
why
we're
building
this
platform
is
that
you
can
actually
get
quite
far
with
these
tools.
B
So
if
you,
if
you
want
to
have
a
look
at
this
I,
encourage
you
to
go
to
the
landscape
website
and
have
a
look
around
and
finally
for
people,
like
kind
of
like
with
my
profile
working
more
in
more
and
more
in
ml,
apps
I
think
it's
it's
always
come
down
to.
How
can
you
enable
your
teams
to
to
be
more
efficient,
always
try
to
keep
that
in
mind?
One
thing
that
worked
well
for
us
was
trying
to
like
talk
to
a
team
and
say:
ok,
what
is
your
biggest
pain
point
right
now?
B
What
can
we
do
to
help
and
and
then
focus
on
that
pain
points,
build
small
proof
of
concepts,
and
if
you
see
that
it's
working,
then
you
can
apply
it
Brodeur
in
the
company
and
served
more
teams,
that's
what
we
did
for
the
clinical
validation.
Now
we
we
are
involving
more
teams
to
run
their
own
clinical
validation
or
cluster
and
yeah.
That's
don't
think,
that's
working
break
for
us.
B
B
We
we
did
a
few.
We
merge
a
few
people
request
for
Q
Pro,
that's
something
that
we
want
to
do
more
and
more.
We
were
very
limited
in
resources
to
build
this
platform
at
first.
So
now
the
team
is
getting
bigger
and
we'll
have
more
time
to
actually
contribute
back,
but
but
yeah.
If
we
can
find
a
use
case
which
is
not
there
and
that
works
for
us
and
yes,
absolutely.
D
B
So
we
so
we
created
this
platform,
so
the
proof-of-concept
I
mentioned
before,
like
at
the
beginning,
was
last
year
we
started
creating
this
platform
in
January
and
by
around
February
March.
We
had
our
first
users
trying
the
training
jobs
and
all
of
that
that
was
when
the
whole
platform
was
only
running
on
github.
So
we
were.
We
ran
a
lot
of
workshops
with
researchers
trying
to
understand
what
they
needed
and
the
clinical
validation
parts
we
made.
B
It
run
about
I,
think
in
April
or
June,
and
what
was
actually
right
is
that
the
team
now
is
actually
managing
this
part.
So
we
worked
with
them
and
now
they
are
improving
it
more
and
more
on
themselves,
so
they
have
control
over
what
runs,
and
so
they
build
their
own
UI
and
all
of
that
and
it's
it's
actually
quite
cool
but
yeah
by
I,
would
say
by
by
summer
time
we
had
a
good
enough
platform
and
then
we
spend
the
rest
of
the
year
running
more
security,
things
and
networking
protection.
B
We
we
did
build
some
theories
and
so
the
one
project
that
I
talked
about,
we
so
the
the
queue
flow
Q
flow
project,
actually
has
a
lot
of
shoddy.
So
everything
for
notebooks
and
tends
to
flow
jobs,
Python
jobs,
catnip
all
of
that
comes
with
yeah
DS
already,
but
there
are
some
additional
tools
like,
for
example,
wisdom
or
tensor
board,
where
I
think
tends
to
bomb
now,
there's
this
yard
in
coop
flow,
but
we
built
one
for
his
dumb.
For
example,
it's
it's!