►
Description
Bringing Kubernetes into Salesforce [B] - Steve Sandke, Salesforce
I'll be talking about our journey to using Kubernetes at Salesforce, in the World's Largest Enterprise Cloud Infrastructure. Topics will include:
-- Our motivations for choosing Kubernetes, and how we built confidence in the technology, solution and approach.
-- Our choice to build a lightweight abstraction atop Kubernetes artifacts, and lessons we've learned based on that. This includes our use of a fully declarative vs, API-driven deployment mechanism, which comes with pros and cons.
-- Lessons learned from bringing up Kubernetes in bare metal environments.
-- Securing communications between Kubernetes- and non-Kubernetes hosted workloads.
-- General observations around how shifting to the Kubernetes world requires changes to host-centric tooling.
About Steve Sandke
Steve is a Principal Architect at Salesforce, based in San Francisco, CA. He’s spent the last few years building out infrastructure aimed at easing service deployment at Salesforce. Prior to that, he held various positions at Proofpoint, Microsoft, Loudcloud, WebTV, and others.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
first
off
cs4
says
that
I
need
to
say
this
to
you
that,
even
though
it's
so
cool
we're,
building,
cobranet
we're
using
kubernetes
and
here
at
this
conference,
you
shouldn't
base
any
purchasing
decisions
based
on
that.
So,
like
I
said
we're
going
to
start
with
motivations.
Why
are
we
doing
this
and
then
get
into
how
we
got
people
to
buy
off
on
this
sort
of
thing?
Some
big
technical
decisions
that
we
made
upfront
what
we
built
and
then
where
we
are
and
where
we're
going
of
course
come
on
motivations.
A
Sorry
pardon
me:
don't
swear
at
the
audience
unintentionally,
all
right,
so
I'm
just
going
to
start
with
a
couple
of
the
slides
that
we
present
to
our
corporate
customers,
actually
because
it
turns
out
that
they
care
very
much
about
understanding
how
we
manage
our
systems
because
we're
the
back
end
to
a
bunch
of
companies
whose
names
you
may
have
heard
of
so
in
the
book
between
2014
and
2016,
we
doubled
our
instance
count.
That
is
what
others
would
call
a
cluster
actually
internally.
A
Salesforce
calls
calls
these
things
pods,
which
means
that
the
kubernetes
choice
of
the
term
pod
is
the
worst
possible
thing,
because
every
time
I
say
pod,
people
get
confused.
So
we
actually
internally
when
we're
talking
about
pods,
we
refer
to
k,
pods,
double
the
number
of
transactions
doubled.
The
number
of
data
centers
the
net
here
is
the
business
is
growing
like
a
weed,
and
we
have
a
a
lot
of
work
just
to
keep
up.
A
Nice
all
right,
you're
getting
previews
of
some
slides
there
we
go
so
our
customers
say
hey.
What
are
you
doing?
How
are
you
making
this
work?
I'll
just
highlight
a
couple
of
things
here
from
the
standard
free,
so
we
go
to
give
customers
service
ownership.
We
are
pushing
really
hard
to
move
from
a
monolithic
software
model
where
a
large
number
of
people
all
right
code
into
a
multi
gigabyte
jar
file-
hypothetically
there's
not
really
it's
hard
to
have
a
sense
of
service
ownership.
A
In
that
case
to
a
model
where
we
have,
we
were
breaking
them
up
into
smaller
services,
micro
services.
If
you
will
and
the
team's
own
the
software
they
create
we're
pushing
scale
out
architectures,
which
means
it's
easy
to
just
add
new
hardware
when
you
need,
when
you
need
to
that's
all
I'll
highlight
there
and
then
the
picture
we
present
outside
is
we
talk
about
software-defined
everything
which
I
think
would
play
reasonably
well
to
the
room
here.
A
Does
everything
they've
moved
to
what
we
call
DevOps,
which
I
don't
think,
is
what
the
the
industry
would
call
DevOps,
necessarily
where
the
human
still
involved,
but
you
do
get
a
little
bit
of
machine
help
with
some
scripting
and
such
we're
looking
to
go
autonomous.
What
we
call
no
ops
where
the
humans
job
is
simply
to
set
the
goal,
and
the
software
does
the
rest
unless
the
world
literally
goes
to
hell
and
you're
operating
outside
of
the
parameters
say
like
your
entire
data
center
I
think
I
need
to
reboot
my
entire
data
center.
A
Then
we
get
humans
involved.
So
a
couple
of
years
ago
we
decided
we
were
looking
for
a
new
way
to
run
software.
We
wanted
to
build
a
system
out.
That
would
make
it
easy
to
do.
Micro-Services,
so
the
things
we
were
looking
for
easy
onboarding
for
these
new
services.
We
want
to
make
it
quick
and
simple
for
people
to
come
on
and
bring
up
their
service.
One
of
the
business
metrics
that
we
are
going
to
measure
ourselves
by
is
how
long
does
it
take
from
the
time?
A
I
have
an
idea
for
a
service
to
the
time
when
it's
running
in
production,
we
want
a
high
fidelity,
B,
produc
products
and
non
production
environments.
I,
don't
know
if
we're
the
only
ones,
but
there
are
cases
where
our
production
environments,
don't
exactly
look
like
the
integration
environments.
Don't
exactly
look
like
the
developer's
environments
to
the
point
where,
in
some
cases
you
have
to
build
two
or
three
different
release
mechanisms
and
deployments
deployment
controls,
we
don't
want
that.
We
want
production
to
look
just
like
integration
and
such
and,
to
the
greatest
extent
possible.
A
The
developers
workstation
should
look
the
same
declarative
rather
than
imperative
interface.
This
goes
right
back
to
the
autonomous
nature
of
this.
We
don't
tell
me
how
to
get
the
site
to
the
state.
You
want
tell
me
the
state
you
want
and
I
will
I
will
get
it
there
and
keep
it
there
secure
by
default.
I'm
not
going
to
talk
about
this
too
much,
but
the
notion
is
there's
a
lot
of
security
practices.
A
We
would
like
to
push
through
the
org
and,
to
the
extent
that
we
can
do
that
underneath
or
help
people
make
the
right
choices.
We'd,
like
two
fully
automated
I,
think
I've
covered
that
needs
to
be
usable
across
public
cloud
and
private
cloud.
Most
of
our
stuff
is
in
private
data
centers,
but
we
do
have
some
public
cloud
investments
as
well,
so
we
decided
we
wanted
a
simple
model
deployment
artifacts
would
be
containers,
you
provide
a
health
probe.
The
health
probe
is
used
in
a
couple
of
ways.
One
kubernetes
obviously
uses
it
spoiler
alert.
A
We
chose
kubernetes
uses
that
for
your
container
health,
but
we
also
use
it
to
monitor
your
application
health
because
we're
doing
things
like
trying
to
move
applications
safely
from
one
version
to
the
next
and
declare
you
to
desire
deployment
state.
We
ended
up
calling
it
the
Salesforce
application
model,
which
is
a
bit
of
a
grandiose
term
frankly,
but
as
our
one
of
our
technical
founders
coin,
the
term
I
said
yes,
sir,
if
that
will
make
you
greenlight
the
project
you're
on.
A
So
we
chose
kubernetes
again
shocking
what
drove
us
that
way?
Well,
we
went
up
to
OS
Con
for
the
1o
launch
and
spent
time
talking
to
them
talking
to
the
community
and
talking
to
the
people
and
we're
frankly
blown
away.
A
lot
of
the
people
on
my
team
have
done
this
sort
of
work
before
for
hotmail
back
at
Microsoft
and
other
companies,
and
we
recognized
people
who
had
the
same
basic
model
for
how
you
run
these
sorts
of
things
open
source
container
based
the
development
velocity
was
incredible.
A
Even
then
it
looked
like
we
would
have
a
good
opportunity
to
affect
the
direction
of
the
the
project
if
we
needed
to
so
far.
We
have
not
an
opportunity
to
help
build
it,
and
there
we've
done
a
little
bit.
The
community
seemed
really
broad-based
and
welcoming.
We
did
have
to
make
sure
that
we
looked
around
because
we
were
sitting
there
drinking
beer
with
them,
everybody's
welcoming
at
that
point,
and
the
vision
was
very
much
aligned
with
our
own.
A
The
way
we
wanted
to
go-
and
you
know
a
lot
of
this-
a
lot
of
the
people
were
coming
out
of
work
right.
These
people
clearly
knew
what
they
were
doing
so
to
get
from
decision
to
actually
building
the
goddamn
thing.
First
off
we
needed
executive
support.
This
took
a
couple
of
months
and
involved.
Things
like
I
alluded
to
horse
trading
with
the
founder
over
the
name
in
the
end,
but
this
was
a
very.
A
This
is
a
100
of
an
open-source
project
and
there
was
some
worry
about
getting
there,
but
we
felt
that
showing
good
demos
and
such
we
got
enough.
We
got
it
across.
We
then
needed
to
convince
an
interestingly
large
portion
of
the
company
to
ride
along,
because
this
isn't
I
mean
deploying
kubernetes,
isn't
isn't
enough.
We
had
to
bring
up
a
full
docker
pipeline,
a
trusted
promotion
mechanism
to
get
it
out
there.
We
had
to
get
the
CI
folks
lined
up
on
this.
We
needed
the
right
OS
deployed.
A
We
needed
a
docker
version
out
there
and
get
all
this
past
security
minor
problem
well.
Lastly,
we
need
to
launch
partners,
and
I
will
talk
a
little
bit
more
about
that,
so
finding
the
right
launch
partners
was
actually
kind
of
hard.
These
people
needed
to
be
comfortable
with
ambiguity.
The
these
slides
should
be
up
on
the
on
the
website
as
well
comfortable.
They
have
big
UT
people
would
come
to
me
and
say
Steve.
How
are
we
going
to
do?
X
and
I'd
say?
A
A
We
additionally
wanted
people
who
were
interested
in
actually
defining
the
platform
with
us
that
helped
a
lot.
People
who
are
comfortable
with
ambiguity
in
general
are
willing
to
define
new
platform
as
well,
so
they
were,
it
took
some
work.
We
wanted
to
pick
teams
that
covered
a
spectrum
across
the
org
buddy
would
feel
brought
in
or
bought
in.
They
need
to
be
willing
to
wait
to
ship.
At
this
point
we
were
about.
This
was
a
little
over
a
year
ago
on
kubernetes
one
one
I
think
one
two
was
just
about
to
happen.
A
So
I'll
talk
about
a
couple
of
the
big
decisions
we
made
as
we
went
into
this
number
one.
We
built
an
abstraction
on
top
of
kubernetes
at
the
time.
I
was
a
little
worried
about
this.
Actually,
even
here
seeing
other
people
talk
about
their
abstractions,
the
digitalocean
guy
yesterday
was
talking
about
an
abstraction,
basically
doing
a
talk
that
have
you
squinted
and
our
systems
are
different,
but
have
you
squinted,
hey,
okay,
yeah,
we're
doing
the
same
thing?
A
Why
were
we
doing
this?
Guardrails
are
important.
Kubernetes
is
incredibly
powerful.
You
can
do
so
many
different
things.
There's
lots
of
fine
grained
stuff.
You
can
do
and
coarse
grain
stuff
you
can
do
and
as
an
engineer
and
during
the
prototyping,
we
have
blast
with
it.
However,
we
don't
want
to
foist
that
upon
a
several
thousand
upper
organization
because
they
will
go
off
and
do
stuff
and
they
will
go
off
the
rails,
so
we're
putting
up
guardrails
also-
and
this
is
based
on
bitter
past
experience-
building
systems
like
this.
A
It
is
far
easier
to
expose
a
new
feature
than
it
is
to
take
one
away.
We
also
knew
that
we
were
going
to
build
infrastructure
related
extensions
into
this
anyway,
so
we
ended
up
with
the
concept
of
what
we
call
a
deployment
manifest
no
direct
relationship,
but
there's
an
indirect
relationship
to
the
kubernetes.
One
I'll
show
you
one
in
a
minute.
It
looks
like
a
unified
set
of
deployment
and
service
packs.
A
Next
up,
where
do
these
deployment
manifests?
Live
we
put
them
and
get
and
get
drives
the
deployment
the
deployment
controls
so
nice
thing
about
this
is
so
we
want
service
owners
who
are
developers
to
manage
this
stuff
turns
out.
Developers
are
reasonably
good
at
using
git
and
they're
comfortable
with
using
things
like
pull
requests
to
look
at
the
difference
in
a
proposed
deployment.
You
also
get
a
full
history
of
all
these
production
deployments.
A
It
can
take
a
while
to
learn,
I,
don't
know
if
you've
ever
just
given
up
and
removed
a
repo
and
cloned
it
again,
because
you
couldn't
figure
out
how
to
dodge
the
damn
thing.
But
in
these
use
cases
we
don't
have
that
problem.
So,
basically,
when
you
do
a
deployment
in
our
system,
if
I
want
to
deploy
a
new
version
of
software,
I
just
check
out
a
manifest
make
my
changes
and
I
submit
a
pull
request
to
approve
that
deployment.
A
There's
some
auditing
and
process
associated
with
this,
that
our
customers
were
demanding
or
customer
contracts,
and
so
we
do
two
factor
auth.
So
I
can
do
a
strong
audit
that
says
Lincoln
over
their
Salesforce
employee.
Shout
out
approve
this
request,
so
it's
all
good.
Next
up,
we
chose
to
put
all
the
manifests
in
one
repo
for
all
of
production.
A
This
was
an
interesting
one,
but
in
the
end
we
felt
that
it's
best
to
put
it
there,
you
can
see
everything
in
one
place.
You
can
audit
it's
actually
easier
for
our
tooling
to
consume
and
I
swear
that
that
was
not
the
main
reason
that
we
did
that
we
are
finding,
even
now
with
limited
number
of
customers,
that
the
repository
is
a
bit
noisy.
Well,
that's!
Okay!
A
Last
thing
is
interesting:
going
to
some
of
the
security
talks
today,
where
they
talked
about,
like
we
think
by
1/8,
might
have
been
1
7
you'll
be
ready
to
run
multi-tenant
fully
multi-tenant
in
a
kubernetes
cluster.
Well,
we
have
one
kubernetes
cluster
in
each
data
center
and
we're
running
fully
multi-tenant.
A
So
big
decision
was
we
isolate
security
profiles
pretty
pretty
dramatically
what
do
I
mean
by
a
security
profile
so
for
a
given
service
who's
calling
me
Who
am
I
calling
and,
most
importantly,
what
kind
of
data
there's
other
pieces
too,
but
these
are
the
main
ones.
So,
for
example,
one
of
the
services
that
we
ended
up
bringing
onboard
is
a
Redis
based
caching
layer,
that's
used
by
our
front-ends.
It
turns
out
that
has
customer
data
in
there
and
it
talks
to
some
services
that
are
pretty
close
to
the
Internet.
A
That's
one
security
profile,
another
service
that
we
brought
on
the
backend
monitoring
services.
It
just
takes
metrics
and
such
from
different
applications.
That's
a
different
security
profile.
So
we
put
we
put
the
pods
associated
with
a
given
circuity
profile
alone
on
their
nodes.
You
can't
have
we
don't
share
right
now
and
we
take
further
actions
to
to
isolate
these
things.
There
are
some
network
boundaries.
There
are
some,
let's
just
say
our
back
was
an
interesting
thing
for
us
to
see
the
the
community
deliver
them.
A
We've
done
some
stuff
ourselves
in
the
absence
of
that
right
now.
Does
this
decrease
the
efficiency,
the
efficiency
of
our
usage
yeah,
but
we
need
to
at
the
moment-
and
the
nice
thing
is,
this
is
entirely
hidden
from
the
application
developer,
as
as,
for
example,
I
saw
some
hypervisor
based,
so
I
could
run
some
of
these
things
in
a
hypervisor
or
we
get
better
at
security
or
our
security
team
me
gets
more
comfortable.
We
can
slide
all
this
underneath
customers
will
never
see
those
are
the
main
decisions.
So
what
have
we
build?
A
Well,
first,
off
I
will
show
you
a
deployment
manifest
at
the
beginning.
We
were
actually
thinking
we
were
going
to
abstract
kubernetes
away
pretty
completely
over
time.
We
decided.
We
didn't
really
need
to
do
that,
so
that
you
can
see
what
we
call
a
function.
Raise
your
hand
if
you
think
it
looks
like
a
deployment.
That's
because
it
is,
we
expose
most
of
the
pieces
in
the
deployment,
the
de
port
of
deployment,
artifact
fun
fact
we
were
at
the
time
we
started.
Building
this.
A
There
was
no
deployment
thing.
We
thought
we
were
gonna
have
to
build,
build
the
equivalent
of
a
deployment
to
manage
replica
sets
and
all
that
fun
stuff.
But
the
community
has
been
ahead
of
us,
we're
just
now
getting
to
the
point
where
we
feel
like
there's
things
that
we
need
that
aren't
there,
which
is
good
I,
guess
it
took
us
a
while
to
deliver
this
load
balancers.
A
They
just
map
into
services
right
now,
eventually
I'm
not
showing
it
here,
but
we
imagine
we're
gonna
put
in
information
so
that
we
can
bind
into
our
existing
Salesforce
alerting
systems
and
such
we
do
not
have
the
luxury
of
just
sail.
Prometheus
everything's
great
we've
got
a
bunch
of
infrastructure
that
we
have
to
bind
into
so
the
overall
flow.
A
laser
pointer,
would
be
nice
here
at
the
upper
right.
You
see
our
beautiful
service
owner,
they
write
code,
they
check
it
in
the
build
pipelines,
run
docker
images
get
created
and
they
sit
there.
A
The
big
the
the
thick
dash
line
has
but
the
difference
between
R&D
and
production
right,
so
these
pipelines
run
CI
can
happen
all
of
that
when
they
owner
service
owner
wants
to
actually
do
a
deployment
they
check
in
a
change
to
get
over
there.
Once
that,
pull
request
is
approved.
A
couple
of
things
happen
first
off.
If
you
have
a
deployment
manifest
that
has
been
that's
been
approved,
any
docker
image
that's
referenced
in
there
is
approved
for
promotion
that
into
production.
A
We
do
not
want
images
that
haven't
been
approved
for
deployment
to
be
out
there
where
they
could
run
attorney.
I
mean
it's
really
easy
to
docker
run,
it's
really
easy
to
start
bringing
things
up
at
that
time.
We
actually
also
sign
them,
because
we
do
signature,
verification
that
out
in
production,
to
verify
that
only
trusted
trusted
signed
images
are
out
there.
The
same
time,
the
manifest
goes
out
to
a
dedicated,
manifest
repo,
where
our
little
controller,
which
is
not
much
code,
actually
looks
at.
A
Those
manifests,
like
you
saw
before,
translates
them
into
the
kubernetes
artifacts
talks
to
the
kubernetes
controller
and
watches
the
deployment
move
forward.
The
deployment
object
handles
most
of
that,
but
if
it
stalls
or
there
are
errors,
then
the
the
then
our
controller
is
Racal
is
responsible
for
pulling
it
back.
It
also
talks
to
some
of
the
other
Salesforce
infrastructure.
A
We
run
that
controller
we've
thought
about
making
that
a
third-party
resource
they
weren't
really
available.
At
the
time
we
wrote
that,
but
for
the
moment
it's
just
it
just
runs
as
a
pod
inside
the
cluster,
with
a
with
a
count
of
one
one
thing:
I
will
note
on
the
quote:
Federation
thing
deployments
go
in
parallel
if
I
check
in
a
manifest
that
says
that
says
this
should
go
into
DC
one
two
and
three
that
pipeline
runs
in
parallel
and
those
things
go
out
and
they
are
deployed
completely
in
parallel
on
all
of
them.
A
We're
not
using
Federation.
Here
a
Federation
wasn't
available
at
the
time.
We
were
doing
this
and
B
the
the
call
direction.
Yeah.
This
picture
doesn't
quite
work
that
way.
Sorry
I'm
debugging
my
picture
as
I
describe
that,
but
the
call
Direction
doesn't
actually
work
with
our
security
model
for
Federation
in
this
form.
A
So
that
is
what
we
built
as
I
said:
it's
pretty
high
level,
so
I'm
going
to
talk
about
where
we
are
so
right
now.
The
coolest
stat
for
me
is
30
minutes
after
checking
in
or
approving
the
manifest
change.
Your
systems
are
running,
live
in
production
with
no
humans
involved
whatsoever.
Now,
I
think
a
lot
of
us
have
worked
in
environments
worth
like
30
minutes.
A
Ok,
that's
exciting
I'll
just
say:
that's
a
significant
improvement
both
in
terms
of
human
involvement
and
especially
in
terms
of
time
for
these
deployments
and
the
nice
thing
is
the
service
owners.
Just
do
it
and
we
don't
actually
even
know.
Sometimes,
we've
got
a
couple
of
services
like
three
services
in
production
right
now:
caching,
a
monitoring
service
and
then
a
little
feature
flag
service.
We
expect
by
the
end
of
the
year,
we'll
have
at
least
20,
if
not
more,
we're
running
100,
maybe
a
little
more
nodes
in
production
almost
entirely
on
bare
metal.
A
A
All
of
the
data
centers
will
be
going
by
the
end
of
the
year,
and
while
we
have
ten
clusters
right
now,
several
for
development
we'll
be
running
at
least
20
by
the
end,
and
whereas
we
have
dozens
of
daily
service
deployments
right
now,
we
expect
hundreds
or
even
more-
and
one
are
actually
somebody
corrected
me
last
night,
kubernetes
contributions.
We
hope
to
have
more.
A
A
We
need
to
get
better
at
pulling
in
more
infrastructure
things.
We
do.
We
want
to
do
certificate
management
with
automatic
certificate
generation
so
that
these
services
can
do
mutual
TLS,
for
example,
and
we're
looking
to
hook
that
in
secrets,
kubernetes
has
secrets
mechanisms
Salesforce
as
a
whole,
which
it
turns
out,
is
running
most
of
the
Salesforce
infrastructure
doesn't
run
on
our
stuff
and
they
have
a
secret
secrets
mechanism.
So
we
have
to
tie
that
in
we're
looking
to
support
clustered
applications,
I
used
Redis,
both
and
Sentinel
and
cluster
mode
as
an
example.
A
Caching,
which
at
least
one
of
our
services
really
needs,
that's
all
they
need,
and
they
don't
actually
care
if
it
goes
away,
but
they
would
just
prefer
that
it
not.
We
just
do
this
for
ourselves
with
a
local
agent
which
provisions
the
disk
as
I
as
a
pod
lands.
Their
persistent
volumes
is
a
big
focus
of
work
for
us
right
now
this
year,
which
is
stateful
sets
and
it
it
sounds
like
we
will
get
automatic
upgrades
before
we
have
to
go,
do
it
ourselves
and
that
would
be
wonderful.
A
That's
actually
all
I
have
we're
using
kubernetes
in
other
areas
as
well.
I
just
picked
the
the
project
that
I
run
here,
one
other
one
I'll
mention
is
a
kubernetes
hosted
streaming
spark
clusters
on
demand,
got
kubernetes
deployed
and
hosting
live
services,
and
our
service
owners
are
really
excited.
We
are
turning
away
far
more
people
than
we're
taking
right.
A
Now,
just
because,
honestly,
we're
kind
of
afraid,
like
I,
want
to
find
problems
with
four
or
five
customers,
rather
than
find
that
same
problem
where
maybe
a
bunch
of
manual
labor
is
required
to
get
out
of
it
when
you've
got
50,
and
this
would
not
be
a
Tech
Talk
at
a
tech
conference.
If
I
didn't
say
this
sounds
interesting,
we
are
hiring.
A
A
So
the
question
is:
how
do
we
sync
docker
containers
across
the
data
centers?
We
not
my
team,
let's
see
so.
Basically,
it's
a
pull
from
the
data
center,
so
we're
actually
doing
a
there's
a
signal
to
there's
a
there's,
a
trusted
registry
which
is
where
we
stage
and
then
there's
registries
out
in
production
that
are
monitoring
that
and
just
pulling
them
down.
A
A
Question-Is
sounds
like
a
big
project:
what's
the
size
of
the
team,
it
was
eight
people.
Well,
so
it
depends
the
core
kubernetes
stuff,
eight
to
ten
people.
It's
ten
now
we're
expanding,
but
we
pulled
in.
It
was
probably
60
or
70
people,
I
would
say
we're
involved.
In
some
way,
I
mean
we
had
people
who
had
to
bring
Santo,
s7
and
figure
out.
Okay,
what
does
a
sent
to
a
seven
image?
Look
like
for
this?
A
To
do
this,
oh
so,
okay!
So
the
question
is:
how
do
we
convince
developers
to
change
their
minds
on
which
access?
Because
on
the
one
axis,
the
hard
thing
is
service
ownership?
It's
like
you're
gonna,
be
on
call
turns
out
developers,
don't
like
that
in
some
cases,
especially
in
an
organization
where
previously
they
didn't
have
to
cuz,
they
were
kind
of
tossing
stuff
over
the
wall
to
a
dedicated,
sre
org
in
terms
of
using
our
stuff,
I
will
say
that
what
we've
deployed
is
so
much
easier
to
use
then
than
the
way
it
was
done.
A
A
The
distance
from
code
check
in
to
production
is
so
much
less
under
this
that
that's
something
that
we
really
needed.
We
understood
that
the
processes
that
we
had
weren't
scaling
as
we
tried
to
break
break
apart
some
of
our
more
monolithic
apps,
so
the
execs
knew
they
needed
to
do
something
like
this.
A
Oh,
our
abstraction
yeah
we
actually
win
and,
like
I,
said
we
kind
of
expected.
We
were
going
to
abstract
it
away
more
than
we
did
so
we
actually
started
with
the
DSL.
We
started
with
our
abstraction.
You
saw
those
functions
and
load
balancers
at
the
beginning.
It
didn't
look
anything
like
kubernetes,
but
over
time,
we're
slowly.
Exposing
more
and
more
honestly.
I
expect
that
a
year
or
two
from
now
we'll
have
a
subtly
different
format.
A
If
I
were
gonna,
do
it
again,
what
I
am
what
I
have
an
abstraction,
absolutely
I
mean,
fundamentally
because
we
are
actually
are
very
much
trying
to
draw
a
box
and
say
you
get
to
you
get
to
color
inside
this
box.
They
do
not
want
4000
developers
going
and
experimenting
with
kubernetes.
Hey
sorry,
I'm
gonna
revoke
my
engineering
credentials,
but
no
actually
he
was
next.
Oh.
A
So
we
hope
so
we
have
a.
We
have
a
couple
of
secret
services,
but
the
main
one
is
just
a
vault
that
lives
in
the
in
the
data
center
and
you
can
go
query
your
secrets,
and
so
you
know
you
can
pull
them
out
of
puppet
and
such
right.
Now
we
use
puppet
to
manage
some
of
our
other
infrastructure
and
what
we're
trying
to
do
here
is
say
you
just
mention
your
secrets
in
your
manifest
and
we
will
go
pull
it
out
and
drop
it
into
we're
thinking
the
file
system.
A
A
Question
is:
are
the
customers
deployed
on
VMs
or
officially
on
there
straight
bare
metal
straight
bare
metal,
which
is
what
go
ahead?
Storage
in
what
sense?
What
storage
are
we
using
I
mean
so
at
the
moment,
we're
not
exposing
storage
to
people
we're
looking
at
SEF
right
now,
as
the
as
the
storage
mechanism
that
we
do
end
up.
Exposing
right
I
mean
we
did
stateless
apps.
First,
don't
make
sense,
we're
using
s
CD
as
as
part
of
kubernetes,
but
that's
it.
A
A
Yes,
I
think
I
was
in
it.
I
was
in
a
helm,
talk
just
now
actually
staring
at
him.
Like,
yes,
I
think
when
we
were
there,
when
we
were
just
starting
this
out,
helm
was
not
fully
out
there
yet
I
think
I
mean
they
were
going
along
around
the
same
time
as
us.
I
remember
them,
presenting
it
the
first
coupe
cut
or
being
at
the
first
coupe
con.
But
at
that
point
honestly
and
I
think
this
is
on
me.
We
had
our
abstraction
and
we
were.
A
A
I'd
say
we're
exploring
it
I
mean
we've
got
some
efforts
of
foot
to
try
and
try
and
bring
them
up.
The
truth
is
I
mean
for
the
moment
Salesforce,
which,
if
you
don't
know,
is
just
this
massive
Oracle
database
with
business
logic.
On
top,
it's
incredibly
complex
and
cool,
we're
pretty
good
at
running
databases.
So
that's
not
the
the
first
thing
we're
trying
to
do.
There's
somebody
over
there.
Next.
A
See
again,
we
wanted
to
make
the
development
environment
look
as
close
to
production
as
possible
and,
on
the
one
hand,
maybe
that's
mini
cube
with
a
little
bit
of
our
abstraction
on
top
the
harder
problem
which
is
actually
not
related
to
this,
but
more
around
micro-services
in
general
is
I.
Have
40
micro
services
out
there
I'm
not
going
to
run
all
40
of
those
on
my
dev
box,
and
so
we
have
teams
I'm,
not
part
of
my
teams
that
I'm
enabling
to
run
micro
services.
A
We
have
people
trying
to
figure
out
how
best
to
do
that
from
a
developer
perspective
in
general.
The
idea
is,
if
I
have
a
cluster
of
them.
I
have
39
of
the
services
running
over
here
and
I
can
configure
discovery
such
that
my
service
is
the
one
that
you
talk
to
in
a
development
environment.
Then
that's
fine.
Does
that
make
sense.
A
So
they
can
build
the
containers
and
run
them
directly,
which
works
for
a
lot
of
the
use
cases.
We've
got
mini,
cube,
mostly
up
and
running,
and
that,
but
the
main
thing
we
have
is
a
a
cluster
where
you
can
just
run
your
stuff,
which
I
think
which
I
find
gets
you
most
of
the
way
there.
And
then
there
are
some
cases
where
an
integration
environments
we're
actually
building
up
bespoke
bespoke
capacity
for
them.
A
A
A
The
fear
of
escape
from
containers
is
greater
than
the
fear
of
escape
from
vm's
I.
Think
you
can
pop
both
of
them
in
some
senses
at
present
were
early
enough,
that
we're
we're
picking
and
choosing
what
we're
close.
We
run
to
ones
that
that
they're,
ok
with
and
we're
expecting
that
as
the
as
the
community,
matures
the
security
stuff
like.
A
If
and
if
you
went
to
the
Google
Talk,
there's
a
bunch
of
capabilities
there
and
we're
looking
at
how
we
can
basically
give
people
confidence
that
it's
unlikely
that
you
break
out
I
mean
the
other
thing
is
we
are.
These
are
friendly
developers
in
that,
like
I'm,
not
running
a
pass
and
so
and
that's
amending
up
has
internal
pads
and
I'm
not
running
up
has
where
any
Tom,
Dick
or
Jane
can
come
and
run
their
code,
but
that
that
said,
no
I
mean
bare
metal,
bare
metal.
A
For
now-
and
you
said
you're
working
on
environmental
and
did
you
have
to
implement
your
own
ingress
or
load
balancers
or
as
I
call
it
with
a
better
mental?
It's
pretty.
Our
networking
continues
to
evolve.
Our
Sdn
capabilities
are
maturing,
shall
I
say
so
we
have
at
the
moment
and
so
we're
working
on
both
Sdn
and
alb
SLB
deployments,
which
will
help
there.
In
the
meantime,
there
is
some
f5
programming
going
on
yeah.
A
How
do
we
handle
rolling
upgrades
for
nodes
and
masters,
so
we
have
a
little
deployer
that
basically
does
that
right.
So
it's
it's
basically
just
one
two
three
and
are
we?
We
run
three
master
nodes
and
then
for
the
actual
nodes.
We
just
do
it
slowly.
It's
simple:
it's
something
we
could
do
a
lot
more
of
I
think
we
could.
We
could
do
more
there,
but
for
the
moment
main
thing
is
for
patching
because
we
patched
the
OSS
underneath
on
a
somewhat
regular
basis,.
A
As
we
put
the
developers
on
call,
what
have
we
done
with
the
sres,
whose
jobs
are
no
longer
needed?
They're?
Not
so
first
off
I
mean
we
only
have
a
few
services
right
now
in
general.
The
notion
is
that
we
want
us
Ari's
to
be
broader,
maybe
they're,
the
first
tier
in
some
cases,
but
but
as
I
show
that
the
business
is
scaling
so
fast
that
I
we
can't
keep
up
in
that
sense,
so
we
haven't
had
any
that
problem.
Yet
I
have
no
idea
how
much
time
is
left.