►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Managing Developer Workflows with the Kubernetes API - Colin Murphy, Adobe
Write your own Kubernetes client to keep application teams from breaking their products. Kubernetes is incredibly powerful. It's extensibility allows for limitless varieties of architecture. But how do you get hundreds of engineers to follow the same conventions when creating their applications? For us at Adobe Document Cloud, the answer was to write our own Kubernetes client. It's easy to get started and well worth the investment for larger organizations. Writing your own client allows you to have much greater control over the deployment process, including secrets and change management. It also allows you to integrate with your existing deployment systems and reduces the cognitive load on your engineering teams.
https://sched.co/ekAX
A
Hello
and
welcome
to
managing
developer
workflows
with
the
kubernetes
api.
My
name
is
colin
murphy.
I'm
an
engineering
manager
at
adobe
first
thing,
I'd
like
to
do
is
cover
why
we
chose
to
write
our
own
kubernetes
client
and
then
I'll
show
you
how
we
did
it.
As
I
said,
I'm
an
engineering
manager.
My
current
responsibility
is
the
infrastructure
for
document
cloud,
both
the
architecture
and
the
software.
A
A
A
So
we
needed
to
go
from
those
configurations
onto
kubernetes
in
a
seamless
manner,
so
the
first
one
is
is
really
we
want
to
profound
outages
right.
We
need
to
exert
a
greater
control
over
what
our
application
teams
can
do
than
just
saying
you
know:
here's
cube
cuddle
here
are
some
yaml
files
go
have
at
it
right?
You
do
that
you're
going
to
have
you're
going
to
have
outages,
because
people
just
don't
understand,
kubernetes
and
I'll
get
to
that
in
a
second,
but
you
also
want
to
limit
the
kubernetes
objects.
A
I
should
point
out
that
it's
not
a
substitution,
for
you,
know:
rbac
network
network
segmentation,
open
policy
agent,
other
controls
on
the
on
the
control
plane
side
that
you'd
wanted
to
implement
to
protect.
So
this
isn't
this
isn't
a
replacement
for
those.
It's
just
a
it's
just
a
nice
supplementary
thing.
A
So
we
also,
as
I
mentioned
it,
was
it's
it's
not
a
green
field.
Right
we
had
an
existing
deployment
system
called
moonbeam
it.
It
actually
made
the
the
problem
statement
a
little
easier,
because
we
knew
we
just
had
a
shaw
and
that's
all
we
had
to
do
with
deployment
so
a
git
repo,
a
shaw,
and
that
that
made
it
kind
of
easy
to
start
as
a
nice
place
to
start.
A
So
so
you
know
the
other
big
thing
right.
We
want
to
reduce
cognitive
load
application
engineers.
They
have
a
deep
technical
knowledge
right.
They
understand,
you
know
how
to
manipulate
a
pdf
or
a
echo
sign
agreement,
or
maybe
some
sort
of
machine
learning.
They
don't
really
have
time
to
do.
Kubernetes
right
I
mean,
if
there's
no
such
thing
as
a
person
who
fully
understands
everything,
that's
required
in
a
in
a
web
application
right.
Then
we
had
our
site,
reliability,
engineers
and
and
really
they
had
the
same
story
right.
They
have
there's
a
huge
legacy
application.
A
A
A
We
can't
we
can't
always
have
a
connection
to
some
sort
of
from
a
cluster
to
a
secrets
management,
so
we
really
had
to
find
a
way
to
package
up
all
the
secrets
an
application
could
need
from
within
our
internal
network
and
and
push
that
in
a
safe
way
out
to
kubernetes
cluster.
In
a
way
that
would
you
know,
allow.
A
Allow
for
roll
back
make
sure
there
wasn't
a
way
that
it
would
interfere
with
with
secrets
in
the
next
release,
that
kind
of
thing,
and
then
lastly,
it's
really
it's
just
not
that
hard.
It's
it's
very
simple!
It's
fairly!
Well
documented!
If
you
understand,
if
you
know
kubernetes,
if
you're
familiar
with
the
concepts
of
kubernetes,
you
can
write
a
kubernetes
client
I
mean
go
is
a
little
bit
of
a
learning
curve.
A
A
A
Now,
in
addition
to
that,
we
needed
teams
to
just
to
immediate
teams
to
migrate,
right
away
or
or
at
least
migrate
without
talking
into
each
and
every
single
one
of
them
and
treating
them
is
each
is
a
special
little
snowflake.
So
we
had,
we
did
a
lot
of
work
around
automating
automation,
generating
the
right
files
for
t
for
teams
automatically,
and
that's
still
true
now,
so
our
onboarding
system
will
give
will
stub
out
the
directories
required
and
teams
can
kind
of
just
go
from
there.
There's
there
there
really
it's
don't
we
don't
have.
A
A
You
know,
through
through
kubernetes
objects,
we'll
touch
on
a
little
bit,
because
that
that
does
cause
some
problems
but-
and
we
certainly
don't
allow
them
to
do
whatever
they
want,
but
but
certainly
a
lot
more
freedom
than
they
had
before,
there's
no
other,
they
don't
have
to
learn
some
other
system
of
you
know
cloud
formation,
templates
or
some
internal
system.
We
have
that
generates
infrastructure
code.
That
kind
of
thing
it's
all
completely
in
their
control
and
also
you
know
sometimes
a
previous
system
we
had
you'd
have
to
go
to
some
other
service.
A
If
you
wanted
to
make
a
certain
kind
of
change
to
your
service,
so
another
internal
web,
ui
and
you'd
have
to
click
through
it,
and
you
know
it
would
take
some
time,
and
so
this
this
that
was
another
thing
we
really
wanted
to
get
rid
of
was
was
anything
like
that
right.
It's
just
there's
a
single
deployment
system,
as
I
showed
a
few
slides
before
you
can
make
any
change
to
your
service
using
that
system
and
and
for
change
management
that
system
we.
A
We
rely
on
that
system
to
to
verify
that
you
know
the
appropriate
approvals
have
been
made
and
reviews,
and
things
like
that,
there's
a
concept.
This
is
a
little
bit
of
a
stretch,
but
there's
this
concept
of
hermetic
builds
from
the
sre
manual
that
that
I
think,
hopefully
most
of
you
have
already
already
built
that
way,
which
is
you
know
no
to
build
dependencies
on
the
on
the
build
system
right
so
easiest
way
to
do
that
is
to
build
in
docker.
A
A
Has
the
shaw,
labeled
and,
and
it
ended
we
have
we
inject
into
it-
config
map
and
secrets
kubernetes
secret
with
that
shaw.
So
if
we
ever
roll
back,
even
if
you
know
our
that
secret
was
changed
somewhere
in
in
hashicorp
vault,
we're
still
getting
those
those
secrets
and
we
keep
up
to
five.
So
you
can't
roll.
You
know.
A
And
then
one
last
thing
is
we.
You
know,
I
might
have
kind
of
inferred
this,
but
we
deploy
to
multiple
cloud
providers
and
and
some
applications
actually
deployed
to
data
centers,
and
we
wanted
to
abstract
that
away
from
from
the
service
team.
We
didn't
want
the
service
team
to
say.
Oh,
I
need
you
know
a
different
domain
name
because
I'm
in
this-
or
this
is
hipaa
compliant
or
oh,
it's
fedramp.
So
I
have
to
you
know
to
make
sure
my
kubernetes
configuration
is
a
little
different.
A
A
A
Okay,
so
moving
on
to
how
the
port
of
kubernetes
works,
we
take
the
shaw
we
clone
the
service
repository
and
and
we
kind
of
go
from
there.
We
read
in
a
bunch
of
environment
variables
and
secret
references
I'll,
just
I'll
show
you
what
those
look
like
in
a
second,
then
we
read
the
cluster
configuration
from
the
cluster
itself
and
then
we
compare
to
what
we
have
for
up
for
an
update
and
we
update
the.
A
A
A
A
A
We
also
have
the
ability
to
override
cluster
settings
so
say
you
had
a
different
certificate
and
you
needed
a
different
domain
name
for
tls
cert
right
for
or
if
you
know
you
didn't
want
to,
you
didn't
want
to
use
the
same
gateway
for
istio
gateway
or
contouring
grass.
That
kind
of
thing
as
the
other
services
in
your
namespace.
A
And
you
know
it's
it's
always
it's
a
constant
fight,
we're
always
always
getting
requests
for
more
features,
we'll
probably
have
to
expand
it
a
little
bit
either
some
kind
of
annotations
structure
or
something
like
that,
because
it
it
gets
very
complicated
with
multiple
ingress
controllers.
Things
like
that.
A
So
this
I
wanted
to
kind
of
show
this.
This
is,
I
guess,
the
first
code.
It's
it's.
I
was
surprised
how
difficult
it
is
to
show
code
in
a
presentation.
I
just
want
to
credit,
katherine
cox,
boude.
I
read
her
book
concurrency
and
go
when
I
got
started.
She
had
this
concept
of
pipelines
that
she
talked
about
that
I
had
never
seen
in
another
language.
A
Maybe
all
of
you
are
really
familiar
with
it,
but
but
basically
you're
just
passing
a
channel
here
from
in
a
nested
kind
of
function
structure,
and
it's
really
efficient
and
clever
way.
To
to
it's,
not
it's
it's!
It
is,
you
know
parallel
computing,
but
it's
it's
not
it's
not.
You
know
obvious,
embarrassingly
parallel,
but
you're,
passing
through
each
region
individually
in
a
in
a
kind
of
a
in
a
elegant
way.
A
A
So
what
we're
doing
here
is
we're
we're
getting
that
configuration
we're
fetching
the
secrets,
we're
copying
we're
reading
in
the
objects,
we're
copying
the
objects,
doing
a
deep
copy
because
we're
going
to
modify
them.
That
was
a
lesson
learned
right.
You
don't
you
have
a
pointer
to
a
struct.
If
you
modify
it
for
one
region,
it's
going
to
modify
it
for
all
regions,
so
deep
copy
function,
method
actually
and
then-
and
then
we
get
this
this,
this
big
struct
that
has
all
the
information
we
need
to
do
our
deployment.
A
The
way
we
handle
cube
conf
configs
is
we
have
just.
We
have
a
file
that
lists
that's
common
to
all
of
our
services
and
it
just
lists
out
all
of
the
all
of
the
cube
configs
in
their
locations
involved.
We
act.
We
use
aws,
iam
authentication
for
our
eks
clusters.
We
use
active
directory,
authentication,
azure
active
directory
authentication
for
our
azure
clusters.
So
it's
not
really
they're,
not
really
secrets,
but
it's
just
a
it's
it's.
It
makes
all
the
security
people
happy
right.
It's
getting
warm
fuzzy
feeling
all
of
our
cube.
A
A
A
A
A
If
they
have
a
horizontal
plot
of
the
scalar,
we
we
always.
We
want
to
make
sure
that
when
they
deploy,
they
don't
reset
the
number
of
replicas.
So
just
you
know
something
like
this
really
basic.
We
just
we
can.
We
don't
have
to
transverse
say
a
map
of
of
strings
to
interface
right.
We
just
spell
out
the
whole
the.
A
We
don't
want
somebody
to
make
it
a
horizontal
part
of
the
scaler
that
that
modifies
someone
else's
deployment
that
you
know
that
happens,
people
copy
from
one
another
as
much
as
we
try
to
you
know
automatically
stub
things
out,
just
it's
just
a
reality.
When
you
have
this
many
services
right,
hundreds
of
hundreds
of
engineers
and-
and
you
know,
over
a
hundred
services
so
and
as
we
expand
this
actually
out
to
more
people
in
dobie.
A
Cluster
settings,
so
these
are
these
are
when
I
I
talked
before
I
mentioned
that
there
is
a
cluster-specific
information
and
it's
we've.
We've
done
a
kind
of
a
good
job
limiting
it,
because
you
could
really
go
crazy
here,
but
these
the
the
real
you
know
the
the
kind
of
the
obvious
thing
so
registry.
We
have
registries
in
every
region.
You
know
we
don't
want
an
outage
say
of
aws
in
eu
central
one
knocking
out
our
azure
in
you
know
singapore
right.
So
so
we
we
always
want
in
our
deployment
strategy.
A
A
They
kind
of
just
they're,
just
kind
of
whatever
you
know
my
registry
and
then
and
then
the
name
of
their
their
docker
image
for
the
for
so
so
that
that
just
gets
overwritten
it
doesn't
it
doesn't
that
you
know
portrait
kubernetes
can
handle
that,
whether
the
whether
the
cluster
has
istio
or
not
the
domain
name
of
the
services
in
that
namespace,
the
istio
gateway,
if
there's
istio,
so
the
a
minimum
number
for
the
hpas,
we
typically
you
know.
If
it's
in
production,
you
need,
you
need
two.
Some
services
need.
A
A
A
It's
you
know
it's
kind
of
if,
if
you're
familiar
with
go,
this
is
kind
of,
probably
not
that
interesting.
But
if
you're
not
this,
this
is
you
know.
This
is
basically
how
we
do
it.
There's
an
update
function
defined
for
every
type
of
kubernetes
object,
because
we
have
those
those
type
client
sets
and
basically
just
make
a
work,
make
a
weight
group.
We
run
that
function
and
then
once
they're
all
done,
then
we
then
we
we
exit
out
of
this
this
this
updater.
A
A
A
The
first
is
we
have
the
aws
controller
for
kubernetes
and
the
azure
service
operator
we're
working
on
supporting
those
and
they
come
with
this
whole
mess
of
crds.
It's
going
to
be
a
common
problem.
If
anybody
wants
to
write
their
own
kubernetes
client
you're,
going
to
run
into
this
issue
unless
you're
very
specific
in
what
you
want
to
do.
A
As
I
mentioned
previously,
the
dynamic
client
has
a
few
drawbacks,
because
you
don't
have
an
actual
struct
that
you
can
really
pin
down
and
grab
something
and
compare
it
to
something
else
right.
It's
it's
a
it's
a
map
of
a
string
to
an
interface
right,
but
you
know
going
to
try
to
work
through
that
and
and
see
what
we
can
do.
A
I
mean
right
now
we're
just
reading
it
into
an
instruction,
unstructured
type
like
kubernetes,
api,
on
structure
type
and
and
then
using
the
kubernetes
go
client
to
apply
that
we'll
see
we're
not
quite
done
so.
I
don't
really
want
to
show
it
in
action,
especially
if
it
turns
out
to
to
not
work
at
all
and
then
the
other
thing
is
that
we
have
this
situation
where
teams
don't
want
to
deploy
every
object
to
every
namespace
or
they
want
to
have
a
certain
type
of
ingress
that
they're
used
for
that
they
use.
A
You
know
the
region,
the
the
registry,
those
those
those
configurations,
but
but
that's
that
that's
kind
of
a
little
bit
too
blunt
of
an
instrument,
so
so
we're
going
to
work
on
that,
probably
some
type
of
annotation
or
label
that
people
put
in
their
kubernetes
objects
in
their
k-8s
directory
inside
their
inside
their
repository
looks
like
that's
what
we
will
do,
but
you
know
still
haven't
still
haven't
decided
yet
so
I'd
like
to
thank
you
all
for
coming.
Thank
you
for
bearing
with
me
at
the
with
the
lighting,
which
is
super
inconsistent.
A
I
don't
have
any
good.
You
know
lights,
so
I'm
kind
of
using
the
sun
and
that's
kind
of
fickle
and
then
also,
if
you
want
to
reach
out,
ask
me
anything
on
the
kubernetes
slack,
I'm
I'm
there.
I
don't
use
twitter.
I
try
to
maintain
my
sanity,
but
but
please
you
know,
please
try
it
out.
You
know
the.
As
I
said
this
is
all
on
git
it's.
It's
meant
to
be
really
a
reference
for
other
people
to
use,
but
you
could
you
know
you
could
use
it.