►
Description
Programming Kubernetes with the Go SDK [I] - Aaron Schlesinger, Deis
Most people talk to their Kubernetes clusters using the kubectl tool, but we’re seeing more and more cases where engineers are writing their own software to talk to the Kubernetes API itself. The use cases for doing so are many: writing ingress controllers, managing pods and watching logs to name a few.
At Deis, we’re using the Kubernetes Go SDK for quite a few different purposes. We first started by vendoring in the Kubernetes mono-repo to watch pod events. We pushed our code to production and quickly learned all about Kubernetes event stream internals and the server-side event cache. When we got something working reliably, we expanded our usage of this client code to a few other domains. We later migrated to the the client-go package when it was released and now use it in almost all of our projects and infrastructure.
In this talk, we’ll share our use cases and detail what we’ve learned using client-go in production. We’ll also share some best practices and show a demo to illustrate how you can get started with the client yourself.
About Aaron Schlesinger
Sr. Software Engineer, Deis
Aaron Schlesinger is a Sr. Software Engineer at Deis. He leads or contributes to development on the Deis Workflow PaaS projects, Helm and the Kubernetes sig-service-catalog Kubernetes incubator project. He’s been a software developer for 10+ years, and most of that time has focused on distributed systems from large scale data warehousing to real time servers. He's a Golang developer at heart.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hey
guys,
everybody
hear
me
OK
in
the
back
yeah
great.
As
you
said,
my
name
is
Aaron
Schlesinger
I'm
senior
engineer
at
dais
come
down
to
our
booth,
get
some
free
socks.
We'd
love
to
see
you
I
will
be
down
there
after
this
talk,
actually
I
also
sit
on
a
leadership
board
of
the
Service
Catalog
sig,
so
drop
in
there
if
you're
ever
interested
in
what
we're
doing
so
today,
as
mentioned
I,
think
in
the
program,
this
is
a
deep
dive.
A
We're
going
to
actually
go
look
at
some
code
before
we
show
a
demo
before
all
that
we're
going
to
go
through
some
use
cases,
including
some
stuff,
that
we
do
a
dais
with
the
go
client
library
and
then
we're
going
to
wrap
up
with
some
patterns
and
if
there's
some
time
left
over
I
will
run
out
into
the
audience
and
take
questions
from
people
on
the
mic.
Hopefully,
so,
first
of
all,
why
would
you
ever
use
the
go
client
library
ever?
A
Why
not
just
script
cubes
ETL
would
bash
or
just
manually
do
stuff
at
your
computer?
Why
not
just
do
that
well
to
get
there.
Let's
talk
about
how
anyone,
how
I
or
any
one
of
you
guys,
probably
started
with
kubernetes
I
call
it
the
twelve
days
of
kubernetes.
I
came
up
with
that
at
about
2:00
a.m.
this
morning.
In
my
hotel
room,
you
would
probably
start
something
like
this
right.
You
would
start
with
something
I've
called
like
the
Zen
of
kubernetes.
That's
something
like
learning!
A
Why
kubernetes
learning
about
immutable
deployments,
learning
about
pods
learning
about
other
basic
resource
types
services
deployments,
replicas
sets
so
on
and
so
forth.
Obviously,
learning
how
to
use
cubes
ETL,
that's
pretty
important
right.
You
might
even
go
a
step
higher
and
learn
about
helm
or
some
higher-level
tools
like
helm,
I
have
a
soft
spot
in
my
heart
for
helm,
because
that
was
started
at
Deus,
but
there
are
other
plenty
of
other
high
level
tools
for
kubernetes,
then,
on
what
I've
called
the
13th
day,
not
bad
luck.
A
This
is
actually
a
really
good
day
for
you
in
kubernetes,
you're,
probably
going
to
go
start
automating
a
few
things
right.
So
probably
the
primary
thing
is
going
to
be
CI.
How
do
you
get
someone's
code
on
a
development
team
through
a
CI
pipeline
through
all
the
tests
stress
tests
staging
all
that
good
stuff
into
an
image?
You
probably
have
figured
that
out
by
now
by
the
13th
days,
and
then
how
do
you
get
that
image
onto
a
cluster?
A
Maybe
you're,
going
to
go
to
a
staging
cluster
first
or
pre,
prod
cluster
I've
heard
it
called
sometimes
and
then
you're
going
to
promote
to
real
production
for
the
big
show
right.
How
do
you
go
through
that
process?
Well,
first,
step
might
be
to
start
scripting
keep
CTL
write
a
bash
script
or
write
it
in
groovy
in
Jenkins,
or
something
like
that
and
then
you're
going
to
do
a
couple.
Cubes
ETL
commands
to
move
through
this
process
right.
A
You
also
might
start
scripting.
Some
things
like
backup,
run
a
script
to
do
some
cube,
cgl
stuff
inside
of
a
pad
inside
of
a
pod
and
put
that
in
a
cron
job,
for
example,
and
start
doing
some
backup
or
analytics
on
your
cluster
or
troubleshooting,
or
maybe
email
out
a
report
to
your
team
on
how
your
clusters
doing
we're
going
to
talk
about
the
14th
day
here
today.
A
This
is
when
you
start
leveling
up
in
your
automation
and
you've
got
to
do
stuff
that
maybe
bash
or
just
cube
CTL
or
something
at
that
level
is
not
really
going
to
be
enough,
and
you
have
to
start
talking
directly
to
kubernetes
directly
to
the
kubernetes
core
api,
doing
things
like
watching
the
event
stream
and
reacting
to
events
in
a
meaningful,
really
deep
way.
Right.
A
That's
just
one
example,
and
that
happens
to
be
something
we're
going
to
demo
today
and
we're
going
to
see
some
code
for
today
and
there's
plenty
of
other
examples
out
there
right.
So
here
are
some
examples
who
here
has
heard
of
a
third
party
resource
in
kubernetes
Wow
awesome
we're
going
to
touch
on
kubernetes
third
party
resources
today
and
in
the
demo.
I'm
really
really
happy
to
see
everyone
raised
their
hands
here
today.
I
love,
pprs,
I
call
them
CPR.
They
could
have
called
three
PRS
by
some
people
too
or
whatever
other
abbreviation.
A
Who
here
has
heard
of
ingress
controllers?
That's
amazing!
Thank
you.
You
guys
are
a
great
crowd.
Thank
you
very
much
give
yourselves
a
round
of
applause.
Anybody,
yeah,
okay,
awesome!
Awesome!
We
heard
yesterday
from
the
Philips
keynote
at
the
end
of
yesterday,
they
wrote
their
own
horizontal
pod
autoscaler.
One
of
the
easiest
ways
to
get
there
is
to
use
the
go
client
library.
Writing
your
goal.
Writing
your
horizontal
pod
autoscaler
and
go
it's
probably
one
of
the
lowest
friction
ways
to
get
something
meaningful
done.
A
You
could
probably
script
something
there
might
even
be
a
bash
kubernetes
client
out
there
and
if
there
is
come,
find
me
later
we'll
have
a
discussion.
I'd
love
to
see
that,
but
the
go
client
library
is
sort
of
made
for
tasks
like
this
and
then
that
last
one
their
user,
aggregated
services
or
user
aggregated
API.
A
This
is
an
up-and-coming
technology,
we're
actually
using
it
in
Service
Catalog
right
now,
and
this
is
a
way
to
extend
the
kubernetes
api
without
touching
the
kubernetes
kubernetes
core
code
base
at
all
right,
you
can
build
your
own
API
and
starting
in
one
six
I
believe
it's
an
alpha
support.
Now
kubernetes
will
figure
out
how
to
route
traffic
from
the
kubernetes
core
api
to
your
api.
But
your
api
here
is
a
catch.
It
has
to
look
like
kubernetes
api.
A
A
In
this
case,
you
might
be
able
to
write
it
in
some
other
languages
too,
and
there
are
some
up-and-coming
libraries
to
do
that,
but
the
gold
client
library,
as
of
now,
is
probably
the
best
fit,
and
then
let's
go
to
some
things
that
deus
and
others
have
done.
We
have
deus
workflow,
that's
an
open
source
project
where
we've
started,
maybe
almost
two
years
ago
now
and
it's
a
path
platform
as
a
service
that
runs
on
top
of
kubernetes.
There
are
a
bunch
of
pods
I
believe
we
have
12
projects
under
the
workflow
umbrella.
A
They
run
a
bunch
of
pods
inside
of
kubernetes
and
they
do
almost
everything
under
the
Sun.
You
could
think
of
the
kubernetes
api
to
make
kubernetes
look
like
something
like
Heroku
right.
You
can
get
pushed
to
this
thing
and
it
will
go
and
build
your
code
with
the
flood
builder
or
if
you
push
something
with
a
docker
file,
it'll
do
a
docker
build
on
it.
A
It
does
do
that
in
a
different
pod
and
then
it'll
figure
out
when
that
pod
done
and
it'll
create
an
image
for
you,
and
it
will
push
that
image
to
a
registry
of
your
choosing.
It
could
be
inside
the
cluster
or
it
can
be
docker
hub
or
quaid
I/o
or
GC
R
or
anything
else
right
and
then
after
that's
done,
then
it
will
somehow
magically.
A
And
then
this
new
team
is
going
to
come
along
and
start
using.
Your
kubernetes
pass
feature
C
and
D,
and
this
is
a
really
great
problem
to
have
right.
I'll
hit
the
wall
soon,
you're
going
to
have
all
these
features
that
people
want,
because
kubernetes
is
so
great
for
deploying
apps
and
your
path
is
going
to
be
the
way
to
build
it
right.
A
Your
developers
are
going
to
want
more
and
more
features
in
the
go
client
library
I
can
almost
like
it
stand
up
here
and
almost
guarantee
that's
going
to
be
the
fastest
way
to
get
these
features
out
the
door
for
your
developers,
and
then
we
have
wizard
levels.
These
are
some
things
we've
done.
A
dataset
have
turned
out
to
be
really
fancy
in
some
cases,
they've
been
overkill,
but
we've
got
things
like
distributed
coordination
right.
This
is
a
primitive,
a
distributed
lock
or
distributed
lease.
A
That's
waiting
for
something
else
to
complete
and
putting
that
in
a
critical
section,
so
that
nobody
else
can
touch
it.
We
got
leader
election
anybody
heard
of
leader,
election
cool
cool,
that's
having
a
process,
do
all
the
work
and
having
a
bunch
of
other
processes,
wait
and
if
that
main
process
fails.
One
of
those
other
guys
would
pick
up.
The
slack
and
they'll
start
doing
the
work,
but
only
one
of
them
is
going
to
do
the
work
at
a
single
time
distributed
latch.
A
That's
again
photo
like
coordination,
that's
waiting
for
someone
else
to
finish
work
elsewhere
in
the
cluster,
and
then
you
continue
your
work.
Okay,
just
some
of
the
use
cases
for
going
in
and
starting
to
look
at
the
client.
Go
library
go
SDK,
let's
go
into
code
mode!
Now,
let's
start
talking
about
what
this
really
looks
like.
Okay,
first
of
all,
what
is
the
SDK?
It's
a
go
package.
You
can
go,
get
it
or
use
your
favorite
go
dependency
manager!
It's
there.
It's
called
Kate's,
Kas,
io,
/
client
go.
A
It
is
focused,
as
the
name
might
imply,
on
building
clients
it's
outside
of
the
mono
repo,
it's
outside
of
kubernetes
flash
kubernetes,
it's
specific
to
clients
right
you
get
full
access
to
the
core
Cunha's
api.
That's
everything!
All
the
resources
watching
all
the
event,
streams,
everything
and
you're
going
to
get
the
best
bang
for
your
buck,
starting
they're,
starting
access
to
the
core
API,
instead
of
using
coop
CTL
to
do
it.
I
think
there
was
coop.
Cuddles
is
another
name:
I'll
use
coop
CTL
today,
but
there's
also
more
right.
A
There's
higher
level
constructs
I'm
not
going
to
be
able
to
show
today,
but
there's
higher
level
constructs
to
build
really
rich
controllers
or
to
do
really
fancy.
Caching
on
the
client
side
and
encourage
you
that's
kind
of
wizard
level,
two,
as
you
start
using
the
client
library,
and
then
you
want
to
go,
build
something
even
more
high-level
go
look
at
those
packages
and
they're
all
inside
of
this
package.
Client
go
so
let's
talk
about
some
of
the
relevant
packages
in
client
go
ok!
A
These
are
the
ones
we're
going
to
see
today,
there's
tons
more
so
first
of
all,
starting
from
the
top
kubernetes
is
the
first
sub
package
that
you'll
hit
when
you
use
client
go
so
the
kubernetes
package
basically
contains
this
big
old
thing,
this
big
old
struct
and
go,
and
that
will
give
you
access
to
everything.
The
whole
gamut
of
resources,
including
you,
get
access
to
the
core,
API
groups,
extensions,
so
on
and
so
forth
and
there's
rest
the
second
one
down
that
contains
a
raw
rest
client.
A
You
can
literally
talk
to
kubernetes
using
get
this
path
or
HTTP
POST
to
this
path
with
a
body,
and
it
also
contains
config
and
you'll
use
it
most
for
config
and
that's
just
how
do
I
locate,
authenticate
and
authorize
with
the
kubernetes
core
API
server.
Another
is
dynamic.
Now
dynamic
is
sort
of
a
smaller
client
that
sits
inside
of
the
kubernetes
master
client
and
that
we're
going
to
use
today
to
talk
with
third-party
resources.
A
I
told
you
I'd
come
back
to
third-party
resources,
we're
going
to
come
back
with
a
vengence
in
a
second
and
then
that
last
one
PKG
api,
that
dot
dot
dot
means
there's
a
whole
bunch
of
stuff
under
PKG
api.
That's
well
you're,
where
you
will
find
all
of
the
resources
that
you
know
of
in
kubernetes.
There
is
a
ghost
struct
for
pod
services,
namespaces
deployments
nodes,
everything,
they're
all
accessible
there
and
they're
all
go
struck,
so
you
can
work
with
them
in
memory
as
you
go
now.
A
A
Okay,
so
we
always
start
with
something
like
hello
world
right
now.
This
is
sort
of
the
simplest
thing
to
show
off
some
of
the
concepts
that
we
can
put
on
paper
on
ePaper
right,
so
that
first
line
there
is
a
really
handy
convenience
function,
and
what
that
is
is
give
me
a
configuration
for
talking
to
the
kubernetes
core
API,
just
given
all
the
stuff
that
any
pod
and
the
kubernetes
cluster
will
already
have
so.
A
A
If
you
come
find
me,
then
that
second
line
given
a
configuration
that
I
got
in
the
first
line,
create
that
new
massive
client
right
create
me
a
client
given
that
configuration
that
I
can
use
to
do
everything
if
I
want
to
and
then
that
third
line,
that's
basically
just
lists
all
the
pods
in
a
given
namespace
right.
That
can
be
pretty
handy.
You
can
also
list
all
the
pods
in
all.
Namespaces
will
see
that
in
use
in
the
code
as
well,
but
there
are
tons
and
tons
of
other
uses
here
right.
A
You
can
call
core
v1
that
that
first
function
on
CL
and
that
will
give
you
access
to
all
the
core
kubernetes
resources
pod
services.
Again,
if
the
list
goes
on
and
on,
you
can
call
extensions
and
extensions
will
give
you
access
to
all
the
things
in
the
API
group
called
extensions
and
so
on
and
so
forth.
A
But
that's
just
hello
world.
Let's
talk
about
something
real,
more
real,
something
you
might
actually
use.
Let's
talk
about
a
program
that
can
backup
the
state
of
a
cluster
right.
Let's
talk
about
something
where
I
can
take,
I
can
tell
it
to
backup
pod
and
when
I
tell
it
that
it's
going
to
grab
all
the
pods,
just
as
we
saw
on
the
previous
slide
that
third
line,
but
it's
going
to
grab
all
the
pods
in
all
the
namespaces
and
it's
going
to
back
them
up
somewhere,
so
that
I
can
see
over
time.
A
What
did
my
cluster
look
like
what
pods
were
running
right?
That
can
be
pretty
useful,
but
furthermore,
I
want
to
make
this
thing
a
little
simple
right:
I,
don't
want
to
have
to
open
up
a
port
and
serve
on
a
REST
API
and
create
a
client
for
that
right.
Api
I
want
to
just
cube.
Cto
creates
F
a
third-party
resource,
just
like
I,
create
anything
else,
including
Eddie's
and
I,
want
that
to
trigger
the
backup
processing.
A
So
that's
the
context.
Let's
talk
about
how
this
thing
will
look
before
we
go
into
the
code.
I
promised
last
slide
before
code,
who
here
was
waiting
for
code
and
saying
shut
up.
Let's
get
to
the
code.
Okay,
fair,
fair
enough,
we'll
get
there.
I
want
to
show
this
before
we
get
into
the
code,
so
you
guys
know
kind
of
what's
going
on
so,
first
of
all
our
main
function,
every
go
program
starts
with
the
main
right,
a
lot
of
other
programming
languages.
Do
that
too.
Now,
that's
the
thing!
A
That's
going
to
create
a
watcher,
the
thing
that
consumes
the
watch
events-
it's
going
to
use
the
dynamic
client
to
do
that.
It's
going
to
send
every
watch
event
down
into
the
watch
event
filter!
That's
just
going
to
decode
stuff
into
those
backups
third
party
resources
that
I'll
show
the
filter
is
going
to
send
those
decoded
events
back
up
into
the
watch
loop
and
the
watch
loop
is
just
going
to
multiplex.
A
A
Good
are
those
thumbs
up
or
make
it
bigger?
Okay,
good
all
right.
So
here's
our
main
this
is
where
it
all
starts.
You
guys
probably
know
that
here's
that
pattern
again
in
cluster
config,
this
thing
is
going
to
work
in
pods
without
me
having
to
figure
out
what
environment
variables
here,
what
file
is
there
same
thing?
Let's
create
that
big
master
client
from
the
config.
Now
we
get
a
little
fancy
from
that
same
config,
let's
create
a
dynamic
client.
This
is
the
thing
that
we're
going
to
use
to
talk
to
third
party
resources.
A
Okay,
next
line,
we
care
about,
let's
create
the
watcher
that
was
the
Box
on
the
left,
the
bottom
left-hand
side
right.
The
watch
is
going
to
be
that
thing
that
watches
these
third-party
resources.
Next,
let's
run
that
loop,
that's
the
loop!
That's
going
to
consume
those
watch
events,
it's
going
to
wait
for
an
event
to
come
in
it's
going
to
multiplex
to
the
right,
set
your
logger
and
then
it's
going
to
call
that
fetcher
logger
to
do
the
work,
the
real
work
that
we
care
about.
Okay.
A
A
You
probably
have
seen
of
the
pod
struct
or
the
namespace
struct
all
of
the
in-memory
kubernetes
object
types
have
these
two
lines
in
them
and
they're
very
convenient,
and
if
you
try
to
write
a
kubernetes
resource
type
and
a
third-party
resource,
for
example,
without
those
you're
going
to
have
a
hard
time,
so
I
recommend
first,
two
lines
always
put
those
and
then
the
thing
that
we
care
about
today
is
resource
type.
That's
the
parameter!
That's
going
to
tell
our
watch
loop.
What
do
I
want
to
back
up?
A
It's
just
like
calling
a
function
and
saying
these
are
the
resource
types
I
want
to
back
up
next,
the
watch
loop
itself.
Excuse
me:
if
you
watcher
itself,
we
create
a
watcher
right
here
from
the
dynamic
client
that
I
showed
we
created
in
the
main.
Ok,
so
we're
watching
backup
the
backup
resource
type
in
a
given
namespace
and
then
down
here.
We're
going
to
send
all
those
watch
events
through
a
filter.
Ok,
let's
go
down
to
the
filter.
This
is
a
big
decode
and
encode
job
right.
A
We
get
objects
coming
in
as
this
unstructured
type.
That's
a
bucket
of
bytes
kubernetes
says
I've,
never
heard
of
a
backup
I'm
just
going
to
dump
you
some
bytes
and
you
figure
out
what
to
do
with
it
and
we
will
figure
out
what
to
do
with
it
on
the
next
line.
Ok,
so
we're
going
to
marshal
and
then
unmarshal
into
our
backup
object
that
I
just
showed
before.
A
Ok,
once
we
do
that,
assuming
all
went
well,
we're
going
to
return
a
new
watch
event
that
looks
pretty
similar
to
the
old
watch
event,
except
this
new
thing
is
going
to
have
a
backup
in
it
and
that's
the
thing
that
we
know
how
to
deal
with
in
our
watch.
Loop
speaking
of
here
is
the
watch
loop
lots
of
code
in
here.
Let's
focus
on
two
things.
First
thing
here:
we're
going
to
consume
from
a
go
channel
that
represents
these
watch.
A
Events
coming
in
right
either
in
memory
watch
events
that
we're
going
to
figure
out
what
to
do.
It
second
thing
up
here
on
line
18:
this
is
an
infinite
for
loop.
That
is
not
a
bug.
Ok,
this
watch
event
on
line
26
can
close
at
any
time.
This
is
a
channel
that
if
the
server
decides
no
more
watch,
events
for
you
it'll
sever
the
connection
and
will
figure
out
that
the
connection
is
severed
by
that
channel
closing.
A
If
that
channel
closes
we're
going
to
bail
out
of
this
inner
for
loop,
we're
going
to
log
and
guess
what
we're
going
to
go
back
up
to
line
18
and
start
all
over
again,
ok
and
then
finally,
there's
a
switch
statement
here,
we're
only
going
to
support
pause
in
this
demo,
I'll
get
to
in
just
a
second,
but
you
had
more
cases
there.
Your
watch,
Lube
thing
and
start
supporting
all
kinds
of
other
resources,
and
then
finally,
here
is
the
fetcher
lager
for
pods.
Remember
this
oops.
A
Remember
that
the
only
difference
between
that
and
what
we
showed
in
our
hello
world
example
is.
This
is
grabbing
pods
from
every
single
namespace.
That's
what
that
open
parentheses,
closed
parentheses
means
you
can
do
that.
That's
valid
in
kubernetes
not
recommended
in
lots
of
clusters,
because
when
you
get
to
let's
say
a
hundred
pod,
that's
going
to
take
forever
lots
of
data
is
going
to
go
over
the
wire.
A
If
you
ask
for
a
bunch
of
pods,
so
I
wouldn't
recommend
doing
that
on
a
production
cluster
instead,
I
would
probably
iterate
through
namespaces
and
do
it
one
namespace
after
another.
All
right.
Let's
see
this
thing
in
action.
Put
your
hands
together,
pray
for
the
demo
God
in
this
case.
If
this
doesn't
work,
it's
a
hundred
percent,
my
fault
alright,
so
you
can
follow
along
with
the
code
here
at
this
github
address.
Everything
is
open-source
everything
get
pushed
to
master
from
my
computer
and.
A
A
It's
going
to
download
this
from
kway
teow
we're
good
thanks
to
Wi-Fi
we're
going
to
look
at
log.
Remember
when
we
get
a
backup
third-party
resource
coming
in
we're
going
to
fetch
all
the
pods
and
print
the
standard
out.
So
when
we
create
that
backup
third
party
resource
we're
going
to
see
a
big,
huge
chunk
of
JSON,
that
represents
all
the
pause
in
our
cluster
and
we're
going
to
see
it
there
just
to
make
sure
we
can
see
it.
Let's
do
that.
A
Okay,
second
step,
let's
install
a
backup
before
we
do
that,
I
want
to
show
you
what
a
backup
looks
like
okay.
This
is
probably
familiar.
I
want
to
point
out
one
thing:
that's
different
from
this
and
a
kubernetes
built-in
resource
that,
because
kubernetes
ever
heard
of
a
backup
resource
have
any
of
you
guys
heard
of
a
backup
resource
in
the
kubernetes
core.
Okay,
that's
good,
believe
it
or
not,
and
the
reason
is
because,
in
the
watcher
I
created
a
third-party
resource
called
backup.
A
A
A
And
there
you
have
it,
there's
five
pod
pod
zero
through
four.
Now
this
is
a
pod
here
that
we
actually
care
about.
But
the
purpose
of
this
thing
was
to
dump
all
the
pods
and
again,
as
I
mentioned
before,
you
might
want
to
extend
this
thing
and
maybe
dump
it
to
persistent
storage
put
in
an
s3,
maybe
send
an
email
out
with
a
report
and
also
it's
not
that
useful
to
print
out
pods
right.
You
might
want
to
print
out
all
the
services
or
the
network,
ingress
policies
and
the
namespaces.
Something
like
that.
A
A
So,
let's
do
a
little
recap
on
what
we
just
wrote:
I've
got
some
patterns
out
of
this
okay,
so
what
we
built
in
kubernetes,
land,
you're,
probably
familiar
with
this
name,
is
called
a
controller
or
a
controller
manager.
This
is
the
thing
that
generally
doesn't
really
have
a
meaningful
network
interface.
You
don't
really
call
like
get
something
on
a
controller.
It's
watching
it's
watching
for
event.
The
kubernetes
has
some
built-in
controllers
that
watch
for
events
on
the
core
api
server
and
then
they
reconcile
space.
We
did
the
same
thing.
A
We
watched
events
on
the
kubernetes
core
api
server
for
those
backup
tt-rs
and
our
reconciling
a
state
was.
We
just
did
another
call
on
the
core
API
to
get
pods,
and
then
we
printed
it
out
reconciled
right.
Also,
remember
those
two
for
loops.
We
were
prepared
with
that
outside
for
loop
that
went
on
forever.
We
were
prepared
for
the
watch
stream
to
close
at
any
time
now.
There's
another
way.
A
We
be
able
to
know
we
built
our
own
resource.
We
built
the
TPR
called
backup.
I
showed
you
the
template
for
that.
That's
a
great
way.
I
think
I
mentioned
this
before
too.
It's
a
great
way
to
super
easily
extend
kubernetes
I'd
recommend
looking
in
that,
if
you
want
to
add
some
custom
functionality
before
you
go
and
look
inside
of
kubernetes
proper
and
submit
a
pull
request
to
file
an
issue
or
anything
like
that.
These
can
go
a
very
long
way.
Okay
forgot
my
little
clicker.
A
So
with
that
I
want
to
say
thanks:
we've
got
7
minutes
and
27
seconds
left
for
questions,
so
I'm
actually
going
to
run
around
the
room.
If
anyone
has
questions
raise
your
hand,
all
right
go.
Do
I'm
going
to
run
to
you
with
this
little
fancy
thing
and
have
you
say
your
question
into
the
microphone
so
I
can
get
it
off
in
my
head.
I
could
do
that,
but
this
is
more
fun.
Okay,.
B
Chase,
thank
you.
So
my
question
is
to
you
and
to
the
audience.
Is:
okay.
Controllers
are
great
things
for
a
synchronously
processing
events,
but
what
if
you
want
to
be
in
line
so,
for
example,
you
want
to
prevent
the
pod,
mostly
spinning
up
for
whatever
conditions
you
want
to
have
or
customer
polarization
whatever
or
whatever
thing
you
want
to
be
in
line
masters
interests.
Thank
you
yeah.
So
that's.
A
A
great
question,
so
the
question
was
I'm
going
to
summarize
what,
if
you
don't
want
to
have
a
controller
running
in
the
background?
What
if
you
want
to
do
something
such
that?
Let's
say
you
cube
ptl's,
do
things
and
it
should
fail
just
right,
then,
in
there
not
not
have
to
go
and
wait
for
the
controller
to
do
some
work
and
then
have
someone
look
in
the
controller
status
field.
My
summarization
was
longer
than
your
question.
So
sorry
about
that.
A
So
what
you're
going
to
have
to
do?
There
is
look
into
the
user
aggregated
service
thing.
You
can,
google,
it
user,
aggregated
services
just
came
out
in
1:6
and
again
just
to
recap
what
those
things
are:
kubernetes
core
the
core,
API
server
will
route
traffic
to
your
API
server
right.
Your
API
server
then
can
do
the
action
to
intercept
a
request,
fail
the
request,
if
necessary,
and
that
gives
that
feedback
back
to
cube.
A
Ctl
writing
a
UAS,
your
user
I
created
service,
that's
probably
another
top,
and
once
it's
tables
to
be
honest,
I'll
probably
submit
a
CFP
for
that,
and
we
can
talk
about
that
next
time.
So
still
in
alpha
so
probably
worth
reading,
but
probably
not
worth
putting
a
production
yet
other
questions.
Oh
okay,
shout
it
out.
B
A
Yep,
the
question
was
shouldn't:
you
just
fail
fast
with
the
watch
stream.
In
our
case,
if
the
watch
stream
is
broken,
why
not
just
fail
fast
and
what
kubernetes
we
started,
assuming
you're
running
inside
of
a
replication
controller
or
a
replica
set
stellar
question
and
yes,
you
could
fail
fast,
but
totally
valid
the
problem
in
both
cases.
Actually
in
my
code
and
the
fail
fast
case
is
that
you'll
you'll
miss
watch
event
right
either
way.
If
this,
if
the
watch
string
gets
broken,
you'll
miss
like
a
couple
hundred
milliseconds,
probably
of
watch
events.
A
If
you
fail
fast,
more
epic
kubernetes,
how
many
wash
events
you'll
miss
and
the
solution
in
either
case
is
that
you
have
to
lift
you
have
to
relist
the
resource
that
you're
looking
for
and
that's
where
we
get
into
the
controller
infrastructure
that
lives
inside
of
client
go.
So
what
the
controller
infrastructure
does.
Is
that
double
for
loop
just
to
make
it
a
tiny
little
bit
more
robust
and
not
have
controllers
continuously
failing,
but
they
periodically
relist,
so
they'll
relist
backups.
A
In
our
case,
and
when
the
relist
happened,
they
then
reconcile
again
they
reconcile
a
base
to
base
on
what
backups
haven't
been
processed.
Yet
so
failing
fast
is
a
great
place
to
start
once
you
go
into
the
controller
infrastructure,
they'll
take
care
of
the
double
four
loop
stuff
for
you,
so
that
seems
to
be
the
de
facto
standard
is
to
kind
of
build
a
little
bit
more
resilience
into
the
app,
but
I
would
actually
recommend
failing
fast
to
start.
So,
yes,
yes,
yep
want
to
shout
it
out.
A
Sir,
yes,
so
the
question
was:
is
there
a
way
to
select
portions
of
a
response?
So,
for
example,
if
your
listing
pause
instead
of
having
a
big
huge
massive
response
body,
can
I
drill
into
just
a
specific
part
of
the
pod
spec
for
the
pod
status?
The
answer
is
yes,
so
when
you
get
Tod
back,
you'll
get
back
a
list
called
a
pod,
less
convenient
name
right.
A
A
A
Gotcha,
so
that's
not
as
easy
to
ask
communities
to
just
give
a
specific
part.
It
will
return
the
whole
pods
back.
The
whole
pod
manifests
actually
to
you
and
that's
where
the
paging
comes
in
or
if
paging
is
too
much
for
you,
just
selecting
all
the
pods
in
a
single
namespace
instead
of
all
the
namespaces.
So
no
support
currently
built
into
client
go
unfortunately
to
do
subpod
resources.
A
Sure
yeah
so
first
question:
was
there
a
way
to
do
label
selectors
as
one
of
the
examples
there
is,
my
slides
are
gone
for
some
reason:
I
think
my
computer
died.
If
you
go
to
the
github
repository
or
go
to
the
slides
which
I'll
post
when
it
lists
all
a
pod,
there's
a
v1
dot
list
option
at
the
end
of
that
call
list.
Options
is
where
you
can
do:
label
selectors
and
there's
a
field
selector
as
well
field.
Selectors
are
start
to
getting
subpod
resource
fields
back.
A
They
don't
go
all
the
way
down
that
road,
but
they
I
imagine
they
will
soon,
probably
in
1.
7.
Second
question
was:
what
do
you
want
or
what,
if
I
want
to
update
client
goes,
submit
a
pull
request
on
it
or
something
like
that
so
client
go
currently
as
it
stands.
Right
now
gets
copied
from
a
staging
directory
inside
of
kubernetes
kubernetes,
so
I
would
recommend,
starting
with
an
issue
or
full
request
on
kubernetes
kubernetes
and
then
waiting
for
the
next
release
to
get
it
copied
over.
A
If
you're
impatient
like
me,
I
would
probably
just
go
into
client.
Go
it
self
submit
a
pull
request
and
then
have
someone
else
figure
out
how
to
get
it
in
faster.
That's
the
best
I
got
for
you
right
now,
so
that's
all
I
have
for
now
for
today.
I,
really,
thank
you
guys
for
coming
thanks
for
switching
rooms
with
me.
It's
been
great.