►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Behind Your PR: How Kubernetes Uses Kubernetes to Run Kubernetes CI - Sen Lu & Benjamin Elder, Google
As a contributor of CNCF/Kubernetes project, you open issues/PRs in one or more of the Kubernetes repos, and the k8s-ci-robot will add proper labels, assign reviewers, trigger presubmits, and automatically merge your PR. This talk will walk you through how Kubernetes CI is automated in Kubernetes clusters, how we use CRDs to manage the presubmit/CI jobs’ lifecycle, and how we automate contributor workflows on Github. By the end of this talk, contributors to Kubernetes project should have a better understanding what happens behind the scenes for Kubernetes CI. Other github projects can also utilize our test-infra for their CI.
To Learn More: https://sched.co/GrSb
A
Hello,
hello,
okay,
let's
get
started
it's
the
first
day,
so
hope
everyone's
still
pumped
up
before
we
go
into
anything.
Let
me
introduce
I
know
a
lot
of
you
have
been
staring
at
the
front,
the
cover
page
for
a
while,
and
we
have
these
little
flushes.
So
this
is
called.
This
is
fine
dogs,
and
the
cover
page
actually
is
the
photo
took
by
Aaron
in
Cuba,
Kong,
Europe,
credit
to
Aaron,
and
it's
our
team
mascot.
Why?
A
Because
we
are
maintainer
of
cumin,
a
DCI
and
whenever,
when
things
goes
bad,
we
just
say
this
is
fine
and
we
start
to
fix
things,
and
you
will
see
this
is
our
team
spirit
and
you
will
see
this
dog
like
coming
coming
out
throughout
the
park.
Okay,
my
name
is
San
Liu
and
I'm
krz
crazy
lazy
on
github,
it's
kind
of
hard
to
pronounce,
and
this
has
been
the
outer
and
bendy
out
there
on
github.
A
B
B
So,
as
Sen
said,
we
already
work
with
kubernetes,
so
it's
kind
of
obvious,
so
we're
going
to
use
it,
but
what
we
want
to
convince
you
is
that
it's
actually
really
excellent
for
a
CI
in
particular,
it
scales
up
to
even
big
projects
like
kubernetes.
It's
very
fault,
tolerant
because
of
the
way
the
controllers
and
the
distributed
system
are
designed,
and
it
already
implements
some
of
the
core
CI
features
itself
that
you
don't
need
to
build
so
scaling
up
to
the
project.
We
run
north
of
10,000
jobs
a
day
for
the
kubernetes
project
ci.
B
We
have
a
lot
of
different
jobs,
a
lot
of
different
projects,
lots
of
config
there's
something
like
150
repos
kubernetes
lets
us
not
worry
about
scaling
this.
It's
very
easy
to
scale
kubernetes,
it's
fault-tolerant,
so
we
can
just
say
this
is
fine.
Most
things
they're
not
actually
really
going
to
break
with
the
way
that
everything
constantly
tries
to
reconcile
the
state
of
the
world
against
the
desired
state.
The
system
fixes
itself.
We
spend
a
lot
less
time.
Maintaining
our
CI
a
lot
more
time
focused
on
maintaining
everything
else.
B
Building
features
and
kubernetes
already
implements
some
of
the
core.
Ci
features
pods,
let
you
run
containers
which
is
great
way
to
run
your
tests.
Ert
is
let
you
do
resource
management,
config,
Maps.
Let
you
roll
out
configuration
secrets.
Let
you
have
managed
credentials.
We
don't
need
to
build
these
things
from
scratch
and
we
don't
need
to
learn
another
system.
We
understand
these
things,
so
your
kerbin
IDs
PR
and
has
Kate
CI
robot
all
over
it
everywhere.
B
All
these
comments,
that's
all
running
on
kubernetes
it's
built
and
tested
on
kubernetes
all
the
automations
cuber
Nettie's,
the
merge,
robot
/
commands
and
importantly
catch
ifs
whenever
you
request
them
all
running
on
kubernetes
and
as
part
of
this
we'd
already
been
container
Eisinger
building
tests.
I
think
a
lot
of
people
here
probably
do
that.
It's
a
really
great
way
to
run
things.
You
get
a
more
hermetic
reproducible,
build
and
test.
You
can
bundle
all
your
dependencies.
It
removes
a
lot
of
headaches
and
if
you're
gonna
run
containers
well,
kubernetes
loves
containers.
B
You
just
need
to
make
a
pod
pods
will
let
you
schedule
container
across
the
whole
cluster.
You
can
let
Coubertin
aids
manage
the
lifecycle
and
you
can
leverage
all
the
advanced
pod
functionality
like
secrets
volumes
and
it
containers
sidecars,
and
you
can
use
all
these
things
for
your
credentials.
Your
cashing
separating
code
check
out
from
the
actual
test
running
sidecar
containers
to
take
things
that
you
ran
and
upload
them.
We
can
do
pod,
presets
2d,
do
power
configuration
and
we
can
use
resource
requests
to
help
schedule.
B
So
a
test
might
look
something
like
this.
We
have
a
little
bit
of
boilerplate
over.
It
should
specify
like
what
repo
it's
running
on
and
when
it
should
run,
but
other
than
that
you
specify
a
container
in
a
normal,
kubernetes
pod
spec.
You
have
an
image,
you
have
a
test
command
and
we
create
a
pot
and
the
pod
will
run
an
init
container
which
does
some
git
checkout,
and
then
it
will
be
in
a
shared
volume.
B
A
So
talking
about
CI
CI
is
about
creating
testing
and
treating
different
resources.
And
how
do
we
manage
resources,
including
Ares?
You
know
a
resourcing.
Kubernetes
can
be
a
pod
or
your
node.
There
are
controlled
by
controllers
and
we
want
to
manage
our
test
jobs
and
how
did
the
OS
customer
resources
in
kubernetes
lets?
You
see
our
DS,
which
is
customer
resource
definition,
I,
believe
you
already
know,
so
we
make
test
jobs,
they
are
these
and
they
can
be
created
and
managed
by
custom
controllers.
A
Basically,
you
wrote
a
controller
that
is
respond
to
the
event
or
watch
the
state
of
the
CRT
and
then
interact
with
the
CR
D.
So
this
is
a
little
bit
of
comparison
between
the
traditional
solutions
and
with
kubernetes.
Traditionally,
a
model
is
mostly
have
a
master
and
agent
and
the
master
will
maintain,
will
maintain
a
queue
of
jobs
and
the
master
will
schedule
the
jobs
on
two
agents.
That
sounds
like
it's
already
been
should
be
done
by
kubernetes
right
with
kubernetes
and
CR
DS.
It's
just
it's
distributed
and
the
API
it's
our
declarative.
A
So
you
just
need
to
create
a
CR
D
and
let
hoobnet
controller
to
schedule
the
pod
and
also
let
the
kubernetes
controller
to
manage
the
state
of
the
CR
DS,
okay.
So
a
core.
A
few
of
the
core
bullet
point
I
want
to
go
through
is
with
customer
controllers.
You
can
have
different
ways
of
triggering
so
with
losing
a
cluster.
You
can
have
different
ways
of
trigger
a
CI
job.
For
example,
it
can
be
a
time-based,
for
example.
You
want
to
have
a
daily
boot
job
running
at
8:00
a.m.
A
in
the
morning,
and
you
specify
a
crown
strain
on
the
CR
D
on
controller
and
I'm.
Sorry
and
the
controller
will
respond
to
the
timely
event
and
create
a
CRT
or
for
project
like
kubernetes.
It
hosts
its
code
on
github,
so
we
can
write
customer
controller
custom
controllers
to
either
interacting
with
the
github
web.
A
A
You
just
read
your
own
custom
controller
that
interacting
with
the
get
to
get
to
that
gate
platform,
and
you
can
either
do
polling
or
listening
and
creating
strategies
based
on
that,
we
also
can
handle
different
ways
of
triggering
jobs
because
sorry
different
ways:
that's
executing
jobs
because
we
store
the
entire
container
and
the
path
back
into
the
CR
decels.
So
when
the
containers
see
a
CRT
needs
execute
it,
it
knows
how
to
talk
to
the
kubernetes
api
server
and
say:
hey,
create
a
pod
onto
a
different
name,
space
or
even
schedule.
A
This
part
onto
a
different
cluster
that
are
can
be
achieved
also
for
utility
layers,
like
monitoring,
garbage,
collecting
and
reporting.
We
can
also
handle
that
with
custom
controllers,
you're
writing
controllers,
which
can,
for
example,
on
left
side.
You
can
watch
the
CRT
status
change
whenever
a
CRT
is
spinning.
Whenever
a
test
job
is
finished,
the
controller
will
report
the
status
back
to
github
and
update
the
contacts
on
github
under
your
PR.
A
A
A
So
previously
give
you
a
little
story
without
using
kubernetes,
we
did
some
dumb
things
that
we,
when
we
configure
jobs
each
job
will
just
bind
to
a
DCP
project,
because
at
the
early
stage
we
were
mainly
testing
kubernetes
on
GC
and
GK.
So
we
say
job
who
need
to
use
this
specific
GCP
project
and
what
problem
we
are
having
is
whenever
someone
comes
in
hey,
can
we
have
a
new
kubernetes
job?
We
say?
Oh
ok
sure,
and
then
one
of
us
need
to
create
a
DCP
project
and
map
it
to
the
map
to
the
job.
A
You
can
see
the
dog
assess.
This
is
fine,
but
everything
is
on
fire.
We
probably
don't
want
to
do
that
test.
Jobs
can
also
be
see.
Are
these
ard
makes
it
easier
to
and
cleaning
up
resources
so,
for
example,
the
kubernetes
Denton
job
one
a
Canary's
cluster
on
a
GCP
project?
So
we
make
the
GCP
project
a
CR
D,
so
we
can
use
cube
tests
and
creating
a
EDD
test
cluster
on
that
C
on
that
testing
project,
and
we
have
a
next
one.
A
We
have
a
little
system
called
Paskus
that
manages
the
state
of
other
testing
projects
because
there
are
CRTs
and
they
basically
cycling
between
the
clean
and
dirty
State.
We,
when
the
ete
job
begins,
the
job
will
trying
to
leasing
a
pod
leasing,
a
project
see
Rd
and
look
for
the
clean
state
and
and
market
has
dirty.
Whenever
the
job
is
finished.
We
also
have
a
janitor
controller
that
will
continuously
picking
up
dirty
resources
and
run
a
clean
up
script
and
set
a
resource
back
to
clean
for
the
next
end-to-end
job
to
use.
A
We
have
a
lot
of
CR
DS,
but
the
next
problem
we
want
to
stop
here
is
CR.
Ds
are
coming
from
come
fix
and
we
as
kubernetes
testing
from
maintain
their.
Currently,
we
are
maintaining
certain
orgs
over
150
repo.
We
have
over
a
thousand
jobs
and
over
10,000
jobs
running
every
day
on
our
cluster,
which
means
we
do.
You
have
a
lot
of
conflict
to
maintain
so
again,
how
do
we?
How
do
we
did
that
before
kubernetes
our
own
car
will
say:
hey
you
have
a
conflict
update.
A
Let
me
just
push
the
config
into
the
cluster
from
my
workstation.
This
is
fine,
and-
and
we
probably
can
do
better
right-
you
can
see
the
bad
come
up
again
and
say:
mm
I
can
probably
help
you
to
automate
this,
so
instead,
so
instead
we
figure
out.
We
should
map
our
config
into
config
maps
and
we
can
build
some
automation
from
the
bots.
A
For
example,
you
can
see
when
someone
changed
a
job
convict
in
the
a
the
best
job
and
once
upon
that
PRS
merged
the
robot
will
come
and
say:
hey,
I'm,
updating
this
key
in
this
config
map
and
the
doc
says
very
happy
good
bot,
and
on
top
of
that
we
don't
want
to
redeploy
our
controllers
whenever
there
is
a
config
change,
that's
too
messy,
and
so
we
build
something
called
config
agents.
For
example,
when
you
see
on
github
when
we
receive
a
merge
event,
the
controller
will
update
the
config
map,
but
living
inside
each
controller.
A
There
is
a
config
agent
which
are,
they
are
all
responding
to
the
config
map
change
event
in
Sapa
whenever
the
config
map
changes
the
confident
we'll
update
the
config
that
stores
in
the
cluster
so
that
whenever
the
config
is
updated
and
merged
in
repo,
our
controllers
get
to
use
the
latest
config.
That
Suns
are
great
to
me,
but
the
doc
says
is:
it
is
questioning
it.
What
we
are
missing
here,
one
more
problem
I
can
think
of.
A
If
we
don't
need
to
redeploy
our
controllers
when
a
config
change,
but
what,
if
we
fix
some
bugs
or
we
add
some
new
features
to
our
controllers,
now
things
we
have
to
deploy,
deploy
it
right?
Ok,
how
did
we
do
this
before
you
might
notice
this
avatar
is
available
and
if
you're
ever
contributed
kubernetes,
this
robot
will
come
in
to
undergo
PR
and
say
slash
retest
or
if
you
have
an
issue,
that's
not
active.
For
90
days
the
bots
were
coming
say:
market,
a
stale,
Marcus
rotten
and
after
150
days
the
bots.
A
They
closed
this
issue.
So
when
fada
was
uncaught,
he
will
trying
to
Butte
and
push
out
prowl
images
from
his
workstation
or
maybe
not
from
the
PR
body.
You
can
see
I'm
using
this
bump
SH
script
from
my
Mac.
Will
that
work,
and
not
yet
but
and
Fela,
will
just
like
trying
to
push
proud
into
when
he's
on
call
at
Friday
afternoon.
That's
a
typical
time.
We
updating
our
cluster
and
Friday
afternoon
in
Seattle,
I'm,
updated
everything.
A
Everything's
broken
like
now
that
the
single
job
is
working.
Why?
But
it
must
be
my
fault.
So
the
next
thing
we'll
do
get
revert,
revert
this
come
here.
We
defy
everything,
feel
exhausted
right
again.
We
can
do
better,
probably
that's
not
what
we
want
to
say
and
we
want
to
introduce
our
robot
friend
and
kubernetes
friend
again
to
help.
So
today
we
are,
if
we
have
evolved
from
there,
which
the
robot
will
compute
and
push
the
controller
images.
Whenever
there
is
a
controller
update,
the
robot
will
make
sure
everything
works.
A
A
Zoom
out,
ok,
so
I've
talked
a
lot
of
lot
of
things
we
doing
in
kubernetes
CI
and
just
show
you
a
big
picture.
This
is
actually
made
by
up
in
the
elder
and
against
the
each
gopher.
Here
you
can
see
you
can
treat
it
as
a
custom
controller
and
each
crate
box
a
test,
a
test
job
and
the
controllers
are
busy
moving
the
test,
jobs
around
make
sure
they
pass
and
we
also
have
a
sale
called
test
grid.
That
shows
you
the
result
of
each
of
each
CI
jobs.
A
B
So
this
is
all
great,
but
it's
kind
of
annoying
managing
resources
that
aren't
actually
under
kubernetes.
So
we've
been
hacking
on
well
what
if
we
could
just
run
those
into
end
tests
but
keep
everything
on
the
cluster.
We
don't
want
to
manage
external
resources,
even
with
the
controllers.
If
things
can
go
wrong,
so
we'd
have
to
run
kubernetes
on
kubernetes.
B
So
we
put
the
containers
in
containers
and
we
get
something
that
looks
kind
of
like
this,
and
the
gist
of
it
is
basically
that
you
have
all
of
kubernetes
running
in
a
container
and
your
nodes
are
containers
instead
of
VMs.
So
we
can
take
that
and
what's
a
little
more
darker
and
darker,
we
can
run
this
in
a
pod,
and
now
we
have
a
job
where
we've
run
the
kubernetes
into
end
tests
and
we
can
conformance
test
kubernetes
entirely
within
our
build
cluster.
B
If
you're
interested
in
more
about
these
things,
there's
going
to
be
a
sick
testing
intro
with
Aaron
and
Steve
here
and
there's
going
to
be
a
sick
testing
deep-dive
with
Cole
and
Kathryn,
and
both
of
these
are
going
again
a
lot
more
details
about
some
of
these
tools,
thanks
so
much
for
coming.
This
is
unbearable.
B
If
you
want
to
contact
us
and
find
out
more,
these
are
the
places
you
can
come,
we're
on
slack
and
sick
testing,
prowl
kind,
which
is
the
container
cluster
thing,
we're
also
in
testing
ops,
where
we're
going.
This
is
fine.
Everything
is
on
fire.
We
have
a
mailing
list
for
sick
testing
and
these
are
the
repos
here.
It's
also
a
weekly
meeting
and
you
can
find
the
slides
of
this
link.
C
B
I
could
take
that
one
where
we're
actually
looking
at
maybe
making
all
the
config
series,
because
CRTs
have
a
lot
of
nice
properties,
but
initially
without
config
config
maps.
I
think
the
biggest
thing
is
that
you're
don't
have
to
really
do
a
lot
of
weave
work
in
your
controller.
Do
this
because
you
can
mount
the
config
map
into
the
pod
and
it
will
be
atomically
updated
by
the
config
map.
Magic
and
kubernetes
makes
it
really
easy
to
configure
roll
out.
B
Some
of
them
we
run
ourselves
if
we
have
resources
for
them.
For
example,
Amazon
now
provides
some
resources
that
we
use
and
Google
does.
In
other
cases,
we
have
some
projects
that
people
can
contribute
and
present
their
own
results
and
the
release
team
will
monitor
those
as
well.
There's
a
test
grid
kept
for
this.
If
you
search
for
that-
and
it
will
tell
you
about
how
you
can
get
involved,
I.
E
B
F
So
if
you
look
at
our
config,
we
have,
we
have
minor
support
for
like
run
this
job
and
then
run
this
other
job,
but
it's
not
really
where
we
want
it
to
be,
and
so
our
plan
is
to
help
out
and
adopt
the
pipeline
C
or
D
at
some
point
to
let
the
pipeline's
here.
Do
you
deal
with
all
the
complexity
of
triggering
a
bunch
of
different
pods
yeah.
G
B
A
problem
there
are
a
few
things
we
need
to
do
mostly
just
to
run
docker
and
docker
at
all,
not
allowing
overlay
file
systems
be
on
top
of
each
other,
but
other
than
that
it
runs
it's
actually
faster
because
we
spend
less
time
creating
clusters.
It's
really
quick
to
boot,
this,
the
pests
that
run
on
GCE.
We
spend
about
20
minutes,
doing
like
up
and
down
of
those
clusters.
We
spend
about
a
minute
or
less
doing
up
and
down
for
this.
B
I
B
Maybe
it
works
pretty
great
it's.
This
is
the
yes.
This
is
fine.
It's
fine,
I
think
the
biggest
concern
you
have
to
remember
there
is
that
today
that
requires
privileged
containers
and
there's
some
considerations
there.
So,
for
example,
when
we
run
our
CI
system,
there's
a
cluster
running
all
of
the
CI
components
and
there's
another
cluster
that
we
schedule
all
the
workloads
to
and
for
some
more
sensitive
workloads
like
the
urban
and
a
security
project
where
they're
working
on
patches
that
uses
an
entirely
separate
cluster
hi.
J
B
B
K
Like
your
talk-
and
it
makes
a
lot
of
sense
for
something
with
the
scale
that
Google
operates
but
I've
been
using
Jenkins
X
and
it
kind
of
does
for
a
small
scale.
It
kind
of
does
a
lot
of
the
same
stuff
and
it
provides
a
lot
of
capabilities
for
me
in
terms
of
orchestration
and
reporting
and
stuff.
So
I
wanted
to
what
are
there
gaps
in
terms
of
if
I
want
to
use.
L
B
M
So
you
have
like
a
lot
of
controller
boxes
on
there
and
they
consume
a
lot
of
different
CR
DS
and
so
I've
written
a
couple
of
controllers
now
and
it
takes
time,
do
you
have
any
tools
or
advice
or
anything
like
that?
You've
learned
from
writing
all
the
different
controllers,
to
like
the
for
the
separation
between
them
or
like
frameworks
that
you've
used
or
just
good
libraries,
or
anything
like
that.
So.
B
Really
early
on
there
wasn't
great
tooling
for
this,
and
we
actually
had
our
own
kubernetes
client
and
our
own
copy
of
the
types
on
our
own
controller
framework.
Don't
do
that!
The
upstream
tooling
is
pretty
great.
Now
I
mean
you
have
to
deal
with
a
little
generated
code,
but
it
will
save
you
from
a
lot
of
work.
We're
moving
everything
towards
that
use,
the
code
generator.