►
From YouTube: Booting 5 K8s Clusters on Every Git Push: How Linkerd Leveled Up Its CI - Andrew Seigner, Buoyant
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Booting 5 K8s Clusters on Every Git Push: How Linkerd Leveled Up Its CI - Andrew Seigner, Buoyant
In mid-2019, the Linkerd project's CI took 45 minutes, all tests were serialized on a single Kubernetes cluster, and multi-hour backups were common. A migration onto one-off Kubernetes in Docker (kind) clusters got CI below 10 minutes, and made it parallelizable. This talk will detail Linkerd's CI journey from a single, persistent Kubernetes cluster to theoretically unlimited one-off kind clusters. This journey includes a few detours on what patterns and tools worked well (and not-so-well) for Linkerd's use case. Attendees can expect a live demonstration of Linkerd's CI workflow, including automated cluster setup, test, and teardown. Folks should walk away from this talk with a grounded, end-user story of how to efficiently test their Kubernetes applications in CI.
https://sched.co/Zeng
A
A
So,
just
a
quick
agenda:
what
we're
going
to
cover
I'm
going
to
touch
initially
just
on
linker
d,
it's
kind
of
an
implementation
detail
of
this
talk,
this
talk's
really
about
evolving
a
build
ci
system,
but
it
helps
to
contextualize
it
by
speaking
to
like
what
were
our
specific
requirements
being
like
an
open
source
project
with
a
relatively
small
team.
It
helps
like
give
some
context
on
why
we
made
the
technology
decisions
that
we
did.
Then
we
get
to
kind
of
the
core
of
the
talk.
A
Take
one
take
two
and
take
three
where
we
go
through
like
three
iterations
of
our
build
ci
system,
we're
gonna
start
with
kind
of
where
we
were
a
year
year
and
a
half
ago.
It
wasn't
a
great
place
and
then
we'll
go
through
like
requirements
gathering
to
rebuild
the
system.
What
tech
we
evaluated
and
then
we'll
look
at
kind
of
our
second
and
third
generation
of
of
ci
and,
as
I
alluded
to
earlier,
the
the
third
generation
is
something
we
just
got
out
and
it
happened
after
this
talk
was
submitted.
A
A
We
have
static
tests
like
linting,
we
have
unit
tests
and
we
have
a
docker
build
and
integration
test
on
kubernetes
and
again,
that's
across
like
testing.
Like
javascript
go
and
and
rust
all
at
once,
we're
going
to
mostly
focus
on
that
box
in
the
lower
left
hand
corner
where
we're
doing
a
docker
build
and
then
we're
running
integration
tests
on
a
live
kubernetes
cluster.
A
So,
to
kind
of
go
back
a
year
year
and
a
half
ago
where
that
docker
building
integration
test
was,
we
were
running
our
whole
ci
system
on
on
travis.
We
were
doing
our
docker
builds
on
the
travis
machine.
Then
we
were
docker
pushing
up
to
gcri.gcr.io
google
container
registry,
and
then
we
were
running
our
integration
test
on
a
gke
cluster.
It
was
like
a
three
node
cluster
and
on
that
cluster
we
would
install
linker
d.
We
would
run
a
bunch
of
integration
tests
and
then
we
would
uninstall
3d
pretty
quickly.
A
We
realized
we
needed
additional
tests
and
we
started
running
helm
tests
as
well.
So
we
install
it
with
helm,
run
our
integration
test,
uninstall
it
with
helm,
and
then
we
found
we
also
needed
to
test
our
upgrade
path.
So
we
installed
an
old
version
of
linker
d.
Then
we
upgrade
it
and
we
run
our
integration
tests.
Then
we
not
install
you
can
see
how
quickly
this
is
starting
to
back
up.
A
So
now
imagine
like
we
want
to
run
five
or
eight
of
these
kind
of
integration
tests
and
we're
doing
it
all
on
one
kubernetes
cluster,
because
we're
a
small
open
source
project.
We
can't
just
spin
up
a
whole
bunch
of
kubernetes
clusters
like
that
and
then
couple
that
with
like.
We
have
lots
of
contributors
coming
into
the
project,
and
so
it
wasn't
uncommon
for
your
pr
to
take
an
hour
or
two
for
you
to
get
through
ci,
that's
a
really
bad
place
because
everybody
was
sharing
this
one
singleton
cluster.
A
So
at
this
point
we
kind
of
took
the
nuclear
option
and
we
pulled
our
integration
tests
out
of
the
main
like
pr
flow,
we
were
still
running
them
when
we
were
merging
into
the
main
branch,
but
of
course,
as
soon
as
you
do,
that
you're
going
to
have
prs
that
are
passing
unit
tests
and
like
static
tests
and
as
soon
as
they're,
they're
merged
they're,
going
to
run
the
integration
test
and
fail.
So
now
the
main
branch
is
failing,
often
because
we're
not
running
our
integration
tests
enough.
A
A
I
should
call
out
on
on
that
slide.
You'll
notice,
like
git,
commit
links
on
the
bottom
you'll,
see
that
throughout
the
talk
and
so
note
that
everything
we're
talking
about
was
all
done
in
the
open,
so
you're
totally
welcome
to
like
click
through,
and
if
you
want
to
see
like
all
of
the
interesting
details
that
I'm
totally
glossing
over
in
this
talk,
they're
all
available
to
check
out.
A
So
we
really
took
a
step
back
and
we
thought
like
what
are
our
requirements
and
we
didn't
start
with
like
we
need
a
new
ci
system.
We
started
with
like
what
do
we
need?
What
do
we
care
about
and
what
kind
of
pain
points
are
we
thinking
about?
And
so
we
kind
of
came
up
with
this
prioritized
list.
We
thought
okay.
A
We
wanted
a
ui
to
like
browse
all
of
the
tests
that
are
running
that
seems
kind
of
a
given
from
oci
systems,
but
we
were
considering
like
anything
like.
Could
we
have
like
offline
jobs
that
like
kick
off
his
scripts,
or
could
we
have
things
that,
like
comment
back
on
pr's
with
just
like
a
thumbs
up
or
a
thumbs
down,
but
we
knew
that
that
was
important
to
be
able
to
like
send
a
link
to
somebody
and
say
hey.
I
need
you
to
look
at
this.
You
know
and
see.
A
A
Maybe
forks
of
the
repo-
and
we
didn't
want
totally
untrusted
code
running
on
like
hardware
that
we
were
paying
for
and
we
didn't
want
to
have
to
deal
with
like
bots,
where
we
like,
approve
or
disapprove
like,
or
we
have
to
like
spot
check
a
pr
before
integration
tests
get
run.
So
we
really
wanted
a
way
to
like
spin
up
a
totally
as
safe
as
possible
environment
and
then
delete
the
whole
thing
when
it's
all
done.
A
We
also
wanted
it
to
be
fast
again,
like
things
were
taking
hours,
so
that
meant,
you
know
not
pushing
docker
images
across
the
internet
for
every
pr
it
meant
doing
incremental
rebuilds
the
difference
between
a
cash
docker
bill
that
takes
five
minutes
and
a
docker
build
from
scratch.
That
takes
an
hour
it's
a
lot
and
we
were
interested
in
a
remote
build.
So
that
meant,
you
know,
can
I
run
a
command
on
my
laptop
and
have
all
the
heavy
lifting
be
done
on
a
remote
vm?
A
A
A
We
spent
a
lot
of
time
working
with
prow,
for
those
of
you
not
familiar
proud
is
like
the
build
ci
system
that
the
kubernetes
community
uses
and
that's
what
really
drew
us
in
in
the
first
place.
We
really
thought
if
it's
good
enough
for
kubernetes,
it's
got
to
be
good
enough
for
us
and
we
had
a
fully
working
end-to-end
system.
You
know
it
was
open
source,
huge
plus
right.
A
So
with
that
in
mind,
spoiler,
here's
the
three
pieces
of
tech
we
settled
on
we
selected
kind,
github
actions
and
packet,
and
I'm
going
to
go
through
each
one
of
those
and
kind
of
talk
about
why
we
selected
them.
So
first
off
is
kind.
This
was
the
first
one
we
zeroed
in
on
and
kind
of
decided
we
wanted
to
use
for
those
not
familiar.
It
stands
for
kubernetes
in
docker.
A
It
allows
you
to
spin
up
a
kubernetes
cluster
anywhere
you're
running
docker,
and
it
runs
the
whole
cluster
in
a
single
docker
container,
similar
to
pro
it's
used
by
the
kubernetes
community.
To
test
kubernetes,
so
the
fact
that
it's
used
by
kubernetes
in
ci
made
it
a
very
compelling
case
right
away,
it's
very
fast
to
like
spin
up
a
cluster
and
and
delete
it
later,
and
it's
really
good
for
local
development.
A
It
also
meant
that
wherever
we
were
doing,
our
docker
builds
was
where
we
could
run
our
kubernetes
cluster.
So
that
meant
no
more
pushing
docker
images
across
the
internet
and
bonus.
It
was
open
source.
So
so
that
was
kind
of
the
first
piece
of
tech
we
we
zeroed
in
on
and
then
we
started
looking
at
like
what
things
can
we
build
around
this
one
piece
of
technology,
the
next
one's
a
bit
of
a
curveball?
A
We,
you
know
we
ended
up
using
a
packet
for
bare
metal
servers
to
do
like
very,
like
fast,
build
and
like
docker
caching
and
running
all
of
our
kind
clusters,
and
the
reason
we
selected
them
was
because
they
have
a
partnership
with
the
cncf
that
provides
free,
compute
to
open
source
cncf
projects.
That's
huge
so
like
shout
out
to
them
like
to
be
able
to
get
like
a
large
bare
metal
machine
like
that
and
to
make
our
you
know.
A
First,
it
was
integrated
it's
already
in
github,
and
that
means
one
less
integration
point
that
we
had
to
be
concerned
with
it
had
a
feature
called
matrix
builds
which
allows
you
to
run
a
bunch
of
similar
builds
based
on
a
parameter.
So,
for
example,
if
you're
booting
eight
kubernetes
clusters
in
a
slightly
different
way,
you
can
do
that
with
a
matrix,
build.
It
has
really
nice
control
over
how
you
define
the
dependencies
between
your
tasks.
A
Maybe
I
want
to
boot
up
a
kubernetes
cluster
and
do
a
docker
build
and
then,
when
both
of
those
are
done,
I
want
to
run
an
integration
test
and
then
do
a
release
and
their
kind
of
config
makes
it
straightforward
to
to
do
that
and
then
it
wasn't
open
source,
but
it
was
free
for
open
source
projects.
So
you
know
shout
out
for
that.
That
kind
of
sealed
it
for
us.
A
So
then
we
we
kind
of
move
forward
with
with
a
full
setup
with
those
three
tech
choices,
and
you
can
see
the
lower
left
has
been
replaced
with
using
kind
packet
and
github
action.
The
whole
thing
is
github
actions
now
note
that,
even
though
we're
doing
docker
builds
and
our
kind
cluster
work
on
packet,
we're
calling
it
out
only
for
non-forks,
so
because
we
were
still
running
hardware
that
we
were
responsible
for.
We
didn't
want
untrusted
code
running
there,
so
that
meant
for
the
maintainers
internally.
A
We
had
this
really
nice
environment,
but
if
you
were
an
external
developer
just
coming
into
the
project-
and
you
submitted
a
pr
through
a
fork
of
yours,
you
would
be
running
all
of
this
work
on
the
github
actions
boxes.
You
wouldn't
get
the
advantage
of
docker
caching,
it
would
be
much
slower
and
that
was
kind
of
a
bummer
that
kind
of
made
like
external
developers,
like
almost
second
class
citizens
when
you're
coming
in,
and
it's
your
first
experience.
A
We
want
it
to
be
really
positive
and
when
the
ci
turnaround
is
that
slow,
that's
that's
kind
of
a
bummer
so
more
on
that
later,
and
just
to
give
kind
of
illustrate
what
this
this
matrix
configuration
is
these.
This
is
a
list
of
the
eight
kubernetes
clusters
that
we
boot
up.
I'm
not
going
to
go
through
all
eight,
but
just
as
an
example,
you'll
see
that
first
one
says
cluster
domain.
A
You
know,
as
some
of
you
know
like
when
you
deploy
kubernetes
cluster,
all
of
your
services
by
default
they're
named
like
something.cluster.local
and
that's
kind
of
the
default
cluster
domain
inside
kubernetes,
and
you
can
configure
that
and
change
it,
and
it's
really
important
that
linkerd
behaves
well
if
you've
changed
that
default.
Cluster
domain
we've
definitely
committed
changes
that
broke
this
behavior.
A
So
next
we're
gonna
jump
into
a
demo
where
you
may
have
noticed
in
in
this
previous
slide,
where
everything's
running
in
github
actions,
but
the
heavy
lifting
is
being
done
in
packet
and
we
figured
out
a
way
to
do
that
by
using
a
docker,
ssh
environment.
Variable
and
some
some
fancy
work
with
conf
how
we
were
using
kind
and
so
we're
going
to
demonstrate
building
docker
images,
creating
a
kubernetes
cluster
and
interacting
with
the
kubernetes
cluster.
A
A
A
I
can't
connect
to
anything,
but
when
I
set
a
dockeros
environment
variable
to
ssh
and
then
the
name
of
the
machine
I'm
connecting
to,
I
can
now
connect
to
docker
and
use
docker
on
that
remote
machine.
Now,
I'm
going
to
create
the
kind
cluster
we're
going
to
call
the
cluster
kubecon
demo
and
here's
the
kind
cluster
booting
up
now.
All
of
this
work
is
being
done
on
the
docker
demon
on
the
remote
vm
and
the
kind
cluster
is
booting
up
over
there.
So
here
it
is.
A
It's
coming
up
now
note
that
I
still
can't
use
cube
cuddle
to
connect
to
that
cluster.
Just
yet
because
again,
the
machine,
the
kubernetes
cluster
is
running
remotely,
but
what
I
can
do
is
I
can
look
at
the
kube
cuddle
config
note,
which
port
it's
running
on
on
the
remote
vm
grab
that
port
and
then
I'm
going
to
forward
that
port
from
my
laptop
to
the
remote
vm
and
if
all
work
correctly,
I
can
use
coop
cuddle
to
connect
to
that
remote
kind
cluster.
Just
as
I
would
if
it
was
running
locally.
A
Okay,
so
we
just
saw
like
we
were
able
to
spin
up
a
kubernetes
cluster
somewhere
else
and
run
it.
This
felt
a
little
weird,
maybe
a
little
hacky,
it's
just
some
kind
of
you
know,
bashful
on
on.
You
know
modifying
kubernetes
configs
on
the
fly
so
before
we
we
went
live
with
this.
I
checked
in
in
the
kind
of
channel
in
kubernetes
slack
just
to
kind
of
see.
A
So
next,
I'm
going
to
take
you
through
one
more
demo,
where
it
kind
of
shows
everything
working
together,
we're
actually
going
to
go
from
git
push
to
eight
kubernetes
clusters.
So
here
we
go
okay,
so
we're
going
to
demo
the
latest
version
of
our
build
ci
system.
So
here
we've
got
the
linker
d.
Repo
we've
got
a
pr,
that's
reporting
that
it's
breaking
in
ci,
so
we're
going
to
dig
in
see
what's
wrong
and
then
try
to
make
a
fix.
So
here
we
see
that
one
of
the
integration
tests
has
failed.
A
These
are
a
kind
integration
test
that
we've
been
talking
about,
so
we
dig
in
we
notice
it's
the
upgrade
test.
It's
the
test
that,
where
we
install
the
old
version
of
linker
d
on
a
kubernetes
cluster,
then
try
to
run
an
upgrade
and,
as
we
look
through
the
log,
we
see
that
some
command
called
upgrade
broken
is
being
run
that
that
seems
wrong.
So
we'll
go
back
into
our.
Our
code
will
find
where
that
incorrect
command
is
being
run
fix.
A
It
up
should
just
be
called
linker
d
upgrade
and
we'll
verify
the
change
that
we're
making
and
then
commit
the
change,
push
it
up
to
github.
This
is
where
going
from
a
git
push
to
booting
eight
kubernetes
clusters
comes
from
so
now
that
we've
pushed
up
the
change.
We
should
be
able
to
go
back
into
github
click,
our
github
actions,
tab
the
checks,
tab
and
note
that
the
new
belt
has
already
kicked
off.
A
You'll,
see
some
of
the
more
simple
tests
kicking
off
right
away
like
the
unit
tests,
the
linting
static
checks
all
that
stuff
and
then
the
docker
build
is,
is
already
going
in
preparation
for
our
our
more
complicated
integration
tests.
I'm
gonna
speed
up
this
whole
section
of
running
the
integration
test
because
it
takes
like
10
minutes.
A
As
the
tests
are
wrapping
up,
we've
verified
that
all
the
things
on
the
left
are
showing
up
as
green,
and
then
we
should
be
able
to
jump
back
to
our
our
pull
request
and
confirm
that
the
build
has
been
fixed.
So,
just
in
a
matter
of
a
few
minutes,
we're
able
to
make
a
change
and
boot
up
all
of
those
clusters
test
it
across
all
those
different
environments
and
work
all
right.
A
I
know
mini
cube,
k3d,
there's
a
number
of
flavors
we've
just
found
it
super
useful
and
not
just
the
tech
itself,
but
the
community,
having
like
a
friendly,
responsive
place
where
you
can
ask
questions,
makes
a
huge
bit
makes
a
huge
difference.
We
try
to
emulate
that
working
on
the
linker
d
project
too.
We
found
it.
It
really
encourages
a
healthy
sense
of
community
cache.
Your
builds,
especially
your
docker
builds
the
docker
host
ssh
pattern
is
really
nice.
I
haven't
run
docker
on
my
laptop
in
in
many
months.
A
It's
great,
so
I
definitely
encourage
you
to
check
that
out.
Docker,
build
x
is
really
nice
for
caching
and
now
we're
starting
to
use
it
to
do
multi-arc
build
so
we're,
I
believe,
either
today
or
this
week,
linkery
will
have
arm,
builds
as
it
goes
out
through
our
normal
release,
workflow
and
a
big
shout
out
to
packing
and
github
actions
for
supporting
open
source
and
and
allowing
us
to
to
use
their
infrastructure,
and
one
more
thing
something
I'd
really
liked
about.
A
Prow
was
that
it
provided
this
really
nice
heat
map
of
all
of
your
build
statuses,
and
just
in
just
recently
one
of
our
linker
d
maintainers
alejandro
built
this
dashboard
in
a
similar
vein.
That
kind
of
gives
us
like
a
timeline
view
of
like
how
are
we
doing
over
time
with
our
build
help
so
check
this
one
out
as
well.
The
links
are
all
in
the
slides,
so
shameless
plug
for
linker
d.
Please
come
check
it
out.
We
use
it
a
lot.
A
A
Hi
folks,
thanks
for
coming
to
the
talk,
I've
been
answering
a
few
of
your
questions
in
the
q,
a
I
think,
there's
a
couple
more
so
I'm
happy
to
answer
stuff
live
as
well.
So,
let's
see,
I
see
one
from
joseph
that
asks
about
how
the
cache
with
docker
build
x
works.
I
didn't
actually
implement
that
which
some
other
members
of
our
team
just
did
in
the
last
month
or
two.
A
But
if
you
go
to
the
linker
d2
repo
and
I
think
there's
some
links
to
it
even
in
these
slides,
but
you
just
google
for
linker
d2
github,
all
of
our
github
actions.
Configs,
are
you
know,
they're
all
in
the
repo
they're,
all
open
source.
You
can
watch
them
running
like
right
now,
so
I
definitely
encourage
you
to
just
like
poke
through
and
see
how
those
jobs
are
are
happening.
It
should
be
all
right
there
anya
asks.
Did
you
ever
notice
issues
that
slip
through
kind
only
showed
up
in
real
clusters?
A
A
So
there's,
I
think,
there's
definitely
a
class
of
like
timing
and
scaling
issues
that
can
happen
on
a
kubernetes
cluster.
That's
maybe
more
fully
loaded
or
you
know,
is
running
in
in
a
different
environment.
So
to
that
end,
in
addition
to
running
all
the
kind
jobs
for
every
like
merge
or
every
pr
at
at
time
of
merge
into
the
main
branch,
I
believe
we
still
run
our
integration
tests
on
a
live
gke
server.
Sometimes
that
can
catch
other
classes
of
issues.
A
It's
on
a
roadmap
where
we
would
like
to
be
able
to
run
like
our
integration
tests
on
gke
on
on
aks
on
eks.
You
know
all
kind
of
all
of
the
different
cloud
providers
digitalocean
all
of
them-
and
I
we've
we've-
definitely
had
issues
that
have
cropped
up
that
only
occur
on
like
one
cloud
provider
and
then
obviously
things
and
that
catches
one
class
dishes.
A
Then
obviously
things
come
up
in
production
that
that
we
didn't
anticipate
and
when
we
see
that
we
try
to
update
our
integration
tests
to
you
know
catch
those
type
of
scenarios
so
yeah.
It's
always
there's
never
like
one
layer
that,
like
catches,
everything
right.
It's
it's
all
about
having
like
many
layers
that
that,
hopefully
they
each
do
their
job
on
catching
things.