►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Migrating Jenkins to Kubernetes Broke Our Brains - Jonathan Hess & Loren Trout, SAP
Trying to migrate Jenkins to Kubernetes broke our brains. We migrated our Jenkins-based build system from plain VMs + Chef to Kubernetes running on VMs in our SAP corporate datacenter. Here is how we did it so that you can too. We will walk through some of the nitty-gritty details to build a Kubernetes cluster from scratch on virtual machines, walking through our open-source example repo: Provisioning the Cluster: * building a solid cluster with kubeadm * preparation of the underlying OS * how to keep the base machines clean * system and application monitoring Running Jenkins on Kubernetes safely: * our helm charts + shell scripts * sizing the pods for builds * persistent volumes and backups Additionally, we will discuss some of the lessons we discovered along the way: * What problems got solved migrating Jenkins to K8s? * Lessons for other legacy apps
To Learn More: https://sched.co/GrSh
B
B
A
So
before
we
get
into
what
we're
up
to,
let's
learn
a
little
bit
more
about
you,
so
I'm
gonna
ask
for
a
bit
of
audience
participation
here,
if
you're
from
Seattle.
If
this
is
your
hometown,
please
raise
your
hand.
Let's
see.
Oh
thank
you
for
coming
all
right,
which
is
the
best
coffee.
Is
it
Starbucks
or
Pete's
or
intelligentsia,
or
anyone
slate.
None
of
the
above
okay
come
find
me
afterwards.
I
need
to
know
which
one's
the
best
one
all
right
wait
hang
on.
A
We
got
a
couple
more
questions
so
who
okay,
who
is
brand
new
to
kubernetes
this
year,
who's
new,
all
right!
Let's
give
these
people
around
applause.
Thank
you
for
coming.
Welcome
to
the
community
or
making
us
all
stronger,
all
right
who
actually
here
is
sitting
in
this
room
bill,
part
of
kubernetes
anyone
raise
your
hands
all
right.
Look
around
folks
now
keep
those
hands
up!
I
see
you!
Everyone
else
see
these
people,
that's
who
you
need
to
talk
to.
They
have
all
the
answers
all
right.
A
B
A
Led
us
to
replace
everything
that
we're
talking
about
here
today
led
us
down
this
journey
into
kubernetes.
Then
we're
going
to
talk
about
the
three
easy
steps.
It
was
simple
to
get
this
Jenkins
running
on
kubernetes,
obviously,
and
then
what
happened
when
we
actually
got
it
right
working,
but
then
we
needed
to
get
the
developers
to
work
with
it
and
our
early
stages
of
getting
it
smoothly
running
into
production.
So
and
if
we're
all
done
at
the
end,
we'll
leave
some
lessons
for
you
and
maybe
some
time
for
questions.
So
alright.
B
B
Top
of
that,
our
build
scripts
were
nested
within
our
Jenkins
jobs,
not
versions.
We
never
knew
what
had
changed
and
we
couldn't
easily
roll
back
really
hard
to
roll
back.
This
is
the
story
of
our
battle
Jenkins,
so
I'm
gonna
ask
the
audience
how
many
of
you
have
a
battle.
Jenkins
back
in
the
office
raise
your
hands.
B
A
As
we
thought
through
what
it
meant
to
have
a
good
Jenkins,
we
realized
there
really
was
one
important
goal
to
our
effort,
and
that
was
to
make
it
simple
enough
that
our
developers
could
manage
it
because
we're
the
DevOps
guys
right-
and
we
just
didn't-
want
to
be
always
answering
questions
from
our
development
team.
You
know
why
is
this
build
broken?
Why
is
this
thing
down?
Why
can't
I?
Could
you
please
fix
it
and
and
so
forth?
B
Also
important
security
we
needed
to
have
it
run
on
our
corporate
network.
It
needed
to
be
free,
so
we're
gonna
reuse,
the
infrastructure
we
already
have
our
Jenkins
need
to
be
repeatable
and
efficient,
so
we
can
run
more
builds
of
the
same
infrastructure,
also
reprovision
from
scratch.
If
it
were
to
break
it
needed
to
be
fast,
we
wanted
to
get
the
build
times
down
to
30
minutes
from
two
hours
three
hours
to
thirty
minutes,
so
we
picked
Jenkins,
2.0
and
kubernetes
cuz.
It
checked
all
those
boxes
and
because
it's
fashionable
and
because.
A
B
A
But
remember
kubernetes
is
open-source
and
so
is
jenkins,
so
this
is
early
2017.
It's
kubernetes
1.6
that
we're
trying
to
build
this
on
right,
so
we're
a
la
there
are
lots
of
bugs
and
there's
very
little
documentation
right
and
everything
about
kubernetes
today
is
much
better.
Everything
about
Jenkins
today
is
much
better,
which
is
fantastic
because
we
were
just
barely
Jenkins
2.0,
barely
able
to
run
kubernetes.
There
was
no
K
admin
when
I
when
we
started
it
was
too
too
early
for
that.
But
what
we
did
is
we
updated
this
talk.
A
B
So
we
suggest
you
check
out
our
example,
repo
the
scripts
and
our
repo
will
enable
you
to
set
up
a
Jenkins
on
a
kubernetes
cluster
and
just
20.
Easy
steps
includes
our
best
stuff
20.
Easy
steps
includes
everything
we
describe
here
today
and
several
the
brain
breakers.
We
didn't
have
time
to
cover
now
Jonathan.
If
we
had
these
scripts
one
and
a
half
years
ago,
our
brains
would
only
be
half.
A
A
This
is
what
we
wanted
to
build.
It
was
sort
of
we
wanted
to
continue
to
use
those
common
developer
infrastructure
pieces,
so
the
developers
could
keep
following
their
current
workflow,
but
we
wanted
to
take
those
Debian,
VMS,
clean
them
off,
get
a
kubernetes
cluster
running
on
them
and
install
Jenkins
on
that,
and
so
that
is
what
we
did.
B
So
you
recall
those
three
easy
steps
right:
how
hard
could
it
be?
This
is
when
our
brains
really
started
to
feel
the
pain
these.
These
are
our
top
three
brain
breakers
that
popped
up.
As
we
tried
to
build
the
kubernetes
cluster
a
year
and
a
half
ago.
Let's
go
through
these
here's.
What
we,
what
you
noticed,
a
vm,
would
suddenly
become
a
responsive
and
completely
unreachable.
It
couldn't
even
peeing
at
responded
that
the
vm's,
why
are
they
slow?
Why
are
they
hanging
right?
This
happened
to
over
and
over
again
with
no
pattern.
B
A
You
might
have
seen
this
before
alright,
okay,
you
know
where
we're
going
here
right
so
though
they
create
all
these
files
on
vara,
Lib
and
Varley
abuse,
Lee
lands
on
the
root
partition
of
your
VM
and,
if
you're,
using
a
VM,
your
root
partition
is
probably
really
small
and
really
slow,
and
so
it
doesn't
take
long
to
fill
it
up
and
even
if
you
ask
to
for
your
people
to
attach
a
disk
to
it,
so
you
can
have
lots
of
space
to
run
your
builds.
That's
not
I'm
gonna
get
mounted
by
default.
A
You
actually
have
to
do
it
yourself,
so
it's
like
you're
driving
a
truck
and
all
the
freight
is
in
the
cab
with
you
and
the
trailer
is
empty.
So
what
do
you
do?
Well,
you
have
to
mount
and
move.
So.
First
of
all,
you
have
to
make
sure
that
you
mount
that
extra
file
system
that
got
attached
the
big,
fast
file
system,
and
this
shows
you
how
to
how
to
do
that.
The
four
unit
Unix
commands
and
then
you
have
to
get
all
get
it.
Excuse
me.
A
You
have
to
set
up
the
paths
for
docker
and
for
couplets
so
that
they
land
on
that
big
file
system.
Here's
a
bit
snip
of
code
that
shows
how
to
do
it
with
the
sim
link.
I'm
sure
this
isn't
new
to
anybody,
but
that
solved
the
problem
for
us
and
we've
got
the
whole
thing
scripted
out
in
our
repo,
so
go
ahead
and
check
out
if
you're
planning
to
do
this
yourself
save
yourself.
Some
brain
pain
problem
solved.
A
B
Next,
what's
up
with
the
networking
not
connecting
right?
So
at
this
point
we
have
our
disc.
Is
she
sure
it's
sorted
out
capacities
looking
good
and
then
we
ran
against
this
networking
issue?
What
we
saw
was
when
kubernetes
tried
to
connect.
Sorry
we
went
where
we
saw
was
when
kubernetes
was
running.
The
nodes
we
never
show
is
ready
right,
okay,
this
is
this
is
quite
peculiar.
This
is
where
a
VM
could
talk
to
a
VM,
but
a
couplet
cannot
talk
to
a
couplet.
A
pod
cannot
talk
to
a
pod.
Our
viens
were
looking
good.
A
So
it
took
us
a
long
time
to
figure
out
that
this
was
actually
the
problem,
but
it
turns
out
the
solution
is
pretty
simple.
You
just
pick
a
different
IP
address
range
from
a
different
private
IP
network
range
and
then
all
right,
everyone
got
pictures,
and
then
you
pass
it
into
the
coop
admin
and
it
function.
It's
as
simple.
As
that
problem
solved,
the
kubernetes
cluster
can
come
together.
B
A
A
First,
we
had
a
complicated
backup
scheme
involving
init
containers
and
git
repositories,
and
we
could
never
get
that
to
be
really
reliable.
Then
we
tried
a
bunch
of
roll-your-own
contestant
volume,
things
like
rook
or
port
works
or
cluster
FS,
and
at
the
time
remember,
this
is
a
year
and
a
half
ago.
None
of
those
were
reliable,
although
we
understand
they're
getting
better.
So
if
anyone
hears
brook
or
cluster
FS
or
any
of
those.
Thank
you
for
doing
the
work.
It's
valuable.
We
want
to
see
more,
but
here's
what
we
ended
up
doing.
A
We
ended
up
using
network
file,
share
it's
good
old,
40
year
old
technology
to
the
rescue
here.
What
we
did
is
we
treated
a
single
node
in
the
cluster
like
an
old
timey
file,
server
right,
so
it
mounts
a
directory.
It
shares
it
in
NFS.
Then
we
can
mount
that
using
the
standard
kubernetes
stuff.
The
benefit
is,
it
is
super
simple
and
it
is
super
stable.
It's.
A
Yeah
drawback
is
it's
an
old-timey
file
share
it's
not
redundant
and
you
just
have
to
treat
it
as
such,
but
good
news,
hey
guess
what
we
have
40
years
of
experience
running
these
kind
of
file
systems,
so
I
think
we
can
figure
this
out.
So
here's
what
you
do
you
first
step
you
see
on
the
left.
You
pin
your
NFS
server
deployment
to
a
particular
node
in
your
cluster.
The
second
thing
is
you
do,
is
you
create
a
service
and
you
have
to
give
it
a
static?
Cluster
IP
address.
A
A
B
Now
we're
ready
to
move
our
Jenkins
jobs
into
our
new
kubernetes
cluster.
We
have
distilled
our
experience
into
these
two
recommendations
for
you
how
and
why
to
benchmark
your
disk
subsystems
and
how
and
why
your
Jenkins
should
be
a
repeatable
configuration
yep
on
benchmarking.
You
want
to.
If
you
want
fast,
builds,
you're
gonna
need
fast
disks.
There
are
two
important
aspects
for
performance:
throughput
and
AI
ops,
throughputs.
How
many?
B
How
fast
can
you
read
and
write
one
big
blob
of
data,
I,
ops,
I
apps
per
second-
is
how
fast
can
you
read
or
write
many
lots
of
tiny
blobs
of
data.
Your
kubernetes
nodes
must
have
fast
disks
that
are
at
least
as
as
fast
or
faster
than
your
old
build
system.
Benchmarking
is
easy
and
here's
how
no
software
to
install
you
can
use
these
simple
unix
commands
to
benchmark
your
disks.
We
have
this
in
our
example,
repo.
B
A
It's
really
easy,
it's
worth
the
time
just
do
it
alright!
The
next
thing
is:
how
do
you
make
the
Jenkins
configuration
repeatable,
and
we
went
through
a
lot
of
trouble
with
this
dealing
with
bugs
and
earlier
Lee
Jenkins
too.
A
lot
of
those
are
shaken
out,
but
we
want
to
share
two
recommendations
with
you
about
how
you
can
handle
Jenkins
configuration.
Our
first
recommendation
is
this:
just
do
the
global
Jenkins
configuration
through
the
Jenkins
user
interface
right?
A
This
is
where
you're
on
your
laptop
you
log
into
Jenkins,
you
go
to
the
config
screen
you
install,
plug-ins,
etc,
etc,
etc.
All
your
changes
get
written
to
Jenkins
home,
and
then
you
have
a
backup
job
that
can
keep
that
backed
up
and
you
have
a
restored
job
that
can
bring
it
back
when
your
cluster
goes
down
or
if
you
need
to
move
a
reprovision
or
do
whatever
you
need
to
do.
The
good
news
is
in
the
example,
repo.
A
We
have
a
backup
job
and
a
restore
job
baked
into
the
helm
chart,
and
so,
if
you
just
pull
it
down,
you
can
see
how
that
works
adapted
to
your
needs
and
do
whatever
you
want
to
do.
It
is
way
simpler
than
trying
to
figure
out
how
to
write
all
those
groovy
scripts
to
do
what
add
menu
I
can
do
in
just
a
few
minutes.
A
Okay,
the
next
thing
we'd
say
is
run
your
build
configuration
in
Jenkins
files
and
check
those
Jenkins
files
in
with
the
code
repo
that
that
way,
the
developers
really
truly
own
their
own
build
process
right.
If
the
build
breaks
they
can
log
in,
they
can
write
code
in
the
Jenkins
file.
They
can
write
scripts,
they
can
patch
around.
They
can
test
out
different,
build
configurations
on
different
branches,
it's
their
problem
and
they
can
have
the
tools
at
their
disposal
to
just
fix
it.
So.
B
That's
what
we
did
and
then
we
let
the
developers
drive
it.
So,
at
this
point
in
the
journey,
kubernetes
is
running
nicely.
We
have
sample
workloads
going
through
flowing
yep,
we're
benchmarked
our
I/o,
so
we're
performing
we've
migrated
our
pipeline
job
over
and
brought
our
old
bad
Jenkins
down
scrapped
and
now
we're
ready
to
turn
over
to
developers.
And
this
is
the
point
like
on
Friday.
We
sent
out
a
reminder
email
to
all
the
development
team
on
Monday
morning
when
you
come
in
you're
gonna
have
a
shiny,
new,
build
system.
Yes,
so
Johnson's.
A
B
Did
it
we
got
through
it,
we've
got
all
our
goals
met
and
then
I'm
gonna
show
you
what
happened
on
Monday
morning
here,
all
right
there.
It
goes
Oh
all
right
here
we
go
we're
barely
working,
we're
barely
working,
and
if
you
see
in
the
back
there
that
red
shirt,
that's
Jonathan
and
I
back
there
we're
putting
some
chains
on
the
bus
to
get
it
out
of
the
snow.
We're
fixing
the
engine
in
the
front
of
the
bus
there.
B
A
All
right
all
right!
Well,
this
is
when
our
brains
were
really
started
to
break,
because
we
didn't
have
a
fallback
plan.
We
had
to
move
forward
and
solve
these
problems.
So
let's
have
a
look
at
the
top
three
brain
breakers.
That
happened
once
the
developers
actually
started
using
this
for
real
first
intermittently.
Failing
builds
we're
good,
alright,.
B
B
Of
memory
people
raise
their
hands.
I
wish
you
guys
were
here
a
year
and
half
ago
could
help
alright.
What
we
saw
was
unresponsive
nodes
intermittently.
Failing,
builds
pods
randomly
turn
terminating
with
room
airs
in
the
logs
and
nodes
from
memory,
starved.
What
we
first
tried
is
adding
more
nodes
to
the
help,
nope
wrong,
add
more
memory,
surely,
and
more
memories
gonna
help
this
problem.
No,
how
about
throttle
the
builds
there
too
many
bills
running
at
once,
nope.
A
A
So
if,
if
you
don't
tell
Java
how
much
memory
it
has
it's
going
to
use
up
every
bit
byte
of
memory
in
your
entire
virtual
machine,
it
completely
ignores
whatever
limits
you
put
on
the
container,
and
so,
if
you've
done
your
kubernetes
right,
you've
set
limits
on
your
pods
as
soon
as
Java
crosses
over
that
8
gigabyte
limit
by
out
of
memory,
all
right,
so
yeah
Java
has
no
respect.
So
what
do
you
do?
It's
disrespectful,
it's
so
disrespectful.
A
A
Ok,
second
always
set
the
explicitly
set
the
Java
memory
limit
on
the
Java
process,
always
because
that's
the
only
way
you
can
be
sure
it's
going
to
stay
within
the
memory
limits
of
the
container
I
know
that
the
Java
folks
have
worked
on
all
kinds
of
extra
economics
and
check
the
C
group
and
blah
blah
blah
I.
Don't
trust
any
of
that
just
set
the
memory
level,
and
that
also
means
if
you're
running,
maven
or
aunt
or
Gradle.
You
have
to
go
dig
through
the
documentation
and
find
those
environment
variables.
B
A
Well,
the
real
problem
is
this:
when
a
docker
container
exits,
it
leaves
some
stuff
behind.
It
drops
some
berries
on
those
partitions
right,
and
that
includes
volumes
that
maybe
someone
referenced.
It
includes
images
that
you
downloaded.
It
includes
container
layers,
especially
if
that
docker
image
crashed
and
so
no
matter
how
big
your
partition
is.
Eventually,
these
things
are
gonna,
build
up
and
build
up
and
build
up
over
time
until
your
partition
is
full.
A
B
Someone
said
it
right
all
right,
so
some
solutions
are
what
we
did
ended
up
doing
a
year
and
half
ago.
Is
we
installed
a
cron
job
to
clean
hourly,
so
we've
created
a
script
that
looks
for
removed,
dangling
docker
berries,
that's
right!
That's
scripts
in
our
repo,
you
guys
can
use
it
to
also.
We
started
monitoring
our
node
partition
usage
with
we're
using
Prometheus
and
grow
fauna
and
we
get
alerts
when
things
are
getting
full.
B
So
that's
that's
a
big
help
and
there's
something
out
there
called
couplet
GC
why
you
should
maybe
have
a
look
at
that,
give
it
a
try.
Yeah
and
some
of
the
lessons
learned.
Are
you
got
dangling
docker
berries
pruning
them
because
they
will
grow
in
your
partitions
and
I
heard
it
can
be
quite
painful
at
times,
yeah.
A
Now
the
builds
are
stable,
but
we're
getting
really
long
waits
for
our
Jenkins
workers
and
we
run
during
what
we
can
do
about
that.
Sometimes
it
takes
like
four
hours
for
a
bill
to
start
because
the
workers
are
all
used.
So
we
said
maybe
we'll
try
a
couple
of
things.
We
will
try
throttling
back
the
number
of
builds
that
run
that
wasn't
great.
We
tried
to
get
the
builds
to
run
faster
that
didn't
solve
the
problem
really
either.
So
what
do
we
do?
Well.
B
We
have
this
thing
called
pod
Tetris
right,
so
this
is
how
to
lose
a
pod
Tetris.
If
you
notice,
we
have
a
24
node
cluster
here
and
we
have
one
size
pod
for
every
step.
Every
stage
of
the
build.
That's,
not
a
good
idea.
If
you
look
at
our
pod
count
on
the
top
left,
you'll
see,
we
only
have
23
pods
running
and
our
cluster
is
100%
used.
There's
no
more!
No
more
pods
can
be
scheduled
on
there.
So
this
this
may
be
only
two
about
two
developer
pipelines
running
every
all.
B
The
other
bills
will
be
queued
up,
so
we're
gonna
show
you
how
to
win
it.
Pod
detrás
right
here,
okay,
so
this
is.
If
you
look
at
this
picture,
you
still
have
23
pods
running
cluster
II
percent
uses
only
it's
down
to
30%.
So
what
we
ended
up
doing
I'll
tell
you,
we
profiled
our
build
stages.
We
watched
the
stages
run
and
identified
how
much
CPU
each
stage
need
how
much
memory,
and
we
saw
that
some
stages
require
very
little
resources
and
others
required
more.
B
So
we
use
the
kubernetes
jenkins
plugin
to
define
three
pod
sizes,
a
tiny
pod
for
checking
out
repos
or
slack
notifications,
a
small
pod
for
build
stages
like
deploy,
sonar
and
unit
tests,
and
then
we
have
some
large
pauses
for
some
intensive
and
in
testing.
So
we
declare
these
these
custom
pod
sizes
in
the
Jenkins
file
in
different
build
stages.
The
results
were
profound.
We
saw
a
5
up
to
a
500%
increase.
In
our
build
rates
per
hour.
B
Additionally,
if
you're
running
in
a
managed
cloud,
this
could
save
you
a
lot
of
money
a
year
too,
because
you
don't
need
so
many
VMs
in
a
managed
cloud.
You
might
be
getting
charged
where
every
milli
core
you
use
or
every
time
you
use
more
ram.
So
this
could
save
you
money
as
well.
You're
gonna
get
more
yield
out
of
your
hardware.
This
is
a
big
win
for
us.
This
may
like
Tetris,
so
we
wanted
to
show
that
yeah.
A
It's
worth
it
to
try
and
figure
this
out.
So
finally,
we
have
Jenkins
all
working.
We
got
those
short
builds
they
down
from
two
hours
to
30
minutes.
We've
eliminated
our
build
queues,
developers,
get
feedback
right
away.
We
can
recover
from
a
broken
cluster
within
a
day.
The
developers
are
starting
to
actually
trust
the
build
results
and
we
are
free
to
work
on
other
things.
A
B
Was
a
big
takeaway
for
us
that
we
had
the
same
exact
infrastructure.
We
started
with
the
battle
jenk,
it's
the
same
VMs
right
and
we
got
our
builds
faster.
We
were
able
to
put
more
builds
through
it.
We
and
it
was
more
reliable.
Repot
we
reliable
repeatable.
All
the
good
vibes
is
because
we
moved
our
Jenkins
into
kubernetes.
We
could
run
everything
way
more
efficient.
We
could
deal
the
hardware,
it
just
wasn't
worth
it.
In
the
end,
it.
A
Was
worth
it
yes,
but
our
brains,
oh
they're,
definitely
broken,
definitely
broken
all
right,
so
let's
bring
this
home.
So
if
you
want
to
do
this
too,
these
are
the
four
things
you
need
to
do.
Number
one
benchmark.
Your
I/o
performance
make
sure
that
your
disks
on
your
new
system
are
gonna,
be
faster
than
the
disks
on
your
old
system.
Okay,
second
Jenkins
files
for
builds,
get
your
developers
to
learn
how
to
manage
it
themselves.
That's
what
they
need.
It'll
that
way.
Jenkins
becomes
everybody's
problem
and
don't
bother
scripting
the
global
config.
A
It's
not
worth
your
time
and
it'll
slow.
You
down.
Third
thing
play
pod
Tetris.
This
is
worth
your
time,
get
more
more
builds
through
the
same
hardware.
That's
what
kubernetes
is
for
and
second
forth,
don't
try
and
build
kubernetes
yourself,
there's
so
many
great
options
out
there,
including
one
from
sa
P
that
I
get
to
tell
you
about
now.
Thank
you
for
your
patience.
It's
called
gardener,
so
here's
gardener
ready
a
gardener
is
an
open
source,
kubernetes
management
tool
from
sa
P.
A
So
if
you
are
running
kubernetes
in
your
organization
and
you
need
to
supply
lots
and
lots
and
lots
and
lots
of
kubernetes
clusters
to
different
teams,
this
is
a
good
solution
for
you.
It
gives
you
the
ability,
a
Ford
for
all
your
kubernetes
clusters.
It
gives
you
user
access
controls,
so
some
teams
can
get
on
some
clusters
and
other
teams
can
get
on
other
clusters
and
it
works
on
Azure,
Google
cloud
AWS
and
OpenStack
and
I
think
Alibaba,
and
a
couple
of
others
are
coming
soon.
A
A
B
A
And
so,
let's
see
stand
up
if
you're
from
sa
P,
you
want
to
wait
wave
your
hands
at
everybody.
We
got
a
couple
of
people
from
sa
P
here
there
they
are.
If
you
want
to
talk
to
these
folks,
please
go
down
to
the
booth,
we're
here,
we're
friendly.
We
like
you
and
s,
AP
source,
of
course,
a
ton
of
open
source
software
and
we'd
love
to
chat
with
you
about
that.
So
thank
you
all
for
coming.
A
A
A
A
C
A
Why
didn't
we
scale
it
up
with
our
dev
teams
a
little
at
a
time?
So
we
did.
We
just
had
a
couple
of
devs
with
really
small
new
projects
and
one
big
old
project
and
the
big
old
project
was
the
real
reason
we
were
doing
this
and
the
small
new
projects
we'd
moved
over
and
they
worked
great.
It
was
no
problem
at
all,
but
then
we
got
this
big
old
project
over
with
all
these
unit
tests
and
system
tests
and
that's
what
really
broke
it.
C
B
A
So
currently
installed
Griffon
and
Prometheus,
and
then
we
brought
up
the
Jenkins
pipeline
in
one
browser
window
and
the
Prometheus
pods
in
the
other
browser
window,
and
then
we
correlated
which
stages
were
running
in
which
pods
and
how
much
memory
and
disk
they
were
using
and
that's
how
he
did
it.
It
was
painless.