►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
Write Less Code, Use More Tools: How we Orchestrate 7M+ Builds/Month with Kubernetes + Nomad - Danielle Tomlinson, CircleCI (Any Skill Level)
When we redesigned CircleCI from the ground-up in 2016 and 2017, we knew we needed a proper scheduling solution to run jobs efficiently across our resources. Proper isolation of CircleCI's infrastructure and individual customer builds was one of the core requirements of this undertaking: for security, scalability, and reliability, we had to make sure to keep our system separate from our users' jobs. In this talk, I'll share why we chose Nomad and Kubernetes to solve these issues over other tools, present the issues and obstacles we solved, and talk about how this setup is running for us in production ~10 months after GA.
About Danielle
Danielle is a Staff Software Engineer at CircleCI where she works on build infrastructure and operator tooling, ensuring fast and stable services for 7m builds/month. Before she started working on cloud infrastructure, she was an iOS developer for 8 years. She is also a maintainer of CocoaPods, a dependency manager for Mac and iOS developers.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
I'm
gonna
be
talking
about
how
we
use
kubernetes
and
nomad
at
Circle
CI
and
how
we
run
one
inside
of
the
other
and
a
little
bit
of
why
we
do
that.
So,
let's
get
started
so
Who
am
I.
I'm
Dani
I
live
in
Berlin
and
I'm,
a
staff
software
engineer
on
the
build
execution
team
at
cycle,
CI
I'm,
also
a
maintainer
of
cocoapods,
a
dependency
manager
from
Mac
and
iOS
applications
and
fast
lane,
which
is
build
and
release
tooling
for
mobile
engineers.
A
So,
almost
a
year
ago
we
launched
the
current
generation
of
the
circle.
Ci
infrastructure
aptly
named
2.0
and
it
was
a
huge
amount
of
work
that
involved
everyone
inside
the
company.
By
the
time
it
was
released,
which
might
sound
a
little
bit
strange
when
we're
a
CI
NCD
company
that
you
know,
we
would
take
a
year
to
go
away
and
build
the
thing
before
we
shipped
it
to
anybody.
But
why
did
we
do
that?
A
Circle
C
I
was
founded
in
2011,
and
this
was
a
very
different
time
for
many
software
companies.
Many
places
were
not
even
starting
to
do
CI
NCD
yet,
and
we
didn't
even
have
docker
yet,
and
so
many
teams
deployed
a
mais
over
running
configuration
management
tools
against
hosts
in
production,
just
sort
of
like
git
pull
and
bundle
exec
rakes
of
hopefully
nothing
that
sad,
but
we've
all
been
there.
A
The
platform
was
originally
designed
around
being
fairly
opinionated
about
your
CI
and
CD
process
should
work
with
explicit
sort
of
demarcations
for
when
you
build
your
testing
you
to
playing-
and
this
was
great
when
we
were
all
writing
like
these
huge
monolithic
rails-
apps
with
relatively
simple
deployment
type
lines,
but
in
the
time
how?
Since
then,
however,
the
industry
has
changed
a
lot.
A
Docker
was
released
in
2013
and
since
then
it's
had
an
incredibly
broad
impact
on
our
Tao
teams
of
building
and
releasing
software.
We
could
see
where
our
own
and
was
the
industry
requirements
around
CI
and
CD
were
going,
but
our
existing
infrastructure
was
rapidly
becoming
a
problem
and
we
couldn't
see
an
incremental
path
to
what
servicing
them,
and
so
we
eventually
decided
that
our
best
option
was
to
tear
everything
down
andrey
platform,
our
core
infrastructure,
to
allow
us
to
service.
These
ever-changing
needs.
A
Reap
platforming
is
terrifying,
like
it's
a
lot
of
work,
you're
taking
an
existing
thing,
that's
working
reasonably
well
and
you're,
trying
to
rebuild
a
better
version
of
it,
and
you
have
to
do
all
of
this
work
without
the
constant
guiding
light
of
all
your
customers.
Feedback
and
our
customers
were
also
scared
because
it
looked
like
we
weren't
really
doing
anything
for
a
year,
but
we
had
the
advantage
of
existing
domain
knowledge
and
understanding
of
what
we
needed
from
our
infrastructure.
A
Running
other
people's
code
on
your
infrastructure
and
on
the
same
servers
as
other
people's
code
is
a
terrifying
thought
like
raise
your
hands.
If
this
is
something
you
would
willingly
want
to
do
on
a
daily
basis,
there's
not
even
a
single
brave
soul
out
there.
That's
wonderful-
and
this
is
a
good
reason,
and
so
because
of
this,
we
have
to
bake
in
strong
isolation
to
everything
we
build
and
we
have
to
do
it
from
the
beginning
and
security
is
not
something
you
can
just
think
about.
A
After
the
fact,
when
you're
running,
like
arbitrary
things
on
your
environment
and
like
there
are
a
few
ways,
we
could
do
isolation
in
a
nobly,
Safeway
and
the
one
we
picked
for
sake.
I.
Wonder,
though,
was
Alex
see.
As
you
can
see,
this
is
a
perfect
representation
of
a
sandbox
as
just
everything
flaps
around
everywhere.
A
We
also
have
some
slightly
interesting
scheduling
concerns
at
our
peak.
We
push
thousands
of
jobs
for
a
minute,
and
many
schedulers
can't
handle
large
volumes
of
very
short-lived
batch
jobs.
Many
job
schedulers,
the
design
for
running
long-running
services
or
processing
large
expensive
tasks
tend
to
have
fairly
slow,
schedulers
that
optimized
for
finding
optimal
placement,
rather
than
trying
to
like
schedule
it
quickly
and
these
types
of
scheduling,
algorithms
tend
to
become
quite
expensive.
When
you're
trying
to
schedule
like
this
many
jobs
on
a
very
large
cluster
of
machines
and
circle,
CIA
isn't
just
a
SAS
product.
A
We
also
offer
an
on-premise
deployment
option.
This
means
that
any
tool
we
deploy
as
part
of
our
product
needs
to
be
operable
by
people
who
are
not
necessarily
domain
experts
and
offer
the
power
to
automate
away
most
of
the
rough
edges,
and
that
means
sometimes
means
we
are
more
sensitive
to
manual
ops,
overhead
than
many
other
organizations,
and
at
the
same
time
we
were
looking
for
a
platform
that
would
enable
us
to
run
our
products.
Services
and
infrastructure,
though
allow
us
to
scale
out
beyond
like
the
single
development
instance.
A
A
A
Stat
stack
provisioning,
our
infrastructure
with
terraform
and
building
our
a.m.
eyes
with
Packer,
so
how
she
copes
still
very
young
Nomad
was
an
interesting
contender
and,
unlike
many
orchestrators
nomadically,
choose
between
a
few
different
scheduling,
algorithms
and
for
batch
jobs
used
as
an
algorithm
based
on
the
berkeley,
sparrow
scheduler.
That
applies
a
power
of
two
choices,
load,
balancing
technique
to
job
scheduling,
and
this
results
in
a
job
scheduler
that
is
incredibly
fast.
A
Nomads
executor
is
also
incredibly
pluggable
with
built-in
support
for
running
various
container
technologies
or
simply
allowing
us
to
execute
an
arbitrary
binary,
allowing
us
to
precisely
control
how
we
set
up
and
execute
our
customer
jobs.
This
is
incredibly
useful
for
us,
as
we
start
supporting
things
like
virtual
machines
or
supporting
running
multiple
services.
A
That
being
said,
this
usefulness
came
with
some
problems,
while
by
there
were
various
stability
and
operational
issues
that
would
force
us
to
ssh
in
and
restart
like
random
processors,
because
things
would
be
wedged
or
somewhat
randomly
failing,
and
so
because
of
our
desire
for
strong
isolation
between
builds
and
internal
services.
We
came
to
an
interesting
realization.
We
didn't
need
to
use
the
same
tool
for
both
our
job
execution
and
our
internal
infrastructure.
A
We
also
members
of
our
Suri
team
that
had
used
it
in
previous
jobs,
and
the
lifespan
and
adoption
of
the
tool
gave
us
confidence
that
others
had
encountered
similar
problems
to
us
and
perhaps
knew
how
to
solve
them
being
able
to
go
to
an
IRC
channel
or
a
mailing
list
or
Twitter,
and
have
people
have
potential
ideas
about
how
to
solve.
The
problem
is
really
valuable.
A
Kubernetes
also
strikes
a
nice
balance
of
giving
you
a
lot
of
valuable
functionality
with
reasonable
default
somewhat
out
of
the
box,
while
allowing
you
to
replace
most
of
the
components
without
having
to
reinvent
your
entire
architecture,
which
is
really
nice
when
you're
spinning
up
a
new
piece
of
infrastructure.
So
you
want
to
last
you
a
while,
and
so
we
ultimately
made
the
distinction
that
Nomad
is
part
of
our
product.
It's
where
we
run
builds
and
it
ships
as
part
of
our
on
premise
product
as
well
as
in
our
suffering.
A
But
kubernetes
is
what
we
use
for
scaling
our
platform
and
operationalizing
our
product.
We
leverage
its
features
to
allow
us
to
run
our
application
layer
services
and
to
provide
abstraction
and
automation
over
our
other
cloud
resources,
and
so
what
I'm
really
saying
is
it's
just
schedulers
all
the
way
down,
because,
yes,
we
do
run
a
scheduler
inside
a
scheduler
and
there's
even
a
meta
scheduler
running
inside
a
scheduler,
the
schedules
on
the
other
scheduler,
and
so
now
we
know
some
of
the
tools
were
working
with.
A
Let's
look
at
a
bit
of
how
we
set
this
up
and
explore
some
of
the
things
that
didn't
necessarily
go
so
well
and
how
we
left
research
tools
to
fix
them,
and
so
this
was
supposed
to
be
a
gift,
but
apparently
it
is
not
a
gift.
It's
fine
from
an
Operations
perspective.
Nomad
is
comprised
of
two
main
components:
the
nomad
server,
which
is
the
nomad
control,
plane
and
nomad
clients
which
execute
the
jobs
provided
by
the
nomad
server.
A
A
So
Nomad
is
an
inherently
stateful
service,
built
on
top
of
Raph
for
handling
blog
replication
and
with
a
couple
of
different
persistence,
operate
options.
Nobody
require
also
requires
some
form
of
service
discovery
for
initial
cluster
bootstrapping,
after
which
standard
gossip
protocol
total
takes
over.
So
how
do
we
manage
that
persistence?
We
handled
nomad
storage
by
having
dedicated
kubernetes
clients
for
Nomad
server
workloads,
I,
which
we
then
mount
the
host
storage
of
into
the
Nomad
server
container,
which
allows
us
to
leverage
our
existing
practices
for
managing
stateful
services
and
simplifies
our
kubernetes
configuration.
A
But
this
was
before
many
of
the
new
sort
of
Nomad
store.
Other
criminally
storage,
API
is
really
existed
and
because
Nomad
is
a
distributed,
stateful
services.
So
this,
it
also
requires
some
way
to
do
discovery
of
other
server
nodes,
and
there
are
a
few
ways
you
can
do
that
how
she
caught
recommend
using
console
for
this,
and
it
works
reasonably
well
with
relatively
low
overhead
to
get
set
up
and
configured
and
it's
a
very
stable
piece
of
software.
A
So
we
started
evaluating
alternatives
before
trying
to
deploy
yet
another
stateful
service
inside
kubernetes,
and
we
eventually
decided
on
switching
Nomad
into
using
a
kubernetes
stateful
set,
which
gave
us
really
predictable,
upgrade
semantics
to
blend
wanting
to
maintain
core
and
also
allowed
us
to
do.
Our
initial
nomad
server
bootstrapping,
using
only
kubernetes
for
service
discovery
and
I'll
talk
a
little
bit
more
about
how
that
works
later
now.
A
My
clients
are,
our
jobs,
actually
execute,
and
these
don't
run
inside
communities,
but
instead
within
a
standard,
AWS,
auto
scaling
group,
we
scale
the
clients
with
a
fairly
simple
autoscaler
that
is
driven
by
our
metrics
and
goes
through
an
alerting
pipeline,
which
then
goes
through
RabbitMQ,
which
don't
get
scaled.
It's
a
wonderful
internet,
Rube
Goldberg
machine,
but
it
comes
with
the
added
side
effect
of
it's
really
easy
to
shut
down
or
override
when
we
want
to
over
provision
for
expected
spikes
and
other
such
changes
or
when
migrating
over
to
a
new
cluster.
A
A
Yeah
a
few
months
ago,
we
also
migrated
to
running
inside
multiple
clusters,
as
I
was
saying,
to
provide
redundancy
and
also
to
make
it
easier
to
canary
new
releases
of
Nomad
and
our
surrounding
tooling,
without
affecting
all
of
our
customers
workloads.
We
leveraged
some
really
nice,
kubernetes
and
terraform
features
for
this.
A
We
deployed
all
of
our
tariff
infrastructure
using
terraform,
because
it
creates
his
great
visibility
and
changes
of
entire
deployment
and
allows
us
to
easily
reason
about
and
change
our
configuration.
One
of
the
first
steps
in
migrating
to
a
multi
cluster
deployment
was
to
modulate
our
terraform
12
hours
to
easily
create
multiple
downstream
clusters
with
minimal
duplication.
A
We
then
migrated
all
of
our
kubernetes
manifests
for
our
deployment
into
helm.
We
actually
packaged
the
entire
deployment
alongside
a
bunch
of
sidecar
services
that
provide
garbage
collection,
draining,
metrics
and
other
services
into
a
single
home
package.
This
makes
it
really
easy.
First
of
all,
to
both
CD.
It
uses
a
standard
CD
flight
plane
to
a
grade
Nomad,
while
also
being
able
to
really
easily
manually
deploy
a
cluster.
A
A
We've
definitely
experienced
some
rough
edges
of
both
nomads
and
kubernetes,
such
as
when
we
were
trying
to
fake
tapes
and
Toleration
zin
old
versions
of
kubernetes
and
also
some
rough
edges
of
nomads.
That
being
said,
we've
been
out
to
overcome
and
walk
around
them
and
in
the
latter
case,
kubernetes
has
been
super
valuable
in
that.
A
So
it's
time
to
talk
about
some
outages.
The
first
major
set
of
production
outages
we
experience
with
nomad,
were
in
January
and
taught
us
a
lot
about
some
less
obvious
behavior
in
nomad
and
resulted
in
some
changes
to
how
we
deploy
our
clusters
kubernetes
and
how
nomad
does
some
things.
So
the
symptoms
were
quite
interesting.
A
We
initially
got
paged
for
the
no
my
job
you're
starting
to
rise
quickly,
despite
us,
having
like
significant
capacity
to
be
able
to
run
users,
jobs
and
all
of
the
nomadic
clients
and
servers
appeared
to
be
opt
and
healthy
and
working
normally
by
all
of
their
metrics.
And
so,
after
digging
into
this.
For
about
like
half
an
hour,
we
were
getting
very
confused
because
we
would
definitely
know
when
there
are
peak
load.
A
But
as
we
look
deeper,
we
saw
that
nodes
were
simply
not
receiving
jobs
anymore,
and
so
we
started
to
take
a
look
into
the
nomad
servers
and
saw
that
all
US.
All
of
that
logging
had
stopped
outside
of
a
significant
number
of
failed
to
delete
job
requests
and
an
attempt
to
unleash
the
hosts.
We
started
just
like
cycling.
A
And
in
follow-up
from
that
we
were
reanalyzing
our
configuration
and
source
code
and
the
source
code
of
nomad,
and
we
noticed
we
were
missing
some
options
that
would
give
us
greater
visibility
into
the
operations
of
its
internal
state
machine.
So
we
quickly
enabled
them
so
we
could
get
more
data.
The
next
time
this
incident
occurred
and,
with
all
things,
computers,
it
turned
out.
We
wouldn't
have
to
wait
so
long
the
next
day,
if
we
were
once
again
grieved
by
the
same
lovely
page,
but
this
time
we
had
the
metrics
we
needed
to
identify
the
issue.
A
We
were
quick
to
deploy,
crump
declare
bankruptcy
after
validating
we
had
the
data
we
needed
and
doing
some
manual
introspection
of
the
running
state,
and
so,
when
analyzing
the
data
we've
collected,
we
eventually
realized.
The
issue
was
occurring
during
garbage
collection
cycles
when
Nomad
was
cycled
through
all
of
the
completed
jobs
and
try
and
remove
them
from
its
graph
database.
A
However,
it
wasn't
quite
perfect
and
around
our
peak
times,
scheduling
would
hole
for
just
like
10
seconds,
while
it
was
going
clearing
up
some
state
which
was
just
slow
enough
to
trip
up
handing
jobs
alerts,
and
so
because
we
only
really
use
the
nomad
API
to
intercept
jobs
that
have
failed.
We
don't
really
need
to
keep
around
jobs
that
are
in
a
successful
terminal
State.