►
Description
Adapting Brownfield Deployments to Cloud-Native Technology - Richard Hartmann
While most people will hopefully agree that a modern greenfield deployment should be based on cloud-native technology, there's a huge installed base of brownfield installations which are hard to impossible to simply switch out over night. This talk is about lessons learned in a company offering everything from carrier and ISP services, over hosting and housing, to architecture and operations, and even web development migrating from a myriad of legacy monitoring systems to One Global Source of Truth: Prometheus. It will also cover technical problems, social challenges, unexpected surprises, and the sheer, distilled joy of removing pager fatigue.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
That's
the
usual
stuff,
I'm
Richard,
we're
always
looking
for
people
a
little
bit
about
who
I
was.
That
might
be
more
important
in
in
this
context.
Here
for
about
seven
years,
I
was
solely
responsible
for
a
quite
large
network
and
literally
I,
without
proper
monitoring
back
at
the
time.
I
wouldn't
be
here
because
being
on
call
for
about
seven
years
is
challenging
in
some
regards
I
I,
really
really
like
monitoring,
I
despise
and
hate
most
monitoring
tools
for
various
reasons.
A
Back
then,
the
only
thing
which
was
really
any
good
and
usable,
at
least
in
my
opinion,
was
cervix,
but
this
is
not
about
settings
a
little
bit
of
context
about
the
company
I'm
coming
from
because
this
is
or
which
I
joined
recently,
because
this
is
one
of
the
prime
examples
of
brownfield
deployments.
It's
actually
the
oldest
commercial
isp
in
germany.
So
there's
a
lot
of
installed
base,
there's
a
lot
of
installed
tools
which
are
legacy
which
were
way
ahead
of
their
time,
but
some
have
aged
better,
some,
not
so
much.
A
A
It's
not
really
dashboard
e,
it's
more
like
just
an
email
for
every
last
piece
of
crap
and
yeah,
and
all
the
teams
they
have
their
own
stuff
on
top,
which
is
better
or
worse,
depending
on
which
team
you
have,
and
you
don't
really
have
anything
of
a
way
to
to
collect
and
correlate
data
and
just
do
useful
things
with
it.
We
can
look
at
it
and
that's
it
and
then
it's
gone
because
it's
not
machine
readable.
This
is
probably
true
for
most
people
who
are
not
designing
it
today.
A
A
A
Everyone
has
a
testing
environment,
some
people
even
have
a
production,
one
yeah,
so
just
small
summary
of
what
promises
actually
isn't.
Why
it's
so
interesting,
because
this
is
surprisingly
unknown
in
in
this
room,
it's
incredibly
efficient.
You
can
look
at
pretty
much.
Every
other
system
which
you
have
as
of
right
now
and
premises
is
more
efficient.
A
This
is
in
part
due
to
it,
focusing
on
metrics
and
basically
ignoring
it
wins,
which
is
a
trade-off.
Also.
It
only
stores
numbers
float64,
which
is
also
trade
off,
but
this
makes
it
incredibly
efficient
in
its
yeah.
You
can
do
things
which
you
can't
do
with
anything
else.
You
don't
have
any
hierarchy
model.
You
basically
apply
labels
as
you
wanted,
and
you
just
slice
and
dice
your
data
whatever,
according
to
whatever
metric
you
need,
you
can
do
it
by
customer.
You
can
do
it
by
my
data
center.
A
You
can
do
it
by
production,
environment
or
development
or
whatever,
and
you
can
just
aggregate
the
data
in
any
way
which
you
want.
Speaking
of
aggregating
data,
if
you
know
our
prom
QL
is
kind
of
issue,
light
are
in
as
much
as
it
does
a
lot
of
vector
magic
automatically.
You
don't
even
really
have
to
think
about
this
being
vector.
Math,
you
just
select
stuff
and
everything
else
happens
automatically.
It's
again
an
incredibly
efficient
language
and
the
same
language
is
used
for
everything.
A
So
if
you
want
to
show
a
graph,
that's
one
language,
it's
exactly
the
same
language
for
alerting,
so
you
can
actually
use
the
same
thing
for
alerting
and
for
graphing,
which
makes
a
lot
of
things
quite
easy
to
to
do,
and
also,
if
you
export
any
data,
you
again
use
the
same
language.
It's
even
a
case
that
if
you
do
graphing
basically
what
happens
in
the
background
is
that
you
get
the
raw
data
and
then
you
just
have
JavaScript,
which
does
the
graphing
for
you.
A
A
So,
basically,
you
don't
store
the
raw
data,
you
take
the
Delta
of
sequential
values
and
then
you
take
the
Delta
of
those
deltas
which
is
fascinating
as
a
concept
and
gives
you
incredible
storage
amounts
or
incredibly
low
surgeons.
There's
a
paper
by
Facebook
called
gorilla
or
about
gorilla.
If
you're
interested
in
this
topic
have
a
look,
it's
really
really
interesting
and
as
it's
go,
you
basically
have
one
binary.
You
have
container
or.
A
So
essa
said
Vieira
de
SP.
We
have
quite
a
few
SNP
devices,
probably
the
most
on
on
earth,
at
least
within
within
Prometheus,
simply
because
we
ran
into
problems
which
no
one
else
had.
So
this
is
kind
of
implies
that
no
one
else
was
using
it
more
than
us.
The
Python
implementation
basically
got
the
whole
system
into
into
a
really
bad
state
and
it
was
more
or
less
unusable,
with
even
only
300
devices.
We
basically
lock
down
the
whole
machine.
It
just
didn't
work.
A
Well,
the
fact
set
of
effective
devices
was
pretty
much
stable,
the
ones
which
came
up
and
down
and
up
and
down.
We
never
found
a
root
case.
We
just
took
some
money,
asked
Brian
to
re-implement
and
go,
and
ever
since
then
it's
been
stable.
There
are
a
few
pitfalls
because
a
lot
of
stuff
in
the
networking
world
I
don't
know
who
of
you,
has
a
lot
of
actual
hardware
and
network
hardware
that
money
yay.
A
There
is
quite
a
few
fishy
things
going
on
in
in
the
SNMP
implementations
around
the
world
and
basically,
if,
if
it's,
if,
if
the
data
is
not
as
the
premises
go
a
library,
you
declined,
I
really
expect
it.
Things
start
to
break
and
it
appears
to
be
rather
hard
to
Verger
on
this.
So,
basically,
what
you
have
to
do
is
you
just
use
the
python
implementation
for
those
kind
of
things
which
is
a
working
burg
around,
but
it's
not
very
nice.
A
There
are
some
caveats
to
the
python
implementation
and
also
some
both,
especially
when
things
are
weird,
but
else
everything
else
in
the
whole
Prometheus
ecosystem
is
more
or
less
brutally
office.
It's
highly
efficient
and
you
are
just
done
days
after
you
started,
so
that's
it
with
basically
having
your
brownfields
monitoring
in
chlorinated
technology.
A
No,
it's
not
because
this
wasn't
a
hard
challenge
as
it
turns
out.
This
was
really
simple.
It
was
a
matter
of
days
and
weeks
and
things
just
worked
and
kept
getting
better
and
better
its
Prometheus
itself.
It's
just
incredible,
but
the
real
challenge
was
to
show
the
social
kind,
because
you've
got
a
lot
of
install
place.
You've
got
a
lot
of
stakeholders
and
these
have
their
own
motivation
and
they
are
very
resistant
to
change.
A
Obviously,
you
have
the
issue
that
most
of
times,
if
you
look
at
operations,
teams
they're
actually
in
sensitized,
to
be
against
change
because
they
want
one
stable
system
which
doesn't
change
and
they
just
keep
it
up
and
running
the
course
change
is
hard.
It's
really
hard
unless
your
processes
actually
embrace
change
and
you
can
even
make
it
manageable
by
computers
not
by
humans,
because
humans
are
slow
and
expensive,
they're,
creative,
but
slow
and
expensive.
A
So
you
have
to
show
the
trade-off
for
the
people
who
actually
run
it
between
what's
new
and
what
they
have.
Why
should
they
embrace
the
change?
Why
did
you
do
this
and
also
simple?
You
diligence?
You
have
to
keep
the
old
systems
running
because
you
can't
just
say:
ok,
everything's
new
and
if
something
breaks
who
cares
you
have
to
have
it
maintained
in
parallel.
A
Unfortunately,
this
introduces
more
toil
foil
if
the
vert
is
basically,
my
definition
of
toil
is,
if
you
have
work
which
is
repeated
and
which
does
not
give
benefit
in
which
it's
manual
and
humans
have
to
do
it.
That's
toil,
and
you
want
to
over
avoid
toil
if
your
teams
are
busy
firefighting
and
just
up
to
here
with
work,
they
can't
really
engineer
your
ecosystem
at
your
company,
and
that
is
one
of
the
most
important
problems
which
I
saw
because
people
don't
have
time
to
actually
do
the
new
stuff
which
will
save
them
time
later.
A
Another
thing,
which
at
least
for
the
operation
skies
is
really
important,
is
to
keep
their
sanity.
So
when
you
redo
things
anyway,
do
it
right,
split
up
importance
and
urgency
and
be
quite
rigorous
about
what
you're
alert
on
if
it's
not
actionable?
If
no
human
can
just
say,
okay
I
need
to
do
this.
This
is
not
an
alert,
don't
alert
on
it
log
it.
A
They
start
to
embrace
this
and
they
actually
work
on
getting
the
new
system
rolled
out
and
cut
off
the
old,
crap,
yeah
and
kind
of
obvious,
but
still
it
has
to
be
said.
If
you
can't
monetary
service,
it
can't
go
into
production
period,
easy
as
that
it
should
be
obvious
way
too
often
abstract.
Unfortunately,
so
one
specific
thing
which
helped
within
the
company
and
helped
quite
well
was:
we
had
an
incident
where
basically,
someone
forgot
to
set
a
certain
flag,
and
so
one
male
server
was
not
behind
the
normal
SEO.
A
Spammers
are
really
good
these
days,
so
they
do
a
stack
at
ramp
up.
They
see.
Okay,
that's
this
machine.
If
the
power
does
it
have
to
bandwidth
to
be
really
useful,
but
they
do
it
slowly
to
see
if
anyone
catches
you
catch
the
stamp
sorry
and
only
then
they
will
really
start
spamming
and
obviously
they
do
it
during
night
time
for
your
local
time
zone,
because
they
know
that's
when
people
won't
be
looking
and
then
they
can
Sentinel
spend.
So
this
is
really
from
engineering
point
of
view.
A
He
would
have
had
to
look
at
various
distributed
systems,
logging
into
ten
different
things,
to
look
at
20
different
things
with
parameters
and
profanity
was
just
one
single
thing
and
he
knew
what's
up
so
all
of
a
sudden
you
have
buying
from
people
who
might
not
be
fully
convinced
before
this
one
incident
convinced
maybe
ten
people
in
various
teams
just
overnight
that,
yes,
this
is
different
for
word
and
before
they
were
more.
Like
I,
don't
know
and
all
of
a
sudden,
they
see
their
own
incentive
and
they
want
to
use
it
but
thing
about
incentives.
A
It's
also
about
perspective,
so
someone
once
said-
and
that's
really
stuck
with
me-
that
it's
quite
easy
to
talk
to
an
engineer
for
hours
about
a
specific
function,
but
the
CEO
will
not
care
not
at
all
and
that's
okay,
because
everyone
has
a
different
perspective
towards
the
same
thing.
They
might
even
agree
in
principle,
but
they
don't
even
realize
it.
So
your
managers
will
care
about
revenue
about
process
execution
about
all
the
high-level
stuff,
but
they
don't
care
about
the
low-level
stuff.
Your
architects
won't
clean
design.
A
So
basically
what
you
need
to
do
is
you
need
to
tell
every
single
person
who
has
a
stake
in
the
whole
system,
what
they
need
to
hear
you
never
lie,
you're
completely
blunt
and
open
and
honest,
but
you
make
them
see
where
their
own
intrinsic
motivation
is
and
how
they
can
apply
this
to
this
whole
big
big
picture,
because
this
is
really
what
you
have
to
do.
You
have
to
have
a
large
big
picture
which
you
draw
on
the
wall,
either
literally
or
proverbial.
A
What's
their
motivation,
what's
their
benefit
from
agreeing
to
this
and
and
helping
you
go
there
where
you
want
to
go,
and
once
you
have
buy-in
people
all
over,
the
company
will
start
to
make
small
incremental
steps
and
every
time
they
have
to
take
a
decision
if
they
want
to
implement
some
policy
or
some
process
or
whatever
in
this
way
or
in
that
way,
they
can
look
to
the
big
picture
and
if
they
all
agree,
they
will
all
make
small
steps
in
the
right
direction
without
having
to
synchronize
again.
This
should
be
obvious.
A
Another
very
nice
thing
which
kind
of
mixes
the
technical
aspect
back
into
the
whole
picture
is
when
you
leverage
this
kind
of
system,
because
the
one
thing
or
one
of
the
many
things
which
Prometheus
does
really
well
is
it
allows
you
to
have
so
much
data
and
to
correlate
so
much
data
in
unexpected
ways
and
in
really
smart
ways
that
you
can
do
things
which
you
couldn't
do
before
simple.
As
that
you
could
not
do
it
before,
at
least
not
with
the
same
resource
usage.
A
A
Coming
from
the
networking
side,
when
you've
got
optical
networks,
which
are
just
just
barely
working-
and
you
have
just
enough
dampening
that
it's
just
working
also,
temperature
can
be
affected
and
if
you're
able
to
actually
map
down
times
in
parts
of
your
network
against
outer
temperatures,
we
had
an
issue
with.
When
there
was
high
traffic
and
a
sharp
decline
or
rise
in
temperature,
then
one
particular
line
would
go
out.
Try
finding
that
with
a
traditional
system
back
then
it
took
us
months
to
realize
what
issue
was
once
we
had
it.
A
So
if
that's
one
feet
goes
down,
UPS
has
an
issue
or
fuses
blown
or
whatever
all
the
systems
are
down
and
then,
when
the
other
take
over
yeah,
all
of
a
sudden
have
way
too
much
load
on
the
other
feet,
because
there's
ramped
up
on
the
other
feet
and
then
you
blow
the
second
fuse
and
then
you're
completely
offline.
So
this
is
something
which
all
the
people
who
were
running
on
one
cloud
service,
don't
really
care
about,
because
it's
done
further,
but
people
who
are
actual
hardware.
A
This
is
something
where
you
can
have
really
really
bad
time
with,
because
this
takes
out
the
whole
site.
So
unless
you
have
a
second
or
third
third
site,
you're
done
and
there's
lots
more
I
suggest
you
look
at
the
talk
of
fabian
about
alerting
it's
pure
genius,
and
you
really
see
what
you
can,
what
you
can
do
with
with
prometheus.
It's
it's
one
of
the
best
talks
of
all
time,
at
least
in
my
opinion.
A
So
all
of
a
sudden,
everyone
agrees
on
the
metrics
and
the
grease
on
what's
happening
because
they
have
the
same
data.
They
don't
have
slightly
different
data,
which
is
slightly
weird
or
something
they
don't
have
invisibility,
where
some
monitoring
systems
have
high
system
load
while
they
collect
and
don't
see
their
own
system
load
morning
would
be
an
example.
So
we
had
those
spikes
in
Prometheus
every
five
minutes
but
morning
didn't
see
him
because
it
was
itself
causing
it,
and
during
this
time
it
wasn't
collecting
any
data.
A
A
A
Dashboards
and
return
are
equally
as
obvious,
something
which
wasn't
obvious
for
us
at
least
at
first,
was
to
have
the
same
dashboards
or
at
least
the
same
graphs
automatically
exported
into
VF,
based
on
our
configuration
management
system
or
our
service
database,
because
then
the
service
manager
basically
just
clicks
a
button.
And
yes,
this
customer
has
ten
new
servers
or
whatever
it's
automatically
part
of
their
SLA
report.
It's
automatically
part
of
whatever
you
have
for
them.
If
you've
got
specific
dashboards,
it
just
happens
but
being
able
to
auto
generate
all
those
reports
is
really
really
nice.
A
You
also
have
quite
an
easy
time
to
give
your
sales
teams
a
whole
new
insight
into
your
data.
It's
a
fair
example
for
us.
We
are
really
really
good
at
keeping
systems
up
and
we
are
really
good
at
exceeding
our
sll.
So
now
we
can
prove
this.
We
can
give
sales
statistics
across
all
our
customers,
obviously
without
naming
them
and
everything
that
across
all
the
customers-
and
we
can
say
ok
for
the
last
quarter
for
the
last
year,
there
was
not
a
single
SLO
violation
on
any
of
the
systems
which
we
run.
A
This
is
gold
for
sales
and
all
of
a
sudden,
you
have
sales
which
and
sales
becomes
excited
about
a
monitoring
system.
This
is
quite
interesting
to
see,
and
also
you
can
put
your
business
data
into
into
Prometheus.
There
was
one
question
earlier
by
some
guy
I,
don't
know
about
keeping
operational
and
business
data
in
the
same
Prometheus
instance.
A
So
for
us,
what
we
still
have
to
do
is
to
basically
merge
all
our
configuration
management,
especially
across
different
teams,
because
UNIX
and
windows
are
using
different
systems
which
made
sense
at
the
time,
but
doesn't
make
sense
anymore,
where
possible.
But
this
again
is
something
where
we
are
kind
of
special,
because
we
have
highly
specific
set
ups
for
some
customers
and
they
might
not
be
actually
brittle,
but
still
very
special
snowflakes.
A
So
it's
kind
of
hard
to
get
those
into
a
stage
where
you
can
auto
scale,
and
this
is
a
hard
technical
problem.
But
as
long
as
the
revenue
is
coming
in
for
those
old
systems,
the
customers
don't
really
want
to
change,
and
after
all
the
customer
decides
what
they
want
to
do.
You
can't
force
them,
you
can
tell
them.
What's
better,
you
can
use
your
technology
to
support
their
case,
but
you
can't
force
them
to
do
anything.
A
A
huge
thing
which
we
need
to
do
is
to
adopt
error
budgets
where,
basically,
you
have
a
feedback
loop
towards
developers
that,
when
their
systems
break
for
this
quarter
for
this
half
year
for
this,
whatever
they
can't
release
any
new
features.
So
all
of
a
sudden
they
they
have
a
real
incentive
to
not
ship
broken
stuff,
because
normally
they
just
ship
it
and
operations.
A
Basically
did
any
that
everyone
get
the
question
or
should
I
repeat:
okay,
so
question
was
how
to
migrate
old
triggers
and
old,
alerting
to
the
new
system,
and
the
answer
is
quite
simple:
keep
everything
in
parallel!
That's
what
I
meant
with
due
diligence.
You
cannot
afford
to
to
have
to
miss
any
important
alert,
because
at
this
point
you
really
start
breaking
things.
So
what
you
have
to
do
for
at
least
some
time
keep
the
old
system
running,
keep
the
new
system
running
and
build
up
the
confidence.
A
Of
course,
all
the
teams
that
this
is
actually
working.
So
for
sometimes
you
have
two
triggers,
but
this
is
also
a
chance,
because
then
people
start
to
realize.
Okay,
this
one
thing
just
carpet
bombs
me
with
emails,
and
this
other
thing
does
it
once
and
does
it
nicely
so
you
actually
increase
incentive
for
for
people
to
change.
But
yes,.
A
So
Ben's
answer
just
for
the
recording
was
to
basically
define
a
class
off
or
a
type
of
class
of
alerts
and
move
by
batches
of
cloud
or
within
the
class
one
type
of
alert
and
then
move
those
over.
But
my
answer
is
actually
one
step
further
backwards,
because
you
should
have
documentation
at
some
point
what
you're
actually
alerting
on.
What
are
your
service
definitions?
What
are
your
cells?
There
should
be
something
which
actually
defines
what
you're
doing
and
why
you're
doing
it?
A
A
A
Yeah
this,
so
the
comment
was
basically
or
the
short
version
of
this
was
that
it's
good
to
have
pine
from
from
marketing
and
from
sales
and
other
stakeholders.
And
yes,
because
normally
you
have
like
this
little
island
of
tech
people
and
you
might
even
have
management,
then
they
kind
of
agree,
but
without
the
whole
company
agreeing
on
this
being
a
good
thing.
You
will
always
have
blockage
at
some
point.
So
try
really
to
think
about
how
to
motivate
people
who
are
not
part
of
a
team.
You
might
not
even
be
the
same
Department.