►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
From Snowflake Servers to Snowflake Clusters - The GitOps Journey - Allison Richardet, Asteris, LLC & Fabio Giannetti, MasterCard
At MasterCard, we have a unique hard requirement of one dedicated cluster per application due to security and other challenges. In an effort to prevent snowflake clusters, we provided a base set of services for logging, monitoring, etc.; however, application teams inevitably want to customize their logging solution or add a monitoring dashboard. We realized this mix of standard and optional services could quickly get out of control and needed normalization. In early 2018, we exposed application teams to Kubernetes manifests; they were excited about control over deployments, but unprepared for the complexity. By late 2018, we needed to minimize complexity and find a way to express high-level application needs, while providing secure, scalable and redundant deployments. This is our journey of how we embraced GitOps, and the challenges we faced making deployment descriptions minimalistic.
https://sched.co/MPc4
A
A
These
are
hand
embroidery
patterns
in
art.
We
want
variety
and
uniqueness,
we
are
drawn
to
imperfection
and
we
want
new
and
different.
However,
we
don't
necessarily
want
this
with
servers,
so
some
issues
with
this
are
they
can
be
difficult
to
reproduce
and
debugging
and
testing
can
be
hard.
Also,
changes
and
upgrades
are
problematic
and
can
cause
breakage.
A
Also
instances
can
become
out
of
sync,
and
then
it
also
can
lead
to
environment
mismatch.
So
typically,
we
want
dev
stage
and
production
to
all
match
and
when
we're
dealing
with
snowflake
servers,
this
typically
can
be
a
problem
and
finally,
there's
also
lack
of
audit
trails,
so
understanding
how
who
and
when
a
change
was
initiated,
so
we're
really
going
to
focus
on
patterns.
That's
a
theme
you'll
see
today
throughout
the
talk
and
how
to
rebalance
similarities
with
uniqueness.
A
Snowflake
servers
are
not
a
thing
of
the
past;
now
we
have
snowflake
clusters
and
with
kubernetes
comes
great
power
and
a
responsibility.
When
we
began
our
journey,
we
spoke
with
application
teams
who
needed
to
move
legacy,
applications
to
containerized
workloads
and
many
needed
a
dedicated
cluster
due
to
several
constraints,
first
of
which
was
regional
requirements
so
with
on
sole
requirements,
an
application
needs
to
ensure
that
all
of
the
data,
all
transactions
and
anything
associated
with
the
application
remains
in
a
specific
region,
and
then
there
are
also
security
concerns.
A
So
with
these
constraints
and
the
number
of
teams
were
working
with
some
requiring
hard
multi-tenancy,
we
need
to
support
tens
to
hundreds
of
clusters
for
environment
and
finally,
we
really
wanted
to
focus
on
developer
experience
and
operator
experience.
This
can
be
difficult
and
we
wanted
to
be
sure
that
we
had
easy
onboarding
for
our
tenants
and
that
upgrades
and
changes
would
not
impact
our
customers
and
we
need
resilient
and
reproducible
infrastructure.
A
So,
with
security,
we
need
tenant
isolation,
we
have
multi-tenancy,
and
in
order
to
do
this,
we
use
namespace
isolation
and
also
utilize
role
based
access
control
and
then
for
those
tenants
who
require
hard
multi-tenancy.
They
have
their
own
dedicated
cluster,
as
this
is
not
currently
possible
with
kubernetes,
and
then
we
have
common
services
that
are
based
on
pattern,
and
these
are
patterns
that
we
found
when
we
were
speaking
with
application
teams-
and
you
know
it's
things
like
ingress
monitoring,
logging,
continuous
deployment
tracing.
A
A
So
how
did
we
make
this
vision
a
reality
based
on
the
patterns?
We
distilled
our
cluster
types
down
to
three
different
types.
First,
we
have
a
control
cluster,
we
have
one
per
environment,
it
manages
deployments
and
it
has
global
monitoring.
Then
we
have
single
tenant
clusters,
which
are
for
teams
who
require
hard
multi-tenancy,
and
finally,
we
have
multi
tenant
clusters
for
teams
who
are
able
to
work
with
soft
multi-tenancy
and
deployment
of
our
services.
We
needed
to
ensure
it
was
automated
and
get
ops
helps
us
to
achieve
this.
A
A
A
Get
is
get
get,
apps
is
basically
a
couple
different
things
go
into
the
definition,
so
git
is
the
source
of
truth,
and
you
have
declarative
definitions
of
environments,
configuration
and
deployments.
So
for
our
case,
we
use
home
floor
template
a
and
we
use
customized
for
manifests.
So
in
that
transition
from
home
to
customize
we
use
helm,
convert
and
customize.
That's
how
we
deal
with
snowflakes.
A
The
patterns
give
us
our
base
services,
and
then
we
use
overlays
to
customize
for
tenants
for
things
like
ingress
and
then
all
of
this
is
stored
in
get
another
part
of
get
ops
is
managing
deployments
from
get
in
an
automated
fashion.
So
we
use
a
continuous
deployment
tool
which
manages
our
automated
deployments.
It
ensures
the
declared
and
actual
states
match
and
the
tool
we've
chosen
is
Argos
CD.
A
A
A
A
We
define
the
number
of
nodes
and
by
default
we
have
three
masters,
so
we're
fully
ajay
the
system.
Namespaces
are
created,
so
cube
system
and
PKS
system.
Pks
system
is
a
pivotal
specific
namespace
and
each
has
its
own
subnet
for
communication
with
the
Gateway,
we
have
the
ability
to
set
IP
ranges
and
set
up
network
rules
as
well,
and
so
the
same
goes
for
any
non
system
namespaces.
B
So
there's
a
lesson
explained:
we
are
building
many
clusters
for
our
different
talent,
so
we
run
all
of
this
on
premise.
So
the
first
problem
we're
facing
is
how
we
gotta
manage
all
these
clusters
right,
because
the
complexity
manage
one
cluster
from
an
Operations
standpoint.
Imagine
start
to
manage
hundreds
of
those
potentially
so
we
introduced
this
concept
of
control
cluster
and
we
use
this
control
cluster
to
basically
have
a
single
place
where
we
can
operate
all
the
other
cluster.
B
B
If
you
are
not
familiar
with
Thomas,
tunnels
is
a
component
that
allows
you
to
do
H
a
Prometheus,
so
you
can
deploy
several
parameters
instances
and
then
tano's
will
reconsolidate
the
data
and
they
duplicate
the
data
for
you,
so
the
tunnels
they
are
deployed-
you
know,
are
the
same
in
all
of
these
clusters:
controller
non
control.
They
will
scrape
data
from
primitives
which
scraped
from
everything
else,
and
then
they
gonna
send
the
data
to
our
object,
storage,
and
in
doing
that,
you
can
also
set
some
of
the
data.
B
You
can
change
the
basically
retention
and
you
can
down
sample
your
data.
So
in
that
way
you
can
basically
have
the
ability
of
retaining
longer
data
for
analytics
later
on,
and
once
you
do
this,
the
tunnels
on
the
control
cluster
also
has
the
ability
to
go
and
read
the
data
from
there
and
provide
through
graph
ahna
a
single
pane
of
glass
for
our
operations.
So
we
have
set
up
graph,
honest
dashboards,
where
you
can
go
and
choose
and
select
all
the
different
clusters.
They
are
part
of
your
environment.
B
So
then
you
can
basically
monitor
everything
from
that
place
and
it
goes
from
both
the
core
services
we
deploy
to
the
tenant
space
is
where
our
applications
are
running
for
logging.
We
did
a
very
similar
thing,
so
we
deploy
a
fluent
bit
for
each
individual
node
in
those
clusters
and
those
fluent
bit
will
relay
data
into
an
elastic
data
store
that
we
have
on
our
control
cluster.
So
all
the
logs
are
going
there.
We
have
some
logs.
We
are
not
representing
ever
some
logs
there
for
compliance
and
and
go
to
Splunk.
B
B
B
So
if
you
do
a
change,
let's
say
an
image
change,
it
will
go
and
check
if
your
deployment
has
the
version
of
the
image
that
you
want
to
deploy
and
if
it
doesn't,
it's
gonna
go
and
update
the
deployment
which
then
in
turn
is
gonna
update
the
pods
downstream.
So
it's
a
very
powerful
tool
and
we
found
it
very
useful
for
automating
all
these
steps.
Ok,
next,
a
little
bit
on
customized
I
know
this
is
now
about
customized.
B
B
You
know,
Yama
file.
You
can
build
it
as
we
do
from
helm
template,
so
we
leverage
the
community
as
much
as
we
can.
You
know,
because
there's
a
lot
of
good
stuff
around
on
you
know
basing
or
foundation
for
deployment
of
applications,
and
then
we
write
an
overlay,
and
you
know
customize
of
many
many
ways
of
doing
this
strategic
manner,
jazz
pace,
Jason,
patch
and
all
sorts
of
ways.
You
can
write
these
overlays
on
top
of
it.
In
this
example,
I
use
a
Jason
patch,
so
it's
very
simple
right.
B
It's
a
JSON
path
where
you
gotta
say
replace
something
somewhere.
So
in
this
case
it's
gonna
go
and
replace
the
cabana
generic
name
into
actually
a
DNS,
fully
qualified
name
that
will
be
used
for
a
hypothetical
cluster.
The
same
thing
can
be
done
with
the
host
itself.
So
then
you
can
write
as
many
as
you
wanted
this
they
can
go
and
change
and
modify
whatever
parts
that
you
want
of
your
deployments
once
you've
done
there.
Basically,
you
do
a
customized,
build
and
actually
customize
now
as
part
of
the
cube.
B
Ctl
has
been
fully
integrated,
but
customized
we'll
do
a
build.
Take
your
base
application
that
your
reference
apply,
the
overlays
they're
rather
relevant
for
what
you
are
deploying
and
build
a
new
yellow
files
or
multiple
Java
files
as
a
result
and
those
would
be
applied
by
your
cube
CTL.
In
our
case,
we
don't
use
cube
CTL,
but
we
use
our
go
so
then
our
go
integrates
as
we
customize.
B
So
basically,
when
are
gone
is
to
check
the
if
an
application
is
up-to-date
or
now
it's
gonna
go
as
customized
customized,
we'll
build
a
thing
for
our
go,
are
just
gonna
check
the
desired
state
versus
the
color
state
and
if
he's
needed
is
going
to
push
those
channels
into
kubernetes
and
and
kubernetes
will
take
from
there.
So
let's
call
it
to
be
more
in
the
details,
so
how
we
build
the
cluster
and
auto
provision.
So
the
goal
here
for
us
is
application
comes
in.
They
want
the
dedicated
cluster.
B
They
can
choose
among
a
kind
of
set
of
Alucard
services
that
we
provide
and
then
they
built
they
put
the
ingot.
They
make
a
get
request
and
from
there
on
is
100%
automated.
We
don't
touch
it.
So,
let's
start
so
they
go
and
make
a
couple
of
files.
They
usually
request
a
cluster
and
then
they
request
a
la
carte
services.
B
So
you
see
these
are
CR
these
customers,
whose
definitions
that
we
introduced
and
we
are
actually
looking
at
Italy
now
at
the
cluster
specs,
which
is
came
out
recently-
and
you
know
we
want
to
follow
them
and
see
if
how
their
progress,
because
we
see
a
lot
of
overlap
with
what
we
are
doing.
But
the
bottom
line
is
that
you
know
they're
going
to
create
the
CR
D
I
know
people
use
here.
The
most
or
most
people
know
CR
DS
in
relationship
wave
operators,
but
we
are
taking
this
with
a
slightly
different
spin.
B
Basically,
the
job
is
gonna
tell
agro.
Then.
Now
this
new
application
exists
inside
the
Geeta
gravel
at
a
point,
Argo
points
those
and,
of
course,
what
is
gonna
do
is
gonna
see
that
none
of
these
applications
are
in
the
cluster.
That's
been
created
because
it's
brand
new,
and
for
that,
like
the
desire
state,
is
not
the
same
as
the
actual
state,
so
it's
gonna
go
and
start
to
deploy
everything
that
has
been
requested.
Okay,.
C
B
Can
replay
deploy
the
core
stuff,
as
well
as
all
the
a
la
carte
services?
So
let's
go
little
bit
more
in
deep
on
in
detail
of
what
we
done
with
the
different
CR
DS.
So,
let's
start
from
the
cluster
spec.
This
is
very
simple:
I
apologize
to
put
llamo
here,
but
I
couldn't
refrain
from
doing
so.
So
it's
very
simple.
Basically
we
define
the
type
of
cluster
you
want.
B
You
get
a
a
name,
you
have
a
URL
for
it,
you
want
to
give
it
and
then
there
is
the
part
very
interesting
beside
you
know
the
plan
and
the
number
of
nodes
you
want
for
this
cluster.
We
define
a
cancer
comforter
of
a
cluster
type
and
the
ability
to
inherit.
So
there
are
other
types
that
will
be
defined.
B
So
let's
say
this
is
a
car
lawyer,
the
fake
example
in
what
you
want
to
be
a
single
tenant
cluster,
which
will
mean
some
of
the
stuff
they
are
part
of
their
core
constituency's
will
be
bringing
into
the
cluster
and
then
you
know
you
deal
with
PII
data
and
you
want
to
be
PCI
compliant
and
that
will
bring
different
type
of
applications
by
the
vault
into
the
cluster.
Before
this
particular
application
team
will
deploy
their
own
workloads.
B
B
So
mapping,
so
this
is
the
other
part
to
the
equation
right.
What
we're
doing
here
we're
saying:
okay,
what
applications?
What
a
la
carte
services
you
want
to
bring
as
part
of
your
own
definition,
and
so
we
have
the
a
la
carte
one
where
these
are
part
of
the
two
original
series
that
the
application
will
submit
and
they
will
basically
say
well
for
my
particular
one:
I
need
this
storage
service
or
two
service
or
whatever
it
is
which
are
part
of
the
core
services.
B
So
when
all
of
this
is
done,
the
customize
is
very
simple,
because
now
what
you
do
you
bring
in
those
base
deployments
you
bring
in
the
cluster
definition
that
they
have
created
and
those
overlays
will
bring
in
the
up
definitions
and
they
may
have
certain
set
of
customized
components
in
it.
That
needs
to
be
adapted
for
that
particular
cluster.
So,
for
instance,
looking
at
logging
what
it
can
happen,
the
Cabana
in
grass
for
that
particular
application,
may
have
a
replacement
operation,
and
this
is
very
similar
to
the
example.
I
show
you
before
on
purpose.
B
Now
you
see,
though,
the
the
real
thing
is
transforming
with
a
bunch
of
variables.
They
are
inherited
from
the
cluster
definition
and
a
mapper
definition.
So
up
named
cluster
name
and
qualified
modified
domain
name
are
coming
from
those
theories,
so
I
don't
have
to
constantly
change
my
deployments
or
make
my
deployments
tailored
to
a
particular
cluster
or
to
a
particular
application
which
we
require
manual
intervention
is
all
taken
from
templates
and
variable
and
is
only
injected
at
the
time
of
deployment.
B
And
lastly,
if
you
guys
are
familiar
with
Argo,
this
is
a
very
typical
definition
of
another
application
and
deployment,
and
again
we
inject
variables
in
it.
We
have
the
application
name,
the
cluster
name,
so
we
can
make
this
application
in
the
metadata
specific.
So
when
we
retrieved
from
the
log
we
will
know
which
application
they
belong
to
which
cluster
they
belong
to,
in
that
particular
environment
and
with
this
I'll
pass
back
to
Allison
for
the
future
plans.
A
Okay,
so
you've
seen
a
bit
of
our
journey
and
where
we've
come
from
on
the
left,
it's
sort
of
grayed
out.
That's
you
know
where
we
are
where
we've
been
and
moving
forward.
We
want
to
share
some
of
our
future
plans
for
our
journey,
so
this
this
quarter,
we've
been
working
a
lot
on
improving
onboarding
and
improving
the
tenant
experience
and
also
making
enhancements
to
automation.
A
We
started
the
talk
with
a
focus
on
security
patterns
and
get-ups.
Hopefully,
you've
seen
this
throughout
the
talk
and
have
learned
some
things
you
can
take
home.
We
think
it's
very
important
to
identify
security
concerns
and
requirements.
Upfront,
as
this
can
have
a
big
impact
on
your
journey
and
where
you
end
up
going,
we
also
focus
on
patterns
and
we've
drawn
on
the
patterns
to
find
unique
needs
and
also
needs
that
are
universal
for
our
application
teams
for
our
customers.
So
these
patterns
have
helped
us
limited.
A
The
types
of
clusters
also
limit
the
differences
between
the
types
of
clusters
and
we've
been
able
to
come
up
with
base
services
that
all
teams
can
benefit
from,
while
also
allowing
them
to
customize
using
overlays
for
services.
So
we
provide
a
platform
and
a
product.
How
can
we
best
enable
our
customers
and
then
get-ups
we're
using
declarative
definitions
in
version
control
and
have
automated
deployments
with
a
built
in
audit
trail?
A
So
if
we
update
an
image
in
the
changed
is
merged
and
then
all
clusters
running
this
are
automatically
updated.
This
is
pretty
powerful
and
then,
with
the
C
and
C
F,
you've
probably
noticed
a
lot
of
the
tools
we
have
discussed
throughout.
This
talk
are
open
source.
Many
of
them
are
part
of
the
CNC
F.
A
C
B
Good
question:
we
we
inject
the
secrets
at
the
beginning,
we
can
coerce
a
little
show
that
part
and
we
basically
had
evolved
for
each
individual
environment,
one
vault
/
environment,
so
we
bootstrap
the
secrets
that
we
use
for
our
own
core
services.
We
don't
manage
right
now
the
secrets
for
the
application
they
managing
themselves.
B
B
Concourse
basically
bootstrap
the
secrets
into
vault
all
right
and
then
the
application
have
a
one-time
token
when
they
come
up,
they
gonna
talk
to
vault,
retrieve
the
secrets
and
we
use
a
path
that
is
similar
to
the
deployment
structuring
customized
to
involve
where
the
secrets
are
stored.
So
if
I
am
up
a
large
bar,
that's
the
equivalent
path
will
be
my
vault
path
to
find
my
own
stuff
yeah.
So.
D
A
B
But
basically
we
have
one
repository
for
environment,
so
the
environments
are
divided
of
sandbox
stage
and
then
inside
there
are
the
classes.
So
we
don't
have
one
repository
pair
environment.
Well,
just
one,
and
then
we
classify
the
different
deployments,
the
different
environments
right
there,
yeah.
E
B
As
Adam
said,
we
are
using
PKS
the
pivotal
solution,
so
pivotal
has
a
basically
it's
built
on
their
boss
foundation
and
through
the
API
you
can
request
the
creation
on
the
cluster,
which
will
talk
to
the
center,
create
events
and
then
deploy
the
core
components
the
kubernetes
pieces
into
it.
So
we
are
not
using
terraform,
it's
all
delivered
as
part
of
the
PKS
platform.
Okay,.
E
D
B
We
are
the
core
service,
so
basically
imagine
that
you
are
coming
to
us
and
you
say:
I
need
a
cluster
to
run
this
application
or
applications,
and
you
are
one
of
the
business
unit,
so
we
will
be
the
cluster
based
on
the
type
of
class
that
you
require.
We
will
provision
the
cluster
and
deploy
all
the
core
application
core
services.
They
are
needed
for
managing
the
cluster.