►
Description
Taking the Helm - Ancestry's Journey to Kubernetes - Paul MacKay, Ancestry.com
Adopting new technologies such as Kubernetes is a big change for traditional enterprise environments. Ancestry was a typical Microsoft .NET shop running in a traditional IT environment. During the past year and a half Ancestry has adopted containerization and micro services orchestrated by Kubernetes. This presentation will describe Ancestry's journey to Kubernetes. We will discuss what we have learned during these past few years from a technical and cultural perspective. This presentation will provide insights to help companies make such a transition.
About Paul MacKay
Paul MacKay is a Software Engineer at Ancestry. He is a leading proponent at Ancestry for Docker and Kubernetes. He helps architect solutions around Kubernetes for engineering solutions at Ancestry.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
My
name
is
Paul
McKay,
I'm
software
engineer
architect
at
ancestry
and
I'm
here
to
discuss
some
of
the
attributes
or
experiences
that
we've
had,
as
are
as
we've
made
our
journey
to
to
kubernetes
in
the
last
year,
or
so
so,
hopefully
you're
in
the
right
place
at
the
right
time,
and
hopefully
there's
some
things
that
you
can
gather
whether
you
are
going
through
the
transition
or
have
gone
through
the
transition
to
kubernetes.
Hopefully
there
are
some
insights
that
you
can.
A
Who
who's
heard
of
ancestry
before
Oh?
Wonderful
great?
Hopefully
it's
been
a
good
experience
for
you.
As
you
know,
we've
been
around
a
bit.
Our
business
is
centered
around
science
and
technology.
That
really
has
a
human
mission,
and
that
mission
is
really
to
bring
people
together
to
have
them
understand
their
their
backgrounds,
their
histories,
anything
that's
related
to
them
to
help
them
understand
themselves,
as
well
as
those
around
them,
and
we
we
started
a
while
back
with
family
history.
A
As
far
as
you
know,
a
lot
of
the
documents
and
and
searching
capabilities,
and
in
the
last
few
years,
we've
also
embarked
upon
DNA
and
the
science
of
genomics
to
also
help
connect
people
together.
One
of
the
things
that
we
found
is
that
as
people
learn
more
and
more
about
themselves,
they
find
that
we're
more
connected
than
not,
and
so
that's
a
nice
thing
about.
Ancestry,
too,
is
connecting
people.
A
Data
drives
our
business.
It's
all
about.
It's
all
about
data.
We
have
billions
and
billions
of
historical
records
that
we
have
created
and
categorized
over
the
years.
We
have
millions
and
millions
of
families,
family,
trees
and
billions
of
profiles
and
again,
as
it
says,
millions
and
millions
of
historical
photos,
documents
and
histories,
and
so
you
can
imagine
this
just
kind
of
accumulates
over
the
years
and
we
have
petabytes
of
data.
A
So
we
truly
are
a
data-driven
company
and
as
our
DNA
efforts
have
grown,
we
we
now
have
one
of
the
largest
consumer
genomic
databases
over
two
and
a
half
million.
That
again
is
useful
in
helping
understand
even
more
connections
with
people,
so
that's
kind
of
where
we
we
come
from.
As
far
as
ancestry
is
a
company
now
I
want
to
talk
about
some
of
the
technologies
that
we
have
used
in
the
past
and
maybe
they're
common
to
what
you
have
in
your
or
your
own
organizations.
A
We
we
come
from
a
Windows
background,
Microsoft
shop,
kind
of
dist
very
quickly.
Our
our
history
started
back
in
1983,
we
were
a
publishing
company,
we
were
publishing
magazines
and
information
about
genealogy
and
family
history
back
in
1996
is
when
you
look
at
ancestry.com
today
that
the
roots
of
that
really
started
in
1996
when
we
introduced
some
of
the
first
online
family
history
resources
available,
and
at
that
time
we
were
kind
of
an
enterprise
window
shop.
A
So
the
majority
of
our
services
are
written
in
net
and
using
a
sequel,
server
is,
and
just
very
traditional
Microsoft
shop,
and
it
served
as
well.
We,
through
the
time
of
several
years,
also
have
accumulated
some
services
in
Java,
no
js'
Python
running
on
Linux,
but
still
the
majority
of
the
technology
has
been
in
Windows
Microsoft
stuff.
A
We
have
our
own
private
data
center,
it's
we
have
thousands
and
thousands
upon
thousands
of
servers.
We
run
our
services
on
top
of
VMs,
and
so
there
are
literally
thousands
of
VMs
running
on
those
thousands
of
servers
and
we
serve
over
75
million
searches
a
day
for
finally
history
information.
So
it's
it's
been
a
great
site.
A
A
We
were
comfortable
with
ways
of
doing
things
and
in
fact,
in
many
ways,
we
think
that
the
current
way
or
consider
the
old
way
is
a
lot
faster
one
because
we
know
about
it
and
and
two
is
just
we're
just
comfortable
with
them,
and
it's
very
hard
for
engineers
for
us
to
really
see
an
advantage
of
change.
Unless
we
can
really
see
a
very
good,
very
significant,
incremental
benefit
for
that
change
and,
and
the
notion
of
change
in
itself
is,
it
has
real
cost.
A
The
notion
of
just
changing
for
changes
sake
is
great,
but
for
a
business
you
just
can't
do
that,
and
so
change
really
takes
away
from
the
developing
of
new
features
to
bring
in
money
right
and
also
change
elongates.
The
schedules
of
developing
those
new
features
so
anything
that
we
do
as
far
as
change.
There
is
a
cost
to
it.
A
Nothing
is
free,
and
so
when
we
talk
about
adopting
new
new
technologies,
it
requires
kind
of
a
new
paradigm
or
a
new
pattern
to
follow,
and
this
is
kind
of
what
we
have
developed
and
it's
very
common,
but
where
we
try
to
understand
how
to
develop
new
technologies
is
the
first
and
foremost
is
understand.
The
current
technologies
and
processes
there's
nothing
new.
You
can't
change
unless
you
know
what
you're
changing
from
and
so
as
we've
just
talked
about
our
technologies
and
Windows
or
deployment
methodologies
with
VMs.
A
It
is
incumbent
upon
us
if
we
want
change
to
make
sure
that
we
understand
the
current
technologies
understand
how
the
developers
are
doing
things
and
such
the
next
is
kind
of
a
an
interesting
approach.
But
you
need
a
patron.
You
need
somebody
that
can
champion
from
the
upper
echelon
down
to
help
change
occur.
You
need
somebody
to
give
you
air
coverage
and
give
you
the
time
and
the
resources
to
actually
investigate
and
make
these
changes.
A
We
all
think
that
grassroots
is
where
it
all
happens,
and
it's
true
we
do
need
to
have
change
come
from
the
grassroots,
but
if
you,
if
we
don't
have
a
patron
or
some
of
this
champion,
that
can
can
help
with
initiatives
to
actually
drive.
You
know
organization
forward
and
then
having
the
grassroots
and
the
upper
initiatives.
Changed
is
not
successful,
and
so
we
find
that
that
has
been
a
very
important
part
of
our
change
as
well.
The
other
is
you
have
to
own
something
to
be
real.
A
If
you
want
to
change
something,
if
you
want
something
to
be
different,
it
has
to
be
something
that
you
will
do
yourself
if
you're
not
willing,
if
we're
not
willing
to
be
a
part
of
that
change
process
and
we're
all
doing
always
just
trying
to
dictate
to
people.
It's
just
not
going
to
happen,
or
if
it
does
happen,
it's
there's
a
lot
of
resentment
and
and
there's
also
some
frustration.
A
So
the
thing
is
is
always
find
out
how
you
can
be
real
by
owning
and
being
part
of
that
change,
and
the
next
thing
is
is
to
create
a
partnership
that,
if
you're
going
to
make
changes,
make
some
make
sure
there's
a
partnership
with
pilot
teams
and
others
so
that,
if
they're,
successful,
you're
successful
if
they're
not
successful,
you're,
not
successful
and
that's
a
very
key
part
in
adopting
new
technologies.
And
we've
kind
of
we've
tried
to
use
this
pattern
as
we
have
adopted
new
technologies
within
the
industry.
A
So
let
me
just
talk
about
kind
of
how
we
started
in
this
transition
from
a
Windows
VM
world
into
containerization
and
then
to
kubernetes
about
three
years
ago.
In
fact,
it
was
three
years
ago
this
week,
I
attended
a
Q
Khan,
which
is
a
great
conference
I'm,
not
sure
they
wouldn't
gand
at
Q
Khan
before
okay,
Q
Khan
is
a
great
conference
for
architects
and
for
others.
That
just
is
a
very
nice
way
of
just
gathering
together
and
learning
new
technologies
and
not
too
much
vendor
stuff,
which
is
very
nice.
A
Three
years
ago,
I
attended,
Q,
Khan
and
docker
kind
of
was
the
little
buzz.
You
know.
Sometimes
you
go
into
conferences
and
here
little
buzzes
as
to
what's
kind
of
coming
out.
It
was
in
beta
and
it
turned
out
to
be
something
that
a
lot
of
people
were
talking
about.
The
time
I
came
home
from
the
conference
and
my
in
previous
life
I
had
worked
in
software.
Virtualization
have
a
large
background
in
the
UNIX
and
Linux,
and
so
containerization
or
the
concepts
there
and
were
familiar
to
me
and
made
sense.
A
This
is
a
lot
easier
way
to
to
deploy
and
the
scale
at
the
services,
and
so
it
was
kind
of
this
grassroot
type
thing,
and
we
could
then
really
prove
to
people
very
demonstrable.
How
much
faster
this
type
of
methodology
is
compared
to
the
current.
You
know,
go
shift
pipeline
type
of
methodologies
that
we're
using
and
also
a
kind
of
as
a
byproduct
is.
A
We
were
able
to
show
a
greater
density
of
putting
services
on
our
current
computing
resources,
which
is
again
is
a
plus
when
it
comes
from
a
business
standpoint,
and
then
we
again,
we
have
to
own
something
to
be
real.
We
had
our
own
services
that
make
up
some
of
the
aspects
of
the
site,
and
so
we
started
deploying
those
as
well
in
docker
now
what's
interesting
about
all.
This
is
some
of
the
early
discoveries.
Okay.
A
So
this
is
great
new
technology
Paul's
coming
in
Dockers,
wonderful,
let's
change
the
world,
but
there's
always
something
that
you
learn
right.
One
is
found
out
that
really
the
normal
Linux
distros
are
just
way
too
big
right,
they're.
Just
this
way
too
big
to
do
the
kind
of
things
that
we
were
wanting
to
do,
they're
not
specifically
built
for
docker.
You
know
it's
kind
of
an
add-on
to
them,
and
so
it
was.
It
became
difficult
as
we
started
to
use
the
corporate
Blessed
image.
A
You
know
for
Linux
to
then
put
docker
on
and
have
it
work
well
and
as
I
mentioned,
it's
just
way
too
large
of
a
footprint.
You
start
not
utilizing
the
resources
that
you
have
in
a
good
way
with
memory
and
all
the
other
things
excuse
me
and
the
notion
for
security
and
this
operations
is
that
they're
just
way
too
many
packages
to
keep
up
to
date
to
worry
about
security
issues,
and
the
other
thing
is:
is
that
docker
really
needs
later
kernel
versions?
A
They
really
almost
need
that
bleeding
edge
of
kernel
versions
with
the
newer
capabilities
and
such
to
perform
well
as
well
as
to
have
all
the
security
and
other
aspects,
then
they
could
make
it
real.
The
other
thing
that
we
found
out,
given
you
know
it's
nuts?
No,
it's
not
a
brainer,
no
brainer,
but
you
know
we
have
had
a
lot
of
Windows
developers
right,
and
so
we
found
early
on
that.
A
Yes,
we're
so
excited
those
that
knew
about
Linux
and
and
containerization
in
general
that
all
of
a
sudden
these
windows
people
were
going
to
like
Oh.
What
is
bi,
you
know,
what
do
you
mean
command-line?
What
does
that?
What
do
you
have
to
do
that
for,
and
so
we
ran
into
that
notion
of
a
cultural
change?
There's
cultural
change,
not
just
technical,
the
cultural
change
in
having
people
learn
and
adopt
Linux
type
concepts,
and
then
we
found
out
I
know
it's
kind
of
surprising.
Orchestration
is
hard
to
do
right.
You
know
we
yeah.
A
A
We
need
something
that's
built
for
docker
and
so
therefore
core
OS
came
into
into
play.
I
evaluated
several
of
them
and
core
OS
turned
out
to
be
what
I
thought
was
something
that
we
should
focus
on
again.
You've.
You
can
talk
to
the
chorus
people,
you
probably
in
fact
who
uses
core
OS
here:
okay,
wonderful,
it's
built
for
docker,
it's
not
us!
It's
a
first-class
citizen,
you
know
running
containers,
I
should
say
in
core
OS.
The
updates
are
holistic.
A
When
you
I
don't
know.
If
anyone
didn't
went
to
the
presentation
last
night
with
Brandon
Phelps
from
core
OS,
we
talked
about
the
dirty
cow
security
issue,
right,
what's
nice
about
core
OS?
Is
this
notion
of
holistic
change,
meaning
you
can
update
push
the
change
to
the
boxes
and
then
just
do
a
quick
reboot,
and
so
the
nice
thing
is
that
you
can
push
them
to
the
machines,
but
I
think
what's
even
nicer
is
that
if
there's
a
problem,
you
can
quickly
rollback
and
it's
a
holistic
change
versus
this
Deb.
A
You
know
rpm,
you
know,
menagerie
of
making
sure
you've
got
the
right
combo
of
dependencies
and
all
that
type
of
thing.
So
very,
very
nice,
as
far
as
in
the
operations
and
management
and
coming
from
the
UNIX
background,
less
is
more,
it
really
is,
and
so,
as
you
have
fewer
packages,
there
is
a
statistical
reality
that
you'll
have
fewer
vulnerabilities,
and
so
that's
what
we
found
as
well.
So
the
operations
of
the
Linux
underneath
is
is
a
lot
easier
to
manage
and
what's
interesting
about
core
OS.
A
Is
it's
really
bare-bones
when
I
say
less
is
more?
It
is
less
on
core
OS,
which
is
great
because
it's
built
to
run
containers,
and
so
because
of
that
it's
we
found
this
very
infrequent
that
we
have
to
get
on
the
boxes,
because
if
we
do
and
want
to
do
anything,
we
have
we've
got
to
bring
in
a
container
to
run
them
down.
So
in
that
essence,
it's
also
very
easy
to
manage.
Well,
so
we've
got
our
Linux.
We
realize
orchestration
is
hard.
A
Then
I
started
looking
at
some
orchestration
opportunities
there
out
there
and
I
thought
hey
kubernetes.
This
is
this
is
this
is
something
that's
kind
of
cool
and
so
I
started
creating
a
sandbox
cluster
a
cluster,
for
there
are
our
group
that
could
be
playing
with
understanding
getting
familiarity
with
the
concepts
of
kubernetes,
and
so
then
we
created
that
for
our
own
type
of
deployments
and
then
we
were
able
to
after
we
after
we
demonstrated
to
management
that
this
was
a
good
idea.
A
What
containerization
and
to
orchestration
through
kubernetes
we've
got
this
air
cover
to
say:
oK
you've
got
some
time
Paul
and
your
inner
team
get
some
teams
to
be
part
of
a
pilot
program
to
see.
If
they
can
then
utilize
this
technology
now,
one
thing
I
can
say
about
this
is
when
you
do
create
these,
these
pilot
teams,
or
when
you
gather
these
private
pilot
teams,
make
sure
you
find
people
that
are
very
vocal
that
have
problems
hard
problems
to
solve.
A
Don't
go
the
easy
route
find
things
that
are
hard
to
do,
small
enough,
that
you
can
cope
with
them,
but
then
find
people
that'll
really
give
you
honest
and
and
and
and
deliberate
feedback,
and
so
that's
one
thing
that
we
were
able
to
do.
We
had
daily
stand-ups
with
them.
We
became
part
of
their
team
as
well,
and
so
we
started
to
understand
their
processes
understand
what
they
they
do
with
their
needs,
their
their
the
problems
that
they
currently
have,
and
that
was
a
good
thing
to
do.
A
The
other
thing
is
that,
like
the
windows,
training
for
Linux,
with
lil
windows
developers
for
Linux
training,
we
then
had
to
create
our
own
docker,
container,
ization
and
kubernetes
training
program
in
which
we
would
educate
help.
Other
people
understand
more
and
more
about
it.
In
the.
In
the
same
vein,
of
course,
we
came
up
with
some
templates
and
some
scripts
along
the
way
that
became
beneficial
to
us
and
to
these
various
teams.
So
now,
where
are
we
on
this
journey?
A
Ok
we've
talked
about
kind
of
where
we
came
from
ancestry,
kind
of
the
things
that
we've
learned
along
the
way,
and
now,
let's
talk
about
where
we,
where
we
are
in
the
journey,
because
the
journey
is
still
going
on,
I
mean
it's
not
something.
We've
we
finished.
Okay,
we're
still
we're
still
in
the
process.
So
where
we
are
today
is
we've
been
able
to
develop
certain
standards
and
conventions
that
we
feel
that
work
for
us,
and
so
where
we
are
today,
is
we
develop
these
these
standards?
A
And
one
of
these
that's
I,
don't
know
if
it's
that
much
different
from
other
companies,
but
we've
concluded
for
us
at
least
that
there
should
be
a
correspondence
between
a
namespace
and
a
service.
There's
a
one-to-one
correspondence,
and
so
I
know
how
many
use
that
pattern
as
well.
Does
anyone
use
that
as
well
kind
of
so
you
know
that
you
can
do
it
very,
very
various
ways,
but
we
find
that
the
namespace
equals
a
service
or
service
equals.
A
A
namespace
helps
us
partition,
secrets,
config,
Maps
tokens
authorization
at
a
very
service
level,
and
the
namespace
adds
has
that
kind
of
boundaries.
So
we
have
ones
namespace
per
service
and
we
kind
of
came
up
with
a
convention
again
again
the
naming
convention.
How
many
love
naming
discussions
right?
You
know,
what
do
we
call
this
right?
I
can't
tell
you
how
many
naming
convention
discussions
I've
had
over
the
last
couple
of
years,
but
we
finally
said:
okay,
let's
find
the
lowest
common
denominator,
and
that
is
DNS
right.
A
Dns
is
always
the
lowest
common
denominator
as
far
as
what
characters
you
can
have
and
the
conventions.
So
we've
come
up
with
this
convention
internally
that
we
would
have
a
functional
or
group
name
and
then
the
service,
so
it's
just
very
readable
and
that
is
used
in
logging.
That's
used
in
course
the
namespace
itself,
as
sees
just
across
the
board.
So,
there's
this
a
very
nice
correlation
between
the
namespace
service
and
all
of
its
attributes.
A
We
have
also,
as
a
convention,
really
concluded
for
us
that
one
container
per
pod-
no
excuse
me,
you
know
you,
we
all
know
we
can
put
as
many
containers
as
we
want
a
pod,
but
it
all
comes
down
to
what
do
you
want
to
deploy?
What
does
what's
changing
a
lot
and
we
find
that
moat
for
most
of
our
services
they
can
fit
in
a
container.
A
Therefore,
it
should
fit
in
a
pod,
and
so
there's
a
one-to-one
and
again
how
many
do
that
type
of
a
convention
as
well
similar
that
thing,
okay,
and
so
also
what
we've
done
when
we
are
today
is
we've
started
out
with
very
broad
privileges:
okay,
our
developers
can
deploy
to
dev
and
stage
and
even
into
prod
all
the
way.
Okay.
A
So
we've
allowed
that
to
be
very
broad
right
now
and
we've
narrowed
it
as
we've
needed
and
where
we've
narrowed
it
and
as
we've
done
currently
is
that
we
make
sure
the
operations
the
security
team
owns
the
search,
the
username
and
passwords
all
that
into
production,
so
they're
the
ones
that
push
into
there,
but
developers
can
stay
scale
up
can
can
scale
down
they
can
deploy.
They
can,
you
know,
do
whatever
they
need
to
even
in
the
production
environments.
A
Now
will
that
change
I,
don't
know
as
an
engineer
hope
not,
but
if
I
think
the
key
is
is
a
freedom
and
responsibility.
It's
kind
of
a
Netflix
mantra
that
we've
also
like
to
adopt
give
freedom,
but
then
also
make
sure
people
are
responsible.
So
we've
continued
to
do
that.
What
another
thing
that
we've
we've
decided
that
works
well
and
well
for
us
right
now
is
a
notion
of
separate
clusters.
A
We
have
a
cluster
for
dev,
we
have
a
cluster
for
stage
and
we
have
a
cluster
for
production
and
for
us
it's
it's
a
very
nice
way
of
making
sure
that
we're
controlling
the
environment.
We
know
what's
really
in
there
and
we
can
also
then
give
privileges
the
ackles
that
are
needed
for
those
environments
and
so
in
our
production
environment
and
how
we've
we
bring
things
together.
We
we
found
that
we
needed
our
own
private
registry.
A
Workloads
as
well,
but
what
I?
What
I
want
to
say
is
the
question
is:
do
we
do
workload
rating
with
Prometheus,
so
individual
teams
and
their
services
have
another
mechanism
where
they're
doing
application,
metrics
and
monitoring,
but
we
use
it
and
make
it
available
to
everybody
to
understand
how
the
cluster
is
performing
and
also
we
do
provide
for
every
namespace
the
current
CPU
memory
bytes
in
those
type
of
things,
as
a
rudimentary
measurement
of
where
things
are
in
the
cluster
now
logging.
Oh,
please.
A
A
Okay,
yeah.
The
question
is
this
notion
of
a
container
per
pod?
Did
we
have
a
need
ever
to
have
a
sidecar
for
logging
and
such
well?
Thank
you.
Well
I'm,
talking
about
logging
right
now.
We
really
have
tried
not
to
do
sidecars
the
reason
why
is
in
looking
at
them
one
you're,
always
making
sure
that
they're
in
sync,
with
everything
that
they're
working
together,
what
we
did
with
logging
is.
We
have
created
a
custom
wide,
a
cluster
wide
logging
forwarder.
We
actually
use
as
a
starting
point.
A
The
glider
labs
log
spout
project
and
we
have
it
as
a
single
log
for
tour
for
the
cluster.
We
have
our
developers
and
the
services
all
right
everything
out
to
standard
out,
so
there
are
no
longer
files
that
we
use
it's
all
standard
out,
and
so
our
log
forwarder
then
reads
in
this
and
it
hooks
into
the
docker
engine
to
the
actual
log
logging
stream,
and
so
we
then
grab
that
automatically
and
then
we
actually
forward
it
off
to
a
Kafka
cluster
that
the
gate
is
consumed
into
and
they'll
cluster
as
well.
A
Logging
is
a
little
bit
interesting
that
answer
ashame.
Maybe
it's
you
have
a
similar
need
or
our
concept.
We
have
forensic
type
logs,
we
have
business
events
or
type
logs,
and
so
this
logger
also
is
able
to
divide
those
up
and
know
where
they
need
to
go
so.
The
business
analytics
team
can
utilize
the
business
events
and
then
developers
can
use
the
forensics.
A
What
were
the
challenges
or
the
not
having
no
sidecar?
Well
so,
in
this
case,
it's
it's
a
it's
it's
a
log
forward
or
for
the
whole
cluster,
okay,
pods
and
the
sort
of
service
developers.
They
don't
know
anything
about
logging
they
don't
have
to
attach
in
their
deployment
a
sidecar
or
anything
of
that
sort.
A
This
is
proving
out
to
be
a
good
thing
to
do
and
with
the
fluent
D
and
some
other
things
that
are
coming
out,
we
will
track
that
progress
to
see
if
it'll
meet
our
needs.
But,
as
I
mentioned,
we
do
have
some
kind
of
custom
things
that
we
do
with
our
business.
In
forensic
logs,
we
also
created
a
namespace
portal,
so
what
we
have
today
is
a
developer
can
go
in.
They
can
log
in
using
their
LD
open.
A
Excuse
me
there
ad
or
LDAP
credentials
they
can
create
a
namespace,
they
can
add
members
to
it
and
what
this
does
is
because
we
have
all
these
separate
clusters.
You
know
a
dev
stage
and
prod.
This
then
actually
creates
the
namespace
across
all
of
these
clusters
and
also
creates
tokens
for
each
developer,
as
well
as
a
robot
token
that,
then,
that
a
CI
CD
mechanisms
can
use,
and
so
this
is
a
way
of
facilitating
the
creation
of
the
management
of
namespaces,
and
then
we
have
a
authorization
webhook
that
Nathan
created
and
and
keeps
on
maintaining.
A
That,
then
does
this
notion
of
who
has
rights
to
this
namespace
and
does
that
check
as
on-the-fly
and
when,
when
events
come
in,
we
also
have
currently
in
place
resource
quotas,
soft,
in
the
sense
that,
if
ed,
if
a
team
needs
to
bump
them
up,
we
do
it.
But
what
we
found-
and
maybe
it's
not
the
case
in
your
organization's,
but
we
found,
is
unless
we
put
a
quota
for
CPU
and
memory,
the
developers
really
don't
know
in
foremost,
most
cases
what
their
CPU
and
memory
needs
are.
They've
been
on.
A
Vm's
they've
been
on
their
own
host,
they've
they've
had
the
whole
environment,
and
so
they
didn't
know
if
they
were
swapping
or
a
little
bit
of
this
or
if
there's
a
memory,
then
that
would
reboot
after
a
while,
you
know,
and
so
the
key
is
this
kind
of
is
a
forcing
function
to
help
people
come
into
the
environment
and
say
wait
a
second.
Do
you
really
need
all
that
memory?
If
your
Java
do
you
really
need?
You
know
this
I'm
a
CPU,
and
if
they
do,
then
we
bump
them
up
question.
A
We
don't
do
the
minimum
at
per
se;
it
would
definitely
the
maximum.
We
do
it
at
the
course
of
the
namespace
level,
and
so
this
has
been
a
forcing
function.
That's
been
actually
quite
good.
We've
we've
actually
found
bugs.
We
found
things
that
we
would
not
have
found
unless
we've
had
these
these
quotas
and
as
I
say,
if
they
really
need
the
resources,
we
just
then
bump
it
up.
Okay,
just
as
a
side
note
and
I
need
to
talk
to
the
kubernetes
and
actually
I
think
it's
a
docker
issue.
A
If
you
are
using
Java
and
you
are
needing
computational,
you
know
really
really
tight
high
computation
computation,
don't
put
anything
in
there
for
CPU
restrictions.
It
turns
out
that
your
performance
will
go
way
down.
It's
very
interesting,
but
we
did
our
machine
learning,
guys
discovered
that
and
so
for
them
we
don't
put
a
CPU,
even
even
just
this
a
even
if
it's
a
really
high
one,
you
just
don't
put
any
CPU
limit
in
there.
I
think
the
doctor
engine
then
does
a
nice
scene
and
with
CPU
and
Java
justice
has
some
interesting
things
with
it.
A
We
also
created
a
QuickStart
tool,
so
it's
called
Barbossa
kind
of
the
you
know
kubernetes
ship
type
of
theme,
and
it's
really
just
a
QuickStart
tool.
It's
just
to
help
people
to
bootstrap,
especially
when
you
come
from
our
environment,
where
a
lot
of
people
are
coming
from
windows
and
such
getting
into
VI
or
Emacs
is
kind
of
you
know,
maduk,
and
so
we've
created
a
tool
that
allows
them
to
actually
deploy
their
systems
very
readily.
And
what's
nice,
too,
is
it
works
across
all
of
our
various
clusters?
So
they
don't
have
to
worry
about.
A
A
One
of
the
big
things
that
was
a
problem
in
current
environments
is
that
this
deployment
thing
was
kind
of
throwing
it
over
the
fence
right,
you
just
kind
of
throw
it
over
and
let
this
the
the
DevOps
kind
of
get
it
to
blow
it
deployed,
and
when
things
happened,
it
was
kind
of
like
very
opaque.
You
didn't
I,
don't
know.
What's
going
on
and
had
to
get
a
DevOps
person,
we've
tried
to
make
sure
we
don't
get
into
that
world.
Okay,
that
we
do
have
a
tool,
but
this
is
very
transparent.
A
You
can
do
a
dry
run
that
spews
out
all
the
resource,
the
animal
files
you
want-
and
you
can
do
everything
you
want
with
Q
control.
So
whatever
tools
you
have
make
it
as
an
option
not
as
the
blocking
is
to
go
through.
Please
the
tool
actually
does
certain
things.
So,
for
instance,
we
just
have
a
very
small
yamo
file
that
is
very
similar
to
all
the
resources
that
you
would
imagine,
but
it
does.
Does
all
the
service
creations
ensures
that
the
node
parts
are
the
same
across
clusters.
The
high
port
deals
with
the
secrets.
A
What
are
some
other
things
Nathan,
just
just
a
lot
of
those
kind
of
conventions
just
to
help
facilitate
that
type
of
a
thing,
and
then
we've
also
kind
of
created
some
scripts
just
raise
rudimentary.
The
reason
why
I
bring
this
up
is
because
managing
secrets
and
certs
is
kind
of
something
that
people
don't
think
about
right
up
front.
But
what
we've
been
able
to
do
is
we
we've
utilized
annotations
and
labels
within
our
secrets
so
that
you
we
can
do
versioning
of
the
secrets.
A
A
We
also
have
created
a
set
of
scripts
for
backing
up
the
cluster
and
restoring
it.
So
we
we
really
can
bring
up
a
cluster
in
just
a
very
short
period
of
time
and
we
can
actually
then
rehydrate
whatever
was
running
in
this
cluster
into
this
one,
and
we
are
constantly
backing
that
up
every
camera
every
hour
or
every
day
or
I,
think
it's
every
hour,
and
so
it
has
all
the
secrets.
This
has
everything
in
it.
A
It's
kind
of
our
backup
vet
CD,
but
in
a
very
specific
kubernetes
way,
and
then
we
also
have
these
scripts
that
we've
used
to
create
our
environments
and
we're
tracking
very
hard
very,
very
closely.
The
efforts
like
the
QB
admin,
the
cube,
AWS
and
kite
and
cops
we're
really
excited
about
those.
So
we
don't
have
to
our
own
our
own,
but
there
were
some
specific
things
that
they
still
don't
do
or
have
not
done
in
the
past
that
we've
had
to
create
our
own
for
both
on
Prem
and
any
cloud
type
clusters.
A
So
our
journey
so
far,
multiple
clusters,
we
have
dev
stage
prod.
We
have
a
Sox
PCI
cluster.
We
have
a
production
support
cluster
for
back
in
non
critical
but
business-related
workloads.
We
have
clusters
all
over
the
place.
Alright,
we
have
hundreds
and
hundreds
of
namespaces
with
hundreds
and
hundreds
of
pods
running
in
them.
We
have
live
production,
we've
been
in
live
for
about
almost
a
year
now
and
what's
interesting
is
that
we
we
now
have
things
that
are
now
being
developed
for
that
environment,
meaning
the
best
where
they
first
deployed.
A
A
So
we
we
slates,
so
let's
see
if
oh
I
see
what
you're
saying
okay.
So
that's
why
we
have
that
namespace
portal
is
that
then
those
that
are
for
the
Sox
or
the
PCI
cluster
have
only
rights
or
tokens
to
go
into
there
most
everybody
has
it
all
the
way
across
the
various
ones,
but
where
there
needs
to
be
governance
or
auditing
type
things.
A
That's
that's
how
we
can
constrain
that
so
at
the
end
of
this,
I
can
say
that
we
found
that
kubernetes
really
has
an
incredible
amount
of
power
now
as
a
programmer
developer
for
those
are
out
there.
We
always
love
this
repple
environment
right
this.
This
read
evaluate
print
and
loop.
You
know
we
love
this
kind
of
feedback
to
understand
or
to
play
with
in
languages,
and
I
would
contend
that
we
now
have
a
cetyl
meaning.
A
Kubernetes
now
gives
us
that
compile,
deploy,
execute
loop,
that
is
extremely
fast,
is
extremely
capable
now
to
deploy
complex
services
micro
services
in
a
very
much
of
a
iterative
approach
we're
down
to
seconds,
if
not
minutes,
for
deployment.
Now,
rather
than
that
tend
to,
you
know,
50
to
an
hour
type
of
deployment,
and
that's
it
any
any
other
questions
let's
go
over
here.
Please.
A
A
It
seems
to
work
out
well
the
for
instance,
that
we're
related
app
on
the
backend
I
think
is
made
up
of
twentyish
services
namespaces
and
again
it's
kind
of
a
nice
division,
as
they
do
micro
services
and
other
things.
So
again
with
the
name,
size
portal
and
some
other
things
internally.
It
seems
to
work
out
pretty
well,
please.
A
So
what
advantages
that
we
have
seen,
or
whatever,
with
this
notion
or
convention
of
one
pod,
one
container
per
pod,
I-
think
for
us.
The
issue
is
just
basically
what
needs
to
be
updated.
You
know
what
needs
to
really
be
deployed,
what
what
is
a
version
of
something
if
you
start
packing
on
a
whole
bunch
of
stuff,
and
you
have
like
something
that
needs
to
be
updated
here
either
because
of
a
security,
vulnerability
or
something
else.
It
gets
cumbersome
that
you're
having
to
do
kind
of
this.
A
Okay,
so
she
asked
we
are
a
data-driven
company.
So
therefore,
what
do
we
use
for
our
data?
Persistence?
Well,
I
didn't
really
mention
that,
and
so
actually
what
we
we
started
on
at
first.
It
said
those
that
come
in
don't
have
needs
for
persistent
storage,
so
meaning
that
we
do
not
use
persistent
volumes.
A
They
would
use
databases,
offline
storage
mechanisms
outside
of
the
cluster
and
again
for
us,
it
was
easier
to
manage
that
in
the
past
we
started
out
with
100
and
we're
now
in
one
four
one,
one
four
four
soon
one
four
five
and
so
really
the
volume
management
the
security
of
that
has
started
to
come
in,
you
know,
influence
of
being
where
it
needs
to
be,
but
for
right
now
we
just
said
no
use
use,
block,
storage
or
object,
store
or
off
off
the
cluster.
At
least
that's
we've
done.
Please.
A
So
the
question
is,
is
in
our
previous
and
that
we
still
have
both
environments
right
where
we
have
Windows
and
others
that
are
being
deployed
in
two
VMs.
Why
is
it
so
slow
or
why
was
it
so
slow
meaning
to
28
50
minutes?
There
are
various
reasons.
Why
that's
the
case?
A
lot
of
it
was
to
to
actually
prove
the
environment
before
we
put
the
stuff
on
every
time
updates.
A
Just
this
convention
checking
all
those
type
of
things
were
just
it
just
takes
time
and
then
just
the
the
notion
of
bringing
them
up
the
VMS
on
our
on
our
hyper-v
and
VMware
type
of
infrastructures.
Just
take
time,
especially
when
we're
scaling.
So
it's
this.
May
it's
mainly
just
physics
of
how
long
it
takes
to
do
things
to
distribute
things.
That's
really
what
the
issue
is
and
I
think
I'm
Way
over,
and
so
we
need
to
cut
off.