►
From YouTube: How Ancestry Got Kubernetes to Run 2x Better Per Dollar Using AI - Darek Gajewski, Ancestry
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Ancestry Got Kubernetes to Run 2x Better Per Dollar Using AI - Darek Gajewski, Ancestry
Darek Gajewski, Principal Infrastructure Analyst for Ancestry.com, relies on Kubernetes to quickly integrate and deploy applications across Ancestry’s website, which receives 50-million visitors a month, and generates more than a billion dollars in revenue. To get optimum performance out of Ancestry’s cloud applications, Ancestry employed artificial intelligence for continuous optimization of the application runtime environment. AI brings continuous optimization (CO) to the CI/CD process. In a PoC, Ancestry used AI to cut the resources of one application by more than 50 percent, with zero drop in performance. In this instance, Ancestry has been able to get two times the performance out of Kubernetes for every dollar spent. AI-powered CO delivers a well-optimized infrastructure personalized to the workload and delivers better reliability, at higher performance, for much lower costs.
https://sched.co/UaZ5
A
So
I'm
sure
everyone's
getting
a
little
tired,
so
I'll
try
to
get
through
these
slides,
pretty
quick.
There's
a
party
happening
at
6:00
right
so
I'm
here
to
talk
about
continuous,
optimization
I,
know:
I
got
you
guys
on
a
hook
that
was
like
how
ancestry
saved
two
times
in
kubernetes
and
that's
true
and
I'll
get
to
that
as
we
kind
of
dive
into
the
infrastructure
at
ancestry,
so
you'll
get
to
learn
a
little
bit
on
how
we
run
what
our
infrastructure
looks
like
and
and
kind
of
where
we're
going
with
this
continuous
optimization
aspects.
A
So
just
quickly
a
little
bit
about
me.
I
started
my
career
over
at
research
in
motion.
If
you
don't
know
that,
is
we
rebranded
eventually
to
blackberry
kind
of
invented,
the
smartphone
we
played
at
scale
right?
We
had
dozens
of
data
centers
globally,
we
had
dozen
more
network
nodes
globally
from
Singapore
to
Amsterdam
to
North
America
and
I
managed
the
global
capacity
team
there.
After
that,
I
moved
over
to
ancestry,
as
you
obviously
know,
and
there
I
run
the
cost
governance
team.
The
cloud
cost
governance
team,
which
is
responsible
for
changing
the
culture
at
ancestry.
A
A
Well,
we
implemented
some
artificial
intelligence
here
we
work
on
different
tracks
and
different
spaces,
but
yeah
we're
headquarters,
that
of
Lehi
Utah,
and
we
also
have
offices
in
San
Francisco
and
a
few
other
offices
globally.
So
you
know,
if
you're
interested
in
what
we
do
by
all
means
we're
always
hiring
but
you're
here,
because
you
want
to
learn
about
AI.
A
You
manage
these
containers
and
you
want
to
get
the
most
out
of
them
right.
We
all
struggle
with
this
problem
right.
How
much
overhead
do
we
keep?
How
do
you
auto
scale
and
all
that
other
fun
stuff
to
make
sure
that
you're
you
have
100%
uptime,
but
you're
not
wasting
money
in
the
process.
So
what
do
you
do?
Well,
you
watch
graphs
right
or,
to
some
extent,
watch
graphs
to
maintain.
Obviously,
some
capabilities
of
whether
or
not
your
system
has
spikes.
A
You
dampen
those
spikes
when
you
find
those
spikes
or
you
adjust
for
those
spikes
with
extra
capacity.
Back-End
workloads
are
a
little
easier
right.
Your
your
system
goes
up
and
then
it
goes
down.
You
can
scale
through
those
tend
to
be
a
little
easier
to
manage.
But
if
we
get
rid
of
the
engineering
aspect,
what
we
can
do
is
we
can
bring
in
kind
of
a
neural
network,
machine
learning,
sort
of
aspect
to
this
that
both
understands
the
underlying
infrastructure
so
you're,
whether
we're
at
all
in
AWS.
A
So
it's
ec2
and
we
run
on
kubernetes,
so
obviously
our
container
spaces,
and
also
we
do
a
lot
of
infrastructure
as
code
right.
So
we
we
have
more
at
our
disposal
than
just
the
resource
type
that
you're
dealing
with.
So
that
includes
whether
whether
that
socket
connections
to
your
databases
or
Java
heap
size
variables
and
we'll
get
into
a
little
bit
of
that.
A
But
the
idea
here
is
that,
on
a
high
level,
we
connect
our
monitoring
to
the
machine
learning
kind
of
algorithms
and
nai
to
process
what
the
system
is
doing
and
then
make
changes
to
the
system
at
faster
rates
than
humans
can
write
like
you
can
do
it.
There's.
No
there's
no
issues
here
with
any
engineer
doing
this,
except
that
our
problem
at
ancestry
is
that
we
have
hundreds
upon
hundreds
of
applications
that
talk
to
each
other.
A
We
have
80,
plus
agile
teams
at
ancestry,
500
plus
developers
and
I
can't
scale
this
with
my
team
of
four.
So
we
look
at
ways
to
optimize
large
systems
without
having
to
you
know,
embed
ourselves
into
these
teams
and
teach
them
these
practices.
What
we're
trying
to
do
is
get
in
front
of
it.
Allow
them
to
install
these
install
this
application.
We
use
an
application
called
up
tune.
A
It's
from
UPS
Ani
that
basically
embeds
itself
into
our
infrastructure
and
allows
we
allow
it
to
make
changes
to
little
bits
of
our
infrastructure
and
I'll
get
into
the
details
there.
But
the
idea
here
is,
of
course,
if
you
can,
if
you
can
get
the
system
to
understand,
what's
happening,
you
can
then
reduce
the
footprints
and
double
your
performance,
and
by
virtue
of
that
it
decreased
the
amount
of
capacity.
You
need
right,
and
so
that's
that's
in
a
nutshell.
What
I'll
talk
about
and
I'll
layer
in
kind
of
ancestry
and
what
we
do?
A
If
you
don't
know
what
ancestry
does?
We
are
a
genomics
company
where
that
does
family
trees
along
with
DNA,
so
you
might
have
seen
some
of
our
commercials
we're
big
in
the
65
plus
year
Nagori,
but
if
your
grandma's,
on
it
eight
hours
a
day,
the
younger
crowds
a
little
off
but
but
we're
working
on
that
and
we
have
a
lot
of
mobile
stuff
in
the
pipe.
But
this
is
a
representation
of
our
infrastructure.
It's
a
logical
infrastructure,
a
logical
representation,
but
I
tried
to
draw
it
in
such
a
way
that
it
makes
sense.
A
One
is
its
representative
of
the
size
of
the
components
we
have
and
it
allows
me
to
drill
down
into
the
component
levels
so
that
when
we
have
conversations,
if
you
have
questions
about
how
we've
implemented
this,
we
can
get
to
the
aspect.
So,
on
the
one
side,
you'll
see
I
built
a
little
bit
of
a
gradient
and
we'll
pull
back
a
little
bit
on
our
infrastructure.
Here,
if
the
front
facing
wall
there
on
that
side,
is
our
Lao,
our
security
team
right
think
you
tend
to
keep
us
safe.
A
It's
the
level
ever
watchful
eye
of
Sauron
Tony.
Our
CEO
would
not
appreciate
that
joke,
but
inside
we
have
tons
of
services
right,
I,
don't
know
how
any
of
you
are
laid
out,
but
at
the
end
of
the
day
we
have
something
like
50
services.
We
maintain
and
they'll
get
a
little
bit
into
that
as
I
pull
away
our
front
end.
So
the
front
end.
There
was
kind
of
obviously
our
ancestry.com.
We
also
have
other
properties
within
ancestry,
like
newspapers,
calm
and
and
full
three.
A
We
do
collections
work
right,
and
so,
when
we
do
collections,
we
have
some
of
our
content
pipelines.
We
won't
talk
about
that.
We
have
commerce,
we
have
data
warehousing,
we
have
data
science
teams
and
we'll
pull
some
of
those
aspects
away,
because
that's
not
what
we're
talking
about,
but
rest
assured
that
we
spend
millions
within
kubernetes
and
even
saying
that
I'm
not
allowed,
but
we
have
lots
of
money
to
be
saved
within
the
space,
because
when
we
take
a
look
at
our
overall
infrastructure,
we
we
know
we
don't.
A
We
don't
have
90%
CPU
utilization
as
we
wish
we
all
did,
but
we
know
that
there's
room
to
save
it's
just
that
as
we
dive
into
the
system.
You'll
notice
that
there's
a
ton
of
complexity,
some
of
our
larger
systems
are
actually
some
of
the
systems
that
I'll
dive
into
one
is
called
DNA
matches
and
that's
the
that's
the
POC
we
did
but
I.
A
On
top
of
that,
we
have
a
few
systems
called
big
tree
and
trees
and
stuff
that
these
are
systems
on
the
backend
that
help
customers
obviously
build
their
family
trees
and
then
systems
that
obviously
do
DNA,
processing
and
Stitch.
Some
of
those
things
together
right.
We,
our
bread
and
butter,
is
not
that
we
do
contents,
it's
that
we
do
contents
and
we
stitch
that
content
with
other
content
like
genomic
data
and
user-generated
content,
and
that
data
is
constantly
living
in
this
space.
And
that's
why
we
have.
A
We
have
a
fairly
complex
system,
even
though
we're
not
at
the
scale
that
you
know,
Blackberry
or
Google
or
Facebook,
tend
to
act,
but
we
do
have
exponentially
growing
systems
or
near
exponentially
growing
systems
right
you
match
to
your
cousins
through
generations,
and
we
go
as
deep
as
your
eighth
cousin
and
if
you
can
imagine
deep,
an
eighth
cousin
match
is
basically
almost
everyone
right
like
now.
We
tend
to
not
go
down
that
far,
but
that's
that's
the
science
in
it
right
we
can.
A
We
can
actually
figure
out
where
you
match,
with
cousins
down
to
generational
lines.
So
as
we
pull
away
some
of
those
back-end
systems,
we
get
down
to
our
our
our
DNA
matches
that
the
top
box
there
will
actually
represent
our
operational
box,
our
operational
systems.
So
that's
logging
monitoring
our
CI
CD
pipeline.
All
the
things
that
we
tend
to
standardize
and
try
and
generalize
on
will
be
the
top
of
that
that
box
and
as
we
pull
that
away,
we
end
up
with
a
service
called
DNA
matches.
A
This
system
grows
at
near
exponential
rates,
so
much
so
that
at
our
scale,
even
though
we're
we're
only
a
billion
dollar
company,
you
know
we
have
complex
complexity
here
that
rivals
Facebook
right
Facebook
has
2.3
billion
subscribers
with
an
average
of
300
plus
friends.
They
have
less
connections
than
we
do
in
that
DNA
matches
database.
A
So
as
we
pull
away,
our
infrastructure
will
focus
just
on
DNA
matches
it's
a
simple
sort
of
system
at
the
infrastructure
level,
it's
complex
at
the
science
level,
but
we
have
a
database
back-end
and
storage
with
some
compute
to
shard
that
data
keep
it
in
memory,
make
it
searchable
and
accessible
to
customers,
and
then
we
have
our
kubernetes
applications.
On
top
of
that,
then
we
bring
in
one
more
thing:
we
bring
in
the
from
the
infrastructure
piece
we'll
bring
in
our
CI
CD
and
monitoring,
Suites
and
logging.
A
This
is
the
the
matches
portion
of
the
application,
and
this
is,
it
doesn't
do
any
ingesting
of
large
systems
at
what
it
does
is
it
takes
a
request
and
sees
what
a
new
user
is
and
who
they're
matched
to
changes.
Those
in
the
tables
then
changes
those
cousins
and
then
those
people
and-
and
so
it's
processing
constantly.
As
we
add
subscribers
to
the
system
now,
we
run
obviously
kubernetes
on
with
this
application.
A
But
what
we
want
to
do
is
we
want
to
pull
away
some
of
that
information,
because
it's
not
prudent
to
this
current
discussion,
so
we'll
pull
back
and
just
focus
on
this
system.
So
we
have
containers
or
pods
within
this
application
and
we
want
to
go
back.
We
want
to
go
one
level
deeper,
and
that
is
to
the
kind
of
atomic
level,
and
that
is
one
pod
right,
and
so
what
we
do
is
we
actually
pull
out.
A
We
create
a
canary
right,
I,
I'll,
assume
that
most
most
companies,
or
most
of
you
will
have
some
sort
of
deployment
pattern.
We
typically
about
half
our
teams
to
use
a
use,
a
Bluegreen
deployment
and
half
our
teams.
Use
kind
of
a
canary
smoke
deployment
depends
on
the
size
of
the
system
and
how
it
arose
iliyan
it
needs
to
be
and
how
how
those
dev
teams
have
approached
it.
A
We
have
adopted
a
DevOps
mentality,
though
we
do
have
problems
in
the
space
where
some
teams
do
it
one
way,
some
teams
do
it
another
way,
and
so
we
kind
of
have
to
figure
out
multiple
ways
to
solution.
Something
like
this.
So
we
use
a
canary
load
right.
What
we
do
is
we
connect
it
to
our
CI
CD
platform.
We
use
AB
Sahni
to
to
gather
the
the
metrics
for
it.
A
Whether
that's
zabbix
said
we
use
internally
or
another
APM
solution,
I'm
not
going
to
mention,
but
and
and
we
monitor
the
canary
and
then
from
there
once
we
have
that
we
set
up.
You
know,
secrets
management
so
that
it
can
make
changes
to
the
canary.
We
actually
do
this
in
production,
since
we
have
a
hard
time
with
some
of
these
systems
doing
load
performance
tests
due
to
the
fact
that
if
we
had
a
load
performance
test
on
this
system,
we
would
just
literally
have
to
duplicate
the
entire
application.
A
So
we
just
do
it
in
the
in
production.
Now
what
we
can
do
is
actually
move
traffic
from
the
cluster
and
put
a
little
bit
into
the
canary
right.
So,
if
you're
running
ten
ten
pods
will
extract,
one
have
nine
will
move
ten
percent
of
our
traffic
into
the
canary
90
percent
to
the
rest,
and
we
create
a
baseline
right.
So
now
that
we
have
a
baseline,
we
can
get
the
machine
learning
to
work
for
us
and
we
can
a
suit.
A
We
can
then
start
understanding
what
changes,
what
effects
the
changes
have
on
the
canary
against
our
baseline
and
that's
what
we
do
we
make
changes
as
in
and
when
I
say
we
make
changes,
we
actually
let
the
artificial
intelligence
start
making
these
changes
right.
So
the
changes
happen
iteratively
and
multiple
times
an
hour.
We
do
40
to
50
changes
to
the
canary.
Within
a
day.
The
system
monitors
the
system
adjusts
and
the
system
recommends
paths
within
a
set
of
variables
that
we
give
it.
A
We
typically
the
PLC,
started
with
two
main
variables:
right,
that's
CPU
and
memory,
and
then
from
there
we
actually
added
other
variables
to
the
system
and
I'll
get
that
in
a
second
but
yeah.
We
added
things
like
socket
connections,
right
thread,
counts,
Java,
heap,
size,
garbage
collection,
we're
this
applications
on
Java
and
and
so
we're
able
to
actually
tune
more
than
just
the
resources
within
this
space.
We
we
let
the
the
moai
model
take
over
and
actually
start
tuning
multiple
variables
in
unison
right.
A
A
You
could
be
constrained
by
connections
right.
There's,
there's
other
reasons
that
you
have
90%
CPU.
So
that's
not
necessarily
the
most
efficient
path.
You
could
potentially
gain
more
performance
by
changing
other
things
and
that's
what
we
do.
We
add
more
connections,
is
that
that
does
that
have
downstream
effects,
or
is
it
just
increased
performance
and
we
find
that
obviously
increase
performance
for
us,
and
this
is
the
application
that
we
POC
the
system
on
right.
A
There
are
initial
POC
found
that
we
got
just
over
two
times
performance
out
of
the
system
and
we
were
able
to
scale
back
that
application
save
thousands
of
dollars
a
month
right
now.
We
extend
that-
and
we
have
been
doing
this
over
the
last
few
months,
and
actually
we
continue
to
do
this
while
I
speak,
and
that
is,
we
create
more
Canaries.
We
cover
more
of
the
applications
and
we
continue
the
process
right
now.
A
Now
you
know
this,
isn't
the
only
thing
we
do.
We
obviously
have
to
do
that,
like
it
doesn't
solve
every
single
problem
for
us,
like
it,
doesn't
understand
the
entire
architecture.
On
the
back
end,
it
doesn't
know
what
the
algorithms
are
being
used
and
the
complexity
in
the
science
doesn't
go
away.
We
still
have
to
work
with
our
scientists
and
engineers
to
try
and
resolve
like
guys.
This
is
going
to
be
petabytes
of
data
and
active
RAM
soon.
A
If
we
don't
do
something
about
it,
and
so
we
figure
out
like
what
needs
to
go
in
the
backend
storage.
What
can
get
processed
once
a
week
versus
every
day
and
try
and
reduce
our
overall
capacity
that
way
as
well,
but
in
this
case
we're
a
la
we're
allowing
a
system
to
do
the
little
things,
because
we
have
hundreds
of
little
things
right
and
every
time
I
argue
with
our
engineer
and
development
groups.
They
go
Derek,
that's
$5!
Why?
A
Why
do
I
care
right
like
because
I
go
because
at
scale
I
have
$5
times
a
million
and
I'm
spending
millions
of
dollars
on
this
stuff
and
I
have
to
argue
with
500
developers
why
the
$5?
Actually
matters
right.
So
I
don't
want
you
to
think
that
like
well,
you
know
you
did
a
small
thing.
I
was
like
yeah
the
small
thing,
but
now
we
now
we're
figuring
out
how
to
extend
this
and
make
this
approachable
to
our
entire
organization
and
I
have
a
couple
guys
in
the
back
that
are
going
to
be.
A
Our
container
developers
came
and
hi
they're
going
to
be
helping
us
onboard
the
hundreds
of
developers
and
stacks
that
we
have
to
make
use
of
this
service
so
that
they
don't
have
to
do
the
thing
that
I'm
making
them
do,
and
that
is
save
money,
but
do
the
things
that
product
is
asking
them
to
do,
and
that
is
build.
Awesome
features
because
at
the
end
of
the
day,
I
love
what
we
do,
but
we're
not
always
giving
them
the
time
to
do
the
awesome
things.
A
A
A
We
saved
you,
know
half
the
infrastructure,
but
you
know
overall,
in
some
of
the
systems
we've
spent
tons
of
times
optimizing,
we're
gonna
have
a
little
bit
of
less
return,
but
overall,
that's
still
hundreds
of
thousands
of
dollars
that
we've
been
managing
to
get
out
of
a
handful
of
applications.
Now,
that's
not
where
the
magic
stops.
That's
just
kind
of
where
we're
at
right.
Now.
The
next
step
for
us
is
a
2020
plan
to
start
talking
about
actually
auto
scaling
and
tuning
auto
scaling.
A
The
biggest
problem
we
have
this
year
is
that
we
moved
into
the
cloud
and
we
spent
two
years
optimizing
learning
the
system,
understanding
how
to
deploy
infrastructure,
all
that
fun
stuff
and
we're
all
better
for
it.
But
the
complications
have
not
gone
away
and
in
fact,
like
I,
said
they've
gotten
more
complex,
because
we
now
have
more
systems.
We've
saved
a
ton
of
money.
Our
infrastructure
year
over
year,
we've
been
able
to
shrink
it
at
about
40
percent,
but
we
have
diminishing
returns
in
that
space
right.
A
You
can
squeeze
the
lemon
a
lot
when
you
first
get
into
the
cloud
and
anybody
who's
done.
A
migration
into
the
cloud
will
know
that
you
probably
have
very
expensive
infrastructure
at
first
and
then
you
find
ways
where
there's
wastage
and
the
easy
things.
So
we
we
did
the
low-hanging
fruit
and
now
we're
really
focusing
on
auto
scaling
and
making
sure
that
our
system
behaves
in
unison
with
the
workloads
that
are
coming
in
and
out
of
them
right.
A
So
we
have
front-end
teams
and
we
have
back-end
teams,
but
some
of
that
some
of
those
systems
that
do
continue
to
talk
to
each
other
and
we
want
to
make
sure
that
as
customers
come
in
we're
approaching
our
holiday
season,
which
is
our
biggest
kind
of
uplift
of
traffic
at
ancestry.
We
want
to
make
sure
that
the
system
can
scale
right
and
when
it
needs
to,
we
didn't
want.
We
don't
want
to
warm
up.
A
You
rather
it
be
able
to
scale
as
needed
with
the
demands
and
that's
what
we're
going
to
try
and
in
the
next
year
and
maybe
I'll,
come
back
and
kind
of
do
a
talk
on
how
our
auto
scaling
orchestration
has
worked
and
how
we're
able
to
tune
auto
scaling
without
the
engineers
having
to
be
involved
as
well
right.
We
struggle
with
that
today,
like
I
said
engineers
know
about
auto
scaling,
but
they're
not
comfortable.
They
don't
know
how
to
predict.
A
They
don't
know
what
to
look
at
for
prediction
and
they
don't
know
where
to
scale
and
when
the
scale
and
understand
things
like.
Oh,
my
provisioning
time
is
three
minutes,
so
I
need
this
much
overhead.
So
we're
gonna
work
on
that
and
we're
going
to
create
more
like
machine
learning,
algorithms
to
watch
that
space
and
then
start
tuning
that
in
unison
as
well.
So
you
know
as
I
bring
back
everything
you
know.
That's
that's
in
a
nutshell,
but
my
talk,
our
implementation.
A
How
am
I
for
time,
okay,
okay
and
so
I.
We
have
Paul
here
and
I
have
a
couple
devs
with
me
as
well.
So
if
you
have
any
questions
about
this,
please
feel
free
to
talk.
I
know
this
wasn't
a
very
technical
discussion.
Cube
Khan
is
usually
a
pretty
developer
heavy
open-source
heavy
sort
of
place.
But
this
is
kind
of
a
complicated
problem
for
us
that
that
at
scale
it
tends
to
plague
us.