►
Description
Art and Science of Optimizing Your Cloud Native Java Apps - Siva Balan, GE
Cloud native apps are a beast when it comes to optimization and troubleshooting. There is no silver bullet to achieve them and it requires a multi-dimensional approach. This session will go through some of those approaches, its pros and cons and how to effectively optimize the bits running on CF. It will also showcase some real life use cases of optimization and troubleshooting techniques that helped GE Digital to run its Java apps effectively on Cloudfoundry.
About Siva Balan
Siva has been doing performance engineering for the past 15 years starting with 3-tier architecture to cloud native architecture. Since joining GE Digital 4 years ago, he has been doing performance engineering for apps and services deployed on Cloudfoundry. He has presented at the CF summit 2016 at Frankfurt on "Performance Engineering of services deployed on CF". Current projects include performance engineering of Java apps and services deployed to CF on Azure, DB performance engineering and using Splunk as a dashboard for Performance test results.
A
Hello,
welcome
to
this
session
of
art
and
science
of
optimizing
cloud
native
java
applications.
I'm
sylva
balan
I
work
for
G
digital
I'm,
a
performance
engineer
with
in
the
security
team.
G
did
G
digital
before
we
start
I
think
we
have
to
go
over
the
fire
code
announcement
and
here
it
is
I'm
sure
you
to
memorize
by
now.
So
let
it
be
there
for
a
couple
of
seconds.
A
So
in
this
session,
essentially,
we've
been
running
quite
a
few
services
Java
services
on
cloud
foundry
for
the
past
three
years,
and
we
have
learned
a
lot
of
lot
of
ways
to
deploy
and
not
to
deploy,
run
and
not
to
run
java
applications
on
cloud.
So
this
session
is
essentially
to
give
you
some
idea
as
to
the
lessons
that
we
learned
on
how
we
optimize
our
java
applications
when
we
ran
or
when
you're
running
it
on
cloud
foundry.
So,
let's
dive
in
as
to
what
we're
going
to
look
at.
A
A
You
know
not
having
dependencies
when
the
application
is
coming
up
or
when
it's
actually
running
in
the
cloud,
and
it
makes
very
easy
to
monitor
and
I'll
talk
about
a
little
bit
later
and
how
easy
it
is
to
monitor,
and
we
also,
it
also
make
sure
the
app
lifecycle
is
taken.
Care
of
your
health
checks
are
taken
care
of.
If
your
application
crashes,
that
brings
up
worry,
brings
up,
brings
it
up
very
quickly
and
effectively
as
well,
and,
of
course,
the
health
checks
and
restarts
are
done
automatically
for
you.
A
So
just
a
background
on
what
cloud
foundry
offers
is
to
deploy
our
java
applications
as
cloud
native
java
applications.
Now,
how
do
we
know
that
your
application
is
optimized
to
run
on
the
cloud,
so
some
of
the
things
are
I
would
call
it
a
checklist
that
we
typically
look
for
is
how
is
it
performing
in
the
cloud?
Is
it?
A
Is
it
receiving
the
request
that
it's
supposed
to?
Is
it
processing
in
the
prior
mount
of
time?
How
is
the
response
time
looking
like
and
how
is
your
containers
behaving
when
the
application
is
running
on
the
cloud
and
how
was
your
garbage
collection,
so
it's
best,
especially
when
you
run
java
applications
in
the
cloud.
One
of
the
key
things
to
look
for
is
how
effective
are
garbage
collections
further
down
on
the
slides.
I'll
show
you
a
couple
of
scenarios
where
we
ran
into
some
very
interesting
scenarios
in
in
terms
of
garbage
collections.
A
So
how
is
it
connection
pools
managed
when
I
talk
about
connection
pool
specifically
I'm,
referring
to
your
database
connection,
pooling
I'm
sure
most
of
the
applications
will
talk
to
some
type
of
a
database
on
the
backend,
so
managing
connection
pools
and
database
was
a
big
challenge
for
us,
so
I
will
tell
you
why
as
well
and
how
your
threat
pool
managed.
So
when
you
talk
about
java
applications,
we
have
to
talk
about
threat
poles.
How
do
you
effectively
manage
your
threat
poles?
A
How
do
you
optimize
the
track
poles
to
run
in
on
the
cloud
and
how
is
it
container
resources
used?
So
we
typically
don't
especially
developers.
Don't
give
a
lot
of
thought
about
how
the
container
resources
is
managed
when
they
deploy
their
java
applications
to
the
cloud
so
but
from
a
performance
near
standpoint,
I
have
to
one
of
the
things
I
look
for
is
how
effectively
are
the
resources
managed?
How
is
your
CP
realization
on
the
containers?
A
How
is
your
memory
utilized-
and
we
don't
have
to
worry
about
this,
because
these
are
ephemeral
disk,
so
we
typically
don't
store
anything
in
the
disk,
so
but
memory
and
CPU
are
something
that
we
need
to
watch
out
for
now.
Let's
take
it
one
by
one
in
terms
of
app
performance,
so,
first
and
foremost,
I
always
say
to
every
service
owner
is
that
never
fly
blind,
always
always
have
a
monitoring
tool
in
place
for
it
to
monitor
your
service.
In
our
case,
we,
you
can
pick
anything
of
your
choice.
A
New
Relic,
AppDynamics
dynaTrace
there
quite
a
few
of
the
tools
that
are
available
there
easily
lets.
You
bind
the
monitoring
tool
to
your
service
in
Platt
foundry.
They
have
service
brokers
that
are
available
for
you
to
offer
as
well
so,
but
please
do
make
sure
that
your
service
is
monitored
right
from
the
time
it's
deployed
and
we
have
to
collect
some
key
metrics.
This
can
either
be
done
through
the
monitoring
tool
or
you
can
have
your
own
scripts
or
services
or
apps.
That
would
actually
collect
the
services
for
you.
A
Some
of
the
key
things
that
I
would
suggest
is
heap
usage.
How
is
it
being
utilized
by
your
application?
How
is
the
garbage
collection
pattern
looking
like
and
how
is
your
threads
use
it
so
how
many
number
of
threads
are
being
used,
how
many
active
threads,
how
many
idle
threads?
So
these
are
some
key
metrics
you
need
to
collect,
irrespective
of
whether
your
application
is
running
fine
or
not
running
fine
for
that
matter,
but
I
would
say
just
start
with
these
basics
metrics.
A
That
needs
to
be
collected
now,
just
to
give
you
a
background
on
what
the
garbage
collection
should
look
like
and
what
it
should
not
look
like.
So,
in
our
case,
this
is
a
specific
screenshots
of
one
of
our
services
so
which
one
do
you
think
is
more
optimal,
one
on
the
left
or
one
on
the
right.
So
I
would
say
one
on
the
left
right.
So
essentially,
essentially
it's
the
same
service,
but
using
different,
build
packs.
A
So
these
are
some
things
you
need
to
watch
out
for
when
you're,
when
you
deploy
applications
to
the
cloud
and
just
to
give
you
idea
the
same
slide
that
we
saw
before,
which
is
this
one.
You
can
see
the
garbage,
the
full
garbage
collection
happens
very
periodically
in
the
first
one,
but
the
second
one.
You
can
see
that
it's
off
it's
occurring
more
often,
and
it's
also
taking
more
and
more
of
your
CPU
cycles.
So
you
can
actually
see
the
garbage
collection
in
terms
of
the
percentage
of
CPU
that's
being
utilized.
A
So
you
can
see
that
you
don't
want
your
application
to
be
something
on
the
right,
so
you
need
to
be
something
on
it.
The
pattern
should
be
something
similar
to
what
you're
seeing
on
the
left,
and
this
might
not
be
very
evident
when
you're
actually
running
your
application
on
your
laptop
or
on
your
you
know,
local
environment,
so
you
would
start
in
most
cases,
you'd
start
seeing
this
behavior
once
you've
deployed
your
application
to
the
cloud.
A
Now,
as
I
said,
we
need
to
look
at
the
frequency
of
the
garbage
collection,
especially
the
full
garbage
collection,
and
we
also
need
to
be
very
conscious
about
how
long
it
takes
for
the
garbage
collection
thing
to
to
kick
in
one
of
the
reasons
I'm
saying
this
is
because
it
also
depends
on
how
you
size
your
heap
when
you
deploy
your
application
as
well.
You
don't
want
your
heap
to
be
too
large
or
it
shouldn't
be
too
small.
A
Okay,
in
this
case
now,
you
also
need
to
look
at
how
how
many
objects
are
being
collected
in
each
of
the
collection,
and
you
need
this
to
be
very
consistent
across
the
time
interval
that
you're
looking
at,
let's
say:
you're,
looking
at
your
service
for
the
past
24
hours.
How
is
your
garbage
collection
looking
like?
A
So
if
you
have
time
at
the
end
of
the
session,
I'll
show
you
a
live
graph
of
how
the
garbage
collection
looks
like
or
a
period
of
24
hours
or
three
days,
so
you
get
an
idea,
but
but
be
conscious
about
how
many
objects
are
being
collected
as
well
database
connection
pool
management.
So
this
is
another
key
factor
that
you
want
to
be
conscious
about
as
well,
because,
especially
when
you're,
if
your
application
is
using
springboard
the
spring
boot
reconfiguration,
whether
you've
noticed
it
or
not,
automatically
we
can
figure.
A
It's
your
max
active
connections
to
four
I'm,
not
sure
why
it
does
that.
But
this
has
been
a
very
painful
problem
for
us
and
this
happens
respective
whether
your
application
is
a
production
application
or
a
development
application
that
you
push
to
the
clock.
So
one
of
the
things
we
had
to
do
in
our
springboard
applications
is
that
we
had
to
override
this
Auto
reconfiguration
and
actually
set
the
max
actor
values
ourselves
and
and-
and
please
watch
out
for
this,
so
we
have
been
in
situations
where
we
ran
out
of
Debbie
connections
for
no
apparent
reason.
A
Another
things
another
thing
to
look
out
for
is
you
want
to
size
your
database
connections
of
your
java
applications
based
on
the
number
of
instances
you're
going
to
run,
for
example,
your
database.
Your
database
itself
has
a
fixed
number
of
connections,
client
connections
that
can
accept.
Typically,
if
you
stake
Postgres
on
my
sequel,
you
probably
have
about
hundred
maybe
five
hundred
or
thousand
connections
at
the
most.
Let's
say
you
size,
your
database
instance
connections
to
say
twenty
five
connections
per
instance.
A
Now,
when
you
start
say
ten
instances,
you're
looking
at
250
connections,
you
start
training
instances
your
500
and
at
40
instances
you're
maxing
out
the
number
of
connections
that
the
database
can
accept.
So
when
you
size
the
database
connection
pooling
you
need
to
be
aware
of
what
the
max
connections
are
or
what
the
min
connections
are.
So
the
min
connections
are,
the
minimum
connections,
are
the
ones
that
it
actually
starts
your
application
with.
A
So
typically,
what
we
do
is
we
have
machination
max
connections
around
25
to
50
min
connections,
probably
around
five
at
the
most,
because
your
application
shouldn't
be
using
a
lot
of
database
connections.
If
it
is
a
transactional
application,
it
probably
would
use
one
to
two
connections
at
any
given
time,
so
if
it
actually
uses
more
than
that,
then
it's
there's
there's
a
problem.
You
need
to
look
at
so
what
kind
of
problems
might
come
up
as
you
act
as
you're,
actively
monitoring
idle
and
active
connections?
A
So
that's
typically
what
you
need
to
look
at
anything
that
takes
more
than
hundred
milliseconds
I,
would
watch
out
and
look
and
make
sure
if
I
can
tune
them,
and
there
might
be
some
situations
where
your
queries
might
have
to
run.
Longer.
That's
perfectly
okay
and
and
and
I
just
want
to
make
sure
I
mean
what
I
want
to
emphasize
is
that
you
want
to
size.
Your
database
can
pulling
based
on
how
your
queries
are
going
to
perform
and
have
make
sure
that
long-running
queries
are
tuned.
A
The
next
one
to
look
out
for
is
track
pool
management.
These
are
specifically
the
Java
threads,
that
your
continued,
that
a
tomcat
or
a
jetty
container
that
you
deploy
to.
How
much
does
so.
You
need
to
be
make
sure
you
need
to
understand
what
the
default
number
of
threads
that
your
container
comes
up
with
and
how
much
are
the
active
threads
and
what
the
idle
threads
are.
A
So
typically,
if
you
deploy
your
tomcat,
your
application,
deploy
on
a
tomcat
container,
you
would
see
about
200
threads
I
by
default,
so
unless
you
go
ahead
and
tune
them,
200
us
is
what
the
default
number
is,
and
that's
perfectly
fine.
But
what
you
need
to
watch
out
for
is
again
monitor
how
many
threads
are
being
act
or
active
and
how
many
times
are
idle.
So
you
want,
as
less
number
of
threads,
to
be
active
as
possible
at
any
given
time,
because
you
threads
are
really.
A
They
should
be
short-lived
in
the
sense
that
your
thread
should
not
be
held
held
on
for
a
long
time
when,
especially
when
you're
running
transactions,
having
large
number
of
idle
threads
is
perfectly
okay.
It
just
takes
up
a
memory
but
but
make
sure
that
it's
it's
not
so
so
one
of
the
things
to
look
for
and
in
terms
of
idle
threads,
which
I've
not
mentioned
here,
is
the
is
a
stack
size.
So
there's
an
XSS
parameter
that
typically
gets
allocated
when
your
java
application
is
pushed,
and
this
parameter
is
allocated
on
per
thread
level.
A
The
other
thing
I
would
look
for
again
using
your
modeling
tool
to
see
how
your
thread
usage
pattern
is
so
maybe,
when
you're
looking
at
the
past
24
hours
or
48
hours,
how
is
your
thread
being
used?
How
many
active
threads
or
when
does
it
go
up?
When
does
it
come
down
or
is
it
allocating
more
than
200,
or
is
it
requiring
more
than
200
threads
by
default?
A
So
these
are
some
of
the
thread
usage
pattern
that
you
want
watch
out
for
if
you
want
to
optimize
your
application
running
on
in
the
cloud
now
one
of
the
things
we
also
come
across
a
stock
thread,
so
I
didn't
many
of
you
have
been
as
well,
and
a
lot
of
these
monitoring
tools
gives
you
options
to
do
thread
dumps
and
I
think
if
you're
using
spring
actuator.
It
also
gives
you
end
points
where
you
can
take
thread
dumps
and
heap
dumps
as
well,
so
make
use
of
them.
A
I
mean
there
are
tools
out
there
that
helps
you
look
into
your
application
and
how
and
lets
you
know
how
it's
performing
so
I
would
watch
out
for
struck
threads
and
make
sure
that
it's
especially
when
you're
looking
at
thread
dams,
you
probably
want
to
look
at
taking
multiple
thread
dumps.
Let's
say
a
5-minute
interval
or
one
minute
interval
and
see
how
the
threads
are
moving
or
they
even
processing
something.
Is
it
moving
or
is
it
just
stuck
at
one
place?
A
So
those
would
give
you
some
idea
as
to
how
the
threads
are
moving
around,
how
it's
being
processed
and
the
other
thing
is
container
resource
management.
So
this,
where
this
comes
up,
is
that
mostly,
when
developers
push
their
applications
to
cloud
foundry,
they
are
not
very
much
concerned
about
how
the
container
being
the
resources
of
the
container
being
used.
So
this
mostly
resides
either
on
the
operators,
are
performance
engineers
or
actually
looking
at
the
performance
of
the
application
itself.
A
So
some
of
the
key
things
we
look
for,
we
should
be
looking
for
when,
when
pushing
an
app
running
an
app
in
production
as
well,
is
you
prop?
You
want
to
be
able
to
monitor
the
CPU
and
memory
usage
of
the
container?
This
can
be
either.
Typically,
when
you
do
a
CF,
app
and
list
out
all
the
instances,
you
would
be
able
to
see
the
numbers
there
are
again
most
of
monitoring
tools
will
give
you
as
part
of
the
jmx
metrics,
your
CP
realization
and
a
memory
utilization
as
well.
A
It
may
not
give
you
the
actual
container
memory
itself,
whether
the
jmx
metrics
would
probably
give
you
just
the
heap
metrics
which
the
heap
is
being
used.
It
doesn't
have
access
to
the
memory
outside
of
the
heap.
So
in
some
cases
what
you
probably
want
to
do
is
to
write
some
custom
scripts
that
will
actually
get
the
values
from
some
of
the
CF
api's,
which
should
give
you
metrics
off
the
containers
itself.
So
we've
seen
situations
where.
A
In
older
built
back,
especially
if
you're
using
bill
packs
three
dot
X
and
not
built
back
for
our
our
for
our
X,
we've
seen
that
if
your
applications
or
especially
springboard
applications,
sometimes
require
more
native
memory
than
its
allocated
by
the
Bill
pack.
So
we've
seen
that
it's
when
you,
when
you
try
it
when
you
don't
specify
when
you
don't
allocate
enough
native
memory,
that's
which
means
that
your
you
have.
So,
let's
say
you
deploy
your
application
with
one
gigabyte
of
container
size.
A
If
container
size
is
1
gigabyte,
it
typically
allocates
70%
to
heap
or
which
I
think
5
or
10%
is
to
native
memory.
If
your
native
memory
requires
more
than
the
10%,
then
you
are,
unless
there
is
enough
memory
left
in
the
container
for
delicate
it's
going
to
eventually
core
do
a
memory
memory
out
of
memory
exceptions.
So
what
we
ended
up
doing
in
the
past
is
to
set
memory
limit
in
our
application,
where
we
reduce
the
amount
of
memory
given
to
the
heap
from
the
default
of
70%
to
say
less
than
50%.
A
So
we
give
more
memory
to
the
native
objects,
so
we
may
have
to
play
around
with
memory
a
little
bit
in
some
cases.
It
may
not
be
true
for
all
the
applications
but
watch
out,
for
it
is
what
what
I'm
trying
to
say
here,
implement
auto
scaling.
Auto
scaling
has
been
a
big
big
help
for
us
in
many
many
instances
so,
and
there
is
I
think
recently,
there
was
a
incubator
project
for
autoscaler
that
came
out
as
open
source.
So
please
look
into
it
and
make
use
of
it.
A
It
really
helped
us
a
lot
and
I
can
show
you
some.
If
you
have
some
time,
I
can
show
you
exactly
where,
where
it
helped
us
again,
you
probably
want
to
look
at
how
the
memory
management
is
done.
Using
the
Java
bail
pack,
they've
done
a
lot
of
improvements
with
the
memory
calculator
in
the
recent
versions
of
the
build
pack
which
is
being
used
and
4.8
and
4.9
so
understand
how
your
memory
memory
is
calculated
and
size
and
memory
accordingly
size
your
container
memory
accordingly.
A
So
it
took
us
a
lot
of
iterations
to
come
to
a
place
where
we
are
able
to
run
our
applications
without
having
to
encounter
any
out
of
memory,
exceptions
so
and
understanding
the
way
memory
is
calculated
helps
lock
heap
dumps.
It's
again,
I
learned
yesterday
that
the
actuator
actually
helps
you
they
have
endpoints
to
take
heap
dumps.
We
are
not
using
that
at
this
time.
A
I
wish
we
were,
but
the
way
we
do
heap
dumps
is
we
have
to
log
into
the
container,
get
the
heap
terms
and
we
have
to
take
help
from
the
operators
to
help
us
do
it.
So
it
was
not
a
very
straightforward.
It
is
not
a
very
straightforward
process
for
us
at
this
time,
but
we're
trying
to
more
streamline
that,
but
heap
dump
helps
a
lot
in
understanding
situations
where
you
see
memory
leaks
and
especially
the
graph
on
the
right
that
we
showed
you
where
the
memory
was
looking
like
it
was
leaking.
A
So
apologize
has
taught
me
if
you,
if
I'm,
if
I'm
going
too
fast
one
of
the
things
that
really
helps,
which
the
developers
may
not
be
aware
of,
is
doing
some
capacity
planning
before
you
deploy
application.
As
I
said
before
size,
your
container
do
some
tests
a
non
functional
test
size
of
container
based
on
the
heap
usage.
You
want
to
make
sure
that
you're
not
over
utilizing
the
memory
and
running
into
situations
where
you're
running
into
our
memory
exceptions
a
number
of
instances.
A
So
you
want
to
you-
want
to
first
run
tests
that
would
identify
how
many
transactions
a
particular
or
one
instance
of
your
app
can,
can
serve.
And
then
you
scale
it
based
on
the
number
of
transactions
that
your
service
is
going
to
expect.
So,
let's
assume
your
service
can
sustain
a
throughput
of
thousand
requests
per
minute
for
for
one
instance.
Now,
if
you're
expecting
10,000
requests,
I
would
add,
please
size
it
for
12
instances.
A
A
You
know
a
scalability
test
to
see
if
it's
actually
scales
out
to
the
number
of
instances
or
the
throughput
that
we
wanted
to
serve,
and
then
endurance
is
basically
to
run
a
very
long
running
task
to
say
for
a
week
or
ten
days
to
understand
memory,
leaks,
resource
leaks
and
so
on.
So
these
tests
really
help
us
to
to
make
sure
that
these,
the
the
java
application
that
we're
deploying
to
production
is
well
tested
in
all
in
all
fronts,
understand
your
failure
points
it
also
it.
A
It
helps
to
understand
how
much
your
application
can
withstand
and
how
does
it
recover?
So
that's,
that's
the
key
part,
so
I'm,
every
application
at
some
point
is
going
to
face
a
situation
where
it's
going
to
crash.
Now
you
need
to
be
able
to
understand
when
it
crashes,
how
does
it
recover
how
gracefully
it
recovers?
How
long
does
it
take
what
kind
of
errors
you
can
foresee
so
this
testing
these
would
actually
help
you
in
you
know
when
you
get
paged
at
the
middle
of
the
night
to
understand
where
the
problems
are
so.
A
This
has
really
helped
us
to
actually
get
to
a
point
where
the
application
fails
and
then
recovers
from
when
the
load
comes
down.
You've
done
some
chaos
monkey
testing.
This
is
the
term
coined
by
Netflix
and
we
randomly
bring
down
instances
to
understand
how
it
would
be
how
the
application
behaves
if
time
permits
I
would
highly
recommend
doing
some
testing
for
any
java
application
and
for
that
matter,
any
application
itself
and
in
cloud
foundry
as
a
platform,
actually
gives
you
capabilities
to
automatically
bring
up
your
instances.
A
A
A
You
see
just
the
older
generation
of
the
heap,
so
you
can
see
that
the
committed
heap
start
going
down,
which
means
that
the
objects
in
the
community
that
the
heap
are
not
getting
collected,
and
ideally,
in
my
experience
when
it
comes
to
a
point
where
your
used
heap
and
your
committed
heap
gets
together.
That's
where
you
see
out
of
memory
exceptions
which
means
it.
But
in
this
case
you
can
see
the
application
was
running
fine
for
the
rest,
well,
another
that
for
two
day
two
days
I
mean
it
was
still
continuing
to
run.
A
Fine
I
just
grabbed
it
for
three
days,
but
in
spite
of
the
fact
that
you're
committed
heap
and
used
heap
are
almost
the
same,
which
means
there's
very
little
memory,
that's
available
for
the
heap
to
allocate
for
new
objects
in
the
old
generation.
You
can
still
see
that
it's
it's
running,
fine,
it's
collecting
the
garbage.
It's
doing
garbage
collection,
it's
running
fine
and.
A
What
you
need
to
understand
is
that
there,
the
way
JVM
behaves
in
certain
instances
are
situations
where
it
might
look
like
there
might.
There
is
a
leak,
but
it
typically
may
not
be
a
leak
at
all
and
and
the
fact
it's
not
a
leak
is
because,
even
though
you're
coming
at
heap
and
used
heap
are
exactly
the
same
or
very
close
to
each
other,
it
still
doesn't
thrown
out
of
an
exception.
A
That
runs
fine,
but
the
side
effect
of
this
is
that
you
would
see
a
lot
more
full
garbage
collections
happening,
which
means
a
lot
more
resources
or
CPU
is
being
utilized
in
application.
It's
just
the
behavior
of
the
JVM
in
the
case,
I
think
it's
JVM,
1.8
101
I
believe
as
a
minor
diversion,
but
it
probably
is
it's
just
a
way.
Davian
behaves
or
the
way
your
application
behaves.
A
Taking
a
heap
dump
will
help.
We
took
some
heap
domes
in
this
case.
We
analyzed
the
heap
domes.
We
couldn't
find
any
leak
suspects,
so
it
probably
is
not
a
League
just
the
way
that
we,
the
JVM,
behaves
so
as
I
said,
there's
no
leak
suspect.
In
this
case
the
older
built
back
was
working
fine
in
a
sense
that
the
older
built
bag
did
not
exhibit
this
behavior.
So
we're
still
trying
to
find
out
what
is
new
in
the
4
dot.
X
version
of
the
bill
pack.
A
So
this
is
another
very
peculiar
situation,
so
we
were
running
a
service
with
multiple
instances
offered
so
on
the
left.
You
see
instance,
one
on
the
right.
You
see,
instance
zero.
So
the
two
instances
so
on
the
left.
You
can
see
a
very
nice
graph
that
you
typically
want
to
see
on
the
right.
You
see
that
it's
again
we're
seeing
a
situation
where
the
amount
of
memory
is
going
down.
It's
the
exact
same
service,
same
codebase,
except
running
on
two
separate
instances
of
off
the
app
now
I.
A
Don't
have
an
answer
for
this,
so
I
was
hoping.
One
of
you
can
give
me
some
idea
as
to
what
might
be
causing
this
excuse
me.
So
we
took
some
heap
dumps.
There
were
no
leak
suspects
so
and
we're
trying
to
figure
out
what
was
causing
and
we're
still
scratching.
It
has
to
figure
out
why
one
instance
might
behave
like
this.
A
The
last
case
study
that
I
wanted
to
show
is
we
run
an
endurance
test
for
three
days
and
it
was
a
very
small
memory
leak
that
we
observed
on
all
the
instances
and
the
heap
dump
was
not
very
helpful
in
identifying
the
leak
suspect
and
what
ended
up
happening
happening
was
one
of
the
external
dependents
that
we
had
was
leaking
memory,
because
the
external
dependency
was,
it
was
part
of
the
the
JVM
it
was.
It
was
doing
a
bytecode
analysis
essentially,
and
that
was
Carl
eking
memory
in
some
cases.
A
So
in
the
situation,
what
I'm
trying
to
tell
you
is
that
it
is
possible
that
your
code,
that
you've
written
may
not
be
the
problem,
because
we
have
external
dependencies
that
we
rely
upon
when
deploying
that
might
be
causing.
So
one
of
the
ways
to
troubleshoot
your
problem
is
that
if
you
can't
find
a
leak
suspect
in
your
code,
try
removing
the
external
dependencies
and
see
if
that
was
also
a
problem,
so
that
in
in
maybe
four
out
of
ten
situations,
that
might
be
a
cause
of
problem
as
well.
A
A
Yes,
one
of
the
tools
we
looked
at
was
sensu
or
graphite
and
refiner
as
well
like
using
elasticsearch,
and
we
did
use
it
as
well
as
a
backup
and
it
works
fine
I'm,
not
saying
no,
so
we
we
use
elastic
search
for
quite
you
can
actually
write
your
own
HTTP
request,
get
the
jvm
metric,
split,
elastic,
search
and
graph
it
through
graph
on
R
or
Cubana.
It
works
fine
and.
A
Yeah
yeah,
it
works
fine,
exactly
it
just
gives
you.
These
SAS
tools
gives
you
a
little
bit
more.
You
pay
money,
you
get
a
little
bit
more
functionality
of
it,
that's
pretty
much
it,
but,
yes,
you
can
yeah
sure.
So
this
is
this.
Shows
you,
the
autoscaler
service
that
we're
using.
So
you
can
see
it
scales
up
and
down,
depending
on
the
amount
of
so
I
think
we
have.
A
We
had
used
CPU
as
our
metric
to
determine
auto
scaling,
so
between
hundred
and
three
hundred
percent,
so
I
think
every
time
it
hits
three
percent
will
scale.
Another
instance
when
it
comes
to
100
percent
it'll
will
scale
down
on
instance.
In
our
case,
you
can
score
go
up
to
eight
hundred
percent
because
it
was
an
eight
core,
I
believe
so
so
yeah
auto
scaling
works
really
well
for
us,
so
please
make
use
of
it.
The
other
thing
I
wanted
to
show
you
was.
This
is
a
live
application.
A
B
A
A
A
So
one
of
the
things
I
would
look
at
is
I
will,
of
course,
take
a
heap
them
to
figure
out.
What
is
what
is
what
objects
are
in
the
heap
which
actually
makes
it
grow,
bigger
and
bigger,
so
you're
saying
the
amount
of
time
it
takes,
grows
larger
and
larger,
or
the
amount
of
heat
oh
I,
see
okay,
so
which
means
that
it's
taking
longer
and
longer
for
it
to
clear
the
guy,
the
garbage
collection.
But
do
you
also
see
the
fact
that
it's
the
amount
of
heap
is
the
same?
In
that
case.