►
Description
Speaker: Aleks Seovic
Scaling stateless services is easy, but scaling their stateful data stores, not so much. This is true whether you are using an “old fashioned" relational database, or one of the popular, “modern" KV data stores, such as MongoDB or Redis.
In this presentation we will discuss some of the issues with state management in elastic microservices today, and look into how Oracle Coherence, with its Helidon and Eclipse MicroProfile integration, provides a better alternative you can use tomorrow.
Subscribe: https://www.youtube.com/user/EclipseFdn
Follow us on Twitter: https://twitter.com/JakartaEE
Like us on Facebook: https://www.facebook.com/JakartaEE/
Join us on LinkedIn: https://www.linkedin.com/groups/13597
Tag us: #JakartaEE
Want more great Eclipse Community webinars? Become a member of our Jakarta Tech Talks Meetup: https://www.meetup.com/jakartatechtalks
A
A
A
Thank
you.
Thank
you
for
the
introduction
and
thank
you
everyone
for
joining.
As
some
of
you
know,
back
in
june,
we
have
released
two
things.
We
have
released
helidon2o,
which
is
the
next
major
version
of
helidone
microservices
framework,
and
we
have
also
released
and
announced
coherence,
community
edition,
which
is
a
an
open
source
edition
of
oracle
coherence
in
memory
data
grid,
which
has
been
around
for
almost
20
years.
Now.
A
As
part
of
that
effort,
we
have
also
built
an
application
that
is
used
to
demonstrate
how
to
do
how
to
integrate
and
how
they
work
together
and
how
they
make
building
stateful
microservices
easier,
while
still
allowing
them
to
to
scale
pretty
much
as
much
as
you
need
to
so.
First
of
all
introduce
the
application
that
we
built.
We
didn't
actually
build
the
original
application.
I
took
the
origin,
the
the
existing
microservices
demo,
application
that
was
created
by
weworks
back
in
2016.
A
A
So
we
we
started
with
that
and
the
original
the
architecture
of
that
application.
It
looks,
looks
something
like
this,
so
it
has
a
number
of
independent
services,
they're
all
packaged
into
docker
images,
and
then
the
original
demo
had
a
number
of
different
ways
to
run
it
because
back
then
kubernetes
was
not
the
fact.
A
A
They
use
combination
of
mongodb
and
mysql
and
rabbit
mq,
and
then
the
front
end
also
had
the
option
of
using
redis
for
for
session
state
management.
So,
let's
take
a
look.
I
actually
have
the
original
demo
running
here
and
I'm
already
logged
in
so
basically
the
application.
When
you
run
it
looks
something
like
this.
You
can
look
at
the
home
page,
you
can
go
in
browse
product
catalog,
you
can
search
for
items
in
the
catalog,
add
something
to
cart,
and
then
you
can
look
at
the
cart
and
check
out.
A
So
when
you
check
out
the
order,
gets
created
and
that's
pretty
much-
that's
pretty
much
it
so
you
can
view
the
order
and
see
what
you
order.
So
it's
your
typical
e-commerce
application
right.
I
mean
it
and
it's
a
it's
a
good
example
to
kind
of
show
how
all
of
these
services
work
together,
but
it
does
have
some
underlying
issues
that
are
kind
of
hard
to
avoid
unless
you
change
the
architecture
a
bit.
A
So
the
problem
with
this
is:
are
these
services
scalable
right
and
and
when
I
say,
are
they
scalable?
It's
not
enough
to
scale?
Obviously
we
have
you
know
stateless
spring
boot
services.
We
have
stateless,
go
services,
it's
easy
to
scale
those
it's!
You
know
you
can
simply
tell
kubernetes
to
to
add
more
pods
and
it's
all
good,
but
you
also
need
to
be
able
to
scale
the
back
ends
right.
You
need
to
scale
mongodb,
you
need
to
scale
my
sequel,
you
need
to
scale
private
mq.
A
Those
things
are
typically
not
as
easy.
I
mean
they're
managing
state,
they're
stateful,
so
they're
not
as
easy
to
scale
with
or
without
kubernetes.
Right
I
mean
there.
There
are
ways
to
do
it
there,
but,
but
it's
it's
a
project
I
mean
you
have
to
set
things
up.
You
have
to
set
up
their
clustering
support.
Usually
there
is
an
operator
involved
that
you
need
to
configure,
and
sometimes
those
operators
even
for
open
source
products,
are
commercial
offerings
and
part
of
only
enterprise,
editions,
etc,
etc.
So,
while
it's
possible,
it's
definitely
not
trivial.
A
The
second
question
is:
are
those
things
highly
available?
Well
until
you
scale
them
they're,
not
right,
because
if
we
have
just
a
single
data
story,
if
that
data
store
goes
down,
your
your
application
is
down
right
and
then
the
last
question
is
how
easy
it
is
to
monitor
each
of
these
services
and
again.
This
is
where
the
complexity
of
the
architecture
comes
into
the
picture,
because
the
more
moving
components
you
have
there
are
more
things
to
monitor.
A
There
are
more
things
to
manage:
there
are
more
things
to
scale,
et
cetera,
et
cetera,
so
and
then
you're
dealing
with
different
platforms,
which,
which
is
fine,
I
mean
micro,
polyglot
micro
services
are
perfectly
acceptable,
but
you
better
have
a
good
reason
for
it,
because
each
platform
will
require
a
different
way
of
monitoring
and
management,
and
you
know
you
have
to
embed
things
like
open,
tracing
and
metrics,
et
cetera,
et
cetera,
using
whatever,
whatever
libraries,
whatever
features
the
application.
The
platform
you're
building
on
provides
so
looking
at
this
application.
A
Let's
see
why
why
this
is
an
issue,
so
this
is
what
we
end
up.
You
know
when
this
application
is
deployed.
This
is
what
we
end
up
having
on
the
back
end
and
I'll,
show
you
the
pods
that
are
running
and
then
I'll
show
you.
What
is
is
backing
those
pods.
You
can
see
that
you
know
based
on
the
pod
names,
you
can
pretty
much
see
that
all
of
these
are
backed
by
the
deployment
set.
A
And
we
can
show
that
really
quickly
and
you
can
see
that
there
are
separate
deployments.
You
know
what
I
mentioned
earlier.
The
separate
deployments
for
the
service
itself,
for
example,
card
service-
that's
our
spring
boot
stateless
front
end,
but
then
there
is
also
corresponding
database
deployment
and
again
for
a
simple
demo
just
showing
that
this
work.
This
is
fine,
but
you
would
never
actually
run
this
database
this
way,
you
would
never
run
it
as
a
kubernetes
deployment,
because
it's
ephemeral.
A
Essentially
if
this
pod
goes
down,
you're
lost,
you
lost
all
your
data
and
you
also,
you
know
you
can
you
can
scale
this?
I
can
do
cube
cuddle
scale,
carts
deployment
to
as
many
nodes
as
they
want,
but
I
can't
really
do
something
similar
with
cards
db,
because
in
order
to
scale
this,
which
in
this
case
mongodb
instance,
I
need
to
configure
for
scalability,
I
need
to
tell
how
to
discover
other
nodes
there
is.
You
know
there
is
all
kinds
of
additional
configuration
that
need
to
need
to
happen.
A
So
none
of
this
catalog
is
backed
by
my
sql
same
thing.
It's
not
it's
not
a
trivial
thing
to
scale.
You
need
to
consider
configure
masters
and
replicas
et
cetera,
et
cetera
same
thing
would
be
with
tradies
right.
You
need
to
configure
something
else.
You
need
to
configure
this
cluster,
you
need
to
configure
where
master
copy
of
the
data
goes
and
then
configure
one
or
more
replicas
for
each
etc
cetera.
A
A
You
know
like
we
all
you
know
well,
maybe
not
all,
but
many
of
us
who
are
old
enough
remember
from
the
early
java
80
days
when
we
started
writing
java
ejb
based
applications,
you
know
and
everything
runs
smooth
as
long
as
you
have
a
single
node,
because
that
node
is
able
to
cache
a
lot.
It
knows
it's
the
only
node
in
the
cluster.
There
is
nobody
else
talking
to
the
database.
A
There
is
no
reason
to
synchronize
data,
so
so
early
java
servers
were
able
to
cache
a
lot
as
long
as
they're
in
a
single
server
mode,
but
you
add
a
couple
extra
nodes
and
all
of
the
sudden
that
that
caching
gets
turned
off
and
your
database
load
increases.
So
how
far
can
you
increase
the
load
without
your
data
store
falling
over
right?
If
you
cannot
handle
the
load
created
by
10
stateless
instances,
does
it
matter
if
you
can
have
thousands
of
them?
It
doesn't.
A
I
mean
you're
just
going
to
make
things
worse,
you're
going
to
you're,
going
to
reduce
performance
you're,
going
to
eventually
make
your
data
store,
keel
over
and
and
your
application
will
be
completely
unresponsive.
Similarly,
if,
if
your
data
store
is
down,
are
your
stateless
services
able
to
process
requests?
They're,
not
I
mean
they
they
depend.
A
So
for
those
of
you
who
are
not
familiar
with
either
helidom
or
coherence,
I'll
just
do
a
quick
overview.
I
mean
this
talk
is
not
about
well,
it
isn't
a
design.
It's
not
necessarily
about
low
level
details
about
either
technology,
but
it
is
about
how
you
can
use
them
to
to
build
application
in
these
applications
in
an
easier
fashion
than
than
most
people
do
these
days.
So
halidon
is
a
micro
profile,
2
implementation.
A
It
also
supports
bunch
of
jakarta
e
components
that
that
we
need
and
that
microprofile
needs
things
like
cdi
and
jax
rs,
etc
jpa,
and
then
it
also
has
some
additions
that
that
some
of
the
other
implementations
don't
and
those
are
kind
of
important.
We
do
use
them
for
communication
between
order
service
and
payment
and
shipping
service
in
in
in
the
application.
A
So
one
thing
that
halidon
adds
is
support
for
grpc,
so
you
can
write
not
only
rest
services,
but
you
can
also
write
grpc
services,
which
is
you
know,
I'm
hoping
we
can
standardize
someday.
But
at
the
moment
it's
not
part
of
the
micro
profile
and
it's
basically
held
on
proprietary
extension
in
order
to
support
grpc
microservices.
A
And
then
coherence
at
a
high
level,
you
can
really
think
of
it
as
a
distributed.
Persistent
kv
data
store
it's
kind
of
similar
to
mongodb
already
it
can
be
used
for
the
exact
same
types
of
applications,
but
it
is
much
easier
to
scale
out.
I
mean
there
is
major
difference
between
some
of
these
technologies.
Well,
the
features
are
the
same.
A
Coherence
was
designed
from
the
get-go
from
the
very
early
days
to
be
a
distributed
technology
while
and
and
redis,
for
example,
they
have
added.
They
were
designed
as
a
single
process
servers,
and
then
they
were
kind
of
extended
it
was.
It
was
a
bolt-on
to
to
allow
them
to
be
clustered
and
to
allow
them
to
support
aha
and
scale
out
environments
so
they're.
You
know
this
can
be
seen
once
we
see
how
easy
it
is
to
scale
coherence.
It
becomes
obvious
that
you
know
one
thing
was
designed
for
scale
out
and
the
other
one
isn't.
A
If
you
are
trying
to
scale
already,
it
is
elastic
scalable
and
highly
available.
It
partitions
data
across
cluster
members,
regardless
of
how
big
the
cluster
is
and
ensures.
There
is
backup
copy
for
each
data.
Entry
and
what's
important
in
this
in
this
refactoring
is
embeddable
into
any
java
application.
Coherency
is
effectively
a
library.
A
We
do
have
a
server
that
you
can
run
as
a
standalone
thing
and
you
can
use
different
apis
and
and
write
clients
that
talk
to
that
server
and
store
data
and
get
and
read
data
et
cetera,
et
cetera,
but
for
most
applications,
and
in
this
particular
example,
application
coherence
is
really
just
a
library
that
can
be
embedded.
But
while
it's
embedded
in
every
single
process,
it
actually
allows
all
those
processes
to
cluster
together
and
to
talk
to
each
other
and
to
basically
partition
data
across
them.
A
It
is
as
easy
to
use
as
java
util
map,
at
least
for
basic
functionality,
coherence
named
map
interface,
which
is
our
core
interface
when
you're
dealing
with
it.
You
know
for
dealing
with
the
data
store,
actually
extends
java,
util
map
and
provides.
You
know
the
extensions
that
we
provide
are
very
useful
for
to
implement
some
some
of
the
distributed
computing
constructs.
A
You
know
the
ability
to
send
us
processing
into
the
cluster
and
to
process
data
in
place
to
aggregate
data
in
parallel
across
the
cluster
to
to
execute
queries
against
the
map,
et
cetera,
et
cetera.
So
those
are
those
are
the
types
of
extensions,
but
for
basic
operations
as
you'll
see
in
some
of
the
code.
Examples
later
it's
really.
It
really
is.
You
know
you
can
do
map
get
or
name
map
get
and
put
and
yeah
compute,
etc,
etc.
A
So,
we'll
see
some
of
the
examples
later
and
as
I
mentioned
in
the
opening
as
of
june,
coherence
is
free
and
open
source.
Well,
community
edition
is,
and-
and-
and
I
do
want
to
point
out
that
you
know-
unlike
some
of
the
some
of
our
competitors
in
the
memory
data
grid
space
in
particular
and
even
some
of
the
competitors
in
the
larger.
You
know-
data
store
space.
We
don't
really
what
we
try
to
do
with
the
community
edition.
I
mean
coherence
has
been
around
for
almost
20
years.
A
It
has
been
around
since
2001
and
it
has
powered
michigan
mission
critical
systems
across
the
planet
for
for
almost
20
years.
But
when
we
decided
to
release
community
edition,
we
did
not
want
to
put
keratin
a
stick
out.
We
wanted
to
release
something
that
can
be
truly
usable
on
its
own.
The
features
that
we
left
out,
the
features
that
are
commercial
only
are
related
to
federation
and
hot
cache,
which
is
replication
from
a
from
a
database.
A
You
know
they're
kind
of
advanced
features,
but
what
we
give
you
out
of
the
box,
both
from
development
perspective
from
what
you
can
actually
do
in
production
and
from
the
management
monitoring
perspective.
All
of
those
things
are
free
and
part
of
community
edition.
The
operator
is
completely
free.
The
the
monitoring
extensions
that
I'll
show
in
a
minute
for
grafana
and
prometheus
are
basically
out
in
the
open
source
and
free
as
well.
So
you
can.
You
can
do
a
lot
everything
I
did
in
this
application.
A
A
And
then
I
want
to
talk
about
how
coherence
and
halidon
mp
fit
together.
So
you
know
once
we
learned
about
micro
profile
and
started,
seeing
that
micro
profile
is
gaining
traction,
we
wanted
to
integrate
with
it,
and
you
know
I'm
using
helidon
in
this
example,
but
helen
microprofile
integration
is
really
a
micro
profile,
eclipse
micro
profile
integration,
so
we
use
standard
micro
profile
apis.
We
just
happen
for
this
demo
to
use
helidon
as
a
backend
and
because
we
need
grpc
services.
A
We
need
to
use
helidom
because
that's
something
that's
that's
specific
to
helidone
and
no
other
microprofile
implementations
have
it,
but
for
the
most
part
everything
that's
on
this
slide
would
work
with
any
microprofile
or
should
work
with
any
microprofile
implementation.
I'm
not
sure
we
haven't
really
tested
it
with
every
implementation,
but
it
should.
The
integration
is,
is
in
these
four
in
these
four
boxes
that
are
kind
of
highlighted
in
the
lower
right
corner,
so
I'll
start
with
cdi,
because
it's
kind
of
the
cornerstone
component
of
it.
So
we
use
cdi
to
bootstrap
coherence.
A
A
We
also
integrate
with
microprofile
metrics,
so
coherence
has
a
fairly
extensive
set
of
metrics
that
are
internally
managed
as
in
using
jmx,
but
we
did
implement
a
couple
years
ago.
We
implemented
support
for
metrics
registries
for
metrics
adapters.
Basically,
and
we
have
written
as
part
of
this
effort.
We
have
written
a
microprofile,
metrics
adapter.
It
allows
us
to
publish
all
of
those
jmx
managed
metrics
into
any
micro
profile,
metrics
registry,
so
they're
get
they
get
exposed
the
same,
slash,
matrix
endpoint
with
all
the
other
application
and
and
application
server.
A
Metrics
we've
also
implemented
integration
with
microprofile
configurations.
So
we
you
can
use
microprofile
config
to
configure
coherence
itself.
There
are
a
number
of
options
that
coherence
allows
you
to
specify.
There
is
java
system
is
typically
as
java
system
properties
or
in
a
conf
in
a
coherent,
specific
config
files.
But
now
you
can
use
any
micro
profile,
config
source,
to
provide
those
things.
So
you
know
the
same
system
properties
that
you
were
always
able
to
provide.
A
You
can
also
provide
with
using
using
environment
variables
or
helidon's
application,
yaml
or
microprofile
config
properties,
or
you
know
any
any
other
mechanism
that
your
mp
deployment
supports
and
then
finally,
we've
added
support
for
open
tracing
which
allows
us
to
see
the
request.
All
the
way
from
the
api
endpoint
down
into
the
data
store
down
into
the
cuts
of
the
data
store
cross
member
is
a
cross
cluster,
etcetera
and
we'll
see
that
we'll
see
that
in
a
minute.
A
Let's
look
at
how
this
application
works
so
again
we're
using
the
same
front
end.
So
there
shouldn't
be
any
difference.
You
may
see
that
the
order
ids
will
become
a
little
bit
shorter
because
in
in
our
new
implementations,
we've
shortened
them
a
bit.
But
apart
from
that,
there
should
be
no
difference
in
functionality,
so
you
can
obviously
browse
the
catalog
you
can
search.
A
You
can
add
products
to
cart
and
check
out
and
again,
as
I
said,
the
order
number
is
a
little
bit
shorter
and
you
can
still
go
in
and
view
the
order.
So
that's
pretty
much
it
functionally.
The
application
is
exactly
the
same,
but
you
know
the
front
end
is
exactly
the
same.
The
front
I'm
using
actually
the
same
docker
docker
image
for
the
front
end
container
that
the
original
application
uses,
but
the
back
end
is
completely
different.
A
Let's
see
what
what
we
actually
have
here.
So
if
I,
if
I
look
at
a
list
of
pods
running
kubernetes,
for
the
coherence
application,
you
will
see
that
there
is
basically
only
one
pod
per
service,
so
there
is
card
0,
catalog,
zero
order,
zero.
There
is
front
end
which
still
uses
the
original
deployment,
as
you
can
see
from
the
mangled
gui
at
the
end,
but
all
of
the
all
of
the
backend
services
are
have
zero
behind
them
and
there
is
only
a
single
one
of
them.
A
A
Stateful
says
that
I
have
six
of
them
and
there
is
one
pod
in
each
one,
and
because
we
have
coherence
operator
running
here,
we
can
also
list
coherence
deployments
which
are:
what
is
what
what
creates
those
data
sets
and
when
you
do
that,
you
can
actually
see
a
bit
more.
You
can
see
that
cards
deployment
is
belongs
to
a
sock
shop
cluster.
You
know
we
chose
to
run
all
of
them
in
the
same
cluster
as
different
roles.
A
A
A
How
is
this?
How
is
this
better
right?
Well,
you
haven't
seen
it
yet,
but
these
services
are
stateful,
they
don't
need
separate
data
store
and,
as
you'll
see
in
a
second,
they
are
as
easy
to
scale
as
the
stateless
one.
There
is
easy
to
scale
as
the
spring
boot
or
go
stateless
services
from
the
original
application.
A
High
availability
is
built
into
the
architecture
again,
not
when
you
run
with
single
pod
per
service,
but
it
will
be
in
it
will
be
in
a
second
support
for
monitoring
is
built
in
and
it's
also
extensible
and
the
implementation
is
significantly
simpler.
So
so,
let's
look
at
some
of
those
some
of
those
things
so
I'll
start
with
looking
at
the
monitoring.
A
So
this
is
grafana
dashboard
that
you
get
effectively
out
of
the
box
that
allows
you
to
see
what's
going
on
in
a
coherence
cluster.
So
this
is
our
main
dashboard.
You
can
see
that
we
have
six
cluster
members
one
for
each
service,
but
you
can
also
see
here
in
red
that
we
have
12
endangered
services
and
total
cluster
heap
is
at
the
moment
19
gigabytes
or
so
so
we
have
six
members
running
each
one
of
them,
I
believe,
is
set
to
three
gigabytes,
three,
three,
something
so
that
that's
about
right.
A
We
can
also
look
at
services
dashboard
and
we
can
see
that
aha
status
for
each
service
at
the
moment
is
endangered,
so
we
have
cart
service,
catalog
service
order,
service
et
cetera.
These
are
these
are
not
our
micro
services.
These
are
at
the
storage
level,
coherent
services
that
are
used
to
store
the
data
for
each
one
of
them.
So
how
do
we?
How
do
we
fix
that?
And
then
you
know
we'll
and
before
we
fix
it?
Actually,
let's
look
at
this
I've.
A
I've
created
some
orders
earlier
and
I
created
one
just
now,
so
you
can
see
here
that
you
know
we
have
different
caches
one.
You
know
on
each
service.
We
have
users,
you
know
we
have
five
users
at
the
moment:
consuming
4.5
kilobytes.
We
have
six
shipments
for
six
orders
and
six
payments
for
six
orders,
so
those
things
match
and
they
should
match
going
forward,
and
then
we
have
catalog
service
with
only
nine
products
in
a
catalog.
But
that's
that's
good
enough
for
for
what
we
need
here.
A
So
this
shows
you
basically
how
much
data
you
have
in
the
cluster,
how
much
data
you
have
in
each
individual
cache?
Well,
how
many
objects
and
how
much
memory
they're
consuming.
It
also
shows
you
the
indexes
that
you
have,
and
this
case
only
payment
service
has
an
index
which
is
consuming
kilobyte,
etc,
etc.
So,.
A
A
And
this
will
take
a
minute
and
see
that
kubernetes
should
start
to
kubernetes
should
scale
each
of
these
services.
And
if
we
look
at
set
of
pods,
we'll
see
that
the
new
pods
for
each
service
will
see
that,
in
addition
to
pod
zero
we'll,
we
should
see
pod
one
and
two
and
they
should
be
in
some
stage
of
of
starting.
A
A
The
services
are
converting
from
one
to
three
members
and
they're
marking
their
status
has
site
safe
and
what
what
site
safe
means?
I
mean
coherence,
has
different
levels
of
of
safety.
It
can
be
endangered,
which
was,
which
is
what
we
were
initially.
Then
you
can
go
from
in
danger
to
node
safe,
which
means
there
is
another
jvm
on
the
same
machine
that
is
storing
a
backup
copy
of
the
data,
and
then
you
can
go
to
machine
safe,
which
is
you
know.
A
A
Site
safe
is
kind
of
the
highest
level
of
safety,
and
it
means
that
the
backups
are
actually
stored
in
a
different
data
center.
In
this
case,
in
a
different
availability
domain
within
the
within
the
okay
cluster
of
kubernetes
cluster,
I'm
running
on
oci
so
because-
and
all
of
this
happens
automatically,
there
was
nothing
I
had
to
configure
to
make
coherence
create
backups
as
far
away
from
as
far
away
as
possible.
A
I
mean
it
basically
is
aware
of
network
topology
and
where
each
node
is
running,
it
gets
configured
automatically
by
kubernetes
operator
using
kubernetes
metadata
about
which
availability
domain
each
member
is
running
in
and
then
it
ensures
that
if
I
have
multi-level
domain
when
it
creates
backups
for
the
data,
it
creates
them
as
far
away
as
possible.
So
if
we
look
at
caches
summary,
nothing
has
changed
right.
We've
added
a
bunch
of
nodes.
A
The
data
got
rebalanced
across
across
them
across
different
data
centers,
but
we
still
have
you
know:
six
orders
and
payments
and
shipments
nine
products
etc.
And
if
we
look
at
the
main
dashboard
which
doesn't
seem
to
have
refr
okay,
so
now
it
refresh
so
we
have
18
cluster
members.
We
have
zero
endangered
services.
We
have
now
58
gigabytes
of
heap
to
store
the
data
in
and
we
obviously
have
18
application
servers
that
can
process
our
press
and
grpc
requests
right.
So.
A
High
availability
is
built
in
if
any
of
those,
if
any
of
those
pods
goes
down
if
a
whole
data
center
goes
down
in
this
case,
if
a
whole
list
domain
goes
down,
so
we're
not
going
to
lose
any
data
because
all
of
our
data
is
site.
Safe
monitoring
is
built
in
I've
shown.
One
component
of
monitoring,
which
is
our
grafana
dashboards,
I'll
show
another
one
which
is
tracing
support
in
a
minute,
and
then
we
look
at
the
implementation.
A
So
before
we
do,
that,
you
know
scaling
to
three
nodes,
is
fun
and
everything
right,
but
that's
not
what
real
scale
out
architectures
are
all
about,
so
god
willing,
because
somebody
seems
to
be
using
michael
burnett's
cluster
but
I'll,
give
it
a
try
anyway.
Let's,
let's
change
this
and
let's,
let's
really
scale
this
thing
out,
so
let's
scale
to
50
knots
per
service,
and
this
will
take
a
bit
and
it
takes
time
for
kubernetes
to
spin
all
these
parts.
A
But
but
when
it's
all
said
and
done,
we
should
end
up
with
a
300
node
cluster,
storing
almost
terabyte
of
data
or
with
a
capacity
of
about
terabyte
on
heap
and
spread
across
six
kubernetes
notes
that
I
have
so
this
will.
This
will
take
a
minute,
but
if
we
look
at
it
here,
you
can
see
that
there
is.
There
is
something
happening
and
the
is
increasing,
but
we'll
come
back
to
that
in
a
second.
In
the
meantime,
we
can
look
at
some
of
the
tracing
functionality,
as
I
mentioned
earlier,.
A
A
So
in
this
case
we
see
that
there
was
http
request
coming
in
as
http
post
to
the
order's
endpoint
and
that
order
service
then
did
couple
things
it
it
that
it
executed
this
jax
rs
method
orders
api
new
order,
which
then
called
out
to
couple
other
services.
It
called
to
the
cart
service
to
get
shopping,
cart,
details,
and
then
it
called
to
the
user
service
multiple
times
to
get
address
for
the
user
and
to
get
credit
card
information.
A
And
to
get
the
actual
customer
information
as
well
and
then
finally
it
called
into
coherence,
so
you
can
see
that
in
this
case
the
this
is.
This
is
just
the
cdi
being
event
driven
order,
processor
process
order
method
was
called
which
sub
you
know,
which
called
save
order,
method
and
the
coherence
order,
repository
and
then
ended
up
in
coherence.
So
this
this
is
where
effectively
helidone
application
stops
and
coherence
begins.
So
so
now
we
are
looking
at
traces
in
coherence
itself,
so
you
can
see
that.
A
We
submitted
put
request
on
member
5
and
that
request
was
processed,
because
in
this
case
this
was
done
when
I
only
had
single
member
for
each
service.
This
request
was
also
processed
by
member
5
by
member
5
and
on
partition
147
on
this
particular
worker
thread.
So
again,
this
is
just
a
simple
example
that
shows
how
you
get
tracing
information
throughout
the
stack
all
the
way
from
the
api
endpoint
down
into
the
dial
into
the
into
the
data
store
itself,
and
then
we
have
some
other
traces
that
we
can
look
at.
A
For
example,
this
is
aggregate
request
that
that
I
executed
again
when
I
had
a
single
node,
so
request
was
executed
on
this
node
and
this
it
came
from
the
http
request
to
catalog
endpoints
only
to
get
number
of
items
in
the
catalog
to
show
on
the
ui
and
that
executed.
The
creation
request,
called
into
get
saw
count
method
which
called
coherence,
catalog
repository,
which
called
which
executed
aggregation,
request
against
the
data
store,
and
you
can
see
that
there
is
only
single
member
that
process
now.
A
A
A
We
basically
have
many
many
many
oh
this
is.
This
is
different,
but
this
is
this
is
the
aggregate
request,
so
the
aggregate
request
is
sent
to
the
aggregate
request
is
sent
to
a
bunch
of
nodes
and
executed
in
parallel.
You
can
see
from
the
from
the
trace
here.
It
doesn't
change
the
aggregation.
Obviously,
everything
is
running
a
little
bit
slower,
because
system
is
in
a
state
of
flux.
It's
scaling
up
at
the
moment,
so
it's
not
fully
stable.
I
mean
there
is
a.
A
A
So
we'll
see
that
card
service
already
scaled
out
to
50
notes,
50
members
order
series
scaled
out
to
50
the
other
four
are
still
in
progress,
so
let's
go
back
to
the
deck
and
look
at
some
of
the
implementation
details.
So
the
implementation
for
this
whole
thing-
and
actually
you
know
for
the
whole
example-
is
on
github.
It's
available.
Everybody
can
use
it.
If
you
go
to
github.comshop,
that's
that's
the
main
repo,
where
kubernetes
deployments,
file
and
and
root
of
the
documentation
is,
and
then
you
can
drill
deeper
into
individual
services.
A
I
mean
the
ones
that
we
built
as
part
of
the
ci
process
are
actually
deployed
to
docker
hub.
So
you
can,
you
can
just
use
those
you
don't
need
to
build
it
necessarily
in
order
to
run
it
in
your
own
kubernetes
cluster.
You
can
just
run
the
kubernetes
script,
so
you
can
go
there
and
get
a
documentation,
and
I
I
do
want
to
point
out.
While
you
know
the
focus
of
this
talk
is
on
how
coherence
makes
things
simpler
and
how
allows
you
to
manage
state
in
a
scalable
way
and
build
these
elastic
microservices.
A
The
the
actual
implementation
on
github
has
multiple
backends,
so
coherence
is
only
one
of
them.
There
is
also
backend.
There
is
redis
backend,
which
is
somewhat
not
quite
what
what
it
should
be,
because
it
oversimplifies
some
things,
and
you
know
it
works
with
a
with
a
small
data
set.
We
have
in
this
particular
application,
but
it
wouldn't
work
for
anything
larger
and
there
is
also
mysql
jpa
based
implementation.
So
you
can
take
a
look
at
any
of
them.
We
hope
you're
going
to
choose
coherence
for
the
reasons
that
I
just
described.
A
So,
let's,
let's
look
at
some
of
the
things
that
make
it
easier
when
you're
using
coherence,
so
there
is
no
no
impedance
mismatch.
First
of
all
I
mean
jpa
tends
to
complicate
things
quite
a
bit
because
you
need
to
go
from
object
to
relational
world
coherence
is
just
those
java
objects.
Obviously
it
serializes
them
and
there
are
a
couple
different
serialization
mechanisms
that
are
supported,
including
basic
java
sterilization,
but
they're
a
lot
more
efficient.
A
A
The
other
thing
that's
important
is
coherence.
Allows
you
to
stall,
reaches
to
store
rich
domain
models.
We
can
actually
perform
operations
on
your
rich
domain
objects
inside
of
the
cluster,
using
in
this
case
in
place
processing.
So
in
the
in
the
code
sample
at
the
bottom,
you
see
that
you
know
we
use
cdi
integration
to
inject
coherence
named
map
into
into
this
jax
rs
resource
and
then
add
item
is
well.
You
can't
really
see
it's
juxta
research,
but
trust
me.
A
A
A
So
while
this
looks
like
what
you
would
do
with
it
with
a
regular
map,
there
is
a
actually
a
very
subtle
but
very
important
difference
here.
This
lambda
that
is
highlighted
that
is
sent
to
the
compute
method,
will
actually
execute
somewhere
in
the
cluster.
We
don't
know
where
and
we
don't
care
where
it
will
execute
whatever.
Whatever
the
card
with
this
given
card
id
exists,
it
will
be
sent
to
the
cluster.
A
It
will
be
executed
on
the
storage
node
that
owns
this
particular
shopping,
cart
and
then
the
result
will
be
returned
to
this
member,
which
will
then
return
it
to
the
front
end
and
front
end.
Will
this
display
updated
card?
So
it's
a
very
subtle
but
very,
very
important
difference.
It's
one
of
those
distributed
computing
extensions
that
I
mentioned
earlier.
You
know
it's
much
more
efficient
to
send
processing
to
the
storage
node
that
owns
a
piece
of
data
and
update
data
in
place.
A
Then,
to
you
know
what
we
would
typically
see
in
applications
that
that
use
conventional
data
stories.
You
would
have
some
kind
of
read
from
a
data
store
to
get
a
shopping
cart.
You
would
update
it
locally
on
this
particular
application
server
where
this
api
endpoint
was
hit,
and
then
you
would
write
it
back.
I
mean
that
that
works,
but
it's
not
very
efficient.
It's
just
you
know,
you're,
sending
a
lot
more
data
than
you
need
to
over
the
wire
twice
you're
doing,
processing
so
putting
additional
load
in
your
application,
servers,
etc,
etc.
A
Then
another
thing
that
we
used
in
the
example
that
I
would
like
to
highlight
is
that
the
support
for
rich
event,
model
coherence,
provides
everything
in
coherence
is
reactive
internally.
Everything
is
everything
you
know
many
many.
There
are
events
for
many
things
like
their
events,
in
this
case,
I'm
listening
for
event
on
a
given
map,
which
are
probably
one
of
the
most
useful
event
types.
You
want
to
listen
when
things
in
the
map
change,
so
I'm
listening.
A
You
know,
I
created
one
order
change
and
I
use
basically
cdi
async
observer
on
on
map
orders
for
inserted
and
updated
events,
and
when,
whenever
an
entry
is
inserted
or
updated
into
these
order
maps,
I
will
receive
an
entry
event
and
from
that
entry
event
I
can
get
the
value
from
the
value.
I
can
get
the
status
of
the
order.
So
in
this
case,
if
the
status
is
order
was
created,
then
I
can
submit
it
for
payment
processing
and
basically
be
done
with
it
and
then
in
the
next
iteration.
A
A
And
then
something
that's
not
really
coherent
specific,
but
I
did
mention
earlier
something
that
makes
helidon
somewhat
unique
in
the
in
the
micro
profile
implementation.
Space
kalidon
has
built-in
support
for
for
a
grpc
server,
and
it
was
pretty
much
designed
based
on
you,
know,
kind
of
inspired
by
jax
rs,
obviously
with
the
with
the
grpc
slant.
So
you
can
mark
a
class.
You
know
you,
you
can
still
do
things
the
way
you
would
normally
do
with
your
pc.
You
can
start
with
a
profile.
A
You
can
build
your
stubs,
you
can
implement
them
and
we'll
happily
run
service
implemented
the
conventional
grpc
way
for
you,
but
we
do
provide
a
much
easier
way,
much
easier
framework
for
for
building
grpc
services.
You
can
basically
start
with
a
class
just
like
a
jax
rs
resource.
You
can
mark
it
with
a
grpc
which
which
says
that
this
is
a
grpc
service.
A
A
You
can
easily
specify
different
marshaller
in
this
case,
because
we
already
had
support
for
for
json
because
of
our
rest.
Apis
we
decided
to
use
the
same
json
serialization
for
jrpc
services
instead
of
re-implementing
them
using
protobuf.
We
could
have
used
pof
as
well,
because
we,
you
know
at
least
in
the
coherence
and
then
you
annotate,
just
like
in
jax
rs,
you
annotate,
individual
endpoint
methods
with
you
know
on
what
you
would
where
you
would
use,
get
and
put
and
post
to
annotate
rest
methods
in
in
jxrs.
A
You
use
grpc
equivalence
kind
of
that
you
you
specify
a
method
type,
especially
whether
something
is
unary
method
or
server,
streaming
method
or
bidirectional
method
or
client
streaming
et
cetera,
and
then
you
simply
provide
implementation
for
it,
and
there
is
a
lot
that
goes
under
the
hood
here.
If
you've
ever
done
grpc,
you
know
the
grpc
expect
jpc
methods
take
stream
observer
as
the
second
argument,
and
then
they
call
on
next
on
complete
on
error
on
those
stream
observers.
A
All
of
that
still
happens
under
the
hood,
we're
still
using
standard
grpc
java
implementation
for
for
everything.
But
this
allows
you
not
to
have
to
worry
about
that.
You
know
we
will
take
whatever
you
return
and
we'll
either
stream
it
back.
If,
if
you
mark
this
method
as
server
streaming
and
your
return
collection
of
authorizations,
we
would
stream
that
collection
back.
It
could
also
written
a
stream
of
authorization.
Then
we
would
stream
it
back,
because
it's
marked
the
summary.
A
You
can
still
use
all
of
the
microprofile
metrics
annotations
so
for
in
the
second
example,
authorized
method
is
metered,
so
we
can
see
the
metrics
for
it
and
etc.
So
so
that's
that's
your
server
side.
So
implementation
is
simple.
Oh
yeah
and
another
thing
I
mean
if
we
wanted
to
make
this
implementation
asynchronous,
all
we
would
have
to
do
is
change
the
return
type
from
authorization,
for
example,
in
the
second
case,
from
authorization
to
completion
stage
of
authorization,
and
that
would
effectively
make
it
make
it
async.
A
The
implementation
itself
obviously
needs
to
return
the
right
thing,
but
that's
what
you
have
to
do
from
from
a
developer
perspective.
So
that's
on
the
server
and
then
on
the
client.
It's
kind
of
it's
it's
very
similar
to
what
micro
profile
resclient
does
right.
So
you
define
the
interface
for
the
client,
you
specify
the
method
that
you
want
to
call
and
it
can
be
a
subset
of
the
methods
that
are
exposed
on
the
server.
In
this
case,
you
only
care
about
authorized
method.
A
Here.
We've
used
authorization
here,
because
these
are
different
services.
They
don't
share
anything.
They
don't
share
the
types
I
mean.
These
types
are
compatible
at
the
serialization
level.
They're
json
compatible
effectively
because
we're
using
json
for
serialization
but
they're
not
actually
the
same
class
they're,
not
the
same
type
and
yeah,
because
they
they
shouldn't
be
shared
enough
in
a
proper
microservices
application.
A
A
So
you
can
see
here
on
the
services
dashboard.
We
have
50
members
for
each
service,
they're
still
site
safe.
I
mean
nothing's
changed
once
we
went
past
three
members
because
we
only
have
three
availability
domains,
so
we
were
site
safe
with
with
three
men
with
three
members
per
service,
but
now
we
have
50
members
per
service.
A
A
We
have
almost
a
terabyte
of
heap
in
the
cluster
and
we
are
using
about
two
gigabytes,
so
that
basically
shows
you.
You
know
again.
Scaling
was
as
simple
as
doing
cube
cuddle
scale
with
the
number
of
replicas
that
we
want
and
the
difference
is.
We
have
now
50
pods
for
each
service
and
to
make
it
more
obvious.
Let's,
let's
do
this.
A
You
can
see
that
we
have
a
bunch
of
pods
for
each
service
and
they
all
are
able
they're
all
behind
kubernetes
services
allow
them
to
both
process.
Api
requests
that
come
to
helidon,
whether
they
arrest
or
grpc,
or
to
store
the
data,
so
they're
they're
basically
used
used
for
both,
and
you
know
you
have
quite
a
bit
of
capacity.
We
could.
We
could
scale
this
more.
We
could
make
jvms
larger.
A
We
could
start
more
cluster
members,
but
unfortunately,
this
about
as
much
as
I
can
do
given
the
kubernetes
cluster
they
have
at
my
disposal,
but
I
think
that's
that's
pretty
much
all
I
wanted
to
show
so
to
learn
to
learn
more.
You
can
go,
you
know
to
learn
more
about
halidon,
you
can
go
to
helidone
io.
There
is
also
helium
blog
on
on
medium,
where
there
are
a
number
of
useful
articles
and
then
for
the
source
code
itself.
A
Obviously,
everything
is
on
github
under
github.com
oracle
helidon
and
there
is
have
it
on
slack
channel.
If
you
want
to
ask
questions
that
you
know,
ask
everyone
developers
how
to
do
something.
If
you
run
into
issues,
that's
probably
the
best
best
place
to
start
and
similarly
for
coherence,
if
you
go
to
coherence,
dot
community,
that's
where
coherence
community
edition
is
hosted.
Obviously
there
are
commercial
coherence,
editions
that
are
on
otn
and
you
can
download
them
from
there.
A
If
you
need
some
of
the
advanced
features
that
are
not
available
in
community
edition,
there
is
coherence
blog
on
medium
as
well,
where
harvey
and
I
published
an
introductory
article
in
coherence,
which
is
a
useful
read
if
you
want
to
learn
more
about
how
it
works
and
why
it
works
the
way
it
does
and
what
you
can
do
with
it.
There
is
also
an
article
that
was
recently
published
in
java
magazine
that
shows
a
different
sample
application,
but
kind
of
explains.
You
know
the
reasoning
behind
it.
A
It's
kind
of
similar
to
to
what
I
described
here.
You
can
get
coherent
source
code
on
on
github
as
well.
Under
oracle
coherence,
you
can
watch
number
of
videos
both
for
older
and
and
new
coherence
releases
on
our
youtube
channel
and
then
just
like
with
helen.
If
you
have
any
questions,
if
you
need
help
assistance-
and
you
want
to
talk
directly
to
us-
you
can
join
oracle
coherence,
slack
channel.
A
So
with
that,
I
think
I'm
pretty
much
done
and
I'm
you
know
we
haven't,
had
any
questions
as
we
went
along
so
I'll
open
for
questions.
If
anybody
has
any
yeah,
if
anyone
has
questions,
please
feel
free
to
ask
them
in
either
the
chat
bar
on
the
right
hand,
side
of
the
page
or
in
the
ask
a
question
box
at
the
bottom.