►
From YouTube: Tutorial: Developing Production Ready Cloud Native NATS - Waldemar Quevedo & Colin Sullivan
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Tutorial: Developing Production Ready Cloud Native NATS Applications - Waldemar Quevedo & Colin Sullivan, Synadia (Limited Seating Available - See Description for Details)
Learn how to develop secure and reliable cloud native applications using NATS simple but powerful pub/sub messaging features. In this tutorial, members of the core NATS team will show how to implement from scratch a service that uses NATS with all the recommended best practices for production operation.We will cover how to setup a secure NATS cluster with TLS, how to monitor it, walk through common troubleshooting scenarios as well as how to deploy it on Kubernetes. Requirements previous to the hands-on tutorial:- Go- NATS Server (gnatsd)- Docker- minikube- cfssl (https://github.com/cloudflare/cfssl)
To Learn More: https://sched.co/GrX3
A
A
About
the
same
number
of
hands,
that's
great!
That's
great!
So
before
we
start,
you
guys
know
a
lot
about
messaging
or
a
fair
amount,
but
let's
talk
about
some
general
messaging
features
and
why
we
need
messaging,
so
messaging,
deploying
and
developing
applications
in
distributed
systems.
It's
hard!
It's
hard!
It's
complex
and
difficult
and
all
message
oriented,
middleware
vendors
like
Nats.
They
provide
a
basic
set
of
features
to
make
this
easier.
A
Not
every
vendor
provides
all
these
features,
but
in
general,
they'll
provide
things
like
multiple
messaging
patterns,
bundled
together
in
one
tech,
location,
transparency,
the
decoupling
of
data
producers
and
consumers.
I
don't
need
to
know
exactly
where
my
data
producers
are
as
a
consumer
as
a
producer.
I
don't
need
to
know
where
my
consumers
are
and
I
don't
even
have
to
know
who
sent
the
message
and
also
built-in
load
balancing
as
a
feature
as
well,
and
that
is
extremely
important
and
cloud
deployments.
A
So
what
is
Nats
Nats
as
an
8-year
old
production
proven
cloud
native
messaging
system?
We
made
Nats
for
developers
and
operators
who
just
want
to
do
their
jobs,
who
don't
want
to
have
to
spend
time
worrying
about
how
to
do
messaging
or
how
to
deploy
a
messaging.
Our
DNA
is
performance,
simplicity,
security
and
availability,
and
Nance
is
built
from
the
ground.
Up
to
the
cloud
native.
Its
first
use
case
was
the
underpinnings
of
Cloud
Foundry.
We
offer
multiple
qualities
of
service
at
most
once
at
least
ones.
A
With
streaming
support
for
multiple
communication
patterns
and
over
30
client
languages,
Nats
was
created
by
Derek,
Collison
and
Derek's
been
solving
the
really
hard
messaging
problems
for
over
20
five
years.
He
was
SVT
SVP,
&
chief
architect
at
tip
co
technical
director
at
Google.
He
architected
Cloud,
Foundry,
CTO
and
chief
architect
at
VMware.
He
founded
and
was
CEO
at
EPS
era,
and
now
it's
currently
founder
and
CEO
at
Sanae
dia,
where
many
of
the
Nats
and
maintainer
Tzar
Dericks
built
a
highly
experienced
messaging
team.
A
Around
Nats
and
we've
built
an
engaged
user
community
I
mean
I'm,
hoping
after
koukin
we'll
break
a
thousand
slack
users.
So
a
quick
bit
before
we
dive
into
the
tutorial
about
message
patterns.
Nat
supports
three
message:
patterns:
publish/subscribe
load,
balance,
cue
subscribers
and
request
reply
before
that
we
need
to
understand
a
little
bit
about
subjects.
All
this
subject
is:
is
it's
simply
a
string
representing
interest
in
data?
A
It's
what
you
publish
to
it's,
what
you
subscribe
on,
so
a
simple
subject
is
foo
and
you
can
tokenize
the
subject
hierarchically,
so
you
can
have
food
bar
and
then
what
you
can
do
is
add.
Wild
card
subscriptions,
so
you've
got
a
star
which
matches
one
token
or
greater
than
which
matches
everything
beyond
where
you
specify
greater
than
in
the
subject.
A
We
have
a
publish/subscribe
pattern
and
this
is
this
is
a
radio
station.
This
is
where
subscribers
subscribe
to
a
subject,
and
anybody
publishing
on
that
subject
gets
it
so
think
of
it.
Like
a
radio
station,
where
you
turn
your
dial
and
you're.
Listening
to
that
subject,
you'll
get
those
messages.
This
is
used
for
high
fan-out
or
specialization
or
parallelization
of
work.
So,
for
example,
a
publisher
might
publish
sensor
data
honest
on
a
subject:
one
subscriber
might
log
that
data
another
subscriber,
my
updated
UI,
a
third
subscriber,
might
check
it
and
generate
alerts.
A
We've
got
a
low
balance,
Q
pattern,
so
this
is
similar
to
the
sup,
except
that
what
happens
is
when
the
server
gets
a
message.
You've
got
subscribers
that
you've
told
to
belong
to
a
group,
and
any
subscriber
in
this
group
can
receive
the
message,
but
the
message
will
go
to
only
one
subscriber
effectively
load-balancing.
So
if
you
publish
messages,
1,
2,
&
3
in
this
diagram
here,
you'll
see
that
the
different
subscribers
in
the
group
can
get
this
message.
This
is
used
for
load
balancing
for
auto
scaling
and
for
lame-duck
mode
during
upgrades.
A
So
it's
a
very
smooth
transition
as
you
move
from
one
application
instance
to
another,
and
then
we
have
a
request
reply,
and
this
is
your
conversation.
This
is
your
phone
call.
You
can
you
you
request
the
one
reply,
err
and
get
a
message
back,
but
underneath
the
hood,
everything
in
Nats
is
a
subject.
So
we've
offered
some
optimizations
here.
You
can
request
too
many
and
tell
the
server
I
only
want
one
response
back
many
reply
errs
will
respond.
The
server
will
get
the
first
response
back
and
short-circuit
it
will.
A
It
will
prevent
the
other
responses
from
filtering
through,
and
this
is
a
great
way
to
get
extremely
low,
latency
response
from
mini
reply
errs
or
you
can
respond
with
cue
subscribers
to
the
load
balance
so
quickly
going
back.
Here's
your
cue
subscribers
and
the
messages
are
going
to
be
randomly
distributed
amongst
the
subscribers.
If
you
add
this
in
a
request,
reply
scenario,
you
can
have
many
reply,
errs
their
cue,
subscribers
and
and
they'll
handle
requests
and
randomly
these
requests
will
be
distributed
evenly.
A
You
know
generally
evenly
amongst
the
subscribers,
giving
you
an
improved
quality
of
service
or
I'm.
Sorry
improves
latency
and
groove
response
times,
along
with
that
we've
added
drain
mode,
so
supported
clients
provide
a
drain
mode
to
allow
graceful
shutdown.
So
remember
with
cue
subscribers
I
mentioned
you:
can
you
can
use
that
for
lame
duck
mode
or
upgrades
now?
Clients
can
just
issue
it.
Instead
of
closing
a
connection
draining
a
connection
they
stopped
listening,
stop
subscribing
and
handle
all
the
for
messages
that
they
may
have
received
in
the
mean
time.
A
So
this
provides
you
a
very
graceful
shutdown
to
eliminate
data
loss
allows
you
to
smoothly
downward
scale,
and
it's
great
again
for
upgrades.
We
also
have
in
the
server
auto
discovery,
so
we've
built
Nats
to
be
extremely
resilient
and
low
config.
So
in
the
cloud
this
is
very
important.
What
happens?
Is
you
have
a
Nats
cluster
up
and
running?
You
can
add
a
NAT
server
and
direct
it
to
point
it
to
any
one
of
the
other
servers
in
the
cluster
and
they
exchange
topology
information.
A
So
the
servers
will
talk
to
each
other,
saying:
hey
I've
got
a
new
server
or
hey
I've
lost
a
server.
Here's.
What
the
topology
looks
like
now.
It
goes
further.
The
server's
can
send
these
to
the
clients
and
the
clients
get
updated,
topology
information
about
the
server
cluster,
and
this
is
great
for
failover.
That
means
a
client
can
failover
to
a
server.
It
never
even
originally
knew
about,
and
it's
great
for
rolling
upgrades
in
case.
A
B
B
So
we're
gonna
show
you
some
of
the
newest
features
that
are
part
of
the
server
and
also
from
the
from
the
clients,
and
also
one
question
is
like
how
many
of
you
have
like
mini
cube,
installed
and
ok,
and
they
go
well
basically
go
installed
in
the
ok
yeah.
So
there
is
a
public
repository
here
or
is
it
if
you
want
a
cloning
clone
it
and
follow
along?
There
are
two
branches:
one
is
the
complete
version
of
the
repository
and
by
default
you're
going
to
be
cloning.
B
B
B
I
also
wrote
a
book
on
nets,
we're
gonna
be
giving
away
copies
of
cube
con.
You
use,
don't
stand
sing
by
your
booth.
Has
anyone
got
the
practical,
Nats
book
and
ready?
Okay?
Thank
you!
So
yeah
we
have
many
copies
of
this
book.
If
you
want
to
get
your
hands
on
it
and
some
of
these
materials
well,
this
kind
of
also
covers
newer
features
that
are
not
part
of
the
nut
server
and
we
brought
next
year.
You
need
a
new
edition
of
the
book,
so
this
is
more
recent
metric
material
than
the
book.
B
B
The
clients
will
be
connecting
to
this
HTTP
front-end
then,
but
behind
the
scenes
all
the
communication
is
internal
communication
is
happening
over
Nats,
so
we
have
a
set
of
a
layer
of
nodes
that
are
going
to
be
component,
I
called
and
that's
the
need
service,
and
that
is
basically
the
the
point
where
you
start
using
pops
up
to
make
requests
to
find
an
available
driver.
So,
in
the
example
application
that
I
have
my
share
in
the
repo,
you
can
send
a
request
to
find
a
driver
that
is
going
to
pick
you
up
right.
B
Other
features
can
be:
is
our
left
as
an
exercise
to
the
reader,
so
we're
going
to
cover
how
that
request,
response
flow
works
and
we're
somewhere
were
to
use
solve
the
messaging
patterns
that
calling
briefly
well
previously
mentioned,
and
so
it's
everyone,
okay
with
the
repo
okay,
so
yeah.
This
is
architecture
as
an
API
server.
You
can
is
a
REST
API
that
we
receive
a
request
and
we're
going
to
add
some
discovery,
subjects
as
well,
and
there's
going
to
be
an
ad
agent
installed.
B
That
could
be,
for
example,
in
a
car
that
are
going
to
so
that
it
can
be
discovered
to
add
sub
requests
and
the
kuroh
fits
the
NIST,
an
IFFT
service.
That
is
going
to
be
the
load-balanced
service
that
the
api
service
connecting
to
to
find
an
available
driver
I'll
go
to
the
repo
already
ok
start
sharing
code.
B
B
When
developing
the
application,
one
thing
what
I
like
to
start
is
by
crane
component
reducible
library,
where
that
all
of
the
other
components
from
the
service
are
going
to
be
importing,
and
that
is
where
we
are
going
to
be
centralizing.
Some
of
the
customer,
how
we
customize
the
Nats
connection
and
then
add
adding
some
default
subjects
that
are
helpful,
for
example,
for
discovery
within
the
within
the
micro
service
units,
so
that
you
can
have
some.
B
So
you
can
define
some
default
behavior
for
off
any1.
That's
part
of
this
service
and,
in
this
case,
create
the
component
struct
to
identify
each
one
of
the
components
from
the
service
I'm
going
to
label
them
with
an
ID
which
is
going
to
be
in
this
case
and
that's
unique
identifier.
So
we
as
per
the
Nats
project,
we
have
our
own
library
for
generating
a
very
fast
efficient
and
efficient
and
unique
identifiers,
and
this
is
one
of
the
mechanisms
that
it's
used
for
the
request
response.
B
So
we're
just
using
this
library
once
again
to
generate
a
label
each
one
of
the
components
without
unique
ID
in
teen
identity,
just
for
instead
of
an
user
ID.
For
example,
so
we're
going
to
label
each
one
of
the
components
with
its
own
ID
using
and
and
each
one
of
the
components
can
can
have
a
different
behavior,
so
we're
also
just
and
label
which
kind
of
component
there
are
going
to
be.
B
So
the
first
thing
I'm
going
to
share
is
how
to
customize
the
Nats
connection,
so
we
use
the
one
pattern:
that's
very
popular
for
gold
libraries
that
is
use
them.
It's
called
well
the
friendly
own
function,
I'm
michelle
options.
Basically,
so
when
you
are
defining
a
gnat's
connection
but
not
connect,
the
API
tells
you
that
the
first
parameter
is
a
list
of
the
server's.
B
It
could
be
a
comma
separated
list,
but
they
also
could
just
be
a
very
simple
domain
is
going
to
default,
for,
for
example,
we
could
make
it
connect
to
them
the
more
endpoint
and
just
setting
them
about
not
say
yeah.
Is
it
going
to
connecting
into
the
publicly
available
endpoint
that
we
have
and
it's
going
to
default
to
four
two
two:
two.
B
If
you
want
to
establish
a
TLS
connection,
you
can't
use
the
fine
TLS
of
the
scheme
in
this
case
for
I,
think
it's
4,
3
C
4,
4
3,
and
that
would
basically
start
a
TLS
connect,
secure
connection
against
the
server.
In
this
case
we
are
plumbing
through
the
list
of
servers
in
each
one
of
these
part.
It
is
one
of
these
components.
B
So
you
can
label
your
connection,
and
that
is
going
to
be
very
helpful
for
monitoring
purposes,
because
then
you
can
pull
the
monitoring
endpoint
for
the
server
which
is
hh2
to
you
by
default
and
list,
for
example,
based
on
that
not
only
the
connection
ID
that
you
have,
but
using
that
label
to
quickly
identify
in
the
JSON
blob
that
the
server
returns
to
you,
which
are
there
in
and
out
messages
and
bytes
per
se.
From
this
connection
and
the
list
of
subscriptions,
for
example.
B
Let's
start
the
API,
sir
API
server,
also
some
boilerplate
code
for
setting
up
the
version
4
for
setting
explicitly
what
is
the
Nats
server
that
is
going
to
be
connecting
this
component.
So
we
can
skip
this
here.
We
are
starting
to
use
the
component
library
that
so
I
made
an
constructor
whether
we
can
just
pass
it
a
string,
and
this
is
going
to
set
the
which
kind
of
component.
B
This
server
is
and
a
signal
handler
and
going
to
that
later
and
some
basic
login,
and
so
here
we
are
establishing
a
connection
by
default,
we're
just
using
the
default
parameters
from
nets.
But,
for
example,
here
we
could
now
set
their
own
customizations.
Let's
say,
for
example,
let's
say
that
Matz
reconnects,
we're
gonna
set
it
to
the
5,
and
this
is
just
going
to
be
blown
through
their
Nance
as
a
baryonic
objects
from
the
net
connect.
B
So
I
found
it
this
weight
of
using
the
net,
so
connect
the
PI
helpful
and
we
it
is
important
to
be
able
to
allow
how
to
customize
the
net
connection
to
Nets
very
component
for
because,
for
example,
let's
say
that
back
to
the
architecture,
and
so
these
layers
of
nodes
are
going
to
be
running
in
a
cloud.
Let's
say
one
of
the
kubernetes
cloud
providers
that
we
have
right
now,
but
the
agents
are
going
to
be
running
in
all
the
agents.
B
B
How
can
we
do
this
differently
for
the
agent?
So
this
is
the
agent
and
we
are
also
using
a
list
of
options
for
customize
the
connection
just
for
this
one
and
in
this
case
we're
going
to
set
it
so
that
it
never
stops
reconnecting
to
net
and
to
X.
You
can
express
that
in
the
client
by
sending
2-1,
then
it's
going
to
be
basically
endlessly
reconnecting
the
Nets.
B
B
B
Another
very
important
thing
for
nuts
applications,
and
then
you
definitely
have
to
implement
for
production
scenario,
is
that
are
the
event
callbacks
the
event
callbacks
are
called
whenever
there
is
the
client
changes.
The
state
from
being
connected
to
being
disconnected
closed
is
a
terminal
state.
So
if
the
client
ever
reaches
they
close
the
state,
it
will
never
try
to
reconnect
once
again.
B
So
at
that
point
you
have
to
recreate
an
app
connection
and
you
cannot
continue
using
the
built
in
reconnection
logic
that
the
clients
have
and
there's
also
they
discovered
handler
for
the
case
where
you
have
a
new
notes,
joining
the
cluster
and
then
the
new
IP
port.
From
that
note,
this
gossiped
all
well
all
the
client
ID
bertrise
that
you
have
set
from
that
server.
So
that
way
you
can
trace
whether
you
in
notes
are
whether
this
client
in
particular
has
been
is
perceiving
these
new
notes
that
are
joining
the
cluster.
So
I'll.
C
B
B
B
We
have
a
monitoring,
endpoint
and,
let's
know,
kill
the
server
and
I
started.
We
should
see
now
that
the
clients,
the
law,
to
which
a
component
it
is
reconnecting.
So
in
this
case
we
only
have
a
single
node
cluster,
but
by
default
it
is.
If
you
have
a
multiple
notes,
cluster,
then
it's
going
to
try
to
randomly
reconnect
to
another
one
in
the
pool.
B
B
Okay,
so
now,
if
I
leave
the
the
server
shuts
down
for
for
too
long,
the
clients
will
try
to
reconnect
the
reconnection
attempts.
They
are
not
logged
as
part
of
a
part
of
handlers.
Only
when
you
reconnect
so
eventually
they
will
run
into
the
close
connection,
and
if
we
want,
for
example,
we
could
say
that
whenever
you
disconnect
from
the
server
and
just
terminate
this
process
and
exit.
B
B
B
B
B
It's
important
when
using
nets
in
the
so
for
Nets,
the
payload
is
opaque,
and
so,
whenever
you
publish
a
message,
internets
is
just
that
blob
of
bytes
and
the
Nats
doesn't
transform
that
data
and
it
doesn't
do
any
transformation
to
that
data.
So
in
case
you
want
to
have
something
like
a
request:
ID
identifier
and
and
investigate
across
multiple
components.
How's
that
request
is
being
handled
and,
for
example,
purposes.
You
can
also
continue
to
use
the
fast
Nets
identifiers.
B
B
B
B
Okay,
so
let's
send
the
requests.
We
set
an
identifier
for
each
one
day,
the
requests
and
in
case
that
marshaling
error
failed.
Somehow,
then
we
failed
there,
and
here
is
where
we
are
going
to
sending
the
request
to
the
layer
of
Nats
nodes
where
well
the
clients
that
can
handle
finding
requests
so
the
API
server
responsibility.
It
is
just
to
call
this
order,
Nats
API
internally,
to
find
an
available
driver,
and
it
is
that
wait
well.
This
is
good
for
table
to
separate
the
concerns
of
scaling
up
that
behavior.
B
Defined
the
queue
subscription
that
is
going
to
be
processing
this
we're
using
a
queue
subscription
here
so
that
you
can
each
one
of
the
components
of
this
type
are
going
to
be
randomly
balanced
one
of
the
requests
so
that
way.
Well,
you
can
load
balanced
and
they
work
among
all
the
components
on
that
layer.
B
So
whenever
it,
whenever
the
API
server
sends
this
request
to
the
drivers
dot
find
subject,
this
nifty
server
is
going
to
receive
the
request
and
then
try
to
look
up
for
another
agent
available
for
to
handle
this
request.
So
the
next
step
is
to
define
this
drivers
dot
right
subject
that
is
going
to
be
created
by
the
agents.
So,
let's
go
to
the
agents.
B
Here
we're
using
here
in
a
synchronous
subscription
on
the
drivers,
rights
subject.
This
is
not
a
cue
subscription,
so
all
of
the
agent
components
are
going
to
be
receiving
this
request
and
then
they're
going
to
decide
whether
to
accept
it,
depending
on
some
condition
in
this
case
and
haven't
implemented
so
that
each
one
will
request,
you
can
specify
the
type
of
vehicle
that
you
want
for
this
particular
right.
B
B
This
is
a
synchronous
subscription,
and
something
important
to
highlight
here
is
that
the
internally
that
the
agents
are
going
to
be
receiving
other
requests,
but
they're
going
to
be
handling
them
one
sequentially.
So
in
case
one
of
these
responses
takes
longer
than
the
it
takes
takes
very
long.
Then
it's
going
to
block
the
rest
of
their
requests.
We
can
simulate
that,
for
example,
sending
a
lot
of
traffic,
and
you
can
see
that
you
will
see
in
the
logs
that
some
of
the
responses
are
going
to
be.
One
needs
to
restart
them.
A
Its
source
order
delivery
by
subject
so
if
one
producer
is
producing
on
foo
you're
guaranteed
that
messages
from
that
producer
on
the
consumer
side,
you're
guaranteed
that
messages
from
that
producer
will
arrive
an
order.
Now,
if
you
have
two
producers
on
foo
and
one
subscriber
those
messages
might
be
interlaced.
You
just
simply
can't
predict
that
right,
but
yeah
source
order,
delivery,
guarantees.
A
Yeah
I'm
gonna
rephrase
your
question.
The
question
was:
does
ordering
change
on
wildcards?
Is
that
that
fair
yeah?
So
so?
Even
at
that
you,
you
will
still
get
source
order,
delivery
by
the
publisher
as
a
wild-card.
So
let's
say
you
you've
subscribed
to
food
at
greater
than
and
somebody
somebody's
published
to
food
at
bar
and
food
out
bass.
A
E
B
A
A
We
do
so.
The
question
was:
do
we
support
MQTT
not
right
now,
our
plans,
sometime
in
2019,
is
to
provide
actually
tightly
integrated
support
for
MQTT.
We
we
realize
how
ubiquitous
it
is,
and
also
how
Nats
being
so,
lightweight
works
really
well
on
the
edge.
So
we
are
gonna
build
something
with
mqtt
some
bridge
or
actually
we're
really
looking
at
native
support
inside
the
net
server.
Oh.
A
A
Okay,
the
the
question
was:
will
wildcards
be
available
in
that
streaming
yeah
if
they
aren't
available
today
we're
looking
at
we
internally,
we
call
a
project
jet
stream,
its
NAT
streaming
version
two
and
we're
looking
at
wildcard
support
there.
A
lot
of
people
have
asked
for
that,
and
so
we're
adding
it
sure.
A
Yeah,
the
question
was:
does
snatch
streaming
cover
stateful
aggregations
as
well
as
stateful
topology,
there's
stateless,
yet
NAB
streaming
so
just
taking
a
step
back
net
streaming
is
actually
layer.
Atop
core
Nats.
Well,
while
he's
demonstrating
here,
is
what
we
call
core
Nats.
This
is
your
fire-and-forget.
If
you
lose
a
message.
Well,
your
application
has
to
retry.
That
does
sound
fragile,
but
if
you
think
about
it,
that's
how
the
internet
works
today,
but
then
on
top
of
cornets
we
have
nat
streaming,
which
gives
you
at
least
once
messaging
guarantees.
A
So
you're
gonna
you're
guaranteed
to
get
your
message
and
if
things
go
really
bad,
you
lose,
you
know
you
lose
and
our
connectivity
to
come
up.
There's
a
chance.
You
might
get
the
same
message
again,
but
what
net
streaming
does
is
it
allows
you
to
replay
data
by
by
time?
Give
me
the
last
hours
worth
of
data
by
sequence
number.
You
know
my
last
sequence
number
is
500
or
give
me
everything
from
sequence:
number,
250
back
and
you're
able
to
get
the
sequence
numbers
for
messages
in
that
streaming
and
then
or
last
value
cache
right.
A
My
application
comes
up.
I
need
to
know
what
was
the
last
thing
sent
so
net
streaming
does
give
you
all
of
those
one
other
thing
it
gives
use
durable
subscribers
and
what
durable
subscribers
are.
Are
it's
a
subscription
where,
if
something
bad
happens,
you
have
a
network
partition
whenever
it's
repaired
it
just
picks
up
exactly
where
it
left
off.
There
are
a
few
other
things
we
can.
We
can
go
into
more
detail,
but
that's
a
general
gist.
A
A
A
A
Yeah
there's
a
couple
things
you
can
do
there
and
it
depends
on
the
client,
but
you
in
most
clients
you
could
actually
unsubscribe
from
within
the
callback.
So
you
receive
your
message
and
then
you
unsubscribe
and
then
you're
you're
done.
The
other
option
is
to
do
a
synchronous
subscriber.
So
then
you
create
your
subscription.
You
wait
for
a
message
and
then
and
you're
actually
block
until
the
message
is
received
and
then
you
unsubscribe
did.
That
answer
your
question.
A
So
the
question
was,
you
know:
are
there
any
general
scalability
recommendations
for
Nats?
You
know
this
is
this
is
one
of
these
areas
where
it
really
really
varies
on
your
network,
your
bandwidth,
throughput
we've
had
a
user
with
a
cluster
of
four
NAT
servers,
able
to
query
50
thousand
nodes
and
get
responses
back
in
about
two
seconds
in
less
than
two
seconds.
In
fact,
I
think
it's
less
than
one
now
we
have
some
just
extremely
large
Nats
users,
but
again
it's
really
up
to
your
use
case
and
your
mileage
will
vary.
A
A
The
things
that
will
affect
the
net
server
is
the
sheer
number
of
connections
and
subscribers,
and
that's
where
you
want
to
look
at
scaling
subscribers
effect
memory
more
than
anything
else
connections.
You
know.
Naturally,
it's
a
function
of
the
instance
you're
on
how
many
you
can
handle.
Also,
if
you're
using
TLS
TLS
is
very
CPU
intensive,
particularly
bi-directional
TLS,
so
you
have
to
watch
out
for
thundering
herd
problems.
A
A
So
the
the
question
is:
how
geographically
dispersed
can
we
make
Nance?
This
is
a
great
question
because,
as
the
Nats
maintain
errs,
we've
been
working
on
Nats
to
actually
solve
this
problem
for
Sanae
dia
and
that's
an
idea
we're
offering
a
global
service.
So
we've
worked
a
lot
with
Nats
to
provide
a
concept
of
super
clusters
which
are
clusters
of
clusters
connected
together
that
use
a
spline
based
connection,
spline
base
connection
routing
and
some
some
things
like
optimistic
sense
to
really
create
a
resilient.
Yet.
A
B
Okay,
so
it
works
and
just
want
to
show
you
real,
quick
how
it
now
you
can
trace
the
flow
from
the
request
of
across
the
components
in
this
case
we're
looking
for
an
agent
that
has
the
regular
I'm
regular
bagel.
So
you
can
see
tracing
using
the
new
it's
across
components
which
one
how
long
it
took
the
request
and
I
call.
This
code
is
in
this
repo
and
then
it's
already
took
at
some
time
here.
B
B
This
is
the
result
from
running
this
manifest
here:
reducing
the
or
using
an
edge
version
of
the
server
you
can
enable
Prometheus
sidecar
for
attach
it
to
each
one
of
the
clusters
by
doing
the
using
enable
matrix-
and
we
have
these
cells-
also
permission-
that's
exporter,
that
is
going
to
be
pulling
them
from
the
monitoring,
endpoint
and
then
feeding
into
Prometheus
all
the
data.
So
it's
deploying
Prometheus.
B
B
B
Now
these
requests
are
flowing
through
the
closeness
cluster
that
we
have
in
kubernetes
and
feeding
through
the
Prometheus
dashboards
by
using
the
side,
cars
that
were
deployed
by
the
operator
and
also
another
thing
that
is
helping
with
the
deployment
of
nets
in
kubernetes.
The
operator
helps
you
to
form
the
cluster
and
also,
you
can
just
add
a
field
with
the
certificates
for
a
TLS
secret,
and
then
you
will
create
them
among
the
values.
For
you,
for
example,.
B
B
So
right
now
we
have
two:
it
is
only
going
to
monitoring
trading
clusters
within
the
default
namespace,
so
you
have
to
create
a
right
now
and
operator
for
namespace
if
that
some
people
really
like
that,
and
what,
for
others,
would
like
the
flexibility
of
having
a
single
operator
being
able
to
monitor
all
the
nodes
all
across
namespaces?
So
that's
coming
soon.
B
Something
that
is
okay,
so
this
is
num
within
kubernetes
and
think
there's
a
still
time
to
show
some
of
the
accounts
functionality
that
we
have
in
the
latest
versions
from
the
servers.
So
one
of
the
biggest
features
that
the
server
has
is
that
you
can
now
define
multiple
accounts
to
be
able
to
have
an
for
better
suit,
better
suited
for
multi-tenancy
and
use
cases.
So
I'm
gonna
show
you
how.
B
These
and
how
this
looks
so
in
case.
In
this
case,
we
have
a
new
stance
and
within
the
net
configuration
where
you
can
define
multiple
accounts.
In
this
case,
we're
gonna
have
three
different
accounts.
One
is
for
the
clients
that
are
connecting
to
the
service
another
one
for
the
the
not
service
it.
The
Nifty
service.
That
is,
the
cue
subscription
receiving
the
requests.
B
B
Okay,
so
in
this
new
service
account,
we
have
an
export
that
is,
for
the
drive
is
not
fine
subject,
and
it
also
relies
on
a
couple
of
imports.
Well,
one
import
from
another
accounting,
the
agents.
So
all
the
agents
and
the
drivers
are
going
to
be
on
this
agents
account
and
they're
going
to
be
accepting
requests
on
these
drivers
rights.
Subject.
B
B
And
this
the
other
accounts,
the
customers,
one
that
is
going
to
be
importing
from
the
leaf
service
and
the
drivers
would
find
subject
into
its
account
and
has
a
couple
of
users,
one
for
the
API
and
another
for,
for
example,
a
generic
client.
So,
let's
start
the
server
in
this
mode,
so
to
show
how
it
looks.
B
So
this
how
it
looks
when
the
server
is
based,
essentially
sharing
data
from
from
an
account
to
an
account,
that's
how
and
then
importing
and
service
functionality
do.
The
sharing
between
the
accounts
is
anonymous.
So
in
that
naita
well,
the
native
implementation,
the
clients
for
the
request/response
uses
the
ephemeral
inboxes
and
subscriptions
for
doing
the
request
response
so
locally
for
an
account.
B
B
B
A
That
a
at
a
higher
level
accounts
are
a
great
way
to
take
a
large
deployment
and
ants
globally,
even
and
put
users
and
really
policy
into
different
buckets.
So
from
an
enterprise
perspective,
you
can
set
up
a
number
of
accounts
when
you
know
it's,
let's
take
a
traditional
sales
and
support
and
something
and
something
else
right.
A
What
you
can
do
is
each
of
these
accounts
have
a
their
own
namespace.
So
if
you
have
somebody
publishing
in
the
sales
account
on
foo
and
somebody
publishing
in
the
support
account
on
foo,
those
messages
will
never
cross,
but
if
you
explicitly
export
them
and
import
them,
then
they
cross,
and
what
this
allows
you
to
do
is
to
manage
accounts
separately
and
you've
decentralized
management
of
some
of
this
stuff.
From
a
gnat's
perspective,
although
you
know
you
may
have
different
organizations
in
your
enterprise
managing
their
own
accounts.
B
So
this
is
a
quick
overview
was
what
we
did.
The
accounts
gave
basically
separated
into
three
different
accounts,
depending
on
the
behavior
three
different
components,
and
so
this
way,
for
example,
we
can
add
users
that
are
agents
that
can
accept
the
request
to
basically
the
the
drivers
agents
in
the
applications,
and
all
of
those
permissions
are
pertain
to
data
accountant.
You
can't
mitigate,
for
example,
being
able
to
act
as
a
service,
for
example,
and
so
just
to
summarize
so
far,
some
of
them
and
I
think
so.
B
The
practices
I've
shown
is
very
important
to
fournette
the
recognition
logic.
It's
really
helpful,
but
it's
also
helpful
to
the
book.
How
is
to
check
the
stability
of
the
connection
and
to
nuts
by
using
the
event
handlers
and
a
really
good
feature
from
Nancy?
They
accused
subscribers,
and
so
that
really
helps
for
doing
the
load,
balancing
and
a
lot
of
load
balancing,
which
is
in
a
very
simple
way
and
to
allows
you
to
keep
things
simple.
B
And
there
is,
there
are
two
ways
for
the
touching
from
that.
One
is
to
abruptly
closing
the
connection,
and
there
is
a
that
is
the
close
API
in
this
case.
Let's
see
see
that's
connection
closed.
So
what
this
will
do
is
that
it
will
any
pending
data
that
was
from
the
client
at
that
point
is
going
to
be
flushed
into
the
socket
and
then
close
the
connection
without
waiting
any
further
or
in
this
case,
and
if
there
might
be
some
requests
in
flight
from
the
within
this
system.
B
B
That
will
basically
first
unsubscribe
from
all
the
topics
that
you
were
registered
and
then
internally
in
the
client
dispatch
all
the
callbacks,
with
the
pending
messages
that
you
have
and
process
them,
and
once
you
are
done
flush,
any
remaining
publishing
data
and
and
closing
the
connection,
so
they
also
leave
leaves
you
at
the
closing
estate
from
from
the
server.
This
is
a
good
practice
to
be
able
to
more
gracefully
process.
The
requests,
while
operating,
for
example,
a
version
of
the
components.
B
Clustering
for
fault
tolerance,
and
so
the
operator
helps
a
lot
in
kubernetes.
If
you
have
a
contest
based
deployment
and
the
new
feature
for
the
accounts
really
help,
you
simplify
something
isolating
the
permissions
and
users
and
also
a
TLS,
and
there
might
not
be
enough
time
here
to
cover
it.
But
it's
in
the
repository
we
can
try.
B
Well,
there
are
two
TLS
configuration
that
you
need
to
think
about.
There
is
actually
three
with
the
latest
version
from
the
server
and
you
have
to
establish
TLS
for
the
clients
and
also
you
have
to
set
up
TLS
for
the
internal
routing
from
from
the
Nets
cluster
and
there's
also
the
superclusters
our
functionality
that
we
have
the
gateways.
It's
also
another
port
and
also
endpoint
that
you
have
to
secure,
and
in
this
case.
B
A
The
question
is:
what's
Nance
relationship
with
envoy?
Is
that
fair
or
you
know,
are
we
competing
yeah?
We
really
see
it
as
an
Ann
story.
So
there's
no
reason
an
envoy
can't
actually
use
Nats
under
the
covers
to
do
a
lot
of
the
work.
So
we've
we've
started.
Looking
at
that
I
know,
Wally's
looked
a
little
bit
about
what
it
would
take
to
plug
Nats
into
envoy,
so
yeah
we're
looking
into
that.
But
I,
don't
necessarily
see
this
competing
head
to
head.
I
mean
it's
it's
an
Ann
story,
not
necessarily
nor.
A
Yeah,
you
might
have
some
some
issues
with
TLS,
so
Nats
by
default
opens
up
a
connection
with
TCP,
and
then
it
does
a
little
bit.
It
gets
a
little
bit
of
metadata
from
the
server
giving
it
some
information
do
I
need
to
upgrade
to
TLS.
This
is
something
we
know
can
cause
problems
in
some
environments.
So
we'll
look
at
that,
but.
A
A
A
A
Oh
absolutely
by
default,
gnats
is
payload
agnostic
and
the
API
is
the.
The
basic
API
is
just
take
a
series
of
bytes.
In
the
example
here,
Wally's
marshalling
some
JSON,
you
know
just
easily
pass
fields
in
that
streaming.
We
use
protobuf
/,
/
cornets,
so
yeah
yeah,
the
NASA,
its
payload
agnostic.
You
could
send
whatever
you
want.
A
A
E
A
Okay,
the
question
is:
does
not
support
open
tracing
or
are
there
plans
to
support
open
tracing
we've
looked
at
this,
and
one
of
the
things
we're
gonna
do
is
provide
a
reference
architecture.
It
sits
in
our
near-term
plans
on
how
to
use
open,
tracing
or
or
one
of
the
open
tracing
project
products.
Projects
with
Nats,
but
Nats
cornets
does
not
have
the
concept
of
headers.
So
it's
very
hard
to
take
a
sticky,
unique
ID
in
without
you
know,
doing
something
with
envelope
being
the
payload.
A
So
so
do
that
and
we'll
provide
a
reference
architecture
and
how
to
do
it
and
then
users
can
choose
to
you
know,
do
this
in
their
applications,
one
of
the
one
of
the
really
cool
things
that
you
can
do,
though,
is
remember
when
I
said
you
can
just
subscribe
to
everything
and
Nats
if
you
use
that
wild
card.
So
what
you
can
do
is
have
a
client
that
subscribes
to
all
tracing
data
or
subscribes
to
everything
and
that's
actually
not
interfering
with
your
clients
when
they're
tracing.
F
D
B
Well,
yeah,
so
the
repo
is
there.
If
you
have
any
questions,
you
know
we'll
be
at
the
sign
ad
booth,
so
you
want
to
continue
discussing
some
of
the
new
features
and
the
server.
The
server
has
evolved
a
lot
this
year
and
it
is,
there
are
very,
very
cool
features
for
the
multi-tenancy,
more
graceful
shutdown
mode
from
the
server
know
of
security,
went
into
this
release
and
half
an
you
as
partisan
idea
team.
We
have
a
new
service
tackled
and
NGS
that
were
also.