►
From YouTube: NATS Deep Dive – Colin Sullivan & Waldemar Quevedo, Synadia (Intermediate Skill Level)
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
NATS Deep Dive – Colin Sullivan & Waldemar Quevedo, Synadia (Intermediate Skill Level)
NATS is a high performance messaging server and also one of the latest additions to the CNCF. In this talk, we will make a deep dive to the internals of the project covering its design, protocol, clustering implementation, security and authorization features that make it an attractive solution for microservices and low latency applications. After this talk, attendees should be ready to become contributors and also familiar with the future roadmap of the project. If you are looking for a simple project to contribute or a fan of performance, definitely consider joining!
About Colin
Colin is the product manager of NATS (https://nats.io). He has extensive experience developing messaging products and designing distributed systems. Prior to Synadia, Colin worked at Apcera and TIBCO software.
About Waldemar
Waldemar is a Software Engineer at Synadia Communications, Inc, working on NATS (@nats_io)
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
so
I'll,
just
in
truth
once
again,
is
calling
it's
a
product
manager
from
Nance
and
Wally.
Also,
one
of
the
core
contributors
to
the
project
and
I
have
been
using
nax
based
systems
for
for
a
while.
It
seems
like
around
six
years
already,
so
it's
really
cool
how
Nance
is
really
taking
off
now
joining
the
CNC
AFM,
the
growing
ecosystem.
So
really
thank
you
for
attending
and
this
is
going
to
be
an
extension
from
the
previous
talk.
A
A
So,
let's
start
with
NAT
and
I
mentioned
by
Colleen
earlier,
but
it's
a
high
performance
machine
server
with
Mingo.
It
was
originally
written
in
Ruby
within
the
main
machine.
This
was
in
2011
or
for
Cloud
Foundry.
So
in
there
I
mean
Nance
was
like
well
the
message
board
used
for
the
command
control,
doing
the
command
control
between
the
components
in
Cloud,
Foundry
and
in
terms
of
performance.
A
It
was
still
pretty
fast
had
around
150,000
messages
per
second,
so
it
worked
very
well
like
for
even
those
in
spite
of
those
limitations,
and
the
foundations
are
still
very
solid.
Since
then
I
mean
like
the
protocol
is,
is
pretty
much
the
same.
Still
it
is.
The
implementation
is
an
again
it's
a
pure
poly
subscribe
system
on
top
of
tcp/ip,
and
you
can
use
secure
connections
with
TLS
and
from
the
rewrite
the
binary
name
is
Jeanette
steam.
A
You
have
thought
about
changing
in
today,
not
server,
but
and
this
point
unity
have
stock,
and
this
is
a
simple
example
of
the
protocol.
We
have
an
example,
demo
endpoint,
you
can
just
turn
it
into
it.
Stanley
demo
dotnet
say
you're
in
the
port
for
two
to
two
and,
as
you
start
establish
a
connection
to
to
the
server.
The
first
thing
you
will
get
is
this
infrastructure
and
a
lot
of
important
fields
use
the
maximum
payload
size.
A
So
all
the
messages
that
you
send
to
nets
are
constrained
to
one
megabyte
by
default,
or
you
can
tune
this
in
the
server.
If
you
want
same
like
larger
messages
and
this
some
basic
publish/subscribe
features
from
from
the
server
right
here
here
we
are
making
a
sub
subscription
into
the
hell
of
dot
Copenhagen
subject
and
publishing
on
the
subject
and
five
bytes.
A
This
is
going
to
be
a
just
a
world
message,
and
since
this
client
has
registered
interest
into
this
hello
dot,
Copenhagen
subject,
then
the
server
is
going
to
announce
it
that
you
have
a
one
message
on
this:
hello
dot.
Copenhagen,
that's
matching
the
first
subscription
that
you
made
and
it's
five
bytes
in
total
right.
So
the
server
is
10.
You
have
many
bytes
you
are
should
be
reading
from
the
server
right.
A
So
this
is
what
allows
you
to
the
server
to
be
payload
agnostic
right
so,
and
you
know,
Nance
doesn't
constrain
you
to
any
type
of
an
encoding,
serialization
formats
right.
You
can
use
protocol
buffers,
JSON
message
pack,
whatever
fits
your
needs
and
the
complete
protocol
is
it's
just
these
ten
commands.
You
can
probably
subscribe
messages.
There
is
an
unsubscribe
and
a
comment
where
you
can
define
a
priori.
How
many
messages
you
want
to
receive.
A
For
example,
you
can
say
I
only
want
to
receive
five
messages
on
on
this
subscription
after
that,
just
fade
away
and
stop
receiving
them,
and
you
can't
customize
your
connection
against
the
server
with
using
connect.
I
mean
this
is
important
when
handling
credentials,
for
example,
there's
the
info
protocol,
which
is
used
both
by
sharing
metadata
to
the
clients
and
also
for
gossiping
the
network
topology
from
the
cluster
right.
So
this
is
the
way
and
the
auto
discovery
mechanism.
It
works
right.
So
as
soon
as
a
cluster,
a
new
server
joins
the
cluster.
A
The
these
network
locations
are
gossip
to
the
clients
and
also
well
yeah
message
messages.
The
protocol,
which
is
for
processing
messages
and
there's
a
errors
that
notification
that
can
necessary
consent.
This
plus
okay,
is
for
when
you
have
the
verbose
smoother
on
the
server
which
is
activated
by
disabled
by
default
for
the
clients
right
and
there's
a
ping
and
pong
always
happening
between
the
client
and
the
server
so
periodically.
Every
two
minutes
there's
going
to
be
a
keepalive
happening
so
that
the
connection
is
always
healthy.
A
This
is
a
simple
example
of
using
they
go
client.
The
go
client
is
the
one
that
is
a
get.
The
most
well
is
a
canonical
implementation
for
for
what
Nats
client
well
with
the
behaviors
from
Nats
client,
and
here
we
have
an
example
of
an
asynchronous
subscriber
subscribing
to
this
hello
subject
in
takes
a
callback.
A
It
has
a
very
similar
API
for
all
the
other
client
implementations
that
we
have,
and
as
soon
as
we
publish
a
message
and
on
the
hello
subject,
then
this
well,
it
would
print
that
you
receive
this
message
right
and
the
server
and
it's
again,
seven
megabytes
binary.
No
dependencies,
you
have,
you
can
grab
one
of
the
latest
release
in
github
and
slash
releases.
A
This
is
the
starting
engine
and
see
it's
fairly
straightforward
and
it
has
by
default,
is
going
to
be
listening
on
the
port
and
a
42
to
there
is
an
extra
port
you
can
open
for
receive
gathering
a
matrix
from
server.
This
is
important
for
a
two
to
two
in
a
just
mock
test.
The
connection
yeah
simpler
subscriptions.
A
So
you
have
a
multiple
and
what
messaging
patterns
right,
so
you
can
broadcast
and
1
to
n
communication
by
using
an
but
pure
poly
subscribe,
there's
also
a
request
response
and
that
will
guarantee
like
doing
the
one-to-one
communication
and
also
there's
the
cue
subscription
that
you
can
use
to
for
doing
the
load
balancing
right
so
simply
simplest,
simple
example.
Is
your
side
doing
a
request
reply?
Well,
just
a
periodic
timer
app.
This
is.
A
A
A
A
A
Then,
let's
see
what
happens
if
I
kill
the
server,
you
will
see
that
the
client
in
the
middle
continues
to
publish
messages,
even
though
it
is
still
not
connected
right,
and
the
reason
for
this
is
that
the
client
is
has
an
internal
reconnection
logic
happening.
So
every
two
seconds
is
going
to
try
to
reconnect
to
the
server
and
it
has
like
80
megabytes
buffer,
where
it
will
continue
to
accumulate
a
pending
data
until
it's
able
to
reconnect.
A
A
Okay,
so
now
we
have
twenty
clients
that
are
consuming
all
these
messages,
and
this
will
get
very
chatty
very
very
fast.
So
there
is
a
another
API
that
you
can
use,
which
is
that
you
subscribe
in
this
case.
We
can
modify
this
subscriber
to
use
a
cue
subscription
and
they
name
a
group
for
this
set
of
workers.
So
then,
instead
of
everything,
everyone
receiving
the
message,
what
happens
is
that
the
server
will
randomly
balance
these
two
just
only
one
of
them.
A
A
A
So
by
passing,
the
D
be
flax
into
the
server.
You
can
enable
they're
both
login
to
the
for
the
server,
but
these
are
hits
a
performance
a
little
bit.
So
it's
very
useful
for
production,
buddy
setup
guys
can
increase
the
latency
from
the
request
so
its
case
I'm,
oh
yeah,
there's
no
errors
and
it's
able
to
communicate
I
just
getting
a
100
milliseconds
response
times,
but
it
would
be
faster
if
I
remove
the
bear
boss,
yeah
yeah.
A
So
this
is
like
the
three
main
features
that
you
can
get
from
from
that
one
and
communication
and
probably
subscribe
using
the
load.
Balancing
groups
and
week
use
subscriptions
and
request
response
for
one
to
one
and
the
rest
of
the
features
from
the
from
that
Sand
Gnats
project
just
build
on
top
of
these
main
three
well
building
blocks
so
back
to
the
presentation.
B
A
Moving
on
to
some
of
the
operational
aspects
from
net,
this
is
what
a
usual
production
setup
for
Nets
would
look
like.
You.
Don't
have
a
set
of
Nats
clusters
so
that
you
have
high
availability.
There
are
three
main
ports.
They
would
be
opening
this
42
for
the
clients
to
connect.
There
is
a
6
to
2,
to
4
the
clustering
peers,
to
connect
to
each
other
and
then
for
monitoring
purposes.
A
In
case
you
want
to
just
either
just
pull
yourself,
be
some
like
to
me
like
to
me
like
not
stopped
or
permitted
exporter
for
the
high
availability.
It
is
you
need
to
set
up
the
nuts
clusters
to
be
in
a
full
mesh
and
one
hop
setup
and
as
for
as
a
client,
and
that
client
doesn't
really
need
to
own
in
requires
to
be
connected
to
one
of
the
to
any
node
as
part
of
the
cluster
right.
So
it
doesn't,
then
the
next
cluster
will
be
responsible
of
routing
the
messages.
A
Rest
then,
according
to
the
interest
graph
right.
So
in
the
in
case.
In
this
case,
if
this
client
publishes
a
messaged
to
to
this
server,
I
mean
all
of
the
clients
that
are
connected
to
this
same
and
server
I'm,
going
to
receive
the
message,
and
also
the
ones
that
are
remotely
interested
into
this
message
is
going
to
be,
are
going
to
be
receiving
them.
A
Setting
up
the
Linux
cluster
and
it
can
be
done
by
just
using
the
auto
discovery
mechanisms,
so
in
this
case
the
first
one
note
that
it
doesn't
have
these
routes
and
parameter
so
that
one
is
going
to
become
the
seed
server.
Then
the
second
one
will
join
and
it
knows
that
there
is
at
least
one
seed
server
at
this
at
this
6
2
location
and
then
the
third
note
and
it's
going
to
be
joining
and
then
the
full
mesh
nuts
cluster
will
be
assembled.
So
in
terms
of
the
protocol.
A
How
it
works
internally
is
that
it
is
leveraging
the
info
comment
to
gossip
around
where
the
the
URLs
that
are
available
inside
of
the
cluster.
So
once
these
joints,
then
the
other
node
is
going
to
connect
to
this
other
one
and
that's
how
the
full
mesh
set
up
is
well,
it's
assembled
and
then
all
of
the
clients
that
were
connected
to
these
other
servers
are
going
to
receive
all
so
they're,
not
the
network
location.
So
now
they
will
be.
A
They
will
add
this
end
into
the
their
own
into
the
server
into
the
pool
of
service
that
they
can't
reconnect.
So
in
case
the
server
and
now
fails
it
will.
The
client
will
fall
back
into
the
reconnection
logic
and
then
transparently,
reconnect
into
this
other
server
that
just
joined,
so
it
automatically
fails
over.
A
A
A
Where
and
let's
say
the
for
example,
there
is
a
client
that
is
subscribing
to
us
and
hello
subject,
but
it
is
not
been
able
to
process
all
of
these
messages
that
it
is
receiving
right.
So
if
the
server
starts
accumulating
too
many
of
these
messages,
it
is
going
to
give
it
two
seconds:
a
deadline
to
drain
everything
that
is
been
sent
to
the
client
and
if
it
doesn't
read
all
of
those
bytes,
then
it
will
disconnect
the
client
and
tell
it
that
it's
a
slow
consumer.
A
A
You
could
cap
the
the
length
of
how
many
messages
should
be
pending,
for
example,
by
the
fawley,
65
65
thousand
messages,
but
you
can
lower
those
so
that
you
can
detect
earlier
that
there
is
an
applic
and
slow
consumer
error
and
the
application
layer
at
the
application
level
so
and
nuts,
and
then
next
I
want
to
cover
at
NAT
streaming
and
not
streaming
builds
on
top
of
all
the
previous
business.
It
is
request
response
protocol
using
the
request
response,
an
API
from
Nats
and
because
Nats
is
a
protocol
and
pedal
and
Gnostic.
A
We
need
to
decide
to
use
some
serialization
format.
We
decided
to
use
a
protocol
buffers
in
this
case
and
the
code
name
is
Stan
because
we
call
we.
We
say
that
is
the
complete
opposite
from
net.
So,
while
net
gives
you
the
admins
to
live
and
most
wants
the
liberal
guarantees-
and
it
really
optimized
for
well
keeping
things
simple-
and
there
is
a
little
bit
more
estate
as
part
of
a
standard,
not
streaming
server.
So
you
have
a
persistent
of
messages.
A
You
have
the
storage,
then
all
this
hacking
happening
between
the
clients
and
so
to
ensure
that
they're
consuming
the
message
so
yeah
just
in
the
in
the
client
libraries.
You
would
see
this
name
stand
like
import,
the
stand,
which
is
much
much
shorter
than
and
at
streaming
it
is
essentially
an
application
ads
application.
So
what?
If
you
reuse
this
for
an
already
available
next
cluster,
for
example-
and
it
connects
to
two
nuts
and
use
it
as
a
transport
to
enhance
the
every
mode
for
fournette.
So
in
case
we
have
a
knack
streaming.
A
A
For
now
that
you
have
a
state,
then
the
high
availability
for
not
streaming
itself
comes
also
into
the
picture.
So
one
of
the
ways
to
solve
this
is
by
the
fault.
Tolerance
move
from
not
streaming.
So
in
this
mode,
you
can
have
a
single
net
streaming
server
become
the
active
server
that
is
going
to
be
persisting.
The
messages
on
well
the
active
server
and
you
can
have
a
set
of
standby
in
a
streaming
servers
that
will
well
become
active
in
case
the
the
other
one
fails
and
there's
also
not
streaming
clustering
support
by
using
raft.
A
A
A
And
what
by
default,
if
you
try
an
app
streaming
and
actually
what
happens?
Is
that
not
Suman
is
embedding
in
the
NAP
server?
So
you
you
will
run
that
streaming
with
an
embedded
and
that's
server
where
that
is
connecting
to,
and
then
you
can
also
deploy
it
in
this
way
where
we
don't
having
to
in
case
you
don't
have
another
available
and
that's
closer,
so
moving
forward
solid
pieces
from
Nats
ecosystem,
we
have
available
docker
images
for
both
and
Nats
and
not
streaming.
A
We
have
a
permitted
exporter
and
we
showed
in
the
keynote
where
you
can
configure
it,
to
scrape
and
matrix
from
a
Sara
Lee.
Exposing
the
monitoring,
endpoint
and
now
very
cool
project
is
glue.
I.
This
is
very
sophisticated
way.
It
does.
Is
they
actually
implemented
around
a
c++
clients
for
nets
and
not
streaming
as
part
of
my
and
here's?
A
link
for
the
demo
is
a
it's
a
basically,
it's
like
this
invented
gateway
where
you
can
make
HTTP
request
and
then
publishes
messages
that
Trinette
and.
A
A
B
So
the
question
was
about
offender
offender
who
often
pluggable
authentication
and
authorization.
It's
been
a
long
a
few
days
right
now.
Today
we
don't
have
anything
pluggable,
but
what
we
allow
you
to
do
is
to
rewrite
the
configuration
file
and
then
send
a
signal
to
the
NAT
server
that
will
then
reread
and
apply
all
the
differences.
B
So
if
you
want
to
let's
say
remover
user
or
revoke
their
creds,
then
you
can
just
simply
remove
them
from
the
configuration
file,
send
a
signal,
it'll
reload
and
actually
kill
all
the
connections
that
that
user
is
associated
with
sometime
in
the
future.
We
aren't
exactly
sure
when
we
have
plans
for
a
pluggable
interface.
That
may
happen
sooner
rather
than
later,
but
are
later
rather
than
sooner
I
mean
what
we'll
see
but
some
sort
of
pluggable
interface.
That
will
let
you
call
out
to
whatever
you
need
200
or
you
know,
LDAP
whatever.
B
The
question
was:
what
authorization
to
changes?
Kick
everyone
off?
No,
it
only
kick
off
the
people
that
were
there,
the
user
that
connection
ID.
That
was
affected
and
the
same
with
permissions.
So
let's
say
you
had
somebody,
you
know
yet
Colin
publishing
to
foo,
and
then
you
change
the
permissions
where
Colin
could
no
longer
published
a
foo.
That
would
just
stop.
B
So
the
question
is:
how
big
of
clusters
have
we
tested
with
I?
Believe
there's
been
some
users
that
have
had
clusters
in
their
range
of
50
to
a
hundred
servers
and
it's
kind
of
a
tricky
question
to
answer.
As
far
as
the
penalty,
it
depends
on
how
many
subscriptions
you're
making
what
rate
you're
making
a
subscription.
So,
if
you're,
making
hundreds
of
new
subscriptions
a
second
that
interest
is
propagated
throughout
the
cluster,
you'll
have
a
penalty.
B
B
B
The
question
was:
do
we
support
exactly
once
the
answer's?
No
I?
Don't
really
think
we'll
ever
support
exactly
once.
There's
a
few
reasons
for
that
with
with
newer
applications,
they
tend
to
be
designed
to
be
idempotent
and
when
you
implement
exactly
once,
there's
a
lot
of
mechanics
behind
the
scenes
that
that
really
incur
some
big
penalties.
B
The
other
thing
is
that
exactly
once
doesn't
actually
protect
your
applications
processing.
So
let's
say
you've
received
your
message.
You
should
start
to
process
it.
You
crash
halfway
through
well
you're,
either
gonna
have
to
roll
or
crash
halfway
through
right.
If
you
did
any
any
sort
of
storage
into
a
database
or
anything
like
that,
you're
gonna
have
to
do
some
cleanup
or
fixing
later
on,
in
which
case
most
people
just
use
the
item,
potent
approach.
B
Let's
say
you
detect
an
error,
then
you're
gonna
have
to
roll
back
and
then
not
act
that
message
and
rereceive
it
anyhow.
So
exactly
once,
there's
there's
just
very
few
situations
where
people
really
need
it.
I
get
the
sense.
A
lot
of
users
think
they
think
they
need
it
when
they
don't
and
in
fact
end
up
having
to
make
things
idempotent
anyhow
to
make
their
system
stable.
That's
my
opinion.
You
know
I
could
be
wrong,
but
long
answer
to
you
know
no
and
probably
not.
B
The
question
was
with
streaming
servers:
how
do
I
scale
horizontally
or
make
it
highly
available?
Is
that
yeah?
So
what
what
we
can
do
is
streaming
servers
is
for
high
availability.
You've
got
two
options
you
can
use
raft,
which
is
your
clustering,
that
we
also
have
a
warm
failover
mode,
where
you
might
have
a
few
instances
running
that
aren't
actually
doing
anything,
but
the
ready
to
pick
up
as
soon
as
a
primary
instance
fails,
and
that's
your
high
availability
option
as
far
as
scaling.
B
So
the
question
was
about
buffering
data
and
streaming,
and
when
do
we
act?
How
do
we
manage
that
so
there's
two
modes
in
streaming,
so
we
support
an
F
sync
mode,
which
means
you
do
the
server's
do
not
respond
to
the
raft
log
append
with
an
okay
until
the
the
message
in
question
has
been
f
synched
to
disk.
We
also
offer
a
mode
that
allows
that
message
to
be
buffered
on
to
the
disk,
for
better
performance
and
for
people
that
are
ok
because
you're
gonna
have
a
quorum,
so
you're
gonna
have
maybe
five
instances.
B
So
that's
one
part
of
that
and
then
it's
just
the
nature
of
the
round
protocol
so
before
we
will
send
a
message
to
a
net
streaming
server
and
before
the
Nats
streaming
server
acts
back,
saying,
I'm.
Okay,
with
that
message,
it
makes
sure
that
at
least
a
quorum
at
least
three
of
your
five
servers
have
successfully
persisted
that
message.
B
The
question
was:
if
I
have
multiple
kubernetes
clusters
and
I've
got
Nats
deployments
in
each
one,
but
I
want
to
connect
all
of
them
globally.
What
pattern
would
I
use
today?
You
would
still
need
to
do.
The
full
mesh
was
what
we
have
today.
We
actually
are
working
on
bridging
different
clusters
together
to
solve
that
exact
problem,
and
we
should
have
that
definitely
by
q3.
B
There
is
an
impact
on
performance,
I
mean
you
have
a
quite
a
bit
more
memory
copy
from
from
your
your
socket
into
memory
and
then
back
into
every
other
socket.
So
it
depends
on
a
couple
things
since
it's
a
memory
copy,
it's
going
to
depend
on
some
of
your
CPU.
It's
going
to
depend
on
how
big
your
pipe
is,
what
sort
of
network
you're
working
on
so
the
answer
is
yeah.
There
is
a
performance
impact.
How
much
depends
on
your
resources
at
hand?
B
So
the
question
was:
are
there
headers
and
Nats?
So
I
can
pass
some
things
like
a
correlation
ID
for
a
message.
The
answer
is
no
we've
intentionally
chosen
not
to
include
headers
and
nets,
and
that
is
part
of
the
payload,
so
NASA's
entirely
payload
agnostic
and
to
that
end,
even
with
hetero
Sweden,
we
don't
include
anything,
but
they
can
certainly
be
added
on
and
that's
exactly
what
we
did
that
Wally
mentioned
in
NAT
streaming
with
using
protobuf
because
we
needed
headers.
So
we
just
use
protobuf
for
that.
B
The
question
was:
is
there
any
tracing
facility
built
in
built
into
Nats?
No,
we
actually
have
to
push
that
up
to
the
application
because
we
don't
have
headers,
so
the
application
will
want
to
have
some
sort
of
correlation
ID
or
a
unique
ID
for
tracing
like
open
tracing
with
Yeager's
Afghan,
or
something
like
that
so
yeah.
The
answer
is,
unfortunately,
the
application
has
to
do
that.
B
B
B
Okay,
so
we've
asked
for
numbers
on
the
largest
cluster
we've
ever
seen
and
throughput
so
largest
cluster.
It
was
between
I
think
close
to
a
hundred
nodes
in
the
cluster.
As
far
as
the
throughput
goes,
some
of
our
recent
tests
with
multiple
streams
darica
what
80
million
messages
a
second
aggregate,
one
just
single
pass
through
one
publisher,
one
subscriber
I,
think
we're
about
21
million
messages.
A
second.