►
From YouTube: Deploy Secure and Scalable Service... Derek Collison, Colin Sullivan, Waldemar Quevedo, & Jaime Piña
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deploy Secure and Scalable Services Across Kubernetes Clusters with NATS - Derek Collison, Colin Sullivan, Waldemar Quevedo, & Jaime Piña, Synadia
Services and Streams are the cornerstones of any modern distributed architecture. Communications and observability of modern systems have become just as important as the deployment of the components themselves. In this talk maintainers of the NATS projectwill create a service using NATS as the communication technology. They will show how NATS allows a service application to utilize cutting edge security with the ability to scale up and down, across multiple Kubernetes clusters and cloud deployments. This will be completely observable, with no code changes from the demo code base to global deployment. NATS allows cutting edge modern systems to be built without the additional complexity of load balancers, proxies or sidecars. NATS allows radically easy yet secure deployments across multiple k8s clusters, in any cloud or on-premise environment.
https://sched.co/UakY
A
Thank
you
so
I'm
Colin,
Sullivan,
I'm
product
manager,
Nance
and
today
Derek
Collison,
creator
of
Nats,
wall-e
couvade,
oh
and
hi.
My
opinion
are
going
to
be
presenting
we're
going
to
cover
an
overview,
the
Nance
project,
new
features
and
our
roadmap,
and
then
we've
got
some
demonstrations
for
you,
so
an
overview
of
Nats.
What
is
Nance
Nance
is
a
10-year
old
production
proven
cloud
native
messaging
system.
We
built
it
to
be
cloud
native
from
the
ground.
Up.
Ease
of
use
is
really
really
important
to
us.
A
So
to
that
end
we
built
it
for
developers
and
operators
that
just
want
to
do
their
jobs
that
don't
want
to
have
to
learn
about
messaging.
Don't
want
to
have
to
deal
with
the
details.
We
want
to
spend
more
time
doing
their
work
less
time,
worrying
about
a
distributed
messaging
infrastructure.
Our
DNA
includes
performance,
simplicity,
security
and
availability.
A
A
We've
got
a
great
ecosystem:
we've
got
over
a
thousand
contributors,
a
hundred
of
our
contributors
have
more
than
10
commits,
so
these
are
people
that
consistently
contribute
75
public
repositories
across
our
repositories.
We
have
15,000
github
stars
between
our
two
servers.
We've
got
about
a
hundred
million
docker
hub
pools
of
healthy
community
and
slack
with
over
1,600
members
and
a
good
cadence
of
net
server
releases.
With
about
five
a
year,
Nath's
was
created
in
Derek,
Collis
and
Derek
has
been
solving
the
really
hard
problems
and
distributed
computing
for
the
last
25
years.
A
A
Our
end
users,
if
you
look
at
the
slide,
our
end
users
don't
seem
to
fit
in
one
particular
vertical.
That's
because
Nats
is
a
horizontally
enabling
technology
you
can
use
it
to
solve
many
different
problems.
In
many
business
different
business
cases,
we've
got
a
number
of
net
clients
that
are
maintainer
supported
fairly
popular.
We
get
a
lot
of
contributions
to
our
clients
and
that's
use
cases
include
cloud
messaging,
which
are
micro,
services
bent
and
data
streaming.
This
is
observability
and
analytics.
I
want
to
request
information
from
services.
A
I
want
to
save
data
off
and
replay
it
later
or
command
and
control
control
planes.
We
also
work
well
in
IOT
and
edge
for
telemetry
telemetry
sensor,
data
command
and
control
again
and
we're
finding
more
and
more
people
are
using
Nats
to
augment
or
replace
older
messaging
products
that
don't
work
as
well.
In
the
cloud.
A
Let's
talk
a
bit
about
message
patterns.
We
try
to
put
message
patterns
in
our
discussions
in
two
buckets
streams
and
services.
A
stream
is
a
flow
of
data.
It
may
be
persisted,
it
might
not
be,
but
streams
can
fan
out
or
fan
in
and
then
we
have
services
services.
A
service
is
something
that
just
does
some
work.
You
ask
it
to
do
some
work
and
it
returns
a
result
back
and
we
can
load
balanced
services,
Nats
connect
as
a
layer,
7
load
balancer.
A
Underneath
these
two
buckets
there's
three
patterns
that
you
use
in
your
applications
request
reply,
publish
subscribe
and
load
balanced
queue
subscribers,
but
before
we
get
into
these
patterns,
let's
let's
talk
about
subjects
for
a
second
a
subject
is,
is
simply
a
string
that
represents
interest
in
Nats.
You
don't
address
your
endpoints.
Everything
connects
to
Nats
and
receives
data
based
on
interests
represented
by
a
subject.
So
a
simple
subject
is
just
a
string.
Foo,
you
can
tokenize
subjects
and
when
you
tokenize
subjects
you
can
then
use
wildcards
to
create
some
fairly
complex
subscription.
A
Matching
Nance
is
interesting
in
that
everything,
including
requests
reply.
Your
RPC
style
is
pub/sub.
So
if
you
want
to
look
at
what's
going
on
in
the
entire
system-
and
you
have
permissions,
you
can
actually
subscribe
to
everything
and
watch
everything
that's
happening
in
this
system,
which
is
really
good
for
debugging,
and
then
we
have
unique
subjects,
unique
subjects
that
you
can
generate
that
are
unique
and
that
creates
that
one-to-one
relationship
that
you
might
have
with
a
micro
service.
Here's
a
more
practical
example
of
what
wildcard
subscribers
look
like.
A
So
given
these
subjects
with
sensor,
sensor
data
us
CA,
San,
Diego,
sensor
errors,
us
CA,
San,
Diego
and
then
maybe
some
in
London.
This
demonstrates
how
you
can
subscribe
to
get
all
US
data
by
subscribing
to
sensors
that
data
at
us
that
anything
afterwards.
You
can
subscribe
very
specifically
to
get
data
in
this
case
only
data
from
London.
You
can
get
arrows
worldwide
and
all
errors
and
data
in
the
UK.
A
A
Publish/Subscribe
can
be
a
fan-out,
so
you
can
publish
to
foo,
and
every
single
subscriber,
that's
subscribed
to
foo
will
get
the
data.
This
is
a
fan-out
you
can
also
fan
in.
You
can
have
a
lot
of
publishers
on
foo
and
one
subscriber
on
food
that
then
aggregates
this
data.
We
have
low
balance
queues.
A
When
you
have
subscribers
subscribe
to
the
same
subject,
you
can
optionally,
give
it
a
cue
name
and
that
groups
them
together
and
then
what
the
net
server
does
is
it
load
balances
messages
that
are
going
to
these
cue
subscribers,
so
the
first
message
will
go
to
one
the
second
one
might
go
to
another
third
to
another
and
so
forth.
These
cue
subscribers
are
randomized
which,
over
a
period
of
time,
works
out
to
have
an
even
distribution.
A
Now,
let's
talk
about
performance,
scalability
and
resilience,
performance
Nance
can
perform
I
think
it's
over
18
million
messages
a
second
now,
but
on
some
decent
Hardware
18
million
messages,
a
second
with
one
server
one
data
stream
and
up
to
80
million
messages,
a
second
per
server
with
multiple
streams,
you
might
be
saying
it
well
I,
don't
necessarily
need
that.
But
what
this
translates
into
is
scalability.
A
If
one
server
can
do
18
million
messages
a
second
as
you
grow
your
business
as
you
expand,
you
aren't
gonna
have
to
change
your
server
topology
until
you
really
really
really
grow
it's
going
to
last
a
while.
How
do
we
get
here?
We
get
here
through
decisions
in
what
we
do
in
our
design
and
R&D,
so
we
designed
for
scale.
That
means
let's
say
we're,
adding
a
new
protocol
message.
Instead
of
having
the
verb
be
five
bytes
we
have
it.
A
Two
doesn't
seem
like
a
big
difference,
but
when
you
scale
to
millions
of
messages
a
second,
it
can
have
an
effect.
We
do
a
very,
very
careful
analysis
of
the
fast
path,
so
as
messages
are
flowing
through
the
servers
and
the
clients,
we
make
we're
very,
very
careful
about
how
we
lock
very
careful
about
moving
memory
around
about
these
operations.
That,
in
normal
conditions,
don't
really
make
that
much
of
a
difference,
but
at
scale
at
performance
they
can
actually
add
up
to
make
quite
a
big
difference.
A
We're
talking
millions
of
messages
a
second
by
avoiding
a
few
memory
moves
just
as
important
as
what
we
look
at
to
get.
This
performance
is
what
we
look
at
not
to
implement
in
the
fast
path.
So
exactly
once
message:
delivery
transactions.
We
don't
have
mess
schemas
built-in.
We
don't
have
message
headers
built-in,
that's
not
to
say
we
won't
do
this
someday,
but
they
aren't
going
to
be
part
of
the
fast
path
or
than
that
server.
They're,
probably
going
to
be
some
sort
of
subsystem.
A
Health
and
availability
of
the
system
as
a
whole
is
prioritized,
so
traditional
messaging
and,
and
so
the
legacy
messaging
products
had
takes
that
they
take
effort
to
prop
up
and
try
to
deliver
that
message
to
a
poorly
behaving
consumer
and
when
they
do
this,
this
this
hurts
the
system
as
a
whole.
You're
taking
resources
away
from
the
system
as
a
whole,
so
Nats
has
taken
a
different
approach.
We
have
selfish
optimization
in
the
server's,
so
if
a
poorly
behaving
client
the
clients
not
keeping
up
the
NAT
server
chops
it
off.
A
At
that
point,
someone
in
DevOps
when
I
take
a
look,
determine
that
that
node
might
need
more
resources
to
figure
out,
maybe
there's
something
wrong
with
the
code,
but
that
keeps
the
rest
of
the
system
as
a
whole
performing
well.
We
have
full
mesh
clustering
in
the
NAT
servers
to
make
sure
Nats
is
always
available
and
server
and
client
connections
can
self
heal.
This
creates
a
dial
tone,
a
utility
that
you
can
use
in
your
enterprise.
That's
always
on
and
always
available
for.
Your
communication
needs.
A
Simplicity
is
really
important
to
us,
so
the
servers
are
single
binary.
The
NAT
server
is
about
a
seven
point:
eight
docker
image.
We
have
a
text-based
protocol
that,
let
me
clarify,
there's
there's
sometimes
some
confusion
here.
While
the
protocol
verbs
and
itself
are
text
base,
our
payloads
are
agnostic.
We
can
do
binary
payloads,
but
what's
what's
kind
of
cool
about
the
text-based
protocol
is
it
makes
it
really
easy
to
create
clients,
we're
very
low
configuration?
Clients
only
need
a
Ural
and
credentials
servers
autodiscover.
A
So
you
don't
need
a
lot
of
configuration
in
the
servers
and
then
you
can
share
configuration
files
amongst
servers.
You
can
include
configuration
files
and
other
configuration
files,
making
it
easier
to
configure
and
there's
a
very
simple
and
straightforward
API.
We
have
auto
discovery
built
in
and
ask
what
does
this
mean?
This
means
I,
you,
cluster
net
servers
and
you
add
a
server
into
the
cluster.
The
servers
talk
to
each
other
and
adopt
this
new
server
into
the
cluster,
adding
in
its
full
mesh.
A
There's
no
additional
configuration
you
need
to
do
other
than
when
you're
letting
this
new
server
join
the
cluster.
You
point
it
to
any
other
net
server
in
the
cluster.
This
means
in
the
cloud
you
can
scale
up
or
scale
down
your
cluster
as
you
need
to,
but
there's
more
in
that
this
topology
is
optional,
also
pushed
on
to
the
clients,
so
that
means
a
client
can
failover
to
a
server
it
never
originally
knew
about.
A
These
are
great
for
rolling
upgrades
as
you're
rolling
upgrades
you
take
down
servers
and
put
new
servers
up,
and
the
clients
can
always
fail
over
and
have
another
server
to
backup
to.
We
support
2
delivery
modes
at
most
once,
which
is
Coronets.
This
is
no
guarantee
of
delivery.
Messages
can
be
lost,
applications
must
detect
and
handle
these
lost
messages.
It
sounds
fragile,
but
if
you
think
about
it,
that's
how
the
internet
works
today
and
it
works
pretty
well
right.
Request
reply,
I,
send
a
request:
I
wait
for
a
reply.
A
If
I
don't
get
one,
then
I
just
send
it
again.
That's
actually
pretty
simple
application
logic
and
we've
seen
it
work
extremely
well
and
scalable
and
highly
skilled
systems,
but
then
we
also
have
at
least
once,
which
is
net
streaming
and
jet
stream.
This
is
where
a
message
will
always
be
delivered
in
certain
cases
where
you
have
errors,
a
message
might
be
delivered
twice.
A
Some
systems
offer
exactly
once
off
line,
I'll
be
happy
to
argue
that
it's
unnecessary,
because
it's
very
complex
and
inevitably
this
complexity
makes
it
very,
very
slow
happy
to
talk
about
that
offline.
We
could
spend
hours
talking
about
exactly
once
with
NAT
streaming
and
jet
stream.
Nat
supports
at
least
once
delivery.
You
can
replay
by
time
or
sequence
number
it
can
act
as
a
last
value
cash.
You
have
durable
subscribers,
which
are
subscriptions
that
receive
messages.
And
then,
if
something
happens,
you
lose
your
connection
for
a
while.
A
It
comes
back
up,
it'll
just
pick
up
where
it
left
off.
You
have
rape
matching.
You
have
memory
file
or
database
storage,
high
availability,
we've
got
a
couple
of
different
fault,
tolerant
modes,
and
then
you
can
scale
through
partitioning.
That's
vertical
partitioning,
not
horizontal
across
servers.
So
no
such
thing
as
rebalancing
in
that
Nats
gives
you
a
number
of
building
blocks
to
build
very
complex
or
very
simple
topologies.
Whatever
you
need
at
the
base
is
a
standalone
net
server.
A
The
standalone
net
server
can
be
clustered
to
form
a
cluster
with
a
number
of
other
net
servers.
These
clusters
can
be
connected
together
to
create
what
we
call
a
super
cluster.
These
are
great
for
wide
gos
I'll
go
into
more
details
on
that
in
a
second
and
then
we
also
have
the
concept
of
a
leaf
node,
which
is
a
net
server
off
on
its
own
connected
into
a
super
cluster
or
cluster
net
server
clusters.
A
full
mesh
one
hop
that's
great
for
preventing
circuit
from
loops
messages,
only
Traverse
clusters,
where
there's
interest.
A
We've
got
super
clusters
again:
clusters
of
clusters.
Super
clusters
aren't
full
mesh
like
basic
Nets
clusters.
It's
a
spline
based
technology,
where
each
server
is
connected
to
at
least
one
other
server
and
the
other
cluster,
and
these
connections
are
optimized
for
LAN
links,
so
they're
there
low
chatter.
But
yet
you
still
have
resiliency.
If
you
lose
almost.
If
you
lose
any,
but
one
of
these
connections
you
can
you'll
still
have
connectivity.
A
Multiple
clusters
can
be
linked
together
to
form
super
clusters,
so
you
can
build
just
really
vast
ployment
over
lands
and
locally
you'll,
get
the
performance
and
and
super
high
resilience
and
scalability
within
a
local
cluster.
Yet
have
connectivity
all
over
the
world.
If
you
want
and
then
we
have
leaf
nodes,
a
leaf
node
is
a
single
NAT
server
connected
to
a
cluster
or
remote
server.
They
extend
clusters
via
a
hub-and-spoke
technology,
leaf
nodes.
A
Allow
you
to
bridge
separate
security
domains
in
a
bit
we'll
talk
about
NAT
security,
but
let's
say
you've
got
a
different
type
of
security.
You
need
on
the
edge.
It
can
be
physical.
So,
for
example,
your
clients
can
only
connect
to
a
loopback
address.
That
is
a
leaf
node
that
is
then
connected
to
a
NAT
supercluster.
A
These
are
ideal
for
edge
computing,
IOT
hubs,
data
centers,
what-have-you,
there's,
there's
many
use
cases
for
this,
and
then
they
transparently
bridge
on-premise
and
cloud
deployments.
One
of
the
cool
things
about
Nats
is
your
clients,
don't
care
anything
about
the
server
topology.
All
they
need
is
a
server
to
connect
to.
So
you
can
change
your
back-end
topology
without
touching
your
clients
at
all.
A
There
we
go
here's
a
hypothetical
global
deployment
with
a
cluster
running
at
kubernetes
in
San
Diego,
let's
say
of
a
cluster
in
Berlin
running
on
virtual
machines.
Cluster
in
London
you've
got
a
leaf
node
off
of
Berlin.
This
could
be
running
on
a
small
device.
We've
run
that
servers
on
raspberry
PI's.
You
can
have
a
leaf
node
in
the
local
cluster
elsewhere,
which
might
be
a
backup
data
center.
A
So
you
can
build
so
you
can
build
very
complex
topologies
with
these
building
blocks.
Now
one
of
the
benefits
that
we've
we've
added
in
these
global
topologies
is
geo.
Aware
cue
subscribers,
so
we're
going
to
take
a
very,
very
simple
cue
subscriber
example.
Both
of
these
subscribers
are
subscribed
to
the
foo
they're
part
of
the
same
group,
but
they're
spread
across
two
different
clusters.
A
Message
flow
will
only
go
to
the
one,
because
the
local
queue
subscriber
is
always
preferred.
If
you
need
to
scale
then
locally,
you
just
add
more
queue
subscriber
queue
subscribers.
But
what
happens
when
you
lose
this?
Your
local
keys
subscribers
you've
lost
all
your
accused
subscribers
locally.
Then,
and
only
then,
will
you
go
to
the
remote
cluster.
A
If
there's
multiple
regions,
the
lowest
RTT
is
chosen,
so
we
attempt
to
give
you
the
best
latency
and
when
you
bring
your
local
one
back
message
flow
returns
to
prefer
the
local
queue
subscriber.
So
what
does
this
mean?
You
have
disaster
recovery
with
runtime
scalability.
You
can
change
ad
scale
up
or
down
at
runtime,
with
no
configuration
and
you'll
get
the
best
latency.
A
Security
is
very
important
to
us
and
we
support
all
you
know.
What's
becoming
the
basics,
we've
got
full
TLS
support.
We
support
CAS,
bi-directional
TLS.
We
default
to
the
most
secure
ciphers,
but
if
you
need
to
you
can
add
other
ciphers,
we
support
using
the
DNR
sand
and
certificates
as
an
identity
and
Nats
to
map
that
for
permissions
for
pub/sub
permissions,
we
support
standard
user
password
off.
As
I
mentioned,
we
have
permission,
you
can
decide
what
identities
can
subscribe
to
or
publish
to
what
topics,
what
subjects?
A
And
you
can
change
these-
that
configuration
reload
at
runtime
and
you
just
send
the
net
serve
or
signal
it'll
reload,
all
these
permissions,
so
you
don't
have
to
have
any
downtime
for
this
and
we
have
an
operator
mode.
Back.
Last
year,
Nats
2.0,
we
created
another
security
scheme,
that's
actually
very,
very
powerful.
In
this,
you
have,
you
have
a
thing
called
an
operator
and
operators,
your
enterprise,
that's
it's!
It's
sort
of
your
global
deployment.
You
have
under
your
operator.
A
You
have
accounts
and
under
accounts
you
have
users,
so
an
operator
serves
as
a
root
of
trust.
The
system.
This
is
you
know
again.
Your
enterprise,
your
operators,
create
accounts
for
account,
administrators
and
a
conscious
record
represents
an
organization
with
the
secure
context.
It
could
be
a
group
of
micro
services,
it
could
be
an
organization
in
your
enterprise.
A
It
can
be
really
whatever
you
want
it
to
be,
but
it
provides
a
secure
messaging
context.
I'll
talk
about
this
in
a
minute
and
accounts
can
define
their
own
limits
and
then
they
can
also
securely
expose
services
and
streams
for
cross
account
communication
and
then
under
accounts.
You
have
users,
here's
where
you
can
assign
very
very
specific
credentials
and
permissions
for
user
access
into
the
accounts,
as
an
account
is
an
isolated
communication
context
and
and
what
it
has
done
for
Nats
is
it's
created
true
multi-tenancy
within
Nats,
you
have
your
Nats
deployment.
A
You
have
accounts
configured
on
that
a
an
application
can
connect
anywhere
in
apps
and
where
that
account
lives,
that
application
can
connect
and
it
acts
if
you
will
as
a
container
for
messaging.
This
separates
your
deployment
from
your
business
use
case.
So
now,
if
you
want
a
data
silo,
you
can
define
it
and
it
doesn't
have
anything
to
do
with
the
topology.
A
You
aren't
forced
into
creating
data
silos
because
of
Technology
limitations
of
your
distributed
computing
system,
they're,
easy,
secure
and
cost
effective,
so
you've
got
one
Nats
deployment
for
your
operators
to
manage,
and
then
it's
decentralized,
if
you
let
organizations
have
their
own
account,
they
can
create
users
all
day
and
create
and
revoke
users,
and
your
operations
doesn't
have
to
worry
about
that.
Your
account
owners
do
it's
a
distributed,
decentralized
security,
and
then
you
can
share
data
between
accounts.
A
You
can
create
secure
streams
and
services,
so
if
I
have
an
account
and
and
Derek
has
an
account,
we
want
to
share
some
data.
I
export,
a
subject
that
some
of
my
micro
services
might
publish
on
Derek's
interested
in
that
Derek
imports
that
subject
only
when
we
both
mutually
agree
will
that
data
flow
and
that
again
helps
with
the
centralized
management
of
security.
This
is
going
to
be
very,
very
important
as
IOT
explodes
its
aren
t
exploding
as
IOT
continues
to
grow.
How
are
you
going
to
manage
these
billions
upon
billions
of
devices?
A
Well,
you
have
to
do
it
in
some
sort
of
distributed
fashion.
To
do
a
centrally
is
would
be
quite
a
headache
behind
all
this.
Rn
fees
and
keys
are
8255,
one,
nine
keys
that
Nats
uses
to
sign
and
verify
accounts
and
users,
and
the
operator
there
they're
very
fast
resisted
side-channel
attacks
and
the
way
we
use
them
is
that
is
in
a
way
that
the
NAT
servers
never
ever
ever
need
to
see
a
private
key.
A
The
server
sends
announced
during
a
connect,
Barrett
and
then
the
client
with
the
client
signs,
the
nonce,
with
its
private
key,
sends
it
back
to
the
server
and
then
the
server
verifies
it
through
the
account
key.
This
allows
us
to
begin
to
create
a
zero
trust
system
as
we're
moving
forward.
We're
going
to
add
some
new
features
that
our
angle
is
a
complete,
zero
trust
system,
and
then
we
have
JW
cheese,
JW,
TS
encapsulate
and
hold
all
the
information
about
users
and
accounts
and
permission
sets
and
the
associated
permission
sets
so
in
operator
mode.
A
Jw
TS
are
used
to
represent
a
user
and
account
a
cluster
server.
They
contain
an
account
and
key
the
public
n
key
again.
Private
n
keys
are
kept
by
your
clients,
they're.
Never
read
accessed
or
accessed
by
the
server.
We
have
a
public
n
key,
which
is
a
just
your
subject:
a
friendly
name,
permissions
limits
and
not
before
and
not
after
so
I
create
a
user.
I
say
this
user
won't
be
valid
until
tomorrow
and
then
by
the
way,
this
users
going
to
expire
next
week,.
A
We've
got
a
tool
to
create
JW
TS,
you
can
create
the
operator
accounts
and
users,
you
can
create
imports
and
exports.
You
don't
have
to
mess
around
with
JW
T's.
We've
got
a
nice
client
tool
to
do
this,
and
then
this
tool
can
also
deploy
JW
T's,
that
you
can
upload
to
something
called
an
account
server
or
create
configure
file
configuration
files
with
these
embedded
JW
tees
for
you,
so
you're
gonna
have
all
these
users
and
accounts
floating
around.
You
need
a
way
to
manage
them
in
the
net
server.
You
specify
a
resolver.
A
We've
got
two
flavors
of
the
resolver
right
now.
What's
called
a
memory
resolver,
which
means
basically
just
means.
Look
at
my
own
memory
space
from
what
I've
configured
in
my
configuration
file,
these
are
JW
TS
embedded
in
the
configuration
file.
They're
ideal
for
static
deployments
start
going
to
change
a
lot.
You
don't
have
a
lot
of
accounts
if
you're,
let's
say
you're,
creating
a
SAS
system
using
this,
then
you
may
have
a
lot
of
accounts
being
used
for
a
Nance
used
in
Nats,
and
in
that
case
you
can
use
an
account
server.
A
So
you're
all
your
servers.
Let's
say
a
user
tries
to
connect
the
server
says:
hey
I,
don't
recognize
this
account.
He
goes
and
tries
to
resolve
it
with
an
account
server
that
looks
it
up
feeds
it
back
the
server
caches.
He
account
data
and
you
can
then
manage
millions
upon
millions
of
accounts
without
having
them
configured
in
every
server.
It
supports
mirroring
for
performance
and
backup,
and
these
are
used
when
accounts
are
frequently
created
or
expired.
A
So
some
of
the
new
features
and
that's
we're
introducing
Jetstream
some
of
you
may
have
been
hearing
about
this
for
a
while.
We
built
Jetstream
to
be
the
next
generation
streaming
system
and
we
had
a
number
of
goals.
You
know
it's
like
the
rest
of
that.
It's
got
to
be
easy.
It's
got
to
be
secure
and
work
well
and
that's
Oh
security.
It's
got
a
scale.
It's
got
a
scale
horizontally.
As
ingestion
rates
increase.
We
want
to
be
able
to
scale
out
without
changing
anything
in
the
clients.
We
want
to
support
multiple
use
cases.
A
Self-Heal
always
be
available
like
the
rest
of
Nats.
One
of
the
things
we've
heard
a
lot
from
from
you
guys.
Our
ecosystem
was
that
the
net
streaming
API
was
too
far
from
the
Nats
core
API,
so
the
Nats
streaming
or
the
jet
stream
API
will
be
built
into
the
Nats
core
API
it'll
use
Nats
core.
We
are
also
going
to
support
wildcards.
A
The
system
has
to
allow
Nats
messages
to
be
part
of
any
sort
of
a
message,
grouping
message
set
as
desired
and
we're
going
to
remain
payload
agnostic,
also
no
third-party
dependencies,
so
jet
streams
well
on
its
way
supports
at
least
once
delivery.
Much
of
the
same
things,
those
of
you
that
are
familiar
with
net
streaming
and
what
we
talked
about
earlier
supports
much
of
those
same
things.
It's
going
to
be
embedded
as
an
that
server
subsystem
with
an
option
to
enable
that
means
in
your
deployment.
A
You
might
have
you
know
a
dozen
NAT
servers
and
maybe
four
or
five,
our
Jetstream
enabled
wildcard
support,
support,
snaps
to
old
security
data
at
rest.
Encryption.
One
of
the
things
we've
heard
a
lot
from
the
community
is:
how
do
I
erase
an
individual
message?
This
is
all
driven
by
gdpr.
We're
going
to
support
that
now
want
to
support
where
you
can
write
over
a
message
with
a
random
series
of
bytes
effectively
deleting
it
from
the
system,
horizontal
scalability,
and
then
you
can
persist.
A
Data
via
message
sets
or
replay
with
observables
message
sets:
are
groups
of
persisted
messages
they're
created
at
runtime,
no
need
to
provision
them
ahead
of
time,
and
they
have
various
various
policy
set
per
unique
message
set.
Observables
are
how
you
pull
data
out
of
the
system
they're.
How
they're?
How
you
get
data
out
of
message
sets
and
they've
got
some
policies
that
dictate
how
these
message
sets
are
consumed.
A
A
So
message
sets
support
various
retention
policies
that
they
determine
when
the
data
is
rolled
off.
So
you
have
a
stream,
so
the
stream
just
means
I'm
going
to
set
some
other
limits
and
that's
when
data's
roll
off
I've
get
interest.
Message
are
retained
until
all
other
observables,
either
durable
and
more
ephemeral
have
consumed
the
message
and
then
it's
rolled
off
or
work.you,
which
allows
multiple
consumers
to
pull
data
from
a
message
set.
A
Limits
are
applied
which
is
similar
to
our
streaming
product
today
with
max
messages
max
bytes,
max-age,
and
then
we've
got
the
other
side.
The
observable,
where
we're
gonna
pull
data
from
a
message
set,
we're
gonna
pull
data
from
the
streams
that
we
have
saved
off.
So
you
have
a
delivery
subject
and
act
policy,
durable,
name,
start
position,
start
type,
a
replay
policy
or
a
subset.
So,
let's
see,
if
there's
an
existing
message
set
out
there,
that
has
several
subjects
you're
only
interested
in
one.
You
can
actually
pull
just
a
subset
from
that
message.
A
Set
nak,
wait,
nak
policy,
so
act
policies
dictate
how
the
observable
behaves
when
they're
reading
messages.
I
know
this
is
a
lot
of
information.
I'm
gonna
try
to
distill
it
down,
but
indirectly
it
just
defines
what
a
lost
message
means
and
defines
your
performance
characteristics
that
you
want.
So
none
is
just
turn
on
the
firehose.
A
If
I
miss
a
message
so
be,
it
all
means
that
you're
going
to
give
me
a
I'm
going
to
receive
a
stream
of
messages
and,
let's
say
I,
receive
10
when
I
act
number
10,
it
implicitly
acts
all
messages
before
that.
That's
a
very
performant
way
of
acknowledging
messages,
I've
received
and
then
explicit.
Every
single
message
requires
an
ACK
or
an
ACK.
Resend
me
this
message:
replay
policies
can
determine
the
rate
of
replay.
A
You
have
instant,
which
is
just
replay
the
messages
as
quickly
as
you
can
I
again
turn
on
the
fire
hose
or
you
have
original,
where
you
replay
messages
with
the
same
timing
that
they
originally
arrived
into
the
NAT
system.
This
is
a
pretty
big
for
testing.
You
have
those
those
corner
cases
where
you
might
get
a
burst
of
data.
Some
of
your
applications
behave
weird.
You
can
replay
this
data
to
recreate
that.
A
So
now
that
jet
stream
is
coming
out,
what's
going
to
happen
in
that
streaming,
nat
streaming
continue
to
be
supported
through
june
of
2022.
We
do
have
50
million
doctor
downloads
and
we
know
it's
deployed
in
production
globally,
so
we
aren't
going
to
just
get
rid
of
it.
Security
fixes
and
bug
fixes
will
be
applied,
but
moving
forward
new
nets
enabled
applications
should
prefer
jet
stream
will
provide
a
migration
path,
use
Jetstream.
A
We
are
entirely
sure
what
that's
gonna
look
like
right
now,
but
we
would
love
input
on
that
and
then
also
new
development
in
terms
of
persistence
and
durability
is
going
to
happen
in
jet
stream.
Moving
forward,
we've
also
created
some
reference
clients
for
distributed
tracing
in
Java
and
go,
and
in
this
case
we
use
open
tracing
with
Jaeger
check
out
the
repos.
If
you
want
to
do
tracing
in
your
application
and
see
where
messages
are
going,
how
long
responses
are
taking?
We've
got
that
option.
A
We've
got
a
number
of
integrations
with
spring
we've
got
a
Kafka
bridge,
MQ
series
bridge
we're
willing
to
build
more
bridges.
Just
let
us
know
what
you
need
and
we
have
service
observability,
so
you've
got
the
tracing
and
open
tracing
where
you
enable
that,
in
your
application,
what
we
built
into
Nats
is
observability
in
your
services.
As
messages
flow
through,
so
you
don't
need
to
do
anything
to
your
applications.
You
just
turn
on
this
switch.
When
you
create
a
service
when
you
export
a
service,
you
turn
on
this
switch.
A
You
subscribe
and
you
get
this
latency
data
again.
Your
applications.
Don't
have
to
do
anything
for
this.
You
get
this
for
free.
One
of
the
things
that's
been
happening
is
some
of
the
dance
maintainer
x'
through
cinavia
have
been
going
on
a
lot
of
POCs
and
we're
doing
a
lot
of
global
deployments.
We've
gotten
pretty
good
at
it.
We
can
usually
do
one
within
a
day,
but
one
of
the
things
that's
really
difficult
is
monitoring.
A
You
need
to
set
up
sidecars
exposed
ports
everywhere
in
multiple
data,
centers
open
up
holes
and
firewalls
to
get
all
this
data,
so
we
thought
about
it
and
decided
to
build
in
that
surveyor.
So
nat
surveyor
from
one
place
plugs
into
your
Nats
network
and
with
the
right
credentials.
You
can
monitor
your
entire
deployment
from
the
single
container
process.
It
provides
a
comprehensive
view,
you're
viewing
the
whole
world.
All
you
do
is
run
it
and
connect.
There's
no
side
cards
to
deploy,
you
can
deploy
it
with
kubernetes
dr.
A
Campos,
bare-metal
I
personally
run
it
with
docker
compose
on
my
laptop
and
I
just
plug
in
using
kind
of
restricted
system
credentials
that
can
only
read
and
I
just
connect
to
a
net
server
deployment
and
see
what's
happening.
It
does
require
a
Nats
OH
security
and
some
set
some
level
of
system
credentials.
A
Here's
what
it
looks
like
my
name
is
Griffin
and
Prometheus.
It's
just
a
very
easy
way
to
monitor
we're
doing
or
we're
kind
of
retooling.
Some
of
our
kubernetes
deployments.
We're
gonna
have
a
single
command
line
to
install
multiple
server
cluster
to
install
that
streaming
to
install
surveyor.
So
it's
one-stop
shopping
for
kubernetes.
You
just
run
this
one
line
and
you'll
have
everything
installed.
These
would
be
with
stateful
sets
the
Nats
operators
also
changing
to
use
stateful
sets
internally
and
then
monitoring
as
well.
We've
retooled
our
dock,
so
our
dock
is
now
searchable.
A
A
What
are
we
going
to
be
doing
moving
forward,
so
jet
streams
going
to
be
released?
Surveyor
is
almost
ready
for
an
official
release.
We're
gonna
have
service
abstraction
api's.
So
when
people
are
writing
services,
they
tend
to
do
the
same
thing.
There's
a
number
of
things
that
are
almost
boiler
point.
The
boiler
plate
when
you
write
a
service,
so
we're
gonna
provide
an
API
to
assist
you
with
that.
We're
gonna
do
additional
tutorial
solutions.
A
What
topology
should
I
use
when
how
do
I
create
these
patterns
that
I
might
need,
or
when
would
I
need
a
certain
pattern
versus
another
and
then
moving
forward
beyond
that
or
where
we
also
are
gonna
do
native
MQTT
support?
If
you
think
about
leaf
nodes,
you
can
have
a
global
NASA
deployment.
You
stick
a
leaf.
Node,
that's
mqtt
enabled
you
can
connect.
You
can
talk
to
many
many
different
devices
that
are
already
out
there
already
deployed
already
out
in
the
wild
and
bridge
them
into
Nantz
WebSocket
support.
A
A
So
what
I'm,
gonna
demo
first
I'm
gonna
do
some
demos
and
then
Wally's
going
to
come
up
and
then
Derek's
gonna
come
up
and
demo
jet
stream
and
talk
about
jet
stream.
So
what
I
want
to
show?
First,
is
I've
created
this
I'm
going
to
demonstrate
just
the
geo
aware,
cue
subscribers,
so
I've
got
this
simple
reply
err.
So
this
is
your
service.
This
is
what
you're
it's
something
pretending
to
do
some
work
and
sending
a
response.
I
connect.
A
The
Nats,
like
you,
subscribe,
I
added
a
little
bit
of
jitter
here,
just
to
pretend
I'm
doing
a
little
bit
of
work
and
then
return
the
response.
I've
got
a
requester
there's
requester
I'm
gonna
run
for
my
laptop
this
requesters
a
bit
more
involved
because
it's
prometheus
enabled
I
wanted
to
show
you
something
something
beyond
a
terminal,
so
I've
been
able
dit
with
prometheus
I've
got
a
little
dashboard.
I'm
gonna
show
what's
going
on
with
this,
and
then
here's
a
docker
compose
file,
I'm
gonna
run
locally.
A
What
I've
already
set
up
in
digital
ocean
I've
set
up
a
responder
in
India
responder
in
London
and
responder
in
San
Francisco,
rather
than
set
up
an
entire
global
deployment
for
the
demo
today,
I'm
using
NGS,
which
is
Senay
Diaz
global
deployment,
I
created
a
free
developer,
account
last
night
on
it
and
just
set
this
up
so
makes
it
a
lot
easier.
So.
A
My
service,
what
I've
done
is
I've
taken
that
service
requester
service
reply,
err
uploaded
it
up
into
docker.
I've
got
some
credentials
for
NGS,
then
I'm
going
to
connect
to
and
run
so
I
want
to
start
the
one
in
India
and
this
guy.
This
is
just
listening
for
messages,
and
here
is
my
requester
and
I'm.
Going
to
run
this
via
docker
compose
all
I.
Do
is
I
run
the
docker
compose
that
I've
already
shown
you
with
Prometheus
core
fauna
and
my
requester.
A
So
now,
because
I'm
sending
this
from
my
laptop,
it's
going
up
through
the
Wi-Fi,
probably
not
getting
the
best
latency,
but
going
up
through
my
Wi-Fi
up
through
the
Wi-Fi
here
through
the
convention
center.
It's
finding
the
nearest
local
cluster,
because
NGS
is
geo,
a
fennec
which
is
probably
in
San
Fran.
My
people
might
be
closer
and
then
my
cue
subscriber
in
San
Fran
is
receiving
these
messages.
Note
that
the
cue
subscriber
in
India
and
this
huge
subscriber
in
London
isn't
really
doing
anything
right
now.
A
I'm
gonna
reload,
my
dashboard
and
here
we're
already
getting
some
latency
data.
You
can
see
Layton
sees
following
into
buckets
of
between
50
and
100
microseconds,
not
too
bad,
not
good,
but
what
we're
gonna
do
is
when
we
let
this
run
and
again
remember.
The
data
is
going
through
my
laptop
through
the
Wi-Fi
up
to
San
Fran
right
now.
A
And
there
we
go
London
took
over
that
took
longer
than
I
expected,
but
in
London
took
over
so
now
I'm
sending
data
across
the
pond
to
London,
so
we're
going
halfway
across
the
world
and
back.
If
I,
look
at
what
my
requester
programs
doing
we're
going
to
see
that
latency
spiked
up
to
now
200
and
might
hover
around
Oh
180
milliseconds
we'll
find
out.
A
At
this
point,
it's
retrying
it's
timing,
out,
I,
think
of
a
three
second
timeout,
and
now
here
we
go
Bangla
door
took
over.
So
now
what
just
happened:
I've
had
three
disaster
recovery
services
and
the
local
one
just
died.
It
went
to
the
one
with
the
lowest
latency
in
London,
the
next
lowest
latency
and
then
I
killed
London,
and
then
it
went
all
the
way
to
India.
But
look
when
I
bring
this
back
there
we
go
I'm
back
online
with
my
local
cue
subscriber.
So
this
is
essentially
using
Nats
for
disaster
recovery
for
free.
A
A
A
B
So
my
name
is
Jaime
Pia.
This
is
welcome
video.
We
were
working
more
recently
on
a
gnat's
installer
for
kubernetes,
so
while
I
was
at
the
booth,
I
definitely
got
questions
on.
How
do
you?
How
can
you
get
started
with
Nats?
You
can
run
the
static
binary,
but
we
also
obviously
we're
on
kubernetes
and
I
also
got
some
questions
on
whether
we're
using
staple
sets
or
demon
set
stuff
like
that.
C
So
I've
heard
that
some
of
your
running
in
production
and
how
many
are
using
the
Nats
operator
right
now,
okay,
so
yeah,
we
do
I
start
doing
some
changes
to
Nets
operated,
moving
to
consolidate
more
stateful
sets,
and
this
is
like
you're
not
using
the
operate
already.
This
is
like
a
simple
way
to
just
get.
It
started
and
get
like
an
out-of-the-box
secure
by
default
cluster
and
we
using
then
have
speed
to
accounts
and
accounts
oscillations
and
TLS
setup.
C
So
if
you're
not
comfortable,
just
piping
into
your
shell
account
into
your
shell,
let's
peek
at
the
script-
and
you
will
see
that
the
only
requirement
is
actually
a
cube
Caudill
you
can.
We
will
be
cute
coral
applying
and
make
a
required
policy
to
bootstrap
that
set
up
container
that
if
you
have
enough
policy
to
be
able
to
run
this
and
that
set
up
and
then
you
have
just
basically
I
create
secrets
and
configuration
Maps
and
among
other
and
trained
training,
pots
and
and
deployment.
C
B
It
so
then
we're
just
running
all
those
keep
commands
pipe
in
its
shell
after
we
carefully
inspected
that
we're
not
doing
anything
crazy
and
so
on
the
right
hand,
side
you
can
see
all
the
pods
and
all
that
stuff
coming
up
on
kubernetes
and
the
left
hand
side.
You
can
see
the
the
commands
running
so.
C
Yeah,
it
will
create
a
similar
to
enhance
operators.
I
mean
we're
just
using
a
headless
service
to
for
the
whole
nuts
cluster,
and
we
are
also
deploying
a
surveyor
and
also
has
a
built-in
graphing,
a
dashboard,
and
all
of
this
is
using
their
secure,
built-ins.
The
latest
secure
buildings
from
Nats.
It
gives
you
as
a
result
that
quick
walk,
walk
through
how
to
get
a
start
with
your
cluster.
So
now
that
I
have
seen
that
we
have
okay,
the
whole
cluster
is
ready.
C
We
have
open
source
couple
of
repos
this
week.
One
of
the
is
Nats
I,
okay,
ace,
operand
repository
and
also
an
Xbox,
which
is
kind
of
like
a
busybox
analog.
Some
very
lightweight
container,
with
some
of
the
common
utilities
that
you
use
for
Nantz,
has
a
to
delete
utils
for
Polly,
subscribe,
both
for
two
nights
and
that's
streaming,
and
also
the
NSC,
which
is
the
tool
used
for
managing
accounts
and
users.
C
B
C
And
this
is
that
in
mega,
Lukas
are
in
like
magic,
just
I'm
trying
to
do
in
there
publish
subscribe
here,
but
is
actually
completely
secure.
So
if
I
try
to
just
net
cut
into
the
night
server
and
of
the
4222
I
get
the
initial
info
line
from
the
Nats
protocol,
but
I
would
get
disconnected
right
away
because
and
both
the
earth
authentication
is
required
and
also
I
need
to
do.
The
TLS
upgrade
for
the
protocol
to
work
and.
B
C
So
I'm
the
installer
generating
a
VSO
environment
variable
some
helpers
to
just
do
pops
up
with
some
good
defaults
for
the
cluster,
and
it
has
the
mounted
as
mounted
volumes
the
credentials
that
were
generated
by
the
installer.
So
you
can
start
expecting
some
of
these
credentials.
For
example,
you
can
look
at
this
user
is
only
allowed
to
publish
on
the
test
test,
object
and
test
a
few
wild
card
as
subjects.
C
So,
if
I
try
to
do
the
same
and
publish
on
another
subject,
I
will
get
actually
a
permission:
violation
error,
for
example,
hello,
again,
you'll
have
enough
permissions
and
also
within
the
same
box,
you
have
the
system
account
user,
which
is
like
this,
a
special
user
and
a
special
account
that
you
can
use
to.
You
want
to
inspect
all
of
the
messages
that
are
flowing
through
them
to
the
system.
C
C
C
C
So
in
this
case
I'm
create
a
simple
service
on
the
test
subject
and
send
in
a
request
for
help
and
I
will
receive
in
them.
I
will
receive
the
message
and-
and
there's
also
this
special
subject
that
this
particular
user
is
able
to
subscribe
to,
to
be
able
to
gather
metrics
about
the
latency
from
this
subject
zone
if
I
subscribe
to
it
and
start
receiving
the
some
of
the
requests
metrics.
C
C
B
C
Yeah
and
there's
an
extra
before
the
latency
tracking,
so
you
can
modify
a
surveyor
to
tell
it
to
which
what
other
subject
that
is
going
to
be
for
which
subjective
is
going
to
be
tracking
the
latency.
So
it's
like
a
like
really
fast
way
to
use
time,
got
some
insight
on
the
response
time
from
your
services.
D
Jet
stream
is
our
next-gen
persistence
layer.
It's
come
out
of
a
lot
of
the
experiences
that
the
team
has
had
both
prior
to
Nats.
You
know
20-some
years
ago,
Stan
and
then
really
from
you,
the
community,
that's
kind
of
been
telling
us
what
you
liked
and
what
you
didn't
like,
and
so
jet
stream
has
been
in
deep
code.
You
know
implementation
stage
now
for
I
think
about
two
and
a
half
three
months
or
so
we've
been
designing
it
for
about
eight.
D
But
again
we
already
had
a
good
set
of
understanding
of
what
we
wanted
to
do.
Colin
did
a
great
job
of
explaining
some
of
the
targets
that
we
did
so
I
want
to
show
you
kind
of
where
we
are
what
you
can
expect
and
then
I
can
actually
answer
any
questions
that
you
might
have.
So
that's,
oh,
as
many
of
you
know,
is
account
aware
right.
It
understands
multiple
accounts
and
users
and
sharing
and
all
the
other
stuff,
and
it
has
a
security
mechanism
around
that.
D
However,
a
lot
of
people
still
use
Nats
in
a
pre
2o
type
of
thing,
where
it's
like
it's
for
me.
I
just
want
to
run
one
server
and
be
able
to
do
things,
and
so
the
trick
we
add
with
Nats
oh,
was,
is
to
make
Nats.
Oh
look
like
a
1o,
meaning
any
client
even
from
nine
years
ago,
can
still
run.
We
wanted
to
keep
that
with
jet
stream,
even
though
jet
stream
is
totally
Nats.
D
Oh
we're
bound
to
accounts
all
that
kind
of
fun
stuff,
and
so
this
is
a
normal
that
server
and
so,
if
you're,
just
wanting
to
play
around
with
jet
stream.
This
is
on
a
branch
right
now.
It's
all
open
source,
you
just
put
jsn
and
the
system
will
automatically
figure
out
kind
of
what
it
wants
to
use
potentially
from
a
resource
standpoint,
mostly
around
memory.
So
this
is
a
16
gig
little
MacBook,
so
it
says
I
can
use
up
to
12
gig
of
the
memory
and
then
there's
the
store
directory
from
a
concept.
D
Standpoint
Colin
talked
about
message
sets
right,
which
are
how
we
actually
define
interest,
global
order
and
retention
policies
on
a
group
of
messages,
and
then
there's
observables
how
you
can
actually
pull
those
messages
out
and
what's
interesting
about
observables
I.
Think
from
lessons
learned
is,
is
that
we
have
both
push
bass,
Stan
and
that's
are
all
push-based,
but
we
also
have
pool
based,
meaning
that
you
can
actually
create
a
message
set
and
say:
I
want
to
create
an
observable.
D
That's
just
gonna
send
a
request
and
get
the
next
message
or
get
the
next
ten
messages
or
next
hundred
messages
or
so,
but
we
start
with
message
tests
themselves.
So
this
is
a
little
utility.
It's
on
my
laptop.
All
it's
doing
is
sending
normal
Nats
messages.
On
pardon
me,
on
a
subject
space
that's
been
imported
on
my
clients
by
account,
so
I
might
want
to
add
a
message
set:
I'll
call
it
cube
foo,
you
know
I
always
use
foo.
However,
you
can
have
as
many
as
you
want.
D
You
can
have
wildcards
doesn't
matter
whatever
you
want.
You
can
have
the
limit
that
columns
we're
talking
about
very
similar
to
Stan,
so
I
can
say
unlimited
here
and
little
bit
here,
but
one
minute
and
then
storage
is
memory
or
file.
We
do
support
memory
as
a
first
classes
and
not
just
a
testing
framework
so
that
when
clustering
actually
hits,
if
you
want,
you
can
have
every
replica
in
memory
only
for
very,
very
high
speed
types
of
situations,
but
file
with
SSDs
and
stuff
isn't
pokey.
D
So
we'll
see
that
so
now,
if
I
say
jsm
info,
this
is
telling
me
what
my
accounting,
by
the
way,
it's
all
account
aware
so
I'm,
just
under
the
covers
of
bound
to
the
global
account,
and
that's
oh
that's,
saying
what
my
account
can
use
and
I'm
bound
to
a
single
server
right.
Here,
however,
think
about
something
like
a
super
cluster
that
Colin
was
talking
about.
Your
account
can
have
so
many
resource
and
they'll
be
spread
across
the
whole
pool
of
resources
from
the
super
cluster
itself.
D
And
you
can
say:
there's
nothing
really
there,
and
so
then,
if
we
want
to
go
and
send
some
messages,
but
we
need
to
be
able
to
see
this,
that's
a
little
bit
better.
So
this
is
going
to
go
ahead
and
send
a
million
messages
of
about
a
K
each
and
we're
sending
it
to
one
of
the
subjects
that
the
message
set
has
been
defined
around
so
that
server.
Remember,
there's
no
other
servers
here.
D
It's
just
in
that
server
is
going
to
detect
that
and
actually
store
these
things,
and
so
when
I
go
ahead
and
start
at
the
benchmark.
This
is
a
normal
Nats
bench
by
the
way,
so
it's
a
existing
program
using
all
low-level
and
that
stuff,
no
specialists
and
stuff
for
whatever.
So
it's
a
little
MacBook
on
battery
power,
but
you
know
about
two
hundred
and
fifty
thousand
1k
messages
a
second
we
could
say
info
on
cube
and
you
can
see
that
the
messages
are
there.
D
D
And
I'll
add
another
one:
I'll
call
it
Derek
and
I'll
just
beyond
foo
and
certainly
allowed
ten
messages.
It'll
store
to
disk.
So
now,
I
have
something
that
sits
there
and
has
five
messages,
and
this
is
one
of
the
things
that
Colin
said.
The
community
was
really
talking
a
lot
about,
and
so
I
can
say
that's
pub.
This
is
the
same
thing
that
comes
in
the
Nats
box
that
Wally
and
I
may
were
showing
to
foo
hello,
and
it
says,
published
it
and
I
can
say:
okay
did
I
get
it
in
my
message
set.
D
Yes,
I
actually
got
that
now.
I
can
say
Nats
request
and
the
reason
is
and
a
lot
of
people
who
have
been
our
ecosystem
for
a
while
will
hear
us
talk
about
anti
patterns
of
if
you're
doing
a
request.
Reply,
there's
really
no
reason
from
an
interaction
standpoint
of
the
request,
reply
to
ever
store
message:
right,
you're,
waiting
for
an
answer
and
I
say
to
people
that
you
know
ask
me
I
say
when
you're
actually
sending
a
persistent
message
in
a
system
you're
just
doing
a
request,
reply
to
a
different
entity.
That's
all!
D
So
we've
got
nine
messages
now,
starting
a
sequence,
one
ending
a
sequence,
number
nine
one
of
the
interesting
things
about
observables,
which
I
won't
show
a
lot
they're,
all
coded
and
stuff
they're,
just
not
in
this
tool.
Yet
I
you
can
tell
like
came
under
the
weather.
A
little
bit
so
I
slowed
down,
unfortunately,
is
the
fact
that
observables,
like
we
said,
can
do
push
or
pull
based.
They
can
replay
at
the
original
rate
that
the
system
receives
the
messages
or
instantly.
You
can
do
a
lot
of
very
interesting
things.
D
D
Sorry,
my
original
point
of
desirability,
sorry
was
message:
hats
have
sequences
right.
You
can
see
sequence
number,
one
through
sequence,
number
nine
observables
have
their
own
sequencing
and
observables
always
start
at
sequence,
one
and
they
always
monotonically
go
up.
So
you
always
get
one
two,
three
four
five
or
whatever.
What's
interesting
about
that
and
that
guarantee
that
we
make
is
that
act
mode?
That
Colin
pointed
out
where
it
said
act,
none
and
everyone
kind
of
tilted
their
head
a
little
bit
act.
D
None
actually
works
really
well,
because
the
system
will
tell
you
anytime,
you've
ever
lost
a
message,
even
if
you're
partitioning
a
message
set
or
you're
doing
a
totally
different
class
of
observable,
because
those
sequence
numbers
for
your
observer
more
one,
two,
three,
four,
five
six,
so
if
you
get
eight,
you
know
you
missed
a
message
and
you
can
immediately
request
it
to
come
back
in
that
reply.
Subject
again,
this
is
all
Cornett,
so
there's
no
protocol
buffers
any
type
of
envelope,
eing
and
capsulation
type
thing.
D
The
reply
subject
that
you
get
isn't
reservable
when
I
hand
you.
The
message
gives
you
total
control
of
the
Jetstream
system.
For
that
message,
if
you
actually
look
at
it,
you
can
see
it
has
delivery,
count,
the
store
sequence
and
your
observability
sequence
and
you're
allowed
to
do
things
like
AK
knack
or
one
of
the
big
things
that
we
put
in
was
the
ability
to
say
hold
the
phone
I'm
working
on
it.
D
D
This
is
really
interesting
because
I
can
know
that
I
only
have
a
minute
and
I
can
say
I'm
almost
done
and
just
send
to
that
reply,
subject
with
a
progress
indicator,
saying
I'm
working
on
it
and
it'll
reset
the
timer
internally,
so
it
won't
actually
redeliver
it
unless
you're
actually
really
really
gone.
So
back
to
the
the
message
set,
so
I'm
gonna
delete
a
message
from
Derek:
oh
wait
a
minute!
Let
me
do
this
because
they
might
all
be
gone.
Nope
they're,
still
there,
okay,
so
I'll
delete
number
to
store
Simas
own.
D
What's
interesting
about
this
and
what's
I
think
important
in
a
can.
We
trust
this
type
of
system
is
when
we
did
the
250,000
messages
a
second.
What
I
didn't
say
is
is
that
they
actually
are
all
processed
through
a
hashing
mechanism
around
their
timestamp
sequence.
Number
order
everything,
so
the
whole
persistent
store
that
Jetstream
is
doing
is
tamper.
Proof
you
can't
mess
with
it.
It
knows
if
you've
tampered
with
anything
and
it's
on
by
default.
You
know
Stan
had
the
ability
to
turn
CRC's
off
and
stuff.
D
D
Put
the
sequence
number
to
zero,
which
is
invalid
in
jet
stream
and
it'll,
put
random
data
overtop
of
the
message,
and
it
will
restate
in
so
it's
totally
tamper
proof,
meaning
you
can't
accidentally
put
something
in
there,
and
so
when
we
do
this,
we
got
an
okay
response.
So
when
we
look
at
the
thing
now,
we
only
have
eight
messages
and
two
is
gone.
D
So
if
I
was
doing
the
observability
thing,
you
would
see
one
and
you,
as
the
observer
will
see
1
2,
3
4,
but
sequence
number
you'd,
see
1
3,
four,
five,
six,
seven
or
whatever.
So
we
think
this
is
going
to
be
really
popular
with
places
that
need
things
like
GDP
are
the
purge
command,
which
I
think
I
showed
earlier
is
good
for
someone
just
did
something
and
ran
away
and
filled
up.
All
of
our
you
know:
resources
in
one
of
these.
D
You
don't
have
to
totally
blow
the
channel
or
we
call
these
message
sets
away.
You
can
simply
purge
it
and
move
on.
So
hopefully
that
gives
you
a
little
bit
of
sense
of
it
Israel.
It
exists,
we're
trying
to
get
this
out
and
merge
it
in
probably
in
the
next
couple
of
weeks
for
people
to
start
pounding
on
the
clustering
algorithms
have
been
designed.
D
They
look
great
on
paper,
but
that's
always
the
case,
but
we're
gonna
start
coding,
those
up,
meaning
that
we're
gonna
have
the
clustered
stuff
and
an
RC,
probably
very
early
in
q1,
hopefully
January
February
timeframe.
So
thank
you
happy
to
answer
any
questions.
You
have
for
me,
or
anyone
in
the
team
I
think
they
did
a
great
job.
So
thank
you
guys
appreciate
it.
E
E
D
D
So
what
we
do
is,
as
we
run
through
NS
one,
when
you
hit
that
one
URL
we
actually
figure
out
where
you're
on
your
geo,
we
figure
out
if
you
have
a
preference
for
a
cloud
provider,
meaning
if
you're
coming
from
Google
or
Amazon,
and
we
return
you
a
cname
of
the
local
cluster
that
matches
that.
So
what
if
you
combine
both
of
those
things,
you're
actually
getting
the
fastest
path
into
a
global
super
cluster?
And
then,
if
you
send
the
request,
we're
doing
the
right
things
geographically,
there
does
that
make
sense.
F
D
D
Sure
so
he
was
saying
one
goes
down:
one
comes
back
up,
so
in
general,
Collins
spoke
to
this,
the
server's
gossip
amongst
themselves,
and
they
share
that
gossip
in
that
state
with
clients
automatically
so
let's
say,
I'm
a
client
I
have
his
IP
and
I
connect
to
him.
Jaime
comes
up
as
another
server.
He
points
to
Collin.
They
form
their
own
little
mini
cluster
I'm,
immediately
aware
of
Jaime,
meaning
if
Collin
goes
away,
I
can
fail
over
there.
D
So
most
of
time,
when
you're
scaling
up,
you
just
add
the
server
and
it
actually
takes
care
of
itself
no
dns.
Knowing
that
the
clients
are
all
aware
now
going
down
is
probably
more
interesting.
So
our
servers
understand
the
notion
of
lame
duck
mode,
lame,
duck
mode
means.
That's
came
from
saying
inside
of
Google
when
I
was
there,
it
won't
accept
any
new
connections
and
it
eventually
starts
pruning
off
the
connections.
D
It
has
so
the
first
thing
it's
doing
to
the
clients
is,
it
says,
I'm
not
accepting
anymore,
and
so
it
updates
the
client
Jaime
saying
hey.
If
I
go
away,
don't
try
me
again,
I'm,
not
here,
but
there's
Collins
here
right
and
it'll
slowly
start
pruning
these.
These
connections
now
in
the
future,
probably
in
some
time
in
2020,
we're
going
to
attempt
to
make
some
of
the
clients
extremely
intelligent
and
the
server
can
give
them
a
hint
that
they're
about
to
get
cut
off
and
they
can
proactively
reconnect
without
dropping
any
data.
D
Even
though
you're
doing
a
once
most
pattern,
the
last
piece
of
it
and
then
I
can
answer
any
more
after
this
or
offline
is
the
way
we
do
our
superclusters.
Is
we
actually
bring
up
all
three
of
the
new
version?
We
form
a
six
node
cluster,
the
full
mesh
one
hop
and
then
we
instantaneously
lame-duck
all
of
the
old
ones,
meaning
all
of
the
clients
only
move
once
they
move
over
to
all
the
new
clusters
that
we
actually
drop.
All
of
the
old
servers
and
and
the
new
configuration
is
just
the
three
node
cluster.
D
G
D
So
the
question
was
scalability
and
you've
kind
of
got
to
subscribers
kind
of
doing
two
logistical
different
types
of
operations.
Can
you
scale
one
and
still
have
the
properties
that
you
want,
so
the
properties
of
at
least
once
or
at
most
once
is
Stan
Jetstream
or
just
Matt's
core?
That's
irregardless!
D
When
you
talk
about
multiple
participants,
listening
to
the
the
same
message,
a
couple
of
things
come
into
play:
you
can
run
those
independently,
so
this
can
be
a
cue
subscriber
you
can
put
a
hundred
and
by
the
way,
cue
subscribers.
Now
the
clients
they've
had
this
for
a
while
have
a
really
interesting
property
again
when
you
scale
down
is
instead
of
just
saying,
close
or
quitting
you
say
drain,
which
means
all
of
the
in-flight
messages,
you'll
process
and
then
you'll
exit
after
you're
done
with
that.
So
even
within
most
processing
you
won't.
D
You
know
the
probably
losing
message
is
almost
zero,
its
pre-sleep,
so
those
are
independent.
The
other
thing
that's
some
of
the
common
knowledge,
but
just
so
we're
all
on
the
same
pages
is
that
anytime,
a
message
comes
into
a
NAT
system.
This
is
a
cluster
supercluster.
Doesn't
matter
the
system
is
extremely
aware
and
efficient
at
where
the
interest
lies
throughout.
D
So
if
you
have
those
indexers,
let's
say
over
across
the
route
connection
or
a
gateway
connection,
that
message
is
only
going
to
go
over
once,
even
if
you
have
a
thousand
subscribers
down
the
chain
right,
so
we
try
to
be
extremely
efficient
at
only
sending
messages.
Once
when
there's
interest
and
the
other
thing
we
do,
that
most
people
don't
do
is
distributed.
Systems
are
hard
to
stretch
and
it
make
it
sounds
funny
because
your
regular
service
system,
but
most
distribute
systems.
D
If
we
try
to
stretch
them
where
there's
variable
RT
T's
in
between
components
not
fixed,
you
know,
bandwidth,
you
know
sketchy
type
stuff.
Usually
they
don't
do
well.
So
our
gateway
connections
are
explicitly
designed
for
stretching
a
NAT
system
into
a
super
cluster
and
what
they're
they
care
deeply
about
is
not
only
only
sending
the
message
once
but
pruning
off
interest
graph
chatter.
So
what
I
mean
by
that
is,
let's
say
we're
over
here
and
we're
doing
our
thing
got
published,
poster
subscribers
you're
all
the
way
in
Asia.
D
You
don't
care
about
us
and
we're
never
going
to
connect
over
in
Asia,
so
us
sending
all
the
chatter
about
he's
interested
in
food,
no
he's
interested
for
no
bar.
No,
you
know
we
prune
all
of
that,
and
so
we
actually
have
a
bi-directional
spline,
topology,
half
duplex,
two
different
connections,
and
we
have
two
personalities
of
the
connection
that
we
can
change
based
on
trying
to
make
sure
we
use
that
bandwidth
as
efficiently
as
possible.
From
a
scalper
perspective,
several
people
asked
this
of
me
at
the
booth.
D
The
first
thing
I
look
for
is
bandwidth
our
number
one
well
today
this
this
thing
was
different
number
one
saying,
but
no
more
saying
we
usually
get
is
I
thought
it
would
be
faster
and
I
was
go.
What
are
you
trying
to
do?
You
know
and
I
said
well,
how
you
know
big
is
your
Nick
and
they're,
like
it's
a
10,
gig,
Nick
and
I'm
like
yep,
and
we're
sending
8.9
gigabits,
which
is
pretty
much
the
theoretical
maximum
or
so
right.
So
currently
the
way
we've
tested
it.
D
We
think
we
can
get
to
about
60,
gigabit,
so
I
think
we
can
do.
We
can
flood
a
40,
gigabit
I,
don't
think
we
can
flood
a
100
gigabit
with
a
single
connection,
but
we
are
going
to
do
testing
with
that.
Amazon
now
has
those
hundred
gigabits
and
I.
Think
Google
has
some
as
well.
So
first
look
at
your
bandwidth.
Then
look
at
your
CPUs.
A
lot
of
people
like
to
throw
lots
of
cores
at
Nats,
which
is
good.
The
problem
is,
is
Nats,
is
very
efficient
now,
but
most
machines
when
you
throw
lots.
D
Of
course,
if
you
light
them
all
up,
the
thermal
thing
kicks
in
and
the
CPU
clock
rate
drops
so
a
lot
of
times.
You
can
actually
run
a
benchmark
on
Nats
with
like
go
max.
Proxy
goes
for
and
it'll
run
faster
and
then
the
last
piece
is:
if
you've
got
tens
of
thousands,
hundreds
of
thousands,
you
want
to
do
million
connections.
It's
just
memory,
you
just
gotta
scale.
The
memory
hold
on
one
second,
I'll
come
back
to
yes,.
D
So
the
question
was
hysteria
on
that
and
it's
a
great
question.
So
if
we
look
fundamentally
at
HTTP,
HTTP
is
one-to-one
right,
Nats
is
m2n
and
we
believe
deeply
in
that
pattern.
We
believe
how
its
surfaces
looks
very
familiar
to
one-to-one
or
want
to
end
what
you're
doing
load,
balancing
or
m2m
when
you're
streaming.
But
fundamentally
it
is
m2m
and
we
are
very
specific
about
that.
So
in
general
we
suggest
people
don't
put
anything
in
between
the
client
and
our
server
you.
You
can
sometimes
and
I'll
explain
where
you
you
can't
border.
D
You
can't
today,
most
of
the
time
things
that
Wally
is
running
in
kubernetes
he'll
he'll
yell
at
me,
if
I
get
wrong,
but
we
use
node
port
right
affinity,
host
port,
sorry,
meaning
that
you
as
a
person
inside
of
the
kubernetes
cluster
or,
if
you
expose
it
outside,
are
connecting
directly
to
the
server.
We
don't
need
any
low
bouncer
in
front
of
us.
We
actually
don't
want
it.
We
don't
want
any
and
gestures
and
stuff.
Now,
with
this
diode
people
said
hey,
we
want
to
use
some
of
the
policies
and
stuff
in
this
day.
D
Oh
and
that's
not
a
bad
thing
per
se.
What
happens
is
is
that
if
you're,
a
layer,
7
router
trying
to
pretend
to
be
a
layer,
4
router
but
you're,
really
not
a
layer,
4
router,
bad
things
happen,
and
so
with
this
there,
what
happens
is
is
if
you
notice,
when
Wally
did
the
low-level
telnet
and
we
got
the
info
block
back
in
plain
text.
D
That
was
a
design
decision
I
made
10
years
ago
and
I'd
liked
it,
meaning,
even
if
it
was
supposed
to
be
TLS
or
SSL
back,
then
you
could
get
a
sense
of
what
was
expected
of
you.
Most
people
will
come
and
kind
of
kick
me
in
the
shins
and
say:
what
do
you
mean?
So
you
auto
upgrade
a
connection
and
fly
tomorrow
and
yes,
we
do
and
that's
what
breaks
this
tie,
oh
so
used
to
kiss.
They
are
in
front
of
us
and
you
try
to
make
it
a
TLS.
D
D
Ssl
worse
is
just
a
fixed
plaintext
header
that
it
sends
first
and
people
can
then
bake
that
in
and
say,
oh
I
recognize
that
as
TLS,
but
ours
confuses
a
lot
of
TLS
proxies
that
seven
we're
trying
to
figure
out
a
way
to
work
better
with
more
cloud
native
stuff.
We
have
Prometheus
and
Agri
fauna
and
flu
and
D
and
stuff,
but
we
were
looking
at
OPA
sto.
D
We've
got
a
lot
of
those
questions,
so
we're
gonna
try
to
figure
out
what
makes
the
most
sense
and
work
through
that,
but
when
it
doesn't
work
right
now,
that's.
Why
is
because
it's
saying
it
can
do
kind
of
lair
for
things,
but
it
can
it's
a
layer,
seven,
and
so
when
it
sees
that
plain
text,
it
gets
confused.
So
what
normally
happens
downstream
is
you'll,
see
those
info
block
and
then
it'll
chop
you
off
and
you're
like
what
the
heck
happened
so
and.
D
So,
for
example,
we
have
we've
had
authorization,
primitives
permissioning,
allow
deny
clauses
dynamic
response
permission,
you
can
only
publish
a
response
to
something
that
you
received
service
mesh
then
cares
about
observability.
We
didn't
do
a
good
enough
job,
and
so
we
really
looked
at
ourselves
and
said:
hey.
D
We
need
to
figure
this
out,
and
what
we
found
was
is
that
that
same
bias
we
talked
about
earlier
of
distributed
systems
having
this
bias,
where
they're
actually
very
close
together
and
there's
a
you
know,
fairly
consistent
RTT
and
a
bandwidth
constraint,
rerun
global
systems,
all
of
our
last
four
or
five
pocs-
have
been
multi-region
multi-cloud,
global
type
stuff-
and
I
said
wow.
If
you
just
gave
me
the
latency
of
let's
say
calling
the
responder
that
doesn't
help
me
at
all,
and
so
what?
D
But,
while
he
was
showing
you
is
when
you
export
a
service,
let's
say:
I'm
exporting
the
Colin
service
and
I
go
I
want
to
track
latency
on
that
you
can
sample.
At
any
rate,
you
want
what
we're
actually
providing.
You
is
four
pieces
of
data,
we're
providing
you
with
what
time
it
took
him
to
actually
do
you
know
service
the
request.
We're
gonna,
give
you
the
Nats
subsystem,
which
I'll
talking
about
a
second
and
we're
gonna.
Give
you
how
far
away
the
requester
is
from
the
net
system.
D
How
far
away
he
is
from
the
net
system
and
how
far
the
two
servers,
which
might
not
be
the
same
and
a
large
deployment,
are
from
each
other.
We
give
you
all
of
that
and
our
I.
He
is
here
he
was
playing
with
it
quite
a
bit
and
he
was
kind
of
like
looking
at
me
funny
like
I,
don't
get
it
and
also
then
it
kind
of
popped.
D
It's
like
Oh
and
I
run
a
requester
in
Asia,
and
you
go
Colin
how
you
doing
Colin's
going
I'm
doing
great,
but
also
in
the
total
latency
went
up,
but
it
was
because
a
requester
were
making
requests
from
way
far
away,
and
what
that
would
signal
to
us
is
who
we
might
want
to
take
a
call
on
and
just
spin
him
up
in
Asia
because
he's
already
a
queue
subscriber.
So
we
don't
have
to
do
anything
special
if
he
just
starts
running
over
there.
D
So
look
for
that
to
come
up.
Ok,
yes,
so
we
don't
have
that.
Yet
we
don't
have
traffic
shame
yet,
but
we
are
on
it.
We
care
deeply
about
the
permission,
but
we've
always
had
that
the
observability
I
mistakenly
said
it's
a
pub
subsystem.
Of
course
it's
easy
and
it
wasn't
there's
a
lot
of
things
going
on
and
inside
of
the
code.
D
D
So
so
the
question
was
I,
think
about
sequencing
and
I.
Think
you're
talking
about
jet
stream
or
or
Stan.
So
the
the
answer
is
is
that
in
cornets
this
has
been
true
since
day.
One
ordering
is
guaranteed
by
publisher
on
a
giving
connection.
So
if
Colin's
a
publisher-
and
he
has
a
connection
into
the
system-
and
he
publishes
one-
two-
three
and
I
published
ABC-
those
orders
are
guaranteed
anywhere
within
the
system.
However,
if
you're
subscribing
to
foo
and
bar,
you
might
get
totally
different
interleave
versus
wally
right,
but
our
orders
are
maintained.
D
So
the
question
is:
how
do
you
maintain
a
global
order
and
how
do
you
scale
that
so
there
will
be
in
jet
stream
clustering,
a
leader
election
and,
let's
say
we're
going
to
say
for
message
that
Derek
column
is
leader.
His
job,
though,
is
actually
only
to
do
the
ordering.
So
what
I
mean
by
that
is,
let's
say
we're
all
servers
in
this
cluster,
we're
all
getting
messages
we
can
independently
operate
on
the
information
that's
shared
with
us.
D
That
says
we
need
three
replicas,
so
I
can
get
a
message
and
I
can
actually
solicit
help
to
get
those
three
replicas.
You
can
Swiss
it.
You
can
answer.
You
were
all
doing
this
at
the
same
time
and
when
we
feel
hey,
that
message
was
stored
and
we
use
a
highway
hash
and
a
timestamp
to
kind
of
understand
a
priori.
D
We
all
know
what
that
message
is
I
can
then
say:
oh
it's
replicated
three
times
all
I
do
is
send
them
a
small
metadata
message,
saying
consider
this
one
for
next
in
line,
and
then
he
stamps
the
sequence
number.
Once
that
happens,
a
message
goes
out
that
then
triggers
all
the
observables
who
have
are
waiting
for
the
next
message
to
say:
I
can
send
ten
now
or
eleven.
So
we're
trying
to
you
know.
D
Yes,
the
the
the
global
ordering
from
a
message
set
perspective
does
need
a
leader
right,
there's,
algorithms,
where
we
could
do
different
things
where
we
say
we
don't
have
to
have
a
leader
or
consensus.
The
problem
is
that
in
Nats,
things
are
so
fast
that
we
would
constantly
be
in
conflict
resolution
in
some
of
those
algorithms,
meaning
it
would
be
slower
than
Christmas
so,
but
we're
only
going
to
allow
him
to
say.
Oh
okay,
this
next
up,
okay
sequence,
number
nine
secrets:
number
ten
Samba.
D
D
So
the
question
is:
can
the
account
server
potentially
tie
into
an
existing
system?
And
we've
we've
been
thinking
through
that
quite
a
bit,
and
so
just
a
teeny
bit
of
background
information
in
operator
mo
the
NAT
server
start
up
with
very
little
configuration
it's
who
they
trust,
and
how
do
they
resolve
an
account
that
they've
never
seen
before,
meaning
that
if
Colin
sends
me
or
tries
to
connect
to
me
and
he's
part
of
the
Sanae
dia
account
and
I've
never
seen
it?
D
Now
to
your
exact
question
of
what
about
different
JW,
T's
and
stuff,
we've
got
a
couple
folks
asking
about
that.
Jdb
T's
in
our
world
are
public
claims
and
we
fix
the
signing
key
to
make
them
secure.
Most
people
will
tell
you
J
beauties
are
fine
unless
you
allow
someone
to
pick
the
algorithm.
That
being
said,
we
are
trying
to
look
at
ways
that
we
might
be
able
to
do
something
a
little
different
to
to
tie
into
those
right
now.
A
D
A
You
need
to
be
item
potent
and
the
reason
why
is
even
with
exactly
once.
Let's
say
you
get
halfway
through
processing
the
message
in
your
crash,
then
you
may
have
a
table
out
of
whack.
You
may
have
have
something
going
on.
Let's
say
you
get
all
the
way
through
processing
the
message
and
you
crash
before
you,
AK
you're
gonna.
A
A
D
In
a
prior
life
at
TIBCO,
I
created
EMS,
which
had
exactly
once
and
I
also
had
XA,
meaning
you
who
coordinate
multiple
resources,
we
deployed
it
I
think
at
the
time
it
cost
32
million
dollars.
For
this
customer
and
I
told
him
I
said
I
built
it
for
you,
but
I
said
you're,
not
gonna,
use
it
and
I
said
that's
what
we
wanted.
He
puts
it
in
calls.
D
Less
than
two
weeks
later
says,
come
back,
I
had
to
fly
back
to
the
mid
of
the
country,
spend
six
week
there
and
what
we
did
was
we
designed
a
compensating
transaction
model.
So
in
other
words,
you
never
rolled
back.
You
always
just
kept
rolling
forward,
and
then
you
know
some
of
the
things
that
Colin
was
talking
about,
can
help
there
so
either
idempotent,
meaning
I
can
keep
saying
the
same
thing.
And
it's
your
your
you
know,
side-effect,
free
or
I
would
suggest.
D
D
You
know
when,
when
you're
on
the
web-
and
you
go
to
your
bank
site,
you
don't
have
a
client,
sir
right.
You
authenticate
yourself
to
the
bank
some
other
different
way
and
your
browser
can
trust
this
early.
The
bank
most
of
the
world
exists
that
way,
because
managing
certs
and
actually
being
able
to
embed
permissioning
inside
of
certs
is
a
capital
P
ITA
right.
So
no
one
really
does
it.
However,
it
is
useful
to
say:
hey,
I
want
to
be
able
to
bi-directionally
trust
that
hey
I'm,
trying
to
connect
you
and
you're.
D
Trusting
me
that
I'm
allowed
to
do
that
so
within
nets.
Today,
you
we
do
support
client-side
serves
and
we
do
have
the
ability
to
pull
identity
out
of
that
client
site
serve.
However,
you
still
have
an
authorization
stanza
that
needs
to
live
somewhere.
So
when
people
come
to
us,
we
had
a
gentleman
doing
some
really
cool
stuff
and
he
was
doing
em
LTS
and
he's
like,
oh
and
now,
I'm
going
to
add
permissioning
into
the
JWT
and
stuff
and
I
said:
hey,
you
don't
need
him
TLS,
he
was
just
I.
Do
I
said
no.
D
You
don't
I
said
the
client
cares
about
connecting
to
the
server,
but
the
server
only
cares
about
the
client
if
it
can
properly
authenticate
with
EE
two
five,
five
one
nine
keys,
all
those
stuff
and
that
JWT
that
you
send
in
that
it
we
check
all
the
way
up
to
the
root
of
trust.
The
operator
it
has
all
your
claims,
as
I
can
publish
the
foo
I
can
sleep
on
this
that
kind
of
stuff?
D
So,
if
you're
looking
at
a
surface
patch,
we
have
a
lot
of
folks
doing
where
the
responders
are
Nats
clients
right,
they
listen.
We
definitely
suggest
TLS,
especially
for
client
to
server.
You
can
do
em
TLS
by
the
way
within
clusters.
If
you
want
or
VPC
a
lot
of
times
v,
pcs
are
fine,
but
we
usually
say
if
you
care
deeply
about
permissioning
of
hey.
I
have
a
connection,
but
now
I
want
to
make
sure
I'm
allowed
to
do
things
in
inside
of
that
with
different
subjects.
Use
the
authorization
primitives
that
we
put
together.
D
D
What
you
can
do
is
set
up
ml,
TS
and
then
say:
I'm
gonna
pull
out
the
identity
from
the
certificate
of
who
the
user
is,
let's
say,
Wally,
and
while
he
did
some
of
the
work
so
grab
him.
If
you
need
more
answers,
but
then
you
still
need
to
configure
the
authorization
that
applies
to
Wally,
and
so
that's
either
going
to
be
a
config
file
and
all,
and
so
with
Nats
to
O,
be
totally
decentralized
and
distribute
it.
We're
like
it
should
be
in
this
claim:
Jo
VT
that
you're
sending
in
as
a
client
app.
H
D
D
A
D
Well,
there
there's
an
amazing
amount
of
people
that
use
in
production.
They
run
a
single
server,
don't
even
monitor
it
and
it
runs
for
two
years
and
they
shut
the
whole
system
out
and
they
say
Oh
should
we
upgrade
it
and
they
just
run
it
again
for
two
more
years,
safe
stuff.
So
but
yeah.
If
there's
a
use
case
that
you
think
we
might
be,
you
know
helpful
on.
Try
it
kick
the
tires.
If
you
feedback,
we
try
to
be
extremely
responsive
in
the
slack
channels.
D
We'd
like
to
see
you
use
it,
we
think
it's,
you
know
extremely
helpful.
We've
had
about
five
or
six
customers
who
we've
saved
about
80
to
90
percent
of
their
cloud
bill,
all
right,
so
that
80
million
message
a
second.
Nobody
really
cares
about
that
until
you
have
to
you
know
it
translates
to
how
much
you
pay,
how
big
of
a
machine
do
I
need.
This
thing
will
run.
You
know
if
you
want
to
do
50
thousand
a
hundred
thousand
messages.
A
second
you
can
do
that
on
a
Raspberry
Pi.
D
You
could
do
it
on
a
32,
nano
I
think
and
we
have
benchmarks
that
you
can
just
run
it
yourself
and
test
it,
and
so
we
got
a
lot
of
people
saying
we
just
want
to
concentrate
on
stuff.
That
makes
us
unique
in
our
business
better
and
our
you
know
value
prop
and
not
have
to
worry
about
a
lot
of
that
stuff.
We
fit
really
well
into
that.
Yes,.
D
So
the
so
the
question
is
clusters
and
superclusters
and
I'll
point
you
at
wally
who's
probably
done
the
most
set
up
of
those
you
can
configure
such
that
everybody
knows
about
everyone
else,
and
their
configure
files
are
pretty
lengthy
for
our
super
cluster.
The
gateways
are
the
same
for
every
single
server,
because
if
we
detect
ourselves,
we
drop
the
connection
or
don't
even
attempt
it
so
and
I
think
we
do
full
match,
meaning
the
config
for
a
small
cluster
knows
about
all
of
them.
D
However,
what
we
do
also
is
we
provide
seed
nodes,
so
what
I
mean
by
that
is
take
a
DNS
record
start
one,
maybe
two
three
or
whatever,
and
put
those
in
the
DNS
and
then
every
new
server
that
you
want
to
join.
The
cluster
just
pointed
out
that
DNS
it'll
get
hold
of
one
of
you
three
and
you'll
gossip
to
all
the
other.
One
it'll
immediately
put
itself
together.
It's
really
slick
how
that
works,
Gateway's
kind
of
do
the
same
thing.
If
you
just
get
kind
of
get
over
there,
it'll
all
put
itself
together.
A
D
But
leaf
nodes
are
a
extend
and
embrace
type
of
a
thing
which
again
they're
they're,
really
slick.
They
totally
transparently
bind
into
a
super
cluster,
handle
all
the
credentials
and
all
those
stuff.
So
what
I
meaning
what
I
do
is
I
am
tries
TLS
for
all
the
tools
that
I
run
on
my
laptop
I
have
a
leaf
node
server.
So
that
leave
note
server
knows
my
credentials
connects
into
the
super
cluster
and
then
I
that
server
runs
on
one
27001
know
off.
D
I
have
that
physical
access
to
my
machine,
but
I
can
run
all
the
tools
for
NGS
on
my
thing,
with
nothing
in
there
instant,
because
the
amortization
of
the
TLS
is
already
there,
it's
just
been
stuck
up,
otherwise
it
TLS.
You
know
back
and
forth
credentials,
you
know
the
nonce
and
stuff
and
it
was
feeling
a
little
sluggish.
So
we
did
that.
However,
Colin
brought
up
the
even
better
point
of
extending
those
into
like
an
edge
to
IOT,
and
that
doesn't
have
to
mean
you
know
like
little
plant
sensors.