►
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
NATS-Sync: Secure, Multi-Tenant, Cross Cluster Message Routing - Brian Mason, NetApp
Wouldn’t it be nice to be able to send and receive NATs messages to your home devices or on-prem enterprise systems from your cloud service? Doing it securely but transparently to those pesky firewalls. With an NATS-Sync, a new open-source project, you can send and receive NATs messages across clusters hosted around the globe securely, through firewalls. This talk introduces the audience to NATS-Sync and describes use cases it can solve including IoT and enterprise device control.
A
I
want
to
acknowledge
my
two
co-workers
that
have
been
instrumental
in
helping
me,
create
this
presentation
and,
of
course,
our
major
code
contributors
to
the
nazi
project
itself,
john
and
raghu.
Thank
you
both.
So
here's
a
quick
agenda.
What
we'll
talk
about
today?
Obviously
we're
going
to
talk
about
problem
we're
trying
to
solve
the
design.
We
came
up
with
to
solve
that
problem.
A
A
So
what
we're
really
in
need
of
is
basically
a
secure,
scalable
multi-tenant,
that's
cluster
exchange,
so
we
also
want
this
to
scale
down
for
simplified
development.
So
we
can
keep
that
simple
view
of
just
using
that
and
have
all
the
complexities
of
that
secure,
scalable,
multi-tenant,
environment,
hidden
from
us
or
at
least
taken
care
of
out
of
our
view.
In
short,
we
want
it
all.
A
A
Okay,
let's
jump
into
the
design
the
solution
that
we
came
up
with,
so
your
basics
environment.
We
like
it
where
you
have
some
random
service,
wants
to
emit
and
receive
gnats
messages
to
that's
cluster
that
are
received
by
another
service
out
there.
It's
a
great
model.
We
just
want
to
extend
it,
so
this
works
across
clusters
and
across
private
networks.
A
A
A
We've
got
you
know,
maybe
not
literally,
but
figurative
fire
breathing
dragons
out
there
ready
to
stop
on
your
data,
steal
it
and
do
other
things
that
you
would
not
like.
So
to
make
this
get
through
the
firewalls.
The
client
side
really
needs
to
be
an
outbound
connection.
We
can't
go
inbound
onto
the
private
cluster,
no
one's
going
to
open
up
a
port
for
us
to
do
that,
so
the
ports
have
to
be
outbound
and
we
use
a
hanging
pole
for
that.
So
this
mythical
line
goes
away.
A
All
the
traffic
travels
through
the
sync
client
up
to
the
sync
server
and
the
sync
client
then
request
messages
to
come
back
down
and
replay
on
the
local
cluster,
pretty
simple,
except
when
we
do
this,
we
also
need
to
add
an
auth
server,
because
when
we
register
the
sync
client
with
the
sync
server,
we
really
need
to
have
some
sort
of
authorization.
We'll
talk
in
more
detail
about
that
in
a
minute.
A
Of
course,
all
the
data
data
is
encrypted,
not
only
via
the
https
mechanism
that
we
obviously
use
to
connect
from
the
sync
client
to
the
sync
server.
The
data
is
also
encrypted
separately,
we'll
dive
into
that
deeper
as
well,
and,
of
course,
this
is
all
multi-tenant.
The
server
tracks,
multiple
clients,
we
get
a
multiple
clients
registered
data
cannot
cross
between
clients,
your
basic
multi-tenant
security.
A
A
Okay,
let's
dive
down
a
little
bit
into
some
of
the
weeds,
so
obviously
to
get
the
messages
going
to
the
right
private
cluster.
We
need
to
have
some
sort
of
addressing
scheme,
and
that's
makes
this
really
easy
because
of
the
selectable
subject
that
you
have
in
nats.
We
follow
a
simple
naming
pattern
for
our
subject:
lines
that
allows
the
clients
to
come
in
and
get
the
messages
they
want
and
to
send
messages
back
up
to
the
server
the
pattern
that
we're
using
right.
A
Now
we
want
to
change
this
in
the
future,
but,
right
now
the
pattern
is
for
the
subject:
the
direction.
The
message
is
going
either
northbound
or
southbound
southbound,
meaning
to
the
client
northbound,
meaning
back
up
to
the
server
the
client
id
which
for
southbound
would
be
the
client
id
for
each
client
for
northbound.
A
That
is
a
known,
client
id
to
go
back
up,
but
we
wanted
to
keep
that
client
id
in
the
pattern
and
then
any
specific
data
for
the
for
the
service,
and
this
varies
to
whatever
you
want
it
to
be.
This
would
be
what
you
would
normally
have
for
that's
message
when
you're
not
trying
to
add
in
a
cross-cluster
synchronization.
A
A
When
that
happens,
the
client
will
generate
a
private
public
key
pair.
It's
going
to
store
the
private
it's
going
to
take
the
public
key,
whatever
metadata
that
you've
told
to
associate
with
the
client,
which
is
completely
user
defined
and
whatever
auth
data
you
set
it
to
authorize
its
access
to
the
server.
That's
all
part
of
the
api
call
and
I'll
show
you
that
on
the
demo
and
then
that's
set
up
to
the
server
the
server,
of
course,
double
checks:
the
auth
data.
A
Let's
dig
into
that
authentication
we
talked
about.
The
system,
has
a
plugable
authentication
model,
this
auth
service
actually
is
user
defined.
We
have
a
sample
one
that
we
use
for
testing
and
demoing,
and
it's
just
uses
a
static
user
to
find
password
and
we
might
expand
this
out
in
the
future.
That's
one
of
the
things
we'd
like
to
add
on
is:
is
variations
on
this
auth
service.
A
A
So
after
we
get
registered,
then
we
can
start
exchanging
messages.
A
message
is
put
in
on
that.
If
it's
got
the
right
pattern,
I
it's
got
the
the
northbound
southbound
direction
and
a
client
id
we'll
select
those
and
we'll
sync
them
across
the
network.
When
we
do
that,
we
actually
encrypt
the
data.
Now
I
can
hear
most
people
say:
don't
you
just
use
https
to
exchange
the
data
between
the
client
and
server
yeah
sure
we
have
an
https
connection,
but
we
didn't
consider
that
good
enough.
A
A
A
We
also
wanted
to
make
sure
that
the
data
was
encrypted
at
rest.
Right
now,
we
actually
don't
have
any
specific
spot
where
it's
sitting
stored
in
any
persistent
storage.
We
had
had
prototypes
where
that
was
the
case.
We
don't
now,
but
if
we
do
put
that
in
if
nat
starts
doing
persistent
stores,
which
I
think
they're
working
on
the
data
would
still
be
stored
at
rest
there.
A
So
we
have
a
basic
public
key
infrastructure.
The
server
generates
a
public
private
key
pair
when
it
starts,
and
one
thing
I
forgot
to
mention
during
the
registration
process.
Besides
the
client
id
that's
generated
by
the
server
the
server's
public
key
is
also
returned
to
the
client
at
registration.
A
A
A
The
encrypted
hash
of
the
message
and,
of
course,
the
encrypted
message
itself.
All
this
is
sent
over
the
wire
to
the
client.
Now
you
can
see
is
once
this
message
is
encrypted.
Not
even
the
server
can
decrypt
it
at
that
point,
the
the
we
don't
keep
the
key
around
it's.
It's
thrown
away
the
encrypted
version
of
it
stored
with
the
message,
but
we
can't
decrypt
it
because
it
was
encrypted
with
the
client's
public
key,
so
the
server
can't
decrypt
it.
The
only
person
who
can
decrypt
this
message
is
the
client
itself.
A
A
The
client
knows
it's
from
the
server
because
it's
got
the
hash
of
the
message
signed
by
the
server's
private
key,
so
you
they
can
decrypt
the
hash
see
that
it
matches
the
hash
of
the
plain
text
data.
So
we
know
that
this
data
is
from
the
server.
So
we've
got
a
good
bidirectional
authentication
going
on
here
for
people
interested
in
the
numbers
we're
using
our
rsa2048
we're
using
256,
aes
keys
and
again.
A
So
one
thing
we
had
to
consider
is
even
with
the
https
in
place
once
the
client
is
registered,
we
really
don't
pass
the
authentication
test
anymore.
We
didn't
want
to
keep
those
tokens
around.
We
wanted
to
get
rid
of
them,
but
we
still
need
a
way
to
verify
that
the
requesting
entity
that's
requesting
messages
for
client
idx,
is
indeed
that
client.
A
So
how
we
do
this
is
remember:
we've
exchanged
public
private
key
at
this
point.
We
actually
do
a
cryptographic
challenge
so
every
time
the
client
comes
up
and
asks
for
messages,
there's
a
little
piece
of
encrypted
data.
That
goes
up
with
that
request
that
then
the
server
can
decrypt
with
the
private
key
or
sorry
the
public
key
for
the
client
and
know
that
this
indeed,
is
the
right
client
asking
for
this
client
id.
A
So
I
have
a
check
sum
there
to
make
sure
that
we
don't
have
any
experience
clients
coming
in
and
asking
for
data
that
doesn't
belong
to
them
or
worst
case
trying
to
post
a
piece
of
data
that
doesn't
belong
to
them,
which
would
be
a
message
that
could
be
acted
out
on
some
system.
That
would
be
bad,
so
we
have
this
cryptographic
challenge
to
make
sure
that
that
doesn't
happen.
A
So
one
thing
we
have
to
do
is
also:
we
have
to
store
data.
We
actually
aren't
storing
messages
anywhere
other
than
the
nats
memory
at
this
point,
but
we
of
course,
have
client
registration
with
the
client's
public
keys,
the
client
id
the
metadata
associated
with
that
registration.
All
this
needs
to
be
kept
somewhere,
so
we
have
in
the
code
an
interface
that
is
represents
a
data
store,
and
we
have
two
implementations
that
interface
right
now,
a
file
based
store,
which
is
handy
for
debugging
and
testing.
A
I
wouldn't
use
it
in
production
necessarily,
and
we
have
a
database
store,
that's
probably
more
of
what
we'd
use
in
production,
where
you
can
have
a
h
setup
again.
The
file
is
great,
though,
because,
like
for
the
demo,
I'm
just
running
a
file
system
based
key
store
and
it's
storing
stuff
in
the
container.
So
if
I
want
to
tear
the
whole
system
down
and
bring
it
back
up,
I
don't
have
to
worry
about
any
persistent
storage.
It's
great
for
doing
demos,
it's
great
for
doing
testing
for
production.
A
A
A
A
But,
as
I
have
to
record
this
presentation
almost
over
a
month
before
kubecon,
I
don't
have
those
numbers
handy
and
I
apologize
for
not
having
them
in
a
recording,
hopefully
for
q
and
a
I'll
have
them
we've
tested
both
on
x64
and
arm
instances
up
at
aws
interesting
when
we
did
the
arm,
we
were
playing
around
with
the
arm
for
cost
savings
and
indeed
arm
instances
can
be
cheaper
and
they
performed
pretty
well.
But
we
saw
one
issue
with
the
arm
where
it
started
to
bog
down.
A
Is
we
do
a
lot
of
encryption,
as
I
mentioned
earlier,
the
arm
instances
at
the
time
we
did.
This
testing,
which
was
back
in
march
of
2021,
did
have
encryption
coprocessors,
so
the
x4
instances
that
did
have
the
co-processors
would
perform
quite
a
bit
better,
but
the
arms
performed
pretty
well
just
not
as
good,
but
in
all
honesty
with
what's
our
simulated
load
of,
I
think
we're
simulating
about
a
hundred
to
and
a
thousand
different
clients
a
hundred
once
in
a
thousand
different.
A
We
really
didn't
see
a
load
showing
up
on
the
x64
instances
where
we
had
a
three
three
node
replica
set.
We
I
mean
we
would
see
the
load
bounce
a
little
bit,
but
nothing
of
significance
popping
up
so
again
we're
going
to
get
some
more
scale
testing
in
on
this,
but
we
think
it's
going
to
scale
fairly
decently.
For
us,
this
was
all
fronted
by
your
basic
aws
load
balancer.
We
are
simulating
multiple
clients.
Of
course,
like
I
mentioned,
we've
tested
on
x64
and
arm
linux
containers
is
how
we
ship
it.
A
We've
created
two
apps
one
is
an
echo
app
which
we
generally
deploy
with
the
client
system,
because
this
allows
us
to
from
the
server
via
little
command
line,
app
post
an
ask
message
that
we
can
check
connectivity
all
the
way
down
to
a
particular
client.
So
we
could
say
echo
a
message
out
to
this
client
give
it
the
client
id
and
we
could
trace
the
activity
down
to
that
client
and
we've
actually
for
the
echo
message,
which
is
a
known
message
defined
in
the
code.
A
A
Https
proxy
is
a
fun
little
tool
that
we
created
that
basically
takes
https
traffic.
It's
it
serves
as
an
http
https
proxy.
It
takes
http
s,
requests
bundles
them
as
a
nas
messages
to
be
replayed
onto
the
client
network
and
in
practice
and
again
we'll
show
you
in
demo.
You
can
simply
use
this
as
a
proxy
and
run
curl
commands
or
cube
cuddle
commands
targeting
a
system
on
a
remote
network
and
they
just
pass
on
through
and
you
can
run.
This
is
like
a
security
nightmare
for
your
I.t
security
people.
A
A
Demo,
fun,
so
what
we're
demoing
is
basically
the
two
messaging
systems
that
we
did:
samples
the
echo
client
talking
to
echo
proxy
lip
and
the
proxy
server
talking
again
through
the
net
system
to
the
hd
proxy
lit
on
the
other
side
on
the
client
network.
Again,
this
is
a
simplified
view.
This
is
what
I
call
our
unicorn
and
rainbow
view
of
the
world
where
there's
no
nastinesses.
A
We're
happy
we'll
also
demo
using
the
pro
the
http
proxy
server
to
do
cube
cuddle
commands
to
a
cluster
sitting
behind
a
firewall
hidden
off
on,
in
this
case
my
docker
desktop.
So
here's
kind
of
the
layout.
We
have
a
multi-node
nat's
cluster
and
running
both
our
net,
the
nat
software
and
our
sync
server.
A
A
It's
fronted
with
a
load
balancer.
This
is
just
your
basic
aws
load,
balancer
it'll
take
traffic
to
the
sync
server
and
we
have
running
on
my
desktop
the
sync
client
again,
a
single
node,
that's
and
we're
running
both
the
hd
proxy
lit
and,
of
course,
the
echo
proxy
lit
as
well
up
here.
The
echo
client
is
a
command
line,
client
that
we
post
to
dats
and
then
that
gets
picked
up
by
the
sync
server
and
travels
all
the
way
down
and
we'll
do
that
demo.
A
So,
let's
go
to
what
I've
got
running
here,
so
I've
got
two
command
lines
here.
So
let's
first
look
at
down
here.
This
is
my
server.
This
is
aws
running
ubuntu
and
I've
got
my
docker
desktop
up
here.
So
you
can
see.
I
did
a
quick
logs
here
on
the
sync
client.
It's
just
kind
of
sitting
here
hanging
out
it's
running,
but
it
doesn't
have
a
client
id
because
it's
not
registered.
So
it's
just
sitting
out
waiting
for
one
down
here.
A
A
We've
got
the
echo
proxy
lit
the
http
proxy
nats
and
the
sync
client
running
here
and
I
can
do
down
here.
Server
get
pods,
we've
got
again
that's
running
up
on
the
server
area.
We've
got
the
http
proxy
server,
the
actual
sync
server
and
of
course
we
have
an
auth
server
running
out
here.
This
auth
server
is
our
simple
auth
server,
which
is
run
with
a
static
password.
A
That
can
be
said
when
you
fire
it
off.
It's
just
an
environment
variable
again,
not
a
production
item.
This
is
for
testing
and
demos,
so
we're
gonna
go
to
localhost.
I've
got
the
sync
client
exposed
on
a
node
port,
3020
3281,
so
I
could
access
it
and
mostly
I
want
to
access
this
registration
endpoint.
A
So
we
need
to
register
the
client
with
the
server.
When
I
fired
off
the
client,
I
gave
it
the
server
url
to
talk
to
the
server
it's
in
a
config
map.
So
let's
kick
this
off.
Let's
give
it
some
metadata,
let's
give
it
a
user.
None
of
this
is
mandatory,
but
it
would
make
sense
to
describe
your
client
and,
let's
give
it
a
use.
A
A
So
I
typed
my
json
right.
Let's
see
kick
it
off
and
indeed
yep
I
must
have
because
it
registered.
So
at
this
point
we've
downloaded
the
server's
public
key
we've
uploaded
the
client's
public
key
we've
authenticated
this
request
and
we
have
a
client
id
or
location
id
the
same
thing.
We
renamed
it
to
location
id
and
I
still
call
it
a
client
id
to
do
our
communications
with.
So
I
need
that
because
I
need
to
use
it
on
the
echo
client,
so,
let's
first
test
connectivity
from
the
server
down
to
the
client.
A
So
let's
look
at
this
little
run.
Echo
shell
basically
runs
the
echo
client.
It's
got
the
url
to
our
dats.
Again,
we
only
talk
to
nats.
All
of
the
software
we
write
only
knows
about
that's
the
fact
that
there's
that
sync
in
the
middle
is
irrelevant
to
it.
In
fact,
we
can
test
all
this
without
the
entire
net
sync
infrastructure.
We
can
just
do
gnats
to
mats
and
that's
or
write
direct
mass
communication.
A
We
don't
need
all
this
stuff
to
do
all
of
our
testing
and
development,
but
for
here
on
the
server
we've
got
nats
running
exposed
on
a
local
on
a
local
node
port
and
we
need
the
client
id.
So
let's
run
this
guy
and
we're
going
to
send
an
echo
message
down
to
this
client,
and
indeed
it
went
off
so
you
can
see
we
can
trace
it
back
and
hit
the
netsync
server.
A
It
hit
the
client
it
hit
the
echolit
and
came
back
up
all
the
way
and
returned
some
basic
spec
version
for
us
total
time.
124
milliseconds,
that's
not
too
bad
for
an
echo
if
it
was
a
ping
echo
that
would
be
pretty
hard,
but
for
passing
through
an
entire
messaging
system,
two
nats,
the
sync
client,
the
sync
server
and
coming
back
around
across
the
internet.
A
A
A
Cube
config
here
this
is
a
cube
config
from
my
docker
desktop.
If
you've
ever
looked
at
a
docker
desktop
cube
config,
it's
they're
all
pretty
standard.
Please
don't
copy
my
certificate
and
steal
it
from
me,
although
I'll
reinitialize
this
after
the
demo,
so
I've
got
to
put
in
a
proxy
url
the
only
change
I'm
making
to
this
cubeconfig.
This
is
one
I
just
got
from
my
desktop
uploaded
to
the
internet
and
I'm
putting
a
proxy
url
line
in
here.
A
A
A
A
A
That
made
a
lot
of
sense
when
we
started
working
on
this,
but
as
we've
matured
in
our
thinking,
we
realized
that's
sort
of
silly.
Let's
just
simply
add
a
netsync
prefix
that
will
allow
the
system
to
identify
the
message.
Then
we
can
use
the
client
id
to
decide
if
it
needs
to
stay
local
or
if
it
gets
passed
up
to
appear
the
peer
being
either
a
client
or
a
server.
A
I
think
that
will
enhance
the
system.
This
will
actually
allow
us
to
send
messages
from
a
client
to
another
client,
because
we
can
pass
the
messages
up
to
the
sync
server.
Sync
server
will
see:
hey,
that's
not
really.
For
me.
Let's
put
that
on
to
the
net
system,
then
it
will
get
picked
up
and
set
back
down
to
another
client
that
will
allow
not
only
northbound
southbound
communication
but
peer
communication.
A
A
A
However,
right
now
we
don't
have
any
data
at
rest,
and
even
if
gnats
would
start
storing
data,
which
I
hope
it
does
start
doing,
some
persistent
data.
The
message
is
not
encrypted
till
it
hits
the
sync
server
or
the
sync
client,
so
we'd
like
to
push
the
encryption
back
a
layer
and
have
helper
libraries
that
allow
the
that's
clients
to
to
post
the
message
encrypted
onto
gnats.
This
wouldn't
be
hard
to
do.
A
A
A
A
Another
thing
is
we'd
like
to
do
in
the
future.
Is
we've
looked
around?
There's
some
industry
standards
for
iot
we've
mostly
ignored
them
other
than
acknowledging
their
existence.
It
would
be
good
if
we
start
working
with
these
standards.
Again,
our
goal
was
to
write
some
software
for
netapp,
which
netapp
didn't
allowed
us
to
open
source.
So
the
industry,
iot
standards,
weren't
important
for
us
at
the
time,
but
I
think
as
an
open
source
project
and
making
this
mature
and
be
have
some
sticking
power.
A
So
again,
everyone
who's
worked
on
this
project
so
far
works
for
netapp
netapp
recognized
that
a
messaging
system
is
not
core
to
our
business.
We
don't
want
to
be
in
the
making
the
messaging
system
world
where
we
do
data
management,
so
netapp
has
allowed
us
to
open
sources
project.
Thank
you,
netapp.
A
We
licensed
under
apache
when
you
open
source
a
project
at
netapp.
You
basically
have
two
choices.
You
can
say.
Are
you?
Is
this
going
to
be
an
official
netapp
open
source
project
such
as
like
netapp
trident,
which
is
obviously
key
to
kubernetes
storage?
It's
key
to
netapp,
that
is
a
netapp
open
source
project.
A
We
netapp
supports
that
that
spends
a
lot
of
money
supporting
that
this
particular
project.
We
didn't
open
source
as
an
official
netapp
project
written
and
supported
by
netapp
engineers,
but
netapp
does
not
have
its
name
behind
this.
This
is
just
an
open
source
project
that
we
put
out
in
the
community.
A
It's
a
minor
distinction,
but
it's
important
that
I
point
that
out
that
this
is
not
a
netapp
supported
project.
You
cannot
call
that
up
technical
support
and
get
support
on
this
project,
like
you
can
with
trident,
for
example,
although
again
everyone
who's
worked
on
this
so
far
has
been
a
netapp
employee
and
we're
happy
to
answer
questions
and
give
people
support
it's
just
not.
You
can't
call
that
up
directly.
You
have
to
email
us,
the
developers
which
you
can
see
on
github.
A
This
is
hosted
on
github,
here's,
the
url
for
it
and
the
images
are
hosted
up
on
docker
and
again
the
urls
for
docker.
As
we
add
to
this,
we'll
get
the
readme's
up
better
for
this,
so
you
can
find
the
code
in
the
where
everything's
hosted
the
proxy
servers
hosted
under
a
different
project
up
on
github.
A
It's
just
something
that
personally
I
threw
together
just
for
fun.
It's
not
part
of
the
natsync
specifically,
although
that
is
open
sourced
as
well,
and
anybody
can
grab
it.
It's
just
not
directly
tied
to
the
netsync
project,
but
it's
still
under
the
one
true
way
which
the
otub
otw
is
the
one
true
way
open
source
hosting
area
for
us,
not
an
end-up
thing.
That's
my
personal
space,
where
I
put
my
open
source
projects
so
issues
that
we
could
use
helping
on
we're
testing.
We
could
really
use
some
help
testing.
A
We
could
love
people
to
try
it
out
and
propose
new
topics
you
can
grab.
You
can
add
issues
on
github
you
can
grab.
You
can
fork
the
project
you
can
email
us
get
added
to
the
project,
we're
pretty
much
open
to.
However,
people
want
to
work.
There
obviously
nobody's
going
to
have
direct,
commit
access
to
the
trunk
or
the
main
branch
that
will
still
be
reviewed.
A
A
I
appreciate
you
taking
your
time,
come
listening
if
you
we're
going
to
have
a
q,
a
session
live
here
after
the
talk.
I
hope
you
find
this
interesting.
If
you
love
it
check
it
out,
try
using
it.
Let
us
know
if
you
think
this
is
an
utter
waste
of
time
and
we
totally
missed
some
other
really
cool
piece
of
software
that
we
should
have
been
using.
Let
us
know
about
that
too,
and
if
you
use
it-
and
you
find
bugs-
please
add
issues,
I'm
pretty
sure
that
issue
tracking
is
open
to
the
public.