►
From YouTube: RDMA Enabled Kubernetes for High Performance Computing - Jacob Anders, CSIRO & Feng Pan, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
RDMA Enabled Kubernetes for High Performance Computing - Jacob Anders, CSIRO & Feng Pan, Red Hat
Adoption of Kubernetes in scientific workloads has been hampered by limited support for high speed interconnects used in HPC clusters. Fortunately, we can now solve this problem by enabling RDMA in Kubernetes. In this session, we will describe the HPC use case and requirements from an end user's perspective. We will discuss how RDMA, a highly efficient network transport protocol, can be used to address this challenge. We will then provide an overview of a community driven RDMA implementation for Kubernetes using CNI plugins and SR-IOV. Finally, we will demonstrate real-world applications running in RDMA-enabled Kubernetes environment and provide a performance comparison between standard and RDMA-enabled networking. You will leave this session understanding the state of the art for HPC networking on Kubernetes.
https://sched.co/UacK
A
Res,
unfortunately,
had
to
fly
catch,
his
flight
back
home
early,
serve
and
he's
not
with
us,
and
also
just
to
finish
up.
The
introduction.
I
would
like
to
make
a
disclaimer
that
my
work
presented
here
today
is
based
on
my
own
research
and
ideas
and
opinions
expressed
here
are
my
own
and
do
not
represent
my
employer,
all
right
so
to
just
to
get
us
started.
Let's
talk
a
little
bit
about
high
performance
computing,
just
to
get
an
idea.
What
your
level
of
understanding
of
HPC
is.
A
Can
we
have
a
show
of
hands
how
many
of
you
actively
use
HPC
or
a
familiar
with
HPC
about
a
health,
very
good
for
the
others?
Let's
start
with
a
quick
introduction
to
two
HPC
and
40,000
feet
overview
of
the
topic,
starting
with
a
definition.
High-Performance
computing
refers
to
systems
that,
through
a
combination
of
processing,
capability
and
storage
capacity,
can
record
ly
solve
difficult
computational
problems.
A
In
my
own
words,
I
would
describe
a
typical
and
HPC
system,
as
thousands
or
tens
of
thousands
of
high-performance
nodes
connected
with
non-blocking
fabric,
and
also
a
collection
of
storage
nodes
connected
to
the
very
same
fabric.
By
combination
of
special
software,
typically
MPI
and
the
high
performance
network,
with
all
those
capabilities,
we
can
take
those
thousands
of
nodes
and
make
them
act
as
one,
and
this
allows
us
to
sell
some
solve
some
of
the
most
computational
problems
that
our
scientists
face.
A
A
To
back
this
up
here,
some
pie
charts
from
top
500,
which
is
the
list
of
the
fastest
supercomputers
in
the
world.
From
those
slides,
you
can
see
the
main
players
right
now
are
the
US
and
China
us
holding
title
for
the
fastest
and
second
fastest
supercomputer
in
the
world.
China
running
the
third.
On
the
other
hand,
China
holds
the
biggest
share
of
the
overall
performance
in
the
top
500
system.
A
Given
radical
Cubert,
it
is
come
for
us.
Let's
spend
a
few
minutes
thinking.
How
can
we
apply
Kapernick
this
technology,
converting
these
principles
to
further
assess
high
performance
computing?
What
is
it
that
we
would
like
to
do
today?
That's
difficult
to
achieve
with
traditional
HPC
tools,
such
as
best
be
oriented
processing.
A
First
of
all,
there
are
new
emerging
fields
of
science,
for
example,
machine
learning
and
to
a
degree,
biotechnology
and
bioinformatics,
then
dot
fully
fit
in
traditional
bicycle
paradigm.
Another
such
area
could
be
cybersecurity.
Why
is
it
so,
for
example,
for
bioinformaticians?
Quite
often
they
would
like
to
bring
their
own
environment.
They
might
have
certified
container
images
that
come
from
big
genomics.
Research,
centers
and
those
images
are
certified
to
build
repeatable
and
dependable
results
if
they
were
to
replicate
that
in
house
that
would
break
certification.
A
A
Many
of
the
tools
that
my
users
use
depend
on
building
blocks
that
were
developed
on
Copernicus,
so
I
think
there
is
something
to
it
and
it's
significantly
easier
for
ml
experts
to
uncover
matey's
than
to
try
to
retrieve
it.
What
they
do
and
ever
need
is,
do
a
batch
queue
system.
So
those
three
use
cases
along
in
my
opinion,
justify
an
increased
investment
in
kaepernick
is
from
the
HPC
centers
around
world.
A
Just
to
recap,
what
are
cost
advantages
we're
seeking
and
as
we
mentioned
in
the
context
of
bioinformatics,
we
have
the
portability
and
reproducibility
advantage,
also
the
greater
degree
of
control
over
the
environment
that
might
be
useful
for
for
cyber
security,
but
also
by
adopting
kubernetes.
We
gain
access
to
a
very
significant,
an
active
community.
That's
working
hard
to
develop
carbonate
is
further,
so
we
go
beyond
classic
HPC
and
classic
HPC
tools
and
we've
got
the
whole
new
ecosystem
that
we
can
incorporate
into
our
work
and
and
toolkit
now,
I
believe
this
goes
both
ways.
A
So,
while
there
is
a
whole
lot
that
Copernicus
has
to
refer
to
HPC
users,
there
are
also
things
that
the
HPC
community
can
offer
the
Copernicus
community.
First
of
all,
HPC
centers
are
typically
very
large.
Hpc
state
systems
vary
from
hundreds
to
tens
of
thousands
of
nodes,
so
it's
pretty
impressive
scale
to
run
on
and
test
on.
So
in
many
ways
we
can
run
into
problems
faster
than
usual
users.
A
B
A
Similar
principle
applies
to
performance,
we
we
have
really
fast
hardware
and
we
drive
it
hard,
so
we
quite
often
encounter
performance
limitations
before
anyone
else
tasks.
So
we
conclude
that
back
and
work
with
you
guys
to
to
make
governance
of
the
region
faster.
So
that's
what
I
believe
we
and
to
offer
and
again
we've
got
a
lot
of
bright
minds
and
capable
developers,
so
we
can
make
a
contribution
to
do
carbonates
if
we
see
that
it
can
fit
our
needs
and
help
us
do.
A
Our
fast
and
large-scale
hardware,
so
if
it's
also
good
and
rosy,
why
has
it
happened
yet?
What
is
slowing
down
adoption
of
copernicus
and
high-performance
computing?
In
my
opinion,
it's
not
working.
The
requirements
for
networking
which
we
see
in
cloud
computing
are
very
different
than
networking
requirements
for
HPC.
A
D
A
They're
very
focused
on
making
sure
that
we
can
separate
different
tenants
in
a
secure
and
resilient
manner.
We
also
don't
want
to
be
too
specific
to
the
hardware
running
on,
because
the
same
companies
advise
my
work
on
Juniper
Cisco,
Arista,
anything
we
shouldn't
really
tie
it
to
specific
features.
A
So
those
those
are
the
features
that,
in
my
opinion,
are
the
most
important
in
cloud
computing
interconnects
now
the
side
effect
of
that
is
that
the
performance
might
be
secondary
to
all
of
this,
and
this
is
understandable
if
we
think
in
web-centric
reality
it
shouldn't
really
matter
and
that
much
what
performance
is
because
quite
often
were
being
constrained
by
the
client
everything
now.
This
is
a
completely
different
story,
an
HPC.
A
We
can
essentially
take
those
priorities,
we're
talking
about
you
know,
because
the
graph
of
them
upside
down
so,
yes,
sure
we
care
about
security
to
a
degree,
but
the
end
of
the
day
performance
is
king.
We
want
the
minimum
possible
latency
maximum
possible
bandwidth
and
anything
that
gets
in
the
way
of
achieving
that
will
probably
get
reamed
one
way
or
another.
Also,
what's
very
important
is
that
performance
needs
to
be
very
consistent.
A
As
one
of
the
speakers
said
in
the
earlier
sessions
in
HPC,
everything
runs
at
the
speed
of
the
slowest
node,
so
we
need
to
make
sure
that
there
are
no
slow
and
everything
is
pretty
much
equal
to
performers.
This
isn't
something
very
important,
and
on
top
of
that,
the
politics
that
we
used
in
HBC
are
quite
different
and
what
you
see
in
enterprise
networking,
because
as
much
as
we
can,
we
try
to
support
on
blocking
communications.
A
So
in
extreme
case
it
can
be
a
factory
interconnect
where
any
node
can
talk
to
any
node
at
full
bandwidth
without
any
blocking
in
all
contemporary
machines,
just
because
of
growing
scale.
Quite
often
there
is
talking,
but
it's
irritably
low
compared
to
what
you'd
see
in
the
enterprise
and
as
I
said
performance
is
king.
So
if
there
is
any
security
related
functionality
that
would
get
in
the
way
of
achieving
performance,
it's
probably
going
to
be
ignored
or
limited.
So
you
look
at
this.
A
You
you
realize
that,
probably
to
bring
Copernicus
to
HPC,
we
probably
to
look
into
this
area
and
see
what
improvements
can
we
make
the
bridge
this
gap
retained
security
and
flexibility
of
cloud
interconnects,
but
introduced
support
for
high
performance
at
high
performance.
Computing
requires
I,
briefly
mentioned
using
specific
hardware
for
for
the
interconnection
communications.
So
I'd
like
to
highlight
one
specific
bit,
which
is
very
central
to
the
work
that
we're
presenting
today
and
it's
remote
direct
memory,
access
or
short,
our
VM
I
mean
what
is
our
DNA?
A
It's
a
transport
protocol,
it's
living
in
the
same
and
networking
layer
as
DCPS
or
UDP.
It's
much
newer
and
it's
very
highly
optimized
for
high
performance
computing.
What
does
it
do?
What
can
it
offer
to
us
in
our
workloads?
Essentially,
it
enables
us
to
do
zero
copy
memory
to
memory
communications
completely,
bypassing
the
CPU
and
the
kernel
CPU,
and
can
no
longer
be
involved
to
establish
the
connection
between
two
nodes.
After
the
connection
is
established.
A
Communication
happens
transparently,
so
there
is
no
more
need
to
invoke
the
kernel
or
the
CPU.
This
is
something
that
gives
excellent
latency,
which
is
very
hard
to
achieve
in
any
other
way.
So
that's
the
first
takeaway,
so,
first
of
all,
we're
running
at
low
latency,
high
bandwidth
and
also
we're
saving
CPU
cycles
for
the
real
processing,
multiple
processing
communication,
perhaps
importantly,
RDMA,
was
born
in
the
context
of
InfiniBand
physical
connectivity.
E
E
And
plugins
you
know
that
we'll
talk
about
each
one
and
what
they
do
and
describe
them.
So
we
use
multis
to
enable
attaching
to
different
networks,
and
in
this
case
it
would
be
a
RDMA
network.
We
will
use
a
SRV
device
plugin
to
initialize,
manage
and
allocate
NIC
resources.
We
will
use
AC
and
I,
of
course,
to
plug
those
resources
into
the
pod,
and
then
we
will
of
course,
also
use
coronaries,
CPU
manager
and
topology
manager
for
performance.
Optimizations,
okay
and
I
will
talk
about
each
one
of
them
in
some
detail.
E
Ok,
so
urban,
any
networking
model
a
self
is
fairly
simple.
It
has
a
few
principles
that
must
be
followed
and
not
much
else
right.
Those
are
the
ones
that
listed
there,
so
every
part
will
get
its
own
IP
address.
That
stays
consistent,
whether
it's
someone
else
viewing
the
part
or
the
part
itself
beyond
the
poly
self
right
that
the
IP
should
not
should
not
change.
The
IP
should
be
the
same.
E
Ip
should
be
able
to
talk
to
other
idea
partially
be
able
to
talk
to
other
parts
without
NAT
and
a
node
should
be
able
to
talk
to
all
paths
with
on
that
right.
So,
basically,
that's
in
fact,
in
the
diagram.
The
the
end
effect
architecture
is
that
you
can
think
this
as
a
flappy
network
and
every
part
gets
attached
to
this
to
this
classroom
network
and
that's
that
that
works
fairly
well
in
most
use
cases.
But
of
course,
in
this
particular
one.
E
What
we
want
to
be
able
to
do
is
attach
to
another
network,
in
this
case
our
DMA,
for
a
number
of
my
paths
right,
maybe
not
often
but
some
of
them,
and
so
that's
that's
where
multis
comes
in
so
motifs
itself
is
a
medicine
I
which
in
fact
cause
others
and
ice
to
make
attachment
to
multiple
networks,
and
this
can
be
one
or
more
network,
of
course,
in
the
picture
I'm
depicting
one,
but
there
could
be
others.
So
all
the
paths
still
require
clustering
network
to
follow
the
same
principle
that
we
talked
about
before.
E
E
All
right
and
architectural
II,
the
way
it
kind
of
works
is
urban.
Eddy
will,
instead
of
calling
the
regular
seeing
nice
that
it
does
like
flannel,
for
example,
we
will
replace
that
with
Montes
right,
so
cue
blog
will
first
call
mulch
as
it's
a
CNI
and
we
will
call,
let's
say
flannel
and
then
any
other
number
of
CNS
after
that,
as
specified
in
the
part
itself.
So
in
this
case
we
are
going
to,
of
course,
use
SR,
VC
and
I
to
plug
the
virtual
function
into
the
problem
right.
E
E
Okay,
CNI
right!
So
now
that
we
can
actually
attach
to
multiple
networks.
We
need
a
actual
mechanism
to
do
so
and
a
we
developed,
a
SRV
CNI,
which
in
fact
it
plucks
a
srvv
F
into
a
pod.
Really,
what
that
usually
means
is
the
VF
will
be
placed
into
the
past
namespace.
If
it's
kernel-mode,
we
do
support
deputy
came
out,
in
which
case
you'll
be
running
at
the
ppmd
driver
and
sort
of
something
like
that:
okay
see
that
this
ni
does
or
does
have
the
capability
or
the
option
to
manage
I
Pam
if
needed.
E
You
know
just
like
you
know
others.
So
that's
not
well,
that's
useful!
That's
not
quite
sufficient
because,
as
our
V
V
F
itself
is
a
limited
resource
and
it
is
a
hardware
resource.
We
need
a
way
to
manage
this
resource
and
we
also
implemented
a
SRV
device
plugin,
which
initializes
the
all
necessary
drivers
for
the
Nick,
creates
those
vs
and
does
the
allocation
right
so
device
plug-in
framework
itself
is
used
to
do
this
whole
management,
and
it
will
also
be
responsible
for
configuring
things
like
our
DMA
mode
and
things
like
that.
E
E
E
E
Its
purpose
right
is
to
effectively
allocate
dedicated
resources
right,
like
CPU
to
a
container,
so
that
a
container
right-
this
is
a
container
level
resource,
can
request
any
number
of
dedicated
CPUs
that
it
can
rank
school
safely
without
interruption.
Okay,
CPU
manager
is
both
new
malware
and
hyper
threading
aware,
so
it
does
require
a
static
policy
to
be
set
by
default
in
criminales.
E
It
is
it's
non
policy,
so
everything
is
shared,
but
it
is
possible
to
turn
on
static
policy
on
a
per
node
basis
to
allow
that
Achilles
epo
to
be
allocated
now,
topology
manager
was
introducing
116
as
alpha
feature.
It
is
still
offering
117,
but
we
are
working
on
getting
it
more
mature,
so
topology
managers
purpose
is
to
coordinate
CPU
and
devices
Numa
alignment.
E
It
is
that's
one
of
its
purposes,
so
it
is
able
to
give
hints
as
to
you
know,
when
I
request,
both
CPU
resources,
special
dedicated
CPU
resource
and
like
a
NIC,
we
want
to
make
sure
that
they
are
new,
my
aligned
right.
So
that's
that's
one
of
the
features
that
it
has.
Of
course
it's.
You
know
you
can
kind
of
imagine.
E
This
can
also
apply
to
other
cases
where,
if
I
request,
multiple
devices,
I
may
want
a
specific
topology
to
be
to
be
applied
right
in
in
that
case,
for
example,
two
GPUs
connected
together
or
something
like
that,
so
those
are
fairly
important
features
that
I
think
it
would
be
useful
for
performance,
sensitive
application
in
the
future.
Okay,
so
with
all
of
those
features
together,
we
get
a
prospect
that
looks
something
like
this,
so
the
Mellanox
Mellanox
eyes
are
obvious.
E
Our
DMA
resource
requests
right,
that's
as
one
there,
so
that
indicates
to
our
device
plug-in
that
I
want
a
NIC
right.
The
network's
annotation
for
the
pot
is
the
network
plumbing
working
group,
spec
API,
which
specifies
the
additional
networking
network
has
been
definition
I
want
to
attach
to
so.
In
this
case,
I
say
something
like
our
DMA
in
network,
so
the
this
key
refers
to
a
network
attachment
definition,
CRD
name
which
really
tells
notice
which
scene
I,
for
example,
I
need
to
call
and
how
do
I
actually
make
that
attachment
to
this
network.
E
The
CPU
requires
right
in
there
I
CPU
6
in
both
requested
limits,
in
the
case
the
CPU
manager,
that
this
is
a
guarantee
pot
and
that
we
need
to
have
dedicated
CPUs.
For
for
this,
and
and
of
course,
the
fact
that
we
have
both
CPU
and
device
requested
will
also
trigger
topology
manager
to
make
sure
they
are.
Those
resources
are
aligned
on
aluminum
I,
think,
that's
it
all
right.
So
Jacob
will
talk
about
our
lab
table
setup
and
the
read
performance
results
that
we
got.
A
All
right
so
now
that
we
know
how
the
solution
works,
let's
have
a
look
at
a
practical
demonstration.
This
slide
is
put
together
just
to
give
you
an
idea
what
sort
of
environment
we
run
this
test
on.
However,
the
key
components
are
the
kinetics
v
card
and
d
sx.
There
is
6036,
which
reason
being
is
they
both
support?
Our
do
you
mind.
That's
the
only
hard
requirement
other
than
that
this
technology
stack
would
work
on
a
different
operating
system,
different
hardware,
different
CPU
architecture,
without
that
without
a
problem.
A
So
let's
have
a
look
at
the
company
setup.
Essentially,
we
just
used
a
small
installation
and
based
on
cute
of
a
cube
admin
with
fun
little
networking.
We
then
installed
and
enabled
the
plugins
that
thang
described
so
motifs,
SRV
and
so
on,
and
also
we
had
to
do
a
bit
of
a
post,
install
configuration
and
both
on
the
hardware
operating
system
as
well
as
community,
so
that
involved,
enabling
huge
pages
and
I
bring
a
survey
and
the
BIOS
and
the
carton
and
the
operating
system
and
creating
the
relevant
cover
needs
resources.
A
Thank
has
run
us
through
the
above
definition.
I
will
show
you
a
couple
more
EML
files
to
to
describe
the
other
parts,
so
the
config
map
is
linking
the
hardware
details.
So
specifically,
we've
got
the
vendor
and
device
IDs,
as
well
as
the
interface
name
in
the
operating
system,
and
we
label
that
with
the
resource
9
that
we
can
reference
later
and
then
we
create
an
SRV
network,
which
reference
is
this
plug-in
as
well?
And
this
puts
us
in
a
position
where
we
can
run
some
tests.
A
So,
in
the
first
demonstration,
what
we
do
is
we
create
a
order
using
the
definition
that
Fang
described
so
we're,
starting
with
a
clean
cluster
which
doesn't
have
any
pods.
Then
let's
have
a
quick
look
at
yellow
file,
defines
the
pod
the
first
important
bit.
Is
we
override
the
scheduler
to
make
sure
we're
testing
across
the
nodes
to
running
on
the
same
node?
We
reference
the
network
we
just
created
and
we
request
not
less
than
one
and
not
more
than
one
a
CeraVe
device
for
this
book.
A
A
A
A
A
A
An
interesting
part
that
I
will
show
you
in
the
slides
just
in
the
interest
of
time,
is
comparison
of
Cabernet
T
spots
and
the
burn
metal.
So
the
second
number
was
obtained
by
running
the
very
same
benchmark
on
the
Copernicus
notes.
Natively.
Without
going
into
the
container,
you
can
see
that
the
performance
is
almost
identical
and
to
look
at
this
further.
Let's
have
a
look
at
some
statistical
data
that
we've
gathered
here.
A
You
can
see
the
performance
comparison
across
200
runs
so
the
very
same
benchmarks
that
you've
seen
in
the
demonstration
we
left
that
running
for
two
hundred
iterations
and
then
we
we
run
some
processing
on
that,
and
this
is
the
numbers
we
got.
So
we
can
see
less
than
1%.
Difference
is
pretty
much
99.99
percent
identical
between
the
container
and
governmental
implementation.
A
Another
interesting
comparison
is
running
a
bandwidth
test
on
an
honest,
Arabic
container.
This
is
essentially
run
on
the
very
same
setup
that
we
used
for
the
earth.
The
RT
my
test.
The
only
difference
is
for
those
balls
we
haven't
requested
as
air
of
interfaces,
so
we
just
used
the
out-of-the-box
default
networking
provided
by
flow
now,
and
this
yields
less
than
a
gigabit,
so
over
40
times
less
than
what
we
achieved
with
our
DMA,
which
one
and
obviously
there
are
other
plugins
that
we
could
use.
That
would
feel
that
performance.
A
But
if
we
just
look
at
apples
to
apples,
vanilla,
vanilla
comparison,
this
is
what
we
get,
so
you
can
see
that
the
potential
is
huge
in
terms
of
improving
network
performance
and
cabernet
TS,
and
also
I
want
to
highlight
that
we
do
not
only
get
the
bandwidth,
but
we
also
get
the
efficient
RDMA
transport
that
can
really
benefit
at
performance
and
latest
season
sensitive.
Our
workloads.
A
A
Performance
I
think
that
the
benchmarks
that
we
run
and
the
statistics
we've
gathered
say
it
all
it's
pretty
much
on
par
and
there
isn't
much
room
for
improvement,
at
least
in
terms
of
bandwidth
and
with
this
I
think
we
can
make
Copernicus
I,
really
first
class
citizen
platform
to
run
and
that
work
intensive
applications
with
that.
I
would
like
to
say
that
there
is
a
fair
bit
of
future
work.
We
see
ahead
of
us.
A
First
of
all,
as
I
started
early
in
the
slides,
we
mainly
focused
on
the
rocky
implementation
for
now,
just
because
it's
more
common
to
see
cabernet,
these
clusters
run
on
Ethernet,
not
on
InfiniBand,
but
that
doesn't
mean
that
InfiniBand
support
isn't
coming.
This
is
one
of
the
next
things
we
would
need.
We
would
like
to
look
like,
and
we
need
that
to
support
Gabinius
deployments
on
existing
HPC
hardware.
The
HPC
centers
typically
go
in
Philly,
but
first
they
might
have
some.
A
They
might
have
some
inter
internet
if
they
need
it
for
specific
reasons,
but
the
nhbc
I
would
say:
InfiniBand
is
king,
so
having
the
ability
of
reusing
the
existing
infrastructure
would
be
something
very
valuable.
So
that's
there's
the
first
thing
and
when
we
have
that
going,
the
next
logical
goal
would
be
to
enable
support
for
high-performance
parallel
file
systems
for
cabrón
et.
So,
for
example,
we
can
to
persistent
volumes
on
a
file
system
such
as
GPFS
and
the
cheapest
one
of
the
format's
and
I'll
try
to
show
you
some
of
this.
A
If,
if
we
have
sufficient
file
and
also
a
couple
of
times
loss
earlier
and
errors
has
given
an
excellent
talk
with
I
would
nvd
about
GPU
support,
so
we
believe
putting
the
two
solutions
together.
What
we've
described
good,
Fang
and
GPU
support
that
would
create
a
really
excellent
platform
for
an
ml
and
other
GPU
intensive
workloads
just
because
we
can
leverage
the
really
fast
Network.
At
the
same
time,
we've
got
the
GPUs
and
we've.
If
we
combine
both.
This
can
do
some
serious
processing
power
for
suitable
workers.
A
A
A
It's
a
lot
of
detail
in
this
slide,
but
what
you
really
need
to
look
at
is
this,
so
this
is
actually
a
number
from
an
opus
that
cluster
between
run
that's
using
parallel
file
system
as
a
back-end.
Sorry,
the
little
pop-up
writes
is
getting
in
the
way,
but
this
actually
will
get
20
gigabytes.
A
second
read
and
can
gigabytes
a
second
right
and
it's
in
VMs.
Imagine
what
happens
if
we
connect
that
to
cover
Nate.
Is
it
only
go
faster,
so
yeah,
that's
something
we
would
like
to
try
in
the
near
future.