►
From YouTube: Blockchains in the Cloud - Nodes, Clusters, and Distributed Systems - Casey Kuhlman, Monax
Description
Traditional cloud systems are built to be distributed and (typically) fault tolerant. There is a huge uptake in systems that are distributed and fault tolerant. Blockchains, on the other hand, are decentralized and byzantine fault tolerant. This means that they have different operational considerations than typical distributed systems. At Monax, we've been running, maintaining, operating, upgrading, and monitoring blockchain networks in cloud based systems for over four years. This talk with focus on our lessons learned. It will be focused predominantly on the intersection between Kubernetes and blockchain technology and provide an overview of the key operational considerations that cluster administrators should keep in mind: runtime boot, network boot, runtime upgrades, monitoring and alert management, and log management.
----
Hyperledger – Open Source Blockchain Technologies
Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory
Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter
Follow-us on Twitter @Hyperledger
Learn about Hyperledger Membership: https://www.hyperledger.org/about/join
#Hyperledger #Blockchain
A
How's
everyone
doing
today,
first
sort
of
demo
session
of
the
morning,
I
guess
at
least
the
room
was
empty
when
I
walked
in
here
to
set
up.
So
what
I'm
going
to
talk
about
today
is
how
we
deal
with
with
running
block
chains
in
our
cloud
environments.
So
before
I
get
there
a
little
bit
about
me
and
my
company
and
why
I'm
standing
up
here
so
my
name
is
Kasey
Kuhlman
I'm,
the
CEO
of
mon
acts,
we're
a
hyper
ledger.
A
Member
we've
been
in
this
game
and
doing
this
playing
this
game
of
clouds
and
block
chains
for
a
wee.
While
now
we're
four
year
old
company,
we
were
the
first
to
market
with
a
permission
we'll
block
chain
before
anybody
knew
that
that
was
actually
going
to
be
a
thing
talking
I'm
talking
some
of
that
history
tomorrow
afternoon.
If
people
are
interested,
we
were
also
the
first
to
market
with
scalable
packages
of
smart
contracts.
It
could
be
mixed
and
matched
and
deployed
to
a
range
of
different
use.
A
Cases
over
time
prominently
solidity
based
smart
contracts,
where
the
primary
maintainer
zuv
hyper,
ledger
borough,
and
so
what
I'm
going
to
to
show
for
the
rest
of
this
will
be
a
little
bit
borough
focus,
but
in
general
it
will
be
I'll.
Try
to
abstract
it
as
much
as
I
can
across
different
DLT
frameworks
that
exist.
A
Currently,
what
our
company
does
is
we're
working
on
two
things:
we're
working
on
a
public,
permissioned
blockchain
network
called
the
agreements
network,
which
is
focused
on
being
a
contract,
lifecycle,
management
tool
for
managing
legal
contracts
and
the
core
piece
of
the
agreements.
Network
are
well
there's
two
core
pieces
from
the
blockchain
perspective
to
the
agreements
network,
one
of
which
is
hyper
laser
borough,
which
the
entire
system
is
built
upon
and
the
other
is
a
business
process
execution
engine
written
completely
in
solidity.
A
Currently
we
call
that
that
project-
Blackstone,
it's
I,
think
probably
well
I,
probably
is
the
largest
amount
of
solidity
code
that
has
been
open-source.
We
will
be
looking
to
move
that
into.
We
will
be
submitting
it
for
incubation
early
next
year
into
the
hyper
ledger
project,
because
there
are
a
whole
range
of
blockchain
use
cases
that
leverage
business
process
modeling
as
as
a
core
piece
of
their
use
case.
On
top
of
that,
we
have
a
SAS
based
application.
A
So
this
is
a
very
DevOps
focused,
a
talk
and
and
I'll
be
primarily
focusing
on
a
few
tools
that
will
under
that
lie,
underneath
what
we
do.
In
general,
we
use
terraform,
helm
and
kubernetes
as
our
operational
runtime
and
obviously
with
docker
images
along
the
way.
So
I'm
going
to
be
talking
about
predominantly
how
we
piece
those
things
together
into
a
coherent
operational
environment
I
like
conversation
talks,
so
you
can
see,
there's
no
slides.
This
is
this.
A
So
how
do
we?
How
do
we
do
that?
It's
it's
relatively
simple
to
to
boot,
an
operational
runtime
in
a
kubernetes
in
environment.
However,
managing
those
things
over
time,
if
it's
a
blockchain
system
can
get
a
little
bit
tricky,
you
have
to
be
very
careful
about
how
you
manage
your
persistent
volumes
and,
and
then
also
you
need
to
be
very
careful
with
how
you
manage
the
keys
in
general.
A
In
other
words,
how
do
we
deal
with
the
cryptographic
keys
that
we
are
managing
in
and
and
having
one-to-one
mappings
with
a
particular
runtime
in
early
in
you
know,
more
traditional
fault,
tolerant,
distributed
systems
that
we
run
in
cluster.
We
don't
typically
have
to
identify
a
particular
node
by
a
particular
identity.
However,
with
a
blockchain
system,
pretty
much
every
blockchain
that
I
know
of
has
a
strong
linkage
between
a
particular,
a
particular
public/private
key
pair
and
a
particular
runtime.
A
That
is
then
submitted
to
a
runtime,
and
this
is
different
from
traditional
from
from
earlier
generations
of
fault,
tolerant
distributed
systems
which
you
don't
have
a
strong
identity,
linkage
between
a
runtime,
so
a
particular
node
within
an
elasticsearch
cluster
or
a
particular
node
within
a
net
CD
cluster.
Those
things
are
not
necessarily
tied
to
a
particular
identity,
and
this
is
the
big
difference
in
running
blockchain
network
from
an
operator's
perspective
and
running
other
distributed
systems.
A
A
Ok,
good,
ok,
one
one
person,
great
all
right,
yeah
I
didn't
mean
to
call
out
people,
so
don't
don't
feel
shy.
If
you
don't
know
what
it
is.
Kubernetes
is
a
is
a
cluster
management
framework
that
allows
us
to
take
cluster
of
cloud-based
nodes.
We
have
I,
I,
think
30
nodes
currently
and
we
can
manage
those
essentially
as
one
docker
runtime
and
underneath
the
hood.
It
deals
with
abstraction
that
allow
us
to
manage
at
a
cluster
level
volumes
and
and
runtimes
Cube
conference.
I
think
is.
A
Is
this
week
as
well
and
if
I
was
at
Q
conference,
I
would
not
be
saying
that,
and
maybe
all
of
those
people
will
burn
me
at
the
stake
for
for
that
description,
but
that's
how
I
think
of
what
kubernetes
is,
and
so
this
this
bass
grip.
Essentially
what
it
does
is.
It
removes
our
old
state
and
then
it
sets
up
a
new
state
in
in
config
maps,
and
this
is
the
this
is
the
piece
here
that
that
does
that
job.
A
It
is
a
simple
wrapper
around
boroughs
configuration
system,
so
within
borough
we
have
low-level
tooling.
That
allows
you
to
define
what
your
business
network
is.
It
does
it
does
some
of
the
some
similar
things
to
how
Composer
allows
you
to
set
up
your
your
business
network
I.
Think
they're
called
in
composer
modality,
although
I
think
Dan
was
in
here.
Are
you
here?
A
Dan
is
that
is
that
the
right
terminology,
business
networks,
okay
yeah,
so
within
burrow
we
have,
we
have
two
commands
that
we
use,
one
is
called
spec
and
the
other
is
called
configure
and
spec.
What
that
allows
you
to
do
is
to
say
in
general,
I
want
a
network
that
has
these
types
of
accounts
on
it,
and
these
types
of
accounts
can
be
custom
configured
or
you
can
use
the
off-the-shelf
abstractions
that
we
use,
and
so
you
can
say,
for
example,
I'm
going
to
have
seven
developers
on
this
chain.
A
A
A
So
for
for
this
network,
sorry
well
I'm
not
worried
about
for
this
network.
We
use
kubernetes
secrets
to
to
store
the
key
pairs
for
our
production
system.
We
use
a
different
system,
but
for
our
demo
environments
we
we
use
just
kubernetes
secrets
to
manage
the
keys
because
they're
relatively
simple
to
inject,
into
runtimes
and
and
so
what
you're
seeing
here
are
the
base64
encrypted
or
sorry
base64,
encoded
keys
that
I
just
actually
added
into
into
the
system.
So
we
save
those
as
kubernetes
secrets
and
then
we
also
save
a
different
sequence
of
keys.
A
These
bottom
four
fields
here
know
validators,
know
key
validators
here
at
all
three.
This
is
a
three.
This
is
a
four
node
network,
a
four
validator
know
network
and
those
keys
right.
There
are
different
because
in
Boro
we
use
to
two
sets
of
keys.
We
use
one
as
a
signing
key,
and
that
is
a
edie
two
to
five
one,
nine
based
elliptical
curve
and
that
elliptical
curve
is
not
very
good
for
connecting
over
a
network.
A
In
fact,
it's
actually
really
bad
at
being
able
to
do
peer-to-peer
connections,
and
so
we
use
a
slightly
different
key
pair
for
for
peer-to-peer
communications
to
be
able
to
authenticate
a
particular
node.
We
use
elliptical
curve,
diffie-hellman
key,
that
that
is
there,
and
so
we
we
map
two
sets
of
keys
to
each
particular
node
that
is
running
on
the
network
and
if,
if
I,
just
pull
up
the
configuration
file
for
a
particular
node.
A
B
B
B
A
A
So
well,
let
me
say
it
this
way:
I'm
not
in
the
security
game.
So
I
don't
know
if
Dave
is
in
the
room
or
somebody
else,
others
would
have
much
B,
but
much
better.
To
talk
to
about
that.
I
will
say
that
kubernetes
secrets
are
used
for
really
high
value.
Api
keys
in
production
every
single
day,
I
know
I,
know
many.
Many
systems.
A
A
A
Tear
down
that
container,
the
CI
system
tears
down
tears
down
that
container,
and
so
in
theory,
because
there's
no
I
mean
there
may
be
some
lag
in
there,
but
for
demo
environments,
it's
safe
enough
for
us
yeah
any
other
questions.
So
so
then,
when
we,
when
we
boot
a
then
when
we
boot
a
particular
runtime,
if
you
look
at
these
lines
here,
what
this
is
do?
A
What
this
is
saying
is
to
mount
the
secret
that
I
uploaded
that
had
the
particular
key
pairs
that
is
mounted
into
a
directory
that
the
runtime
knows
is
where,
where
the
keys
should
be
so,
then
the
Boro
runtime
has
access
to
the
keys
that
I
have
giving
it
given
it
and
for
our
demo
environments.
What
we
do
is
to
make
things
a
little
bit
more
simple,
operationally
it
we
we
make,
we
add
all
of
the
keys
into
one
particular
secret,
and
then
we
mount
that
secret
across
all
of
the
runtime.
A
So
all
the
run,
all
the
validator
runtimes
they
actually
have
access
to
all
the
validator
keys.
This
isn't
something
that
you
would
do
in
production
in
production.
You
would
do
a
one-to-one
mapping,
but
depending
on
sort
of
how
you
set
up
your
secret
management
system.
If
you
used
other
systems
or
use
grenade'
secrets,
you
would
you
would
isolate
these
a
little
bit
more,
but
for
a
proof-of-concept
environment,
for
a
demo
environment
and
for
a
pilot
environment.
This
is
this
I
would
say
it
would
be
fine,
I.
A
What
what
you're
only
showing
here
is
the
keys
that
one
one
portion
or
one
entity
of
this
particular
block
chain
has
access
to.
So
this
wouldn't
be
all
the
keys.
This
would
just
be
all
the
keys
within
my
cluster,
and
and
so
we
mount
that
into
the
runtime
and
then
the
second
thing
that
we
have
to
do
is
we
have
to.
A
Then
we
have
to
manage
what
are
my
identities
and
and
I
said
earlier.
There's
there's
two
pieces
from
Burroughs
perspective:
there's
two
pieces
to
its
identity:
there
is
its
signing
identity
and
there
is
its
peer-to-peer
identity,
and
these
are
slight.
These
are
independent
of
one
another.
However,
at
the
at
the
network
level,
within
the
validators
object
within
a
borough
runtime
that
there's
a
mapping
between
the
two.
So
how
do
we
get
it?
A
The
the
validator
key
well
I
I
showed
earlier
how
how
we
do
we
inject
a
particular
validator
key
with
a
number
in
and
with
each
of
our
nodes
in
the
hell
charts
which
I'm
going
to
show
in
a
minute.
We
then
set
up
a
very
simple
mapping
that
all
this
does
is
it
copies
the
particular
node
key,
and
this
is
the
peer-to-peer
key.
We
copy
it
from
the
kubernetes
secrets
into
the
place
within
burrow,
where
we're
burrow
expects
that
key
to
be
and
then
and
then
we
give
it
the
the.
A
A
Does
anybody
in
the
room
not
know
what
helm
is
anybody
willing
to
admit
that
they
don't
know
what
helm
is?
Okay,
how
is
it
it's
how
it
markets
itself
is
as
a
package
management
system
for
kubernetes
and
essentially
what
it
is,
is
a
templating
system
over
kubernetes
configuration
system
so
that
at
a
core
level,
how
kubernetes
works
is
that
it
is
really
a
series
of
yeah,
mol
or
JSON
files
that
allow
you
to
say,
I
want
the
runtimes
to
look
like
this
and
then
under
the
hood.
A
A
Yamo
configuration
files
that
declare
to
the
kubernetes
api
server,
how
you
want
the
world
to
look
and
then
it
allows
for
a
templating
and
management
of
a
whole
range
of
different
systems.
So
if
you
want
to
manage
services
which
are
essentially
cluster
level,
ports
ingress
is
runtimes
which
come
in
a
couple
of
flavors.
All
of
that
can
be
done
as
a
package
around
a
particular
system
that
you
want
to
send
to
into
the
kubernetes
platform.
So.
A
A
This
is
how
the
validator
addresses
look
now.
This
is
this
is
really
frankly,
this
is
really
ugly
code,
but
it
is
about
the
best
that
we
have
been
able
to
get
to
in
terms
of
dealing
with
the
keys.
So
essentially,
what
we
do
is,
if
you
see
on
line
85
before
we
do
the
helm
chart
release,
we
tell
the
helm
chart
what
are
the
public?
A
Sorry,
what
are
the
addresses
of
the
particular
nodes,
and
we
give
it
to
that
in
in
an
order,
and
that
order
then
determines
what
key
anyone
a
validator
on
the
network
uses
and
then
at
the
spec
level
we
have
mapped
that
node
to
the
ability
to
be
a
validator
on
on
the
network,
so
these
home
charts
are
relatively
easy
to
to
use
the
way
that
helm
works
in
general
is
that
it
works
on
this
idea
of
values,
files
and
values.
Files
are
essentially
the
parameters
that
you
want
for
running
this
particular
package.
A
We
have
within
the
chart
and
ability
to
add
extra
seeds,
which
are
at
nodes
external
to
your
cluster,
that
you
want
to
connect
into
you
set
up
the
number
of
nodes
log
level,
other
things,
and
then
you
have
the
validator
addresses
object
and
that
validator
addresses
object
is
what
tells
you
the
addresses
both
the
signing
address,
which
we
call
just
the
address
and
the
peer-to-peer
address,
which
we
call
the
node
address
and
then
some
other
normal
runtime
configuration
such
as.
Do
you
want
to
open
to
open
external
to
the
cluster
various
ports
in
burrow?
A
We
have
a
range
of
ports.
We
of
course,
have
the
peer-to-peer
port.
We
have
a
gr
PC
port.
We
have
an
info
port
which
serves,
which
serves
over
HTTP,
a
very,
very
specific
information
about
how
there
was
a
burrow
runtime,
and
then
we
have
metrics
profiler
and
a
few
other
ports
that
we
use
now
once
once.
A
That
system
is
up
and
operational,
then
what
we
do
so,
if
I
go
back
to
the
CI
system
that
I
triggered
at
the
beginning,
what
this
did
is
it
booted
a
cluster
of
burrow
nodes,
which
is,
are
these
lines
here
so
terraform
outputs
for
us?
What
are
the
public
key
addresses
and
then
we
run
through
a
sequence
of
sending
a
quite
sophisticated
well
over
I
would
say
well
over
22,000
lines
of
solidity
code
that
was
then
deployed
on
to
the
system
and
and
then
we
boot,
our
application-level
runtimes
on
top
of
it.
A
This
is
the
this
is
the
system
here
that
is
booting
that
and
then,
when
we
push
that
together
within
within
this
namespace,
all
of
these
runtimes
were
we're
booted
configured
and
talked
to
one
another
in
in
that
sequence.
So
we
have
some
UI
containers
here
we
have
an
Oracle
container
here
that
does
some
Oracle
stuff
on
this
particular
platform,
notifier
runtime.
We
have
the
Boro
configured
nodes
right
here
and
then
we
have
the
primary
API
containers.
Here
we
have
a
Postgres
that
we
use
for
querying
vent
if
anybody
is
interested.
A
It
also
does
a
pretty
sophisticated
encryption,
and
so
then,
the
next
thing
that
we
really
need
to
look
at
is
log
management,
so
we
use
in
system
one
of
the
advantages
of
leveraging.
Kubernetes
is
that
off-the-shelf
we
can
pull
an
elastic,
searching
an
elk
stack
and
we
run
a
really
quite
standard.
Elk
stack
for
elastic
search,
log,
log,
stash
and
Cabana,
and
this
kibana
here
is-
is
showing
the
dashboard
that
we
have
set
up
for
for
for
the
run
times.
A
The
burro
run
times-
and
you
can
see
here
that
these
are
just
the
the
logs
and
in
in
burro.
We
have
mostly
mostly
everything,
is,
is
relatively
easy
for
an
elastic
search
system
to
manage
all
the
all.
The
logs
are
JSON
formatted
as
long
as
you
choose
that
option,
and
that
makes
it
relatively
quite
easy
and
then
within
burrow
we
have
inbuilt
Prometheus
system
and
Prometheus
is
a
cloud-based
monitoring
system.
It's
housed
within
the
same
project
as
kubernetes,
and-
and
this
is
graph
on
a
graph
on.
A
What's
the
mempool
looking
like
what
are
the
transactions
per
blocks
blocks
per
minute,
quantiles
around
the
blocks,
CP
usage
memory,
usage
and
other
sort
of
low-level
go
primitives
that
we
can
then
track
from
from
our
monitoring
system.
Obviously
what
blocks
are
we
on
as
well,
and
so
this
is
with
five
minutes
left.
This
is
how
we
do
monitoring
and
how
I
wanted
to
kind
of
wrap
up
here.
A
Then
it
makes
your
life
a
whole
lot
easier
because
then,
as
a
cluster
administrator,
because
then
you're,
not
thinking
of
block
chains
as
some
other
thing,
but
rather
are
just
integrated
in
and
an
example
of
this
is
that
when
we
submitted
our
helm
charts
to
the
helm
repo,
you
know
those
the
helm.
Repos
have
a
CI
system.
A
That
manages,
does
this
thing
boot
or
not
essentially,
and
they
can
do
that
for
a
whole
range
of
distributed
systems
and
we
were
the
first
blockchain
to
be
submitted
into
them
and
other
than
and
so
I
had
to
talk
to
the
maintainer,
about
the
differences
between
running
block
chains
and
others,
and
between
us
and
and
and
then
just
after
we
started.
Somebody
from
Microsoft
was
working
on
getting
theorem
charts
in
there
as
well
and
between
the
two
of
us.
A
We
were
able
to
say:
okay
within
the
CI
system,
we
essentially
need
a
way
to
manage
keys,
but
outside
that
the
CI
system
should
run
off
the
but
off
the
bat,
and
so
they
gave
us
an
ability
to
have
essentially
a
testing
mo
and
with
a
single
testing
animal
where
we
could
set
up
keys,
we
were
able
to
instantly
run
and
pass
all
of
the
CI
CI
systems.
So
with
that
I
have
about
five
minutes
left
or
maybe
less
yeah.
A
A
A
Yeah,
so
we
so
this
is
another
difference
in
how
you
deal
with
block
chains
versus
other
systems
in
that
block
chains,
don't
really
horizontally
scale,
and
you
can't
you
can't
do
it.
You
can't
do
it
manual.
I
mean
sorry,
you
can't
do
it
automatically.
So
if
we
just
scale
up
the
number
of
runtimes
within
the
system
that
won't
actually
work.
So
we
do
it
manually.
We
we
would
generate
a
new
key,
a
new
two
sets
of
keys.
We
would
then
upload
those
into
the
particular
place,
depending
if
we
it
was
a
demo
environment
or.
B
A
Be
it
wouldn't
be
too
hard
to
add
I
mean
you
could
run
a
helmet
art.
The
charts
are
pretty
flexible.
They
they
could
be
ran
with
a
different
release.
That
would
then
do
your
new
keys
and
because
of
how
how
manages
it
state
pretty
well
I.
Think.
Actually,
if
you
just
added
a
new
chart
and
reran
on
the
same
release,
it
may
or
may
not
work,
but
we've
never
actually
tested
doing
it.
That
way,
so
yeah.
A
So
is
your
question:
how
do
we
deal
with
node
failover
yeah,
okay,
that
that's
something
that
I
missed?
So
we
struggled
with
this
for
a
little
while
how
we
deal
with
this
and
and
the
the
key
thing
that
you
really
need
to
know
is
if,
if,
if
the
runtime
exits,
so
if
the
runtime
exits
kubernetes
knows
that
it
that
it
exited
and
your
old
boot,
so
that
part
is
fine.
A
The
harder
part
for
us
to
deal
with
is
the
runtime
still
going,
but
for
whatever
reason,
this
runtime
has
become
disconnected
from
the
network
and
this
runtime
isn't
making
blocks,
and
so
within
the
info
port
that
I
talked
about.
We
actually
build
a
system
where
you
can
query
the
runtime
to
say.
Have
you
made
blocks
within
this
last?
You
know
minute
five
minutes.
A
10
minutes
is
configurable,
you
you,
you
can
say,
have
you
made
blocks
within
this
time,
acts
and
so
with
our
liveness
and
readiness
probes,
which
is
how
kubernetes
tells
that
a
runtime
is
still
doing
what
it
should
be
doing.
We
use
this
end
point
and
we
essentially
ask
the
Boro
runtime.
Have
you
made
blocks
within
the
last
three
three
minutes:
I
think
is
how
we
currently
have
it
configured
and
and
the
burrow
node,
if
it
hasn't,
if
it
has
not
made
blocks
within
those
minutes,
will
then
give
a
400
I.
A
Think
it's
a
4
or
500,
but
anyway,
it'll
give
it
it'll,
give
an
error
HTTP
code,
which
will
then
signal
to
kubernetes
to
failover
that
that
particular
node,
which
almost
always
then
reconnects
the
node
and
then
it
will
carry
on
because
obviously
the
state
is
handled
in
volume.
So
the
state
doesn't
go
away
when
you
fail
over
the
run
time.
A
Yeah
yeah
it'll,
just
reboot,
with
the
secrets
mounted
and
and
the
old
blocks
are
the
old
blocks
and
data
are
in
a
persistent
volume,
so
it'll
just
connect
that
and
it'll
say
carry
on
that's
how
we
do
it
with
no
failover,
but
that
was
a
that
was
an
interesting
thing
that
we
actually
had
to
build
a
custom
system
to
do,
because
then,
when
the
runtime
was
working,
if
it
was
on,
then
that
wasn't
really
the
biggest
concern
anybody
else.
I
think
I'm
over
time
anyway.
So
sorry
to
evers
next
yeah.
C
C
A
Are
you
talking
about
the
data
that
is
saved
on
disk
from
the
blocks
and
and
state
yeah?
Okay?
So
how
we
deal
with
that
is
kubernetes
abstraction
called
persistent
volumes.
So
we
save
everything
in
persistent
volumes
which
maps
to
the
persistent
volumes
within
the
cloud
within
the
cloud
providers,
and
so
those
persistent
volumes
can
get
moved
around
and
mounted
on
physical
disks,
I'm,
sorry,
a
actual
machine,
sorry
and
then
and
then
exposed
into
docker
runtimes
kubernetes
handles
that
really
elegantly.
It's
a
one
of
the
primary
things
that
kubernetes
handles.