►
From YouTube: Lessons Learned from Building a Kubernetes Offering for Small Edge Devices- Daniel Fröhlich, Red Hat
Description
Lessons Learned from Building a Kubernetes Offering for Small Edge Devices - Daniel Fröhlich, Red Hat
There is a rising demand to use cloud technologies like container orchestration at the far edge. Far edge usually comes with tight resource constraints like very limited compute power (2 cores, 2GB RAM), limited energy (passive cooled on solar energy), only intermitted any slow network connectivity (e.g. mobile/sat networks). MicroShift.io is a Kubernetes distribution targeting these types of environments. In this talk, we share lessons learned on the challenges of designing MicroShift. Topics we will address: - Selecting CNI and CSI implementations - How to embed container images into the device to avoid pulling from remote - How to handle updates/rollback without bricking a device - Fine details on Conformance testing
A
Yeah
so
welcome
everybody
good
afternoon
here
in
beautiful
Amsterdam.
I
must
admit
it
for
me,
the
first
time
being
in
Amsterdam,
which
is
really
strange,
because
you
hear
from
my
accent
obviously
I'm
a
German
guy
I
live
nearby
cologne,
which
is
just
a
three
hour
ride
and
yeah
I'm
already
today,
I
know
I
will
be
back
because
it's
such
a
beautiful
city
Steve.
Thank
you
for
the
nice
introduction.
You
actually
spared
like
five
minutes
of
mild
production,
on
what
is
Edge
Computing
and
what
are
the
use
cases.
A
So
my
name
is
Daniel
I
work
for
Redhead
since
I
think,
seven
years
now,
I
used
to
be
in
sales
working
a
lot
in
an
open
shift:
sales
selling,
kubernetes
to
standard
I.T
customers
and
recently
I
transitioned
into
the
product
management
role,
so
I'm,
actually
a
product
manager
at
redhead
and,
as
you
see
in
the
slide,
redhead
has
been
born
30
years
ago
in
the
traditional
Data
Center.
And
yes,
we
we
just
celebrated
30
years
of
surviving
with
open
source
in
the
it
space
which
is
I,
think
quite
an
achievement
and
proud
of
that.
A
So
after
winning
the
data
center,
we
went
into
the
cloud.
Harvard
cloud
is
obviously
a
big
business
in
the
last
10
years
now,
but
recently
we
extended
to
Edge
and,
as
Steve
said,
edge
Computing
use
cases
are
from
from
my
personal
definition,
really
easy.
If
you
do
any
kind
of
compute,
not
in
your
data
center,
not
in
your
Cloud
but
anywhere
else
yeah,
it
could
be
a
really
small
Edge
device
running
in
a
point
of
sale,
point
of
information
solution.
A
It
could
be
a
cruise
trip,
it
could
be
a
satellite
flying
around
the
Earth
I'm.
Now
since,
like
five
to
four
years
in
this
Edge
space,
and
if
I
learned
one
thing,
then
it
is
no
one
size
fits
all
yeah.
Everybody
has
different
requirements:
different
sizes
really
from
two
cores:
two
gigabytes
of
RAM,
just
a
Raspberry,
Pi
internet
type
of
device
ranging
to
server
grades
up
to
hey.
Actually,
it's
a
big
rack
and
it
feels
like
a
data
center.
A
So
the
more
aspects
are
like
the
connectivity
power
consumption
that
are
criterias.
If
you
are
constrained
in
any
kind
of
these
domain
domains,
that
is
already
kind
of
an
edge
deployment
right
and
it
could
be
power.
Consumption,
for
example,
yeah
think
about
a
flying
drone.
Think
about
an
oil
rig
where
you
have
just
solar
power
where,
for
example,
arm
is
really
good
because
it
has
a
nice
power
to
compute
ratio
so,
but
still
I
mean
this
is
not
really
new.
A
We
have
been
doing
point
of
sale
Solutions
in
the
last
20
years
already
right.
So
cashier
is
something
like
that.
So
what
what
brings
us
to
the
game-
and
that
is
that
we
want
to
apply
Cloud
native
principles,
Cloud
native
Technologies,
to
these
types
of
edge
locations
right,
because
I
try
to
avoid
the
word
cloud
native
applications,
because
I'm
at
the
edge
I'm,
not
in
the
cloud,
go
away.
A
How
old
is
kubernetes
now
is
yeah,
that's
one
of
the
major
Cloud
Technologies
and
we
want
to
use
that
for
at
Edge
locations,
and
that
is
what
I'm
going
to
talk
about
and
to
give
you
an
example
on
what
I'm
talking
about
or
meaning
is.
Let's
take
a
look
about
the
brief
history
of
red
Hat's,
kubernetes
distribution
and
the
option
Community
project
okd
or
openshift
at
the
Enterprise
supported
product,
and
here
you
can
see
the
trajectory
on
how
we
reduced
it
yeah
when
I
started
like
six
years
ago.
A
With
that
we
were
on
a
trajectory
like
you
have
a
minimum
scale
of
nine
servers,
you
need
yeah
and
it
has.
It
has
to
be
physical
service
anyway.
Yeah
then
getting
virtual
ones
even
as
really
a
big
one.
Yeah
remember
here,
external
LCD.
That
was
the
reason
why
we
needed
nine
nodes,
yeah
free
for
the
control
plane,
free
for
the
hcd
and
then
free
for
the
workload
which
is
by
the
way,
an
insign
ratio
of
overhead
to
what
you
can
actually
use
with
the
platform.
A
A
It
has
to
run
on
a
let's
say,
internet
with
just
four
cores
and
yeah
that
made
us
look
into
this
and
how
to
approach
it
and
actually
I'm
the
product
manager
for
these
two
products
here,
looking
at
the
requirements
and
shaping
the
roadmaps
for
these
before
we
jump
into
the
details
on
how
we
did
this,
let's
talk
about
the
elephant
in
the
room.
Why?
On
Earth,
what
do
you
like
to
do?
Kubernetes
on
a
two
core
system,
right
sing,
a
singer,
note
single
point
of
failure.
A
A
A
The
same
Technologies
methodologies
like
your
CI
CD
chain
and
use
the
same
charts,
maybe
to
deploy
your
workload
even
to
the
edge
location
right.
So
that
might
be
a
good
reason
to
run
kubernetes
on
it
on
that
yeah
and
all
the
goodness
of
Royal,
for
example,
just
a
rolling
update,
if
you
think
about,
for
example,
if
you
just
use
a
Docker
or
Portman
on
a
single
node,
you're
a
little
bit
in
trouble
and
doing
a
rolling
update
while
the
system
is
running,
you
can
do
that,
but
it's
much
uglier
much
more
complicated.
That's
that's!
A
What
kubernetes
is
really
great
at
right
and
still,
you
need
to
add
the
additional
stuff
like
Ingress
routing
security
certificates
and
so
on.
So
actually,
there
are
good
reasons
and
you
have
to
balance
that
yeah,
and
that
is
why
at
redhead
redecided
that
we
actually
approach
this
and
tackle
that
so
there's
a
big
difference
between
our
where
we
are
coming
from
the
traditional
okd
openshift
single
node
distribution,
which
you
can
see
here
on
the
right
hand,
side,
which
is
a
full-blown
kubernetes
distribution.
A
The
design
approach
of
that
is,
we
take
the
full
whole
platform
and
put
it
on
a
diet.
Make
it
slim
remove
components
you
don't
need
remove.
For
example,
all
the
scaled
out
deployments
reduce
it
to
one,
but
still
it
is
a
full-blown
distribution
and
the
design
philosophy
is
it
works
quarks
and
acts
like
the
full-blown
thing.
So
everything
we
have.
You
are
not
you
you're
familiar
with
with
openshift
like
the
console.
A
Everything
is
there
yeah
and
that
brought
us
to
a
dilemma,
because
the
minimum
system
footprint
of
the
regular
openshift
kubernetes
distribution,
is
for
Cores
and
out
of
these
four
cores,
you
can
utilize,
probably
two,
because
the
rest
is
used
by
the
LCD
API,
something
like
that
yeah
much
more
RAM.
So
to
get
into
this
really
two
core
two
gigabytes
of
RAM,
and
you
can
actually
do
something
usable
with
that.
A
We
had
to
switch
design,
approaches
yeah,
and
that
is
why
we
started
on
the
left
hand
side
with
microshift,
where
we
basically
the
design
approaches
we
start
with
nothing.
Then
we
take
the
absolutely
minimum
Linux
distribution
we
can
think
of,
and
then
we
add
just
the
layers
and
pieces,
you
need
to
run
your
kubernetes
workload.
That's
the
design
goal
that
we
have
kubernetes
workload,
portability
that
the
same
workloads.
You
run
on
your
full
openshift
cluster.
You
can
run
on
the
microshift
cluster
yeah.
A
That's
basically
the
idea
so
start
with
nothing,
and
you
can
see
a
lot
of
differences.
For
example,
here
it's
just
networking,
Ingress
storage
and
that's
basically
it
all
the
rest
like
monitoring,
logging
and
so
on,
is
not
there
because
you
don't
need
it
right.
So
it's
really
the
bare
minimum
approach
and
every
time
we
add
anything
to
this
distribution
of
microshift,
we
think
about
how
big
is
it?
How
much
CPU
does
it
consume?
How
much
RAM
does
it
consume?
Can
we
import?
Do
we
want
to
afford
it?
Does
every
customer
need
it?
A
So
in
the
next
couple
of
minutes,
I'm
talking
only
about
microshift
and
these
design
decisions
we
made
to
build
this
kubernetes
distribution
and
once
you
have
the
very
Basics
like
okay,
you
need
where
to
solve
the
kubernetes
state
we
use
LCD
for
that.
That
was
for
us
kind
of
a
no-brainer,
because
that's
the
one
where
we
have
biggest
experience,
and
so
we
keep
with
that.
A
The
kind
of
the
really
next
thing
is
the
networking
provider,
the
cni
implementation.
What's
your
cni
driver,
you
would
like
to
use
so
here
you
can
see.
We
have
come
up
with
a
couple
of
requirements.
We
need
to
take
a
look
at,
for
example,
the
footprint
as
I
mentioned.
What's
the
CPU
Ram
consumptions,
but
there
are
other
requirements
like,
for
example,
this
Dynamic
IP
requirement,
which
is
a
little
bit
unusual
for
example,
especially
if
you're
coming
from
the
takeaway.
A
If
you
think
about
the
take
or
5G
base
station
deployment
that
doesn't
change
as
IP
addresses
yeah
it
has
nice
Network
addresses
everything's
fine.
Now,
if
you
think
about
a
let's
say,
train,
you
have
an
edge
device
running
in
a
train
moving
around
it
switches
networks,
it
switches,
IP
addresses,
so
you
have
to
support
Dynamic
IP
address
changes
which
is
in
the
data
center
Cloud.
It's
not
that
often
yeah
and
the
devil
is
really
in
the
detail.
A
It's
like
your
network
provider
has
to
be
able
to
cover
with
that,
but
also
the
surroundings
like,
for
example,
have
you
thought
about
certificates?
Oh,
yes,
there
are
internal
certificates
because
the
internal
communication
is
secured,
so
you
have
internal
certificates
and
there
might
be
the
IP
address
in
there,
so
you
might
even
as
a
side
effect,
be
able
to
recreate
certificates.
A
So
and
then,
of
course,
we
have
requirements
like
we
have
to
be
able
to
support
it,
supportability,
because
at
the
end
we
are
making
money
of
this
by
providing
Enterprise
support
for
this.
So
can
red
hat
as
a
company
support.
This
technology
is
an
argument
but
also
other
requirements.
For
example,
is
there
the
the
network
policy
support
for
cni
networks?
A
If
you
want
to
isolate
between
namespaces
stuff,
like
that,
so
lots
of
lots
of
different
requirements,
and
then
so
we
took
a
look
into
the
different
providers
or
implementations
and
did
an
assessment
like,
for
example,
we
did
measurements
of
the
footprint,
how
much
CPU
run?
Does
it
use
yeah
and
for
all
the
rest?
And
so
here,
for
example,
the
obvious
outcome?
Is
we
choose
ovnk
yeah
besides,
yeah
footprint
is
a
little
bit
higher,
but
all
the
rest
is
green,
so
microshift
uses
ovnk,
SSC
and
I
provider
yep
same
question.
A
Same
problems
arise
around,
for
example,
storage.
Frequently
we
get
a
question.
Why
is
storage?
You
are
talking
about
modern
applications.
They
should
be
stateless
right.
Why
do
I
need
storage
at
an
edge
device?
Welcome
to
reality,
Steve
told
us
it's
about
data.
It's
about
data
at
the
edge.
You
might
be
not
allowed
to
transfer
that
data
into
the
cloud.
You
might
not
want
to
transfer
it
into
the
cloud.
Still
you
have
to
store
it
think
about,
for
example,
time
series
database.
A
If
you
have
the
ultrasonic
data
from
your
example
or
if
it's
temperature
curves
to
do
this
predictive
maintenance,
digital
twin
type
of
use
cases
you
might
need
to
store
at
least
the
last
week
of
data,
and
you
have
to
do
this
on
disk
right,
because
a
reboot
might
happen.
It
might
be
a
power
outage
so
doing
this
in
Ram
is
not
a
good
idea.
A
So
yes,
having
a
CSI
container
storage
interface
provider
is
a
good
idea
even
on
a
two
core,
two
gigabytes
of
RAM
HD
device,
but
it
has
to
cope,
maybe
with
only
16
gigabytes
of
SD
card
right,
so
yeah
and
again
we
we
did
the
same.
We
had
requirements
like
it
has
to
be
run
unprivileges
here
it
has
to
enforce
storage
size
requests,
that's
what
that
was
a
really
important
one.
That
is,
if
you
have
two
ports
and
they
store
data
via
their
persistent
volume
claim
into
a
PV.
You
don't
want
that.
A
If
part
A
is
misbehaving
and
consuming
all
the
storage
and
filling
the
data
you
do
not
want
to
have
to
that.
An
impact
to
the
other
plot
yeah
that
that
should
be
isolated.
But
if
you
do
really
simple
host
pass
provisioning,
oh
sorry
for
the
abbreviation
HPP
is
host
pass
provisioning.
So
what
you?
What
you
simply
do?
Is
you
map
that
storage
PVC
claim
from
the
Pod
to
the
file
system
of
the
host?
If
you
do
that,
there
is
no
isolation,
because
it's
just
a
directory
yeah
and
in
that
directory
they
share
everything.
A
So
there's
no
isolation.
So
that's
not
a
good
thing.
Yeah!
That's
here!
Protection
against
greedy
workloads,
support
snapshots
snapshots
are
CSI.
Snapshots
are
important
because
that's
the
entry
point
for
each
and
every
kubernetes,
backup
and
Recovery
Solution,
because
the
data
might
be
valuable,
so
you
might
want
to
have
backup
and
Recovery
Solutions
for
that
in
place.
A
So
yeah
we
actually
did
the
same
arm
support.
That
was
also
a
fun
surprise,
because
obviously,
at
the
edge
arm
is
a
good
idea,
and
so
we
did
again
our
due
diligence
and
in
this
case
we
decided
for
our
variation
of
Topo
IVM.
It's
an
upstream
project
which
simply
Maps
persistent
volume,
claims
persistent
volumes
to
logical
volume,
manager,
logical
volumes
if
you're
familiar
with
Linux,
there's
lvm
thing,
which
is
yeah
quite
flexible
in
providing
storage,
and
it's
really
nice
because
it
has
Dynamic
provisioning
and
it
yeah.
A
Yeah
talking
about
another
really
challenge
at
the
edge
is
you
do
not
want
to
break
your
Edge
device?
We
have
customers
who
run
this
stuff
at
a
location
where,
if
you
need
to
get
physical
access,
you
have
to
fly
in
with
a
helicopter,
that's
expensive!
You
really
want
to
avoid
that
now.
How
do
you
do
an
update,
consistently
transactional
without
breaking
your
device
and
the
the
simple
ingredients
for
that
is
first
of
all
use
OS
tree
os3
is
a
different
deployment
type
of
Linux,
where
you
have
kind
of
an
immutable
operating
system.
A
So
it's
not
RPM
based
you
install
your
packages
at
that
device,
but
you
on
your
development
environment,
build
the
distribution.
You
build
your
image,
your
commit
and
there's
everything
baked
into
it,
and
then
you
just
deploy
that
to
your
Edge
device
yeah,
it's
kind
of
a
change
from
a
package
base
to
an
image
based
deployment
yeah.
That
is
something
we
have
in
Linux
for
quite
a
while
out
there,
which
is
good
because
it
brings
you
this
Atomic
update
capability
so
on
a
single
edge
device.
A
A
A
So
that
is
a
really
nice
feature
which
is
out
there
as
or
S3
based
deployments
for
a
couple
of
years,
and
now
we
add
kubernetes
to
that
which
makes
your
life
a
little
bit
more
complicated,
because
you
could
imagine
that
from
version
A1
to
version
A2,
you
actually
have
a
kubernetes
update
in
there.
So
your
LCD
database
might
change
the
physical
layout.
Api
might
changes
all
that
stuff,
and
now
you
roll
back.
A
So
you
have
to
do
a
rollback
of
your
kubernetes
distribution,
and
that
is
for
us,
for
example,
in
in
product
management
and
Engineering
currently.
Hence
the
work
in
progress
sign
here
quite
a
challenge,
and
that
is
what
we
are
currently
focused
on
to
get
this
right,
because
that
is
one
of
the
big
differences
between
our
small
microchip
kubernetes
distribution
and
the
full-blown
okd
distribution.
The
okd
distribution.
Never
ever
rolls
back.
It's
always
path
forward
yeah.
A
A
So
with
that
being
said,
I
think
yeah.
Networking
is
also
topic,
that's
the
last
one,
so
this
has
to
work
in
a
fully
self-contained
air
gap
networked
environment
like,
for
example,
your
your
Edge
device
might
have
not
a
connection
at
all.
And
yes,
we
have
customers
with
a
requirement.
You
walk
up
with
an
USB
stick
to
that
edge
device
plug
it
in
turn.
It
on
wait
and
then,
after
let's
say,
60
seconds,
two
minutes,
the
whole
workload
is
up
and
running.
How
do
you
get
this
done?
Yeah?
A
Usually
you
pull
all
the
images
you
need
from
the
from
your
registry
and
so
on,
and
here
the
idea
is
that
in
that
OS
tree
commit
that
you
build
it's
literally
everything
you
need
to
run.
Your
workload
is
baked
into
that.
Even
your
container
images.
Yes,
we
configure
the
runtime
in
a
way
that
you
have
actually
two
container
image
locations.
One
for
the
static
immutable
part
of
the
OS
3
commit
keep
in
mind.
Os3
commits
are
unusual.
You
cannot
change
them
after
you
have
built
them
and
then
you
might
later
add
Dynamic
ones.
A
So
we
have
actually
both
storages,
and
that
is
what
you
hear
in
this
image.
Builder
blueprint
declare,
for
example,
for
microshift
offline.
You
describe
here,
for
example,
the
microchip
package.
This
actually
brings
you
the
microshift
installation
to
your
Edge
commit,
and
then
you
can
declare
all
the
container
images
like,
for
example,
where
do
we
want
to
put
the
get
these
container
images
from
so
now
at
image
build
time
all
these
images
are
being
pulled?
And
yes,
we
provide,
for
example,
for
all
the
internal
images
Microsoft
requires
for
the
router,
the
networking
and
so
on.
A
We
provide
the
necessary
locations
where
to
find
them,
so
you
can
bake
them
in
and
then
you
have
all
really
fully
safe
content
there.
You
can
just
turn
the
device
on
and
it
works.
I
could
talk
for
hours
about
this
I
had
no
time
to
talk
about,
for
example,
security.
One
of
the
really
really
big
challenges
in
Edge
Computing
is
security,
especially
if
you
can't
guarantee
the
let's
say,
physical
Integrity
of
your
device.
A
If
you
have
to
live
with
the
fact
that
it
can
be
stolen
because
it's
setting
out
on
a
parking
lot
and
it's
just
a
big
display
stuff
like
that,
how
did
it
handle
private
keys
I
think
we
have
a
session
later
for
that
on
how
to
handle
that?
It's
not
that
easy.
So
with
that
I
think
I
am
kind
of
done
with
what
I
wanted
to
say
all
about
this
I
will
be
here
for
the
rest
of
the
day.
I
will
be
here
for
the
rest
of
the
week.