►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Taming Unbounded Resources with Node Feature Discovery - Mark Abrams, SUSE
Although Node Feature Discovery (NFD) can be used in any Kubernetes cluster, it's value really shines at the Edge. Kubernetes has 2 notable differences at the Far Edge versus in the data center. 1) The edge exploits and unbounded number of heterogenous resources 2) Enterprises are reporting cluster deployments defined in numbers from hundreds to 10s of thousands. These two characteristics will require management capabilities beyond what use in the data center today. This presentation and demo uses a small fleet of Kubernetes edge clusters and NFD to show how we can build solutions which expose the availability of any arbitrary resource desired. The capabilities proposed by NFD will provide much needed insights into the unbounded capabilities of the massive number of available resources at the Edge.
A
Hi
welcome
to
the
kubernetes
edge
conference.
This
is
the
session
on
taming
unbounded
resources
with
node
feature
discovery.
My
name
is
mark
abrams
and
I'm
a
field
engineer
and
edge
specialist
at
susa,
formerly
with
rancher.
So,
first
of
all,
I
want
to
take
a
minute
to
say
that
edge
means
different
things
to
different
stakeholders.
A
A
These
are
the
leaf
nodes,
so
the
leaf
clusters,
for
example,
we'll
see
these
edge
use
cases
in
factories
in
retail
scenarios
in
energy,
in
the
automotive
and
aerospace
industries,
we'll
see
it
in
smart
cities
and
there's
lots
of
other
industries
with
that.
Let's
take
a
look
at
node
feature
discovery,
node
feature
discovery
is
really
it's
simply
a
fancy
labeler,
you
can
use
it
to
label
kubernetes
nodes.
A
A
A
A
Do
I
mean
by
unbounded?
Well,
because
these
unique
resources
are
not
they're,
not
infinite
right,
so
I
say
unbounded
because
there's
really
no
limit
to
what
device
might
be
attached
to
your
notes
with
with
each
individual
use
case,
there's
like
a
finite
set
of
resources,
but
overall
for
the
for
the
whole,
you
know
global
edge,
the
it's
there,
it's
unbounded,
so
anyways
with
your
you
know.
A
Without
node
feature
discovery,
and
you
know
massive
number
of
numbers
of
clusters,
it
could
be
like
playing
a
game
of
go
fish.
I
don't
know
if
you
know
this
game,
but
basically
you
take
turns
with
your
opponents
asking
if
they
have
the
resource
you
need.
I
say
to
my
opponent:
do
you
have
any
threes?
They
don't
have
any
in
their
hands,
so
they
say:
go
fish,
it's
a
guessing
game
and
eventually
you
can
use
cash
and
history
to
make
more
accurate
guesses
right.
So
I
have
a
sense
of
what
my
opponent
has.
A
A
A
A
A
There
are
eight
ports,
but
they
only
map
to
four
of
the
seven
compute
modules.
So
not
every
compute
module
has
access
to
usb
and
each
pair
of
usb
actually
only
maps
to
one
of
the
compute
modules
and,
for
example,
here
is
compute
the
compute
module
at
device
slot
number
two
which
has
access
to
the
set
of
usb
connectors
shown.
A
A
A
A
So
the
first
thing
I'm
going
to
do.
I
have
kubernetes
running
on
all
the
on
the
nodes.
As
I
said
on
the
system,
and
so
the
first
thing
I
just
show
you
the
six
nodes
where
the
workers
run,
and
then
I
wrote
this
little
bash
script
that
will
just
grab
the
labels
using
kubectl
dumps
the
labels
for
each
node
into
a
directory.
So
touring
pi01.labels
we'll
have
the
labels
from
touring
pi
zero.
A
A
So,
let's
clear
it
and
then
let's
install
node
feature
discovery
so
I'll
install
it
by
installing
a
couple
of
config
files.
These
config
files
are
there's
a
master
and
a
worker.
I
simply
downloaded
these.
I
didn't
modify
these
well.
I
yeah.
I
changed
these
to
use
an
image
container
that
I
created
for
the
arm
system.
I
couldn't
get
their
arm
containers
working,
so
I
rolled
my
own,
but
basically
these
are
preset
and
ready
to
go
so.
A
Great
okay,
so
you
can
see
that
I
I
now
have
labels
that
have
this
domain
prefix
and
then
I've
got
stuff
for
cpu
for
kernel
for
storage
for
system.
This
is
all
just
defaults.
I
didn't.
I
didn't
change
anything
here.
This
is
just
what
node
feature
discovery
does
out
of
the
box
and
you
can
see
there's
even
one
for
for
usb.
A
This
is
actually
the
ethernet
adapter
for
the
devices.
Okay,
so
we've
got
we've.
We've
now
got
node
feature
discovery
installed.
We
can
see
it's
working
and
what
I
want
to
do
is
actually
modify
the
config
map.
So
the
node
feature.
Discovery
is
managed
by
a
configuration
file
which
tells
nfd
worker,
which
types
of
resources
to
look
for
to
find
so
I'll
go
into
this
config
file,
and
I
will
uncomment
out
the
sources
section
and
then
I'll
find
the
usb
section.
A
I
will
uncomment
out
the
usb
section
and
open
up
the
device
class
white
list.
I
will
enter
just
one
device
class
and
that
is
class
0
3.,
that
is
a
human
interface
device
I'll
make
it
so
that
we
can
see
the
the
class
the
vendor
and
the
device
on
the
label
that
gets
applied
and
I'll
save
this
and
kubernetes
is
going
to
basically
now
label
the
nodes
that
have
a
human
interface
device.
Now
I
have
plugged
in
a
keyboard
so
a
human
interface
device
that
will
show
up
on
one
of
the
usb
ports.
A
I
just
modified
my
script
here.
Well,
I'm
I'm
using
the
script
but
to
get
the
labels,
but
then
I
want
to
show
you
every
node
and
I'm
only
going
to
we're
just
going
to
grep
for
usb
we're
not
going
to
look
at
everything.
I'll
just
show
you
where
the
labels
are
for
usb
now
so
here
you
can
see
that
usb
on
touring,
pi,
5,
node
5
is
has
this
usb
with
a
human
interface
device
label.
A
So
now
I'm
gonna
move
the
the
usb
unplug
it
and
plug
it
into
a
different
port,
and
so
what's
going
to
happen
when
I
do
that,
is
that
it's
going
to
relabel
it's
going
to
remove
the
label
from
the
one
that
no
longer
has
it
and
then
it
adds
it
to
the
one
that
has
it.
So
you
can
see
here,
it's
moved.
Originally
it
was
on
five
and
now
it's
on
three
and
that's
just
by
unplugging
and
plugging.
A
So
you
know
hot
swap
capability
with
node
feature
discovery
figuring
out,
which
features
exist
which
devices
exist
on
which
nodes
great
okay.
So
let's,
let's
so
that's
the
usb.
Let's
take
a
look
at
i2c
for
i2c,
I'm
going
to
use
the
kernel
module.
So
again,
I'm
editing
the
configuration
file,
I'm
going
to
look
for
the
custom
section.
A
I
will
uncomment
custom
and
then
I'll
add
this
block
of
text
that
or
this
this
block,
that
I
call
it
i2c
power
management.
That's
a
name!
I've
chosen
I'm
going
to
match
on
loaded
kernel
modules,
and
this
is
a
feature
of
note
feature
discovery
that
allows
me
to
select
loaded
kernel
modules,
i2c
dev
and
this
other
one.
If
both
of
those
exist
it
will
label
for
i2c,
so
we're
going
to
get
the
labels
again
and
we'll
pick
up
which
ones
have
i2c
on
them.
A
I'm
going
to
take
a
look
this
time
at
you
can
see
I'm
grepping
for
both
usb
and
i2c,
so
we're
going
to
look
at
all
the
nodes
and
just
ignore
everything,
except
for
i2c
and
usb,
and
what
we
should
see
is
that
every
node
has
i2c
because,
as
I
said,
every
node
is
connected
to
that
bus,
and
so
every
node
has
those
kernel
modules
and
therefore
the
labels
get
applied
everywhere.
Plus
we
still
have
on
node
number.
Three.
A
A
A
Now
I
happen
to
know
that
each
of
these
nodes
has
gpio,
but
I'm
only
labeling
touring
pi01
as
having
gpio,
and
I'm
saying
that
on
pin
number
40,
it's
an
input
pin
and
you
can
imagine
that
there
might
be
other
labels.
I
want
to
apply
for
specific
needs
for
each
node,
so
here's
the
file-
I
will
take
this
file,
move
it
into
this
directory
that
I
created
node
feature
discovery
will
notice
the
directory
the
change
in
the
directory.
A
A
A
A
A
A
It
reports
what
it
finds
to
the
node
feature:
discovery
master,
the
master
communicates
with
kubernetes
api
and
is
is
the
component
that
actually
does
the
labeling
of
the
nodes
from
a
human
perspective.
The
operations
team,
of
course
needs
to
know
their
devices
and
the
features
that
they
want
to
expose.
A
A
And
when
the
underlying
config
map
changes
the
worker
reloads
it,
the
config
map
is
that
default.
Config
map
with
that
was
commented
out
that
gets
installed
automatically
with
node
feature
discovery.
There's
a
healthy
bit
of
documentation
on
how
to
use
that
configuration
file
and
I'll
share
a
link
with
you
at
the
end
on
where
you
can
go.
Look
at
it.
A
A
A
A
A
A
A
A
Each
cluster
is
production
at
the
edge.
It's
all
production,
also
in
most
case,
most
of
the
edge
scenarios
that
I've
come
across,
we're
talking
easily
10
or
100
times.
The
number
of
clusters
shown
here
I
have
accounted.
I
have
32
clusters
here.
So
should
we
play
go
fish
with
320
or
32
000
clusters?
Now
I
don't
think
we
want
to
play.
Go
fish
right.
We
want
to
use
node
feature
discovery.
A
How
can
we
do
it
with
this
many
clusters,
given
how
node
feature
discovery
works?
I
think
that
git
ops
is
a
solution
for
this
problem.
One
of
of
these
three
tools,
I'm
really
only
familiar
with
fleet.
However,
I
know
that
argo,
cd
and
flux
are
also
being
used
as
git
ops,
tooling.
There
are
others
azure,
devops
and-
and
you
know,
there's
a
bunch
of
options.
A
A
The
concept
of
cluster
is
it's
a
custom
resource
definition
in
fleet
because
it's
managing
multiple
clusters
right,
but
it's
not
a
concept
in
kubernetes
now.
Fortunately,
there
is
a
kubernetes
special
interest
group
called
cluster
api
and
I
think
that
the
solution
they're
working
on
there
may
hold
the
answer
to
how
we
can
get
our
devops
tools
to
have
a
standardized
model
for
cluster
labeling,
based
on
node
labels
down
downstream.
A
This
is
what
I
think
the
gitops
pipeline
would
look
like.
So
assuming
we
can
get
this
working
simply
install
node
feature
discovery
into
the
pipeline
itself,
make
sure
your
config
is
set
up
either.
Do
that
as
part
of
the
install
or
you
know,
of
course,
you
can
do
it
afterwards
as
well,
and
you
may
end
up
doing
it
multiple
times.
That
is
probably
the
aside
from
upgrading
node
feature
discovery.
The
config
file
is
the
changeable
component.
A
So
that's
a
bit
of
hypothetical
get
ops.
I
did
want
to
also
talk
about
security
with
node
feature
discovery.
It
is
currently
possible
to
configure
tls
between
the
node
feature,
discovery
services
between
the
master
and
the
worker.
So
that's
out
of
the
box,
that's
a
nice
security
enhancement
that
I
have
not
yet
enabled.
A
I,
as
I
was
doing
this.
I
believe
that
allowing
that
local
user
specific
capability,
where
I
put
a
script
into
a
directory
and
let
it
execute
that
directory,
I
believe,
there's
some
risk
associated
with
that,
because
it
will
execute
anything
in
that
directory
now
it
only
executes
it
in
in
the
container.
A
A
In
addition,
it's
possible
probably
to
reduce
the
attack
surface
by
limiting
what
features
are
usable
in
the
configuration
file
in
that
config
file.
There
is
a
section
called
sources
and
we
can
do
things
like
specify
just
usb.
Just
custom
leave
out
local
or
you
know,
only
include
local
on
the
machines,
the
the
nodes
that
need
it,
that
sort
of
thing.
So
that's
a
little
bit
about
security.
A
This
is
a
summary
of
what
we
just
went
through.
Labels
are
applied
to
nodes
in
a
cluster.
I
think
git
ops
will
be
useful
in
labeling.
Large
numbers
of
clusters
features
are
unlocked
using
a
configuration
file.
Node
feature
discovery,
has
a
limited
number
of
out-of-the-box
feature
types
which
does
include
a
kernel
module
discovery.
A
Some
more
summary
items
I
didn't
show
this,
but
there
is
a
mount
in
the
container
for
uniquely
named
configuration
files
instead
of
the
default
nfd
worker
conf.
So
I
was
modifying
the
default
configuration
every
time.
It
is
possible
to
take
snippets
of
that,
put
them
in
uniquely
named
files
and
provide
a
configuration
through
this
other
directory.
A
A
I
know
that
currently
nvidia,
for
example.
Well,
they
don't
I'm
sure
the
config
file
is
available,
but
essentially
their
gpu
operator
use.
This
node
feature
discovery
to
label
nodes.
It
would
be
as
simple
as
finding
that
configuration
that
they
use
and
you
could
label
your
own
gpu
nodes.
Without
the
gpu
operator.
A
A
A
A
A
What
else
so
removable
devices
so
hot
plugging
works,
but
it's
not
immediate
in
the
demo,
I
showed
you,
I
did
it.
It
looked
like
it
was
happening
right
away.
There
was
some
time
lag
between
when
I
unplugged
it
and
replugged
it.
A
fair
amount
of
time.
Node
feature
discovery
needs
time
to
go
through
the
kubernetes
controller
process
of
checking.
Is
the
state
what
I
think
it
should
be?
No,
it's
not
change
it
once
it
changes
the
labels,
there's
going
to
also
be
another
process
where
kubernetes
says.