►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
COSI: The Common Operating System Interface - Steven Borrelli, Mastercard & Andrew Rynhard, Talos-Systems
This talk introduces COSI, the Common Operating System Interface, which defines an API for the configuration of container operating systems. While there has been innovation in the development of minimal Linux distributions for running Kubernetes, the distributions diverge from each other in terms of management and API endpoints. Similar to other projects such as the Container Network Interface (CNI) , COSI has a focus on the configuration of the underlying operating system, providing Protocol Buffer definitions and a gRPC API reference implementation for configuration settings such as DNS, network, and kernel settings By utilizing a flexible plugin system, multiple backend implementations can provide a consistent API to consumers. In this talk we’ll review the configuration and APIs of popular container operating systems and Kubelet-node interactions. We will then review the COSI API, the plugin architecture, and demo node management using backend plugins written in Go and Rust.
A
A
Let's
start
at
the
very
beginning
and
define
what
cozy
is
cozy,
it's
an
acronym
for
the
common
operating
system
interface
and
what
it
concerns
itself
with
is
the
configuration
of
the
nodes
that
run
in
a
kubernetes
cluster.
So
whenever
you
provision
a
new
cluster
first,
you
provision
the
nodes,
whether
they're,
bare
metal
or
vm,
and
then
on
the
next
thing
is
you
actually
have
to
configure
those
nodes
for
things
like
networking
disk,
any
processes
that
you
want
to
run
installing
the
kubernetes
software?
That's
what
cozy
concerns
itself
with.
A
It
also
gave
us
a
chance
to
reimagine
operating
systems,
and
for
this
is
what
we
do
in
operating
systems
in
a
kubernetes
world.
Another
thing
about
cozy,
that's
actually
been
really
fun.
Is
it
given
us
the
chance
to
rethink
exactly
what
it
means
to
be
an
operating
system
in
the
age
of
containers
and
distributed
schedulers?
A
The
first
thing,
one
of
our
highest
priorities,
is
having
an
operating
system
that
is
completely
api
driven,
and
what
this
means
in
practice
is
that
we
want
an
operating
system
that
doesn't
need
to
have
a
shell
installed,
and
we
would
not
like
to
have
people
sshing
into
this
box
to
configure
things.
Everything
on
the
operating
system
should
be
configurable
via
an
api.
A
Another
thing
that
we're
really
looking
to
implement
in
this
operating
system
is
having
configuration
settings
have
a
standardized
model
to
them.
Currently,
in
the
unix
world,
we
have
lots
of
different
file
formats
for
every
single
different
application,
and
this
causes
a
lot
of
issues
in
terms
of
integrating
with
other
tools.
So
one
of
our
goals
is
to
be
similar
to
kubernetes
and
have
a
standardized
model
for
any
kind
of
settings
that
we
want
to
put
into
whether
it's
a
disk
or
dns
settings,
or
anything
like
that.
A
Another
thing
that
was
very
inspiring
to
us
was
the
pattern
of
kubernetes
controllers
and
custom
resource
definitions.
Now,
custom
resource
definitions.
Allow
you
to
extend
the
platform
with
a
flexible
schema,
so
you
can
make
your
own
objects.
Look
like
kubernetes
on
the
back
end,
the
way
it's
decoupled
and
you
could
run
controllers
that
actually
get
to
the
desired
state,
and
you
can
write
these
in
any
language
you
want
and
they
just
continually
run
and
reconcile.
A
So
this
was
something
that
inspired
us,
this
kind
of
running
model
and
then,
finally,
we
wanted
these
systems
to
be
able
to
easily
propagate
events
to
other
applications.
Running
on
the
system,
now
there
are
a
lot
of
eventing
systems
on
linux,
but
we've
seen
in
the
past
that
they're
not
heavily
used,
because
sometimes
the
interfaces
are
difficult
to
use.
A
So
we're
looking
for
something
that
say.
If
someone
pushes
a
power
button
somewhere,
we
could
actually
tell
kubernetes,
if
someone's
done
that,
so,
let's
get
to
the
agenda
for
the
rest
of
the
talk.
First,
we're
going
to
venture
into
the
past
and
talk
about
a
little
bit
of
unix
history,
and
we
hope
this
will
help.
You
understand
where
unix
comes
from
then
we're
going
to
venture
into
the
present
and
talk
about
modern
container
operating
systems
and
then
we'll
be
having
a
deeper
dive
into
cozy.
A
So,
let's
start
by
talking
about
the
history
of
the
unix
system
and
to
assist
us
in
our
history
lesson
today
we'll
be
sourcing
a
video
from
1982
by
bell
labs.
It's
called
the
unix
system,
making
computers
more
productive,
and
I
do
appreciate
the
fancy
graphics.
So
the
first
thing
you
need
to
know
is
that,
if
you're
running
kubernetes
today,
the
odds
are
overwhelming
that
you
are
running
at
an
operating
system
called
linux,
which
is
a
descendant
in
a
very
complex
family
tree.
A
That
goes
all
the
way
back
to
this
earlier
system
called
unix
and
unix
itself
is
very
opinionated.
It's
a
very
simple
operating
system
because
it
descended
from
another
system
that
was
being
developed
in
the
late
60s.
It
was
called
multix
and
multix
due
to
its
complexity,
was
having
a
very
slow
development
rollout.
A
So
the
folks
who
were
developing
unix
at
bell
labs
wanted
simplicity
as
a
core
function
of
the
operating
system.
Another
thing
that
was
very
important
for
the
early
inventors
of
unix,
like
ken
thompson
and
dennis
ritchie,
is
that
they
were
programmers
and
they
wanted
an
environment
that
they
could
enjoy
working
in
in
a
communal
environment.
So
from
the
very
beginning,
this
concept
of
a
common
work
area
that
you
can
log
in
and
program
interactively
was
very
important
to
them,
and
these
kind
of
ideas,
simplicity
and
communal
work,
influence
unix
to
this
day.
A
A
They
would
set
up
a
terminal
hook
it
up
to
the
system
and
as
soon
as
they
did
that
they
would
get
a
program
called
a
shell
that
they
could
issue
interactive
commands
to
so
there's
two
concepts
here
that
we
see
in
unix
that
we
may
not
want
in
the
kubernetes
world.
The
first
one
is:
we
do
not
want
users
directly
logging
into
our
systems
and
we
don't
want
them
sharing
resources.
The
ideal
kubernetes
workload
is
something
that's
highly
isolated
from
every
other
workload.
A
A
The
last
thing
I
want
you
to
know
about
unix
is
fragmentation
when
a
t
developed
unix,
since
they
were
a
monopoly,
they
were
forced
by
the
government
to
license
units
to
other
companies
and
academic
institutions,
and
because
of
this,
we
ended
up
having
many
slightly
incompatible
unix
variants,
and
this
was
the
condition
for
the
first
couple.
Decades
of
the
life
of
unix.
A
Until
in
1991,
an
open
source
operating
system
came
out
called
linux
that
was
mostly
unix
compatible
unix
linux
over
the
next
couple
decades
ended
up,
eliminating
the
market
share
of
almost
every
commercial
unix
and
becoming
dominant.
Of
course,
linux
itself
saw
multiple
distributions
emerge,
each
which
was
slightly
incompatible
with
the
others.
A
In
2013,
the
linux
community
was
transformed
by
the
emergence
of
docker.
Docker
is
built
upon
c
groups
and
name
spaces
in
the
linux
kernel,
and
it
made
it
easier
for
developers
to
package
share
and
deploy
applications
using
docker
means
you
don't
have
to
have
a
system
administrator
install
your
application
on
the
server
itself.
They
could
just
run
your
container
with
the
growing
popularity
of
containers
came
the
emergence
of
cluster
schedulers,
including
our
favorite
one,
which
allowed
organizations
to
run
containerized
workloads
across
a
pool
of
servers.
A
This
combination
of
containers
and
schedulers
caused
many
developers
to
start
rethinking
the
role
of
the
traditional
operating
system,
and
this
led
to
the
emergence
of
the
container
operating
system.
Now
container
operating
systems
are
focused
on
stripping
out
functionality
from
full-featured
linux
distributions
set.
The
only
remaining
software
installed
is
what's
required
to
run
a
containerized
workload
and
the
schedulers.
A
So,
let's
talk
about
modern
container
operating
systems
as
part
of
the
research
into
cozy.
We
surveyed
the
documentation
and
code
for
a
number
of
popular
container
operating
systems,
including
flat
car
bottle,
rocket,
k3os
and
talos.
We
were
really
impressed
with
all
the
operating
systems,
but
there
are
several
things
that
we
noticed,
which
ended
up
influencing
the
design
of
cozy.
A
A
Other
ones
are
designed
not
even
to
be
managed
via
an
api.
You
just
boot
them
and
if
you
need
to
make
changes,
you
just
boot
them
with
different
user
data,
another
difference
between
all
these
api.
All
these
container
operating
systems
is
core
technologies
of
the
four
surveyed
operating
systems.
There
are
three
different
init
systems
now
two
of
these
container
operating
systems
use
system
d,
but
the
way
they
manage
it
is
completely
different.
A
Finally,
even
though
these
operating
systems
share
many
of
the
same
goals,
we
found
that
there
wasn't
a
lot
of
shared
technology
in
the
underlying
implementations
of
these
operating
systems,
so
we're
seeing
the
same
fragmentation,
we're
running
the
risk
of
repeating
unix
history
and
having
multiple
incompatible
operating
systems.
But
here's
the
thing.
What
we've
seen
with
kubernetes
is
that
we
could
define
common
standards
that
still
allow
innovation.
A
A
The
examples
we're
showing
here
are
in
yaml,
because
he
doesn't
care
it
could
be
in
tamil
or
json
as
long
as
it
can
be
serialized
into
the
cozy
poto
buff
message
you
can
use
it.
The
next
thing
that
cozy
defines
is
the
common
rpc
definitions.
If
you're
used
to
kubernetes
these
are
things
like
get
set
list
and
watch
and
others
cozy
also
defines
plug-in
standards,
and
these
are
inspired
by
kubernetes
projects
such
as
the
cni
and
the
csi.
A
The
next
step
in
cozy
is
defining
a
series
of
rpcs,
and
these
are
inspired
and
very
similar
to
kubernetes
rpcs.
You
can
manipulate
resources
by
listing
them
getting
them.
Creating
them
watching
them
and
cozy
tends
to
be
a
declarative
system.
In
terms
of
that,
you
set
the
desired
state
for
a
resource,
and
then
the
cozy
plugins
will
work
to
get
the
system
into
that
state.
A
For
communication,
cozy
uses
bi-directional
grpc
now
as
part
of
the
research
into
cozy.
We
have
investigated
other
internal
ipc
systems
for
linux,
but
for
simplicity
of
implementation
we
are
starting
with
grpc.
Another
important
thing
to
understand
about
cozy
is
that
plugins
are
designed
or
meant
to
be
as
close
to
the
underlying
operating
system
as
possible.
Cozy
is
not
a
configuration
management
system.
We
don't
want
to
wrap
other
commands
and
execute
shell
commands
and
parse
the
output.
A
Finally,
the
things
that
mutate,
the
work
and
do
change
the
system
are
plugins
and
plugins
are
very
similar,
they're
inspired
by
unix,
binaries
and
kubernetes
controllers.
Every
plugin
is
a
separate
executable
like
unix
utility,
but
it
runs
continuously
like
a
controller.
Now,
within
a
plug-in,
you
can
manage
multiple
resources.
So
when
you
bring
up
a
plug-in
it
registers
with
the
engines
what
types
of
resources
it's
responsible
for
managing
and
a
plug-in
internally
could
be
like
a
controller
manager.
It
can
manage
multiple
other
controllers
inside
that
manage
different
parts
of
resources.
A
The
cozy
engine
could
be
responsible
for
the
life
cycle
of
plug-ins,
so
what
you
can
have
is
that
when
cozy
starts
up,
it
can
start
all
the
plug-ins
and
if
a
plug-in
dies,
the
cosy
engine
can
restart
it.
Another
thing
to
know
about
plugins
is
instead
of
standard
output
and
standard
input.
They
take
inputs
and
outputs
over
the
grpc
channel
and
act
on
those.
A
A
special
class
of
plugins
generate
events,
and
these
are
called
generators
and
the
way
generators
are
implemented
in
cozy.
Is
they
use
linux
kernel,
ebpf
infrastructure,
so
a
generator
will
deploy
a
probe
into
the
kernel,
and
this
ebpf
probe
will
generate
kernel
events.
The
generator
is
then
responsible
for
taking
these
kernel
events
and
converting
these
to
cosy
events,
and
then
these
cozy
events
can
be
used
by
other
parts
of
the
system.
B
B
B
B
B
One
of
the
pillars
of
unix
is
the
idea
that
text
streams
are
the
universal
interface.
One
feature
that
this
pillar
is
missing
is
structure
with
cosy.
We
amend
the
third
pillar
of
unix
summarized
by
peter
salas
as
write
programs
to
handle
structured
text
streams,
because
that
is
the
universal
interface.