►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Superpowers for Windows Containers - Deep Debroy & Jean Rouge, Docker
The Windows Operating System does not support privileged operations from inside a container today. Daemon-sets on Windows nodes in Kubernetes clusters that need to perform configuration actions on the node are significantly impacted by the absence of privileged mode support on Windows. In this talk we: 1. Explore the pros and cons of the options the SIG Windows community brainstormed to provide containers running on Windows the ability to perform privileged operations while being managed by Kubernetes. 2. Delve into the specific characteristics of the privileged proxy approach we decided to adopt. 3. Demonstrate how the privileged proxy approach is used to support privileged operations that need to be executed by daemon-sets associated with CSI plugins running on Windows nodes in a Kubernetes cluster.
https://sched.co/Uadg
A
B
So
the
agenda
for
the
session
roughly,
we
are
going
to
go
over
a
very
quick
intro
to
the
history
of
support
for
continued
rise
workloads
in
kubernetes
for
Windows
and
then
go
over
the
concept
of
privileged
operations
and
the
support
for
that
in
Windows
containers,
which
pretty
much
does
not
exist
at
the
moment.
We'll
look
at
a
possible
workaround
that
the
community
has
sort
of
agreed
upon
which
is
based
on
this
concept
of
privileged
proxies.
We
will
look
at
the
specific
design
of
such
a
privileged
proxy
for
the
area
of
persistent
storage.
B
So
very
quick
introduction
to
the
history
of
Windows
containers,
Windows
Server
2016
was
the
first
LTS
release
when
initial
support
for
containers
made
it
to
Windows.
A
lot
of
enhancements
were
made
between
2016
and
2019
through
the
rs3
and
RS
4
releases
and,
along
the
same
time,
a
lot
of
folks
from
say
windows
had
been
constant,
has
been
concentrating
on
getting
the
cubelet
and
other
components
of
kubernetes
up
and
ready
for
Windows
nodes.
B
So,
finally,
around
kubernetes
114,
the
overall
kubernetes
community
and
Segway
nose
was
ready
to
declare
stable
G
as
support
for
Windows
workloads
and,
as
you
guys
probably
heard
in
the
session
right
before
introduction
to
Windows.
A
lot
of
work
is
upcoming
to
build
support
for
container
D
and
things
around
those
lines
in
1903
and
in
future
versions
of
kubernetes.
B
So
here's
a
very
quick
overview
of
how
and
a
very
high-level
overview
of
how
a
kubernetes
clusters
with
Windows
and
Linux
nodes
looks
like
the
control
plane
still
runs
completely
on
Linux
masters,
but
for
the
worker
nodes
you
can
have
a
mix
of
Linux
and
windows,
so
the
windows
nodes
are
going
to
run
the
pods
that
comprise
of
containers
whose
base
images
are
Windows
based
images.
They
would
be
running
your
Windows
workloads,
as
well
as
the
cube
led
to
proxy
and
plugins
necessary
for
storage
and
networking.
The
Linux
model
does
not
change
the
layin.
B
Eps
workers
run
pretty
much
the
same
set
of
stuff
now.
One
interesting
deficiency
or
limitation
for
the
container
support
in
Windows
is
there's
no
concept
of
capabilities.
This
is
something
you
might
be
used
to
in
the
Linux
world,
where
every
container
can
be
assigned
a
specific
set
of
capabilities,
all
the
way
up
to
something
called
privileged,
which
allows
a
container
to
have
full
access
to
the
host
and
perform
a
wide
variety
of
operations
on
the
host
that
otherwise
a
regular
container
cannot
do
in
case
of
Windows.
That
option
is
not
there.
B
So,
as
a
result,
a
lot
of
privileged
operations
cannot
be
easily
performed
by
Windows
containers.
Some
of
these
involve
accessing
and
managing
the
host
registry
managing
host
network
configuration
if
you
want
a
Windows
container
to
be
in
charge
of
networking
for
the
rest
of
the
other
containers
in
the
host,
it's
typically
a
case
for
CNI
plugins
and
the
ability
to
access
and
manage
storage
devices
such
as
disks
volumes
and
so
on
from
the
host.
This
allows
container,
encapsulating,
say
CSI,
node,
plug-in
logic
to
be
able
to
provision
storage
for
other
containers
running
on
the
host.
B
So,
given
this
lack
of
ability,
the
pretty
much
windows
containers
are
a
bit
handicapped
when
it
comes
to
acting
as
full-fledged
kubernetes
daemon
sets.
This
specifically
becomes
a
problem
when
you're
thinking
about
CNI
and
CSI
plugins,
so
the
community
started
thinking
about
that.
We
were
we,
we
discussed
it
quite
a
bit
in
sig
windows
and
storage
was
one
of
the
areas
both
John
and
I
are
interested
in.
B
So
we're
looking
at
looking
at
it
also
from
the
storage
perspective,
and
we
came
up
with
a
bunch
of
different
options
or
workarounds
for
this
lack
of
Rivlin
support.
One
is
just
allowing
a
container
remote
shell
access
into
the
host.
Obviously
this
sounds
very
dangerous,
but
it
could
be
achieved
over
something
like
say,
secure,
shell
or
PowerShell,
but
it's
very
challenging
to
constrain
the
it's
very
challenging
to
constrain
the
set
of
operations
that
a
container
can
perform
on
the
host.
It's
completely
unfettered
access,
so
this
was
too
radical
thing
and
the
community
decided
against
it.
B
Another
process
or
another
approach
that
was
thought
about
was
how
about
that
the
the
base
desire
was
the
ability
to
ship
out
logic
within
containers.
So
what
if
there
was
something
called
a
specialized
runtime
class
that
the
windows
runtime
surfaces
that
allows
a
container
image
to
come
down
into
the
windows
node,
but
the
payload
be
extracted
and
launched
as
a
process.
B
Now
this
could
be
achieved
through
extensible
container
runtime
likes
a
container
D
which
allows
you
to
write
specific
shims,
so
you
can
run
something
called
a
process
shim
that
needs
to
be
designed
that
allows
you
to
extract
a
payload
from
windows
image
and
run
it
directly
as
a
host
process,
with
all
the
privileges
of
say,
a
Windows
service.
However,
this
is
not
a
container,
and
this
again
allows
for
fairly
unfettered
access
into
the
host
and
was
kind
of
considered
a
bit
too
radical
to
start
off
with.
B
So
what
we
eventually
decided
on
is
this
mechanism
called
a
privileged
proxy,
which
is
basically
a
specific
logic.
That's
coded
into
a
binary
that
needs
to
be
run
on
the
host
and
a
bunch
of
different
containers
can
access
the
privileged
process
to
accomplish
the
set
Aubrey
that
they
need,
and
this
is
going
to
be
the
focus
of
this
presentation.
C
So
that
approach-
basically
that's
just
gonna-
be
a
row
binary,
that's
running
on
the
host
itself.
That's
gonna
run
with
admin
access,
and
so
it
can
do
whatever
it
wants
and
then
exposes
the
capabilities
that
it
has
through
name
pipes
so
that
members
can
get
can
then
be
mounted
in
containers
and
containers
can
have
privileged
access
to
the
hosts.
That
way,
so,
hopefully
I
mean
that
this
binary
needs
to
somehow
make
it
to
the
worker
and
somehow
be
run
so
there's
no
good
solution.
C
There's
no
standard
solution
for
that,
but
that's
something
that
admin
clusters
today
are
used
to
dealing
with
on
Windows
because
they
still
have
to
today.
They
still
have
to
make
sure
the
cube.
Ladies
running
the
cube
proxy
is
running
that
duck
ad
is
running,
so
we
don't
have
a
good
solution
for
that.
But
it's
not
it's
not
really
a
new
problem
either.
C
So
whatever
your
dispro
uses
to
get
the
cube
left
on
the
node
and
running
you
can
use
for
the
for
a
proxy
as
well
and
the
good
thing
one
of
the
good
things
about
this
design
is
that
the
supported
operations
by
the
proxy
can
beep.
So
it
can
be
scoped
to
your
OS
subsystems,
and
so
it's
it's
really
easy
for
the
proxy
to
enforce
constraints
and
restriction
as
to
who
has
the
right
to
do
what,
as
opposed
to
what
deep,
with
we're
talking
about
before
having
a
remote
access
to
the
host.
C
C
So
this
this
manifest
file
sure
how
easy
would
be
to
use
that
to
have
privileged
containers
to
do
what
you
need
to
do
and
use
that
for,
for
example,
CSIS
see
and
I
plugins
the
same
it
used
to
Linux
doing
sir
considerations
for
deploying
this
proxy.
That's
what
one
of
the
things
we
brought
down
before
yeah
that
binary
needs
to
be
somehow
run
manage,
not
any
problem
up
to
you.
But
one
of
the
good
things
about
this
proxy
is
its.
It's
really
easy
to
have
control
and
who
does
what?
C
And
one
of
the
way
to
do
that
would
be
using
pod
security
policies
and
when
admission
can
run
web
hooks
to
decide
which
parts
and
which
namespaces
I
actually
have
pretty
little
access
to
the
hosts.
So
let's
look
at
what
that
would
look
like
in
practice.
So
using
port
security
policies,
you
could
have.
C
All
that
youth,
that
pot
security
policy
and
we
bind
that
cluster
role
to
everyone
so
by
default
once
you
do
that,
no
one
can
use
volume
ants
in
your
cluster
anymore,
and
so
that
way
people
can't
use
the
name
pipes
that
the
proxy
exposes
for
the
pods
and
the
services
that
you
do.
You
want
to
have
using
a
proxy.
C
Then
you
create
a
new
signal,
new
pod
security
policy,
which
has
the
house-passed
capability
in
the
volume
mount
you
create
a
cluster
role
to
use
that
security
policy,
and
then
you
bind
that
that
cluster
role
to
whichever
namespace
and
service
accounts
you
want
to.
You
know
to
use
the
proxy
another,
really
big
benefit
of
the
of
the
proxy.
C
The
whole
philosophy
of
the
proxy
idea
is
to
really
expose
the
Royal
cap,
the
most
low
level
possible
capabilities
of
the
us
through
the
proxy,
so
that
we
hope
there
is
that
the
API
should
be
fairly
stable
and
and
then
plug
in
vendors.
Flagging
authors
can
rely
on
somewhat
reliable
and
somewhat
stable.
Api
is
to
have
their
own
relay
cycles.
C
So
we
can
have
like
really
decouple
life
cycles
for
containerize,
plugins
and
the
proxy
itself,
and
so
once
you
have
that
you
can
use
the
proxy
to
to
develop
containerize
plugins
and
do
the
same
as
people
are
used
to
on
the
next
two
days.
So
having
CNI
and
then
CSI
plugins
come
in
as
cube
workloads,
demon,
says
and
whatnot
and
deep
is
gonna.
Go
into
how
we
adapted
this
proxy
idea
to
do
your
storage
thanks.
B
Shawn,
so
basically,
the
first
use
case
that
we
looked
at
was
storage,
CSI
was
becoming
really
popular
and
just
to
give
you
a
quick
introduction
to
how
storage
support
personal
storage
support
works
for
Windows
today
is
it's
based
on
two
models
that
are
being
deprecated
pretty
soon.
One
is
the
entry
plan
model.
The
community
does
not
want
specific
cloud
provider,
logic
or
logic
from
specific
storage
vendors
within
the
kubernetes
codebase,
so
they
want
to
get
rid
of
entry
plugins
from
inside
kubernetes
kubernetes
into
external
plugins
that
are
in
a
completely
separate
codebase.
B
So
the
entry
plugins
that
windows
depended
on
are
kind
of
on
a
path
towards
deprecation.
The
other
model
is
using
flex
volume,
plug-in
scripts,
but
flex
volume
itself
is
also
considered
pretty
much
deprecated
for
linux,
it's
kind
of
like
one
of
the
V's
out
for
Windows,
so
it's
not
completely
removed,
but
it's
there,
but
it's
also
considered
essentially
deprecated.
In
both
of
these
approaches.
B
You
notice
that
the
cubelet
based
approach
for
entry,
plugins
and
the
Flex
volume
there
is
direct
access
to
the
disk
devices
and
the
volume
devices
and
the
host,
because
everything
is
running
directly
on
the
host.
Nothing
has
within
a
container
so
just
to
kind
of
put
it
in
a
diagram.
You
know
cubelet
and
the
Flex
volume
scripts
they're
running
directly
on
the
host.
They
have
direct
access
to
the
device
handles
for
the
disks.
B
There
are
full
capabilities
to
mount
the
SMB
share
remotely
and
then
surface
them
up
to
Windows
pod
or
the
containers
within
the
Windows
Phone.
However,
the
future
is
based
around
the
container
storage
interface,
which
is
CSI
now
CSI
is
a
very
detailed
topic,
so
let's
focus
mainly
on
the
aspect
of
CSI.
Our
concerns
Windows,
there's
a
concept
of
CSI
node
plugins,
where
the
node
plugins
implement
the
logic
to
identify,
say
a
LUN
that
get
service
to
a
node
based
on
the
LAN.
B
Once
it
has
identify
that
it's
a
land
that
belongs
to
the
driver
that
it
implements
it
can
you
know,
partition
at
the
weight,
ones,
format,
the
file
system
and
service
it
up
to
containers.
So
the
node
plugins
essentially
implement
the
modern
CSI
spec,
which
hid
around
113
typically
for
linux.
Today,
the
CSI
node
plugins
are
distributed
as
containerized
binaries
and
they
are
deployed
through
privileged
pods
in
Linux,
so
for
windows.
The
desires
to
have
a
very
similar
mechanism-
and
this
fits
very
nicely
with
the
privileged
proxy
concept.
B
We
won
the
CSI
node
plugins
to
do
a
whole
bunch
of
privilege.
Operations
as
we
went
over
and
the
idea
with
the
CSI
proxy
is
to
have
a
privileged
proxy
that
surfaces
a
bunch
of
API
through
name
pipes
up
to
the
CSI
node
plug-in
containers
so
that
they
can
do
what,
if
whatever
it
is,
that
the
desire
to
do,
for
example,
scan
the
physical
disk
objects
based
on
their
scuzz
scuzz,
EPG,
t3
or
PG
TR.
B
So,
as
you
see
the
model
changes
from
cubelet
and
a
bunch
of
flex
volume
scripts
having
direct
access
to
the
devices
in
the
host
to
actually
a
CSI
node
pod,
which
is
essentially
comprising
of
a
couple
of
containers,
the
CSI
no
driver
registrar,
which
is
used
for
registering
a
plug-in
red
CSI
and
the
CSI
node
plug-in
core
logic.
That's
shipped
by
a
vendor
all
wrapped
up
within
a
windows
within
a
couple
of
windows,
containers
and
shipped
as
part
of
daemon
set
specification.
B
The
the
the
CSI
node
part
of
will
mount
the
name
pipes
exposed
by
the
CSI
proxy
dot
exe
and
through
the
name
pipe
it
can
make.
Api
calls
down
to
the
proxy
to
get
access
to
the
devices
from
the
host
and,
finally,
once
it
has
formatted
the
volumes
or
identify
the
volumes
that
needs
to
surface
up.
It
can
just
create
the
necessary
same
links
in
the
file
system
of
the
windows
pods
to
make
the
volumes
available
in
the
containers.
C
So
one
problem
we
that
was
that
surfaced
when
we
submitted
a
cap
for
the
proxy
for
CSI
what
people
wanted
to
have
a
mechanism
that
would
make
it
easy
for
the
the
proxy
to
evolve
to
add
new
capabilities
and
and
modify
existing
ones.
That
would
be
easy
for
developers
working
the
proxy
and
yet
ensure
stability
for
for,
like
in
authors
and
reduce
the
API.
So
one
one
way
we
came
up
to
come
up
with
some
of
that
problem
is
just
reusing.
C
The
same
same
thing
cube
itself
does
on
its
API,
so
we
recycle
the
notion
of
API
group
and
API
versions.
So
the
capabilities,
the
proxy
exposes
our
group
by
API
groups.
So
typically,
this
is
an
API
group
for
discont
volume
operations,
there's
another
one
for
ice,
CAHSEE
operations
and
so
on
and
so
forth
and
H
API
group
can
have
in
a
given
version
of
CC
a
proxy
for
each
API
group.
You're
gonna
have
one
or
several
versions
of
the
API
that
are
running,
and
then
the
API
versions
are
maintained
and
deprecated
according
to
a
release.
C
Schedule,
that's
also
pretty
similar
to
what
keep
does
internally,
though,
for
maintenance
of
the
proxy.
You
don't
want
people
to
have
to
duplicate
a
bunch
of
code
when
they
create
a
new
API
version.
You
want
to
be
able
to
reuse
most
of
it,
because
it's
just
gonna
change
something
for
your
new
version:
I,
didn't
it
capability
or
changing
existing
behavior.
C
So
internally
for
developers,
you
want
to
have
a
single
internal
representation
of
what
the
version
API
is
use,
so
you
can
have
requests
and
response
types
for
each
API
endpoint
and
each
version
might
have
slightly
different
semantics
for
each
of
these.
But
internally
you
can
have
one
only
one
presentation
for
each
API
group
for
all
these
types
and
every
single
internal
representation
should
be
able
to
contain
all
the
informations,
the
old
information
containing
all
the
virgin
versions
of
the
same
type.
C
C
If
you
want
to
add
a
new
version,
you
need
to
create
a
profile
that
defend
that
defines
what
your
API
version
looks
like
and
then
you
need
to
add
the
right
types
to
to
the
internal
types
that
go
and,
of
course
you
need
to
write
the
the
server's
logic
for
it.
So
how
do
you
take
a
request
and
turn
into
a
response
and
all
the
rest
is
it
generated?
C
So
the
idea
here
is
to
make
it
easy
for
developers
to
contribute
to
the
project.
Of
course,
conversion
auto-generated
conversion
only
only
works
if
you
have
one-to-one
mapping
from
the
version
types
to
the
entire
presentation.
So
that's
where
that's
where
you
might
have
to
do
some
manual
code
to
to
say:
ok,
like
this
version
type,
that's
how
we
turn
it
into
the
internal
presentation,
but
hopefully
the
goal
here
that
you
only
have
to
do
that
when
it's
not
trivial.
C
To
deploy
proxy
versions
to
cluster,
that's
what
plus
your
admin
instant,
a
cluster
admins
need
to
know.
So,
basically,
when
you
mr.
Windows
cluster-
and
you
want
to
use
the
CSI
proxy,
you
need
to
understand
which
API
group
inversions
matter
to
the
plugins
you
want
to
use,
and
you
want
to
ensure
that
the
version
of
the
CSI
proxy
running
actually
caters
to
the
to
the
plugins
that
you
care
about.
C
So
hopefully,
plugins
should
stay
roughly
in
sync,
with
the
latest
version
or
see
a
proxy
if
they
don't
and
you
need
to
run
a
very
old
API
group
inversion.
That's
possible!
It's
possible
to
run
several
versions
of
the
CSI
proxy
at
the
same
time
to
be
able
to
support
everything
you
need
and
that's
a
deep
is
going
to
do
a
little
DEMA
for
us.
B
All
right
so
here
we
have
a
Windows
node,
that's
running
our
CSI
proxy
or
Exe.
Let's
get
this
started
and
what
you
see
here
in
the
good
old
disk
management
UI
is
freshly
formatted
VHD
that
we
would
like
to
create
some
partitions
in
from
inside
a
container.
This
is
something
you
cannot
do
from
a
regular
Windows
container,
but
what
we
did
is
we
created
a
container
called
partition
demo.
B
B
B
Server
that
so
yeah
we
ran
the
container
over.
Here
we
are
running
the
CSI
proxy
and
what
we
did
is
we
had
disk
number
two,
which
was
there
as
a
unpartitioned
disk
initially
and
from
inside
the
container.
It
mounted
the
v1
alpha
one
pipe
with
the
disk
API
set
and
invoked
the
necessary
commands
to
first
check
whether
disk
is
initialized.
If
it
was
not,
it
initialized
it
with,
say
the
MBR
partition
format
or
you
could
do
a
GPT
and
then
check.
B
If
the
disk
was
partitioned
and
says
it
was
not,
it
created
partitions
on
a
created
or
raw
partition
on
this.
So
again,
the
gist
of
this
is
this
could
not
have
been
possible
from
a
regular
container,
but
because
it's
mounting
the
name
pipe
exposed
by
CSI
proxy.
It
talked
over
an
API
with
CSI
proxy
to
make
these
operations
happen.
B
So
storage
is
not
the
only
use
case
for
the
first
use
case,
we
decided
to
look
at
the
same
model
could
be
applied
to
networking.
Cni
plugins
are
an
obvious
candidate
for
the
same
model.
If
there
are
folks
interested
in
working
on
community
managed
proxy
for
H
and
s
API
calls
that
would
be
great
to
have
to
kind
of
get
the
community
moving
in
that
direction.
B
There
are
various
avenues
through
which
you
can
get
involved
if
you're
interested
in
contributing
to
the
CSI
proxy
project.
Here
are
some
of
the
details.
Overall,
you
can
just
stop
by
sig
windows
and
you
know
talk
about
an
idea
and
we
can
help
you
get
maybe
kept
started.
That's
what
we
did
with
CSI
proxy
kind
of
got
it
reviewed
with
architecture,
security
and
storage
besides
Windows
itself,
so
we
can
do
the
same.
For
you
know
other
areas
like
networking,
diagnostics
and
so
forth.
There
are
the
slack
channels,
and
especially
around
CSI
windows.