►
From YouTube: Kubernetes SIG Auth 2022-01-11 KMS #3
Description
Kubernetes Auth Special-Interest-Group (SIG) Meeting 2022-01-11 KMS #3
Meeting Notes/Agenda: https://docs.google.com/document/d/1woLGRoONE3EBVx-wTb4pvp4CI7tmLZ6lS26VTbosLKM/preview
Find out more about SIG Auth here: https://github.com/kubernetes/community/tree/master/sig-auth
A
All
right
so
hey
everyone.
This
is
chemist
meeting
number
three
on
january,
11th
2022,
so
for
this
meeting
we
had
wanted
to
go
over
for
what
the
tracks
work
are
for
ms
and
sort
of
like
what
we
plan
on
doing.
A
So
yesterday
we
had
talked
after
the
issue
triage
meeting
briefly
and
some
of
the
things
we
discussed
there
were
that
we
wanted
well.
We
had
talked
about
these
three
distinct
things.
We
talked
about
the
reference
mms
library.
A
A
Yeah,
so
that
covers
the
the
three
things
rio.
Do
you
mind
maybe
going
over
them
for
damage.
D
Yeah,
maybe
just
before
like.
Could
you
give
me
an
idea
of
the
state
currently
of
the
cab
like
because
we
kind
of
wanted
to
split
it
into
different
topics,
so
observability
performance
and
things
like
that,
and
I
know
we
talked
a
bit
about
performance
in
the
first
place
and
a
lot
about
observability,
but
not
yet
about
recovery.
I
guess
you
give
me
a
heads
up
of
where
we
are
today.
B
Yeah,
so
the
first
meeting
we
talked
about
are
the
different
categories
right
so
recoverability,
observability
and
performance
and
then
also
like.
If
you
looked
at
the
initial
proposal
draft,
we
had
said
we'll
focus
on
observability
and
recoverability
first,
because
those
are
things
that
we
could
get
in
without
any
breaking
changes
and
also
based
on
some
of
the
discussions
that
we've
already
had.
So
we
decided
to
go
with
observability
first,
so
open
a
cap
for
it
and
then
see
what
changes
that
we
can
include
in
the
next
coming
release.
B
So
the
kept
deadlines,
I
think,
john
28.
So
we
want
to
get
kept
in
before
that
and
then
try
to
make
changes
in
this
release
and
the
conversation
for
this.
What
we
thought
was,
maybe
we
could
do
the
audit
id
part
right
so
where
we
can
actually
string
the
whole
audit
id
from
the
user,
creating
the
request
all
the
way
to
the
kms
plugin.
So
in
the
last
meeting
we
decided
that
that's
something
we
could
do,
but
there
were
a
few
outstanding
questions
that
we
had
for
the
cigar
community.
B
One
was
audit,
id
is
user
configurable,
so
the
user
can
actually
overwrite
it
with
whatever
audit
id
they
need.
So
we
needed
to
understand
why
that
is
required,
and
then
we
also
wanted
to
see
if
we
could
get
a
sign
off
from
everyone
on
cigar
and
the
outcome
from
the
call
was
yes.
Audit
id
is
configurable.
We
need
to
understand
more
on
that
behavior.
B
So
that's
why
there's
a
track
item
right
now,
so
that
we
can
understand
why
it's
configurable
and
what
we
need
to
do
and
then
the
second
thing
is
every
request
that
not
all
the
decrypter
encrypt
decrypt
request
comes
to
the
kms
plugin,
because
the
api
server
has
caching.
So
basically
it
caches
the
deck
and
the
encrypted
version
of
the
deck
in
it
in
memory,
and
because
of
that,
there
is
no
guarantee
that
every
request
that
originates
from
the
user
will
actually
hit
the
kms
plugin.
B
So
we
decided
that
there's
no
way
we
can
provide
the
direct
audit
trail
from
the
user
request,
all
the
way
to
kms
plugin.
So
we
wanted
to
focus
on
something
where
we
can
correlate
requests
from
api
server
to
kms
plugin.
So
at
least
that
enables
debugging
easier
than
what
it
is
today
so
today,
if
you
need
to
correlate
the
only
way
you
can
do
that
is
through
timestamps,
you
have
to
look
at
a
timestamp
in
kms
and
then
you
have
to
go
back
and
look
at
timestamping
the
api
server.
B
But
so,
instead
of
that,
we
were
going
through
a
couple
of
options,
and
one
thing
we
decided
to
do
now
was
the
api
server
will
generate
a
unique
id
for
every
request
that
it
sends
the
kms
plugin.
This
will
be
part
of
the
grpc
request,
that's
being
sent
to
the
kms,
and
then
the
api
server
and
kms
plugin
can
log
using
that
uid
and
then
they
can
also
use
that
for
auditing
to
start
off
and
then
in
the
track.
B
Two
that
we're
following
up
with
cigar,
to
understand
why
audit
id
is,
in
the
end
users
control.
So
if
we
can
change
that
to
make
it
less
configurable,
then
maybe
we
can
also
hook
up
that
audit
id
with
the
uid
that
we
are
generating.
So
that's
where
we
are
now
and
the
next
steps
is
we're
going
to
write
a
cap
for
this
part
of
it
and
then
try
to
get
that
in
and
then
start
the
work
on
this
release.
B
D
Yes,
thanks
for
the
summary
I
think
it
covers.
Like
most
of
the
thing
I
was
wondering
the
last
meeting
I
went
to.
Basically
we
had
this
discussion
of.
How
can
we
propagate
the
audit
information,
and
I
think
you
had
two
ideas
behind
that
first
was
to
have
one
id
that
would
be
generated
specifically
for
kms,
like
some
specific
field,
just
for
us
or
use
a
diode
id.
So
I
guess
the
the
approach
you
are
trying
to
take
now
is
to
switch
to
edit
id
with
the
audit
id
chain.
B
Yes,
so
right
now
we're
starting,
which
is
the
uid,
but
I
think
once
we
understand
the
track
two
for
the
audit
id
and
why
it's
user
configurable,
then
we
can
actually
change
it
to
like
once
we
figure
out
that
part
and
then
fix
what
is
it?
What
we
have
can
just
be
chained
toward
the
audit
id,
so
we
can
actually
also
show
that
to
the
users.
D
Okay,
and
so
the
goal
today
is
to
like
kind
of
answer,
the
question
that
the
seagull
people
had
and
the
concern
that
they
had
right.
B
So
at
least
the
start
of
this
call
is
so
we
have
these
three
track
items.
One
is
a
reference
to
mms
library
and
then
the
observability
kept
and
then
also
figuring
out
the
audit
id.
So
we
had
a
small
discussion
yesterday
after
our
triage
call
just
to
understand
these
things
and
then
who
will
pick
it
up?
A
I
wanted
to
give
an
update,
so
I
did
speak
with
tim
yesterday
in
regards
to
the
audit
id
thing
and
what
I
got
from
him
was
so
there
appears
to
have
been
two
use
cases
that
were
thought
of
for
the
audit
id
being
under
control.
One
of
them
was
something
related
to
keep
ttl
and
it
following
redirects
and
like
maintaining
knotted
id
that
it
received
originally
from
the
server
and
continuing
to
pass
it
through.
A
I
think
that
use
case
fell
through
because
they
just
decided
to
ctl
should
never
follow
a
redirect.
It's
just
a
bad
idea.
It
shouldn't,
because
the
server
doesn't
redirect
you
really
in
any
situation
other
than
very,
I
think
I
think,
there's
technically
redirects
in
some
of
the
exact
and
proxy
things,
but
they
led
to
problems
and
other
things.
So
that
wasn't,
I
don't
think
that's
the
case.
We
need
to
worry
about.
The
other
case
is
aggregated
api
servers,
which
we
also
are
not
concerned
about,
because
we
have
an
mtls
connection
to
them.
So
we
can.
A
We
can
use
that
as
our
trust
anchor,
but
the
gist
I
got
from
tim
is
that
we
we
could
consider
ignoring
a
use
end
user
provided
audit
id
when
it's
untrusted
like
we
could
make
such
a
change
right,
the.
So
what
what
I
had
at
what
I
had
thought
that
we
could
do
as
a
next
step
there
is
reach
out
on.
You
know
like
kdev
security
and
take
off
mailing
lists
and
be
like
hey.
Does
anyone
rely
on
this
being
under
your
control?
If
so
like?
Why?
What
are
you
trying
to
do
and
etc?
A
Right
then,
with
that
in
mind,
once
we
get
a
little
bit
of
feedback,
we
could
try
to
write
a
kept
to
describe
the
change
we
want
to
make
right
like
so.
The
the
end
state
would
be
only
trusted
actors
can
set
this
so
aggregated
api
servers
can
or
front
proxies
in
front
of
api
servers
can
set
this
totally
fine.
You
know
they're
they're
root
on
the
cluster.
That's
totally
fine!
A
You
know
you
could
even
imagine
a
authenticated
user
can
set
this
header
if
they
have
some
permission
right
if
they
have
some
some,
if
they
pass
some
subject,
access
review
as
a
for
example,
that
could
totally
be
a
thing
also
right-
and
you
know,
there's
a
variety
of
ways
to
go
about
this.
I
don't
want
to
speculate
too
much
on
that
because,
as
far
as
I
know,
no
one
uses
this
for
anything,
but
that
you
know
this
is
kubernetes
and
everyone
comes
up
with
horrible
things.
A
They
want
to
do
so
that
that
was
my
thought
there
that
we
continue
like
you
know.
We
try
to
reach
out
to
our
community,
see
if
anyone
would
be
impacted
by
a
change
and
based
on
that
feedback.
It
would
tell
us
sort
of
the
scope
of
the
work
for
the
cap.
D
A
A
A
But
you
know
we
don't
want
to
break
anyone
if
it
is
valuable
to
them
right,
because
you
know
at
a
bare
minimum,
any
change
would
need
to
support
aggregated
api
servers
right
because
aggregated
api
servers
should
be
able
to
know
that
the
real
api
server
is
telling
them
here's
the
audio
id
for
this
request
keep
keep
flowing
this
through
your
stack
right.
That
way,
because,
from
the
user's
perspective,
it
is
one
request
it
doesn't
matter,
that's
passing
through
seven
layers
of
proxy
or
whatever
so
yeah.
That's
the
general.
Just
there.
A
E
Yeah,
so,
oh
sorry,
one
question,
so
what
would
happen
when
we
find
because
probability
is,
is
quite
high
that
someone
on
the
world
uses
the
the
ordered
id?
What
would
be
the
kind
of
the
triage
if
it's
not
now?
I
think
that
we
should
not
touch
because
it
breaks
too
much
of
the
infrastructure
in
the
world
or
where
we
say:
okay,
it's
fine,
you
can
deal
with
it
and,
if
not
so,
would
be
there
an
option
to
maybe
add
a
new
header
to
the
request,
like
a
trusted
ordered
id.
A
So
you
know
say
say:
this
is
widely
depended
on
and
it's
really
problematic
to
change
right.
You
know
we
could
we
could.
We
could
add
a
new
flag
to
the
api
server.
A
So
it
knows
if
it
can
trust
the
audit
id,
because
what
we
had
wanted
to
do
is
basically
we
would
go
ahead
and
start
with
a
nice
uid
and
work
on
the
kms
case
and
get
that
the
ability
to
you
know
tie
a
kms
failure
on
the
api
server
to
kms
actual
operations.
We
want
to
get
that
in
and
you
know
in
parallel,
we
work
on
this
audit
id
work
and
if
we
are
successful
in
getting
that
into
a
better
state,
we
can
just
go
ahead
and
optimistically
change
the
audit
id
sorry.
A
The
uid
within
that
kms
call
to
be
the
audit
id
right
that
way
in
the
future.
Without
breaking
any
api
thing,
we
just
give
you
more
information
right,
so,
instead
of
it
just
being
a
randomly
generated
uid,
it
actually
is
the
kubernetes
audio
id.
So
you
in
those
cases,
you
can
actually
tie
it
back
to
the
kubernetes
out
of
the
event
right,
but
it
doesn't.
A
You
know
from
you
know,
a
we're,
not
saying
that
you
know
any
of
these
uids
have
any
schema
or
necessarily
any
meaning
today,
right
we're
just
saying
that
you
can
use
it
for
tracing
right
and
if
we
get
to
a
state
where
we
can
give
you
more,
which
is
yeah.
If
you
you
know,
if
you
happen
to
have
kubernetes
audited,
events
enabled-
and
this
particular
request
did
generate
an
audit
event.
A
You
can
use
the
uid
to
tie
them
together.
That
would
be
nice,
but
that
that
was
sort
of
like
the
final
end
state
that
we
had
kind
of
hoped
for.
But
we
don't
know
right.
We
don't
know
if
we'll
be
able
to
make
the
audit
id
get
into
a
state
where
it
can
be
trusted,
and
if
we
can't
that's
okay,
we
still
want
to
do.
E
Would
it
be
possible
to
kind
of
say,
okay,
no
matter
what
we
just
proceed
with
the
flag,
because
we
would
leave
it
up
to
the
admins
if
they,
if
they
have
some
special,
weird
setup,
where
they
think
they
can
trust
the
client
and
for
the
other,
people
who
are
not
trusting
their
clients
could
just
proceed
with
the
flag
and
then
have
an
implementation
for
for
this?
No
matter
what
and
really
kind
of
push
the
decision
back
to
the
users
yeah.
So
like.
A
A
I
I
would
like
to
push
hard
on
the
ability
for
kubernetes
to
improve
without
end
user
interaction
right,
because
what
that
would
end
up
being
is
no
one
would
be
able
to
set
this
to
true
without
possibly
breaking
their
users
or
customers,
so
they
almost
likely
keep
it
at
false
and
then
maybe
make
some
configuration
in
their.
You
know
infrastructure
right
like
so
you
know
I'm
imagining
gk
or
aks,
adding
more
knobs
in
their
ui
where
you're
like
no,
I
definitely
do
want.
A
E
And
if
you
would
do
the
inverted
way
that
we
that
you
need
to
set
up
in
a
flag
in
order
to
to
make
clients
be
able
to
set
the
audit
id,
then
people
who
so,
if
they're,
not
many
people
or
companies
who
are
using
this
alternative
from
the
client's
starting
point.
You
could
maybe
say:
okay,
we
just
you
need
to
add
this
flag
to
keep
the
previous
functionality.
A
Yeah
and
that
such
you
know,
such
an
approach
is
fine.
It's
more
with
that
approach.
We
would
have
to
have
a
long
stage
rollout
where
the
flag
basically
starts
off,
not
changing
the
behavior,
then,
with
some
amount
of
buffer.
You
know
we
say
cool
the
behavior
is
about
to
change.
You
need
to
set
it
to
whatever
you
want,
and
then
it
changes
and
so
forth
that
you
know
that
that
is
a
choice.
No,
that
was
not
a
bad
approach.
A
You
know
my
personal
desire
would
be
to
see
if
we
can
make
it
so
that
we
just
sort
of
force
the
ecosystem
into
like
no,
you,
you
have
to
make
this
id
trustable.
You
cannot
just
arbitrarily,
allow
it
right
and
only
if
you
get
strong
push
back
in
the
other
direction,
saying
no
like
here's.
My
absurd
use
case
for
why
I
want
to
set
this
id
from
actors
that
are
completely
untrusted
and
have
no
like
attached
to
authentication
or
something
right.
D
If
I
can
chime
in
and
bring
an
idea
to
the
discussion,
so
we
are
like
what
we
are
trying
to
achieve
is
basically
to
have
this
trusted,
and
the
example
that
you
had
with
aggregated
api
and
the
api
server
means
that
if
we
want
that
site
to
be
trusted,
we'll
need
some
kind
of
authorization
to
kind
of
say
that
you're
changing
the
id
between
an
aggregated
api
and
the
api
server
is
valid
and
should
be
done.
It's
something
that
is
trusted.
D
And
if
we
introduce
this
kind
of
trusting
chain
like
this
kind
of
authorization,
why
not
make
it
even
more
generic
and
say
that,
instead
of
having
just
that
piece
of
code
in
the
api
server,
have
it
really
generic
and
allow
users
to
authorize
their
own
application
to
change
the
audit
id
as
per
needed,
like
as
if
it
was,
for
example,
on
their
back
rule,
and
they
would
allow
the
application
to
change
this
particular
value
as
if
it
was
basically
a
negated
api
that
is
talking
to
the
abs
server
like
building?
D
A
Right
whenever
you
have
to
enable
something
you
broke,
somebody
right
that
is
considered
yeah
right,
so
it
can.
It
can't
be
in
that
form,
without
a
giant
rollout
phase
of
some
sort.
So
the
the
one
thing
I'm
trying
to
avoid
here
is
too
much
change
to
aggregated
api
servers,
so
the
front
proxy
mechanism
that
they
use
isn't
really
an
authorization
mechanism.
It's
an
authentication
mechanism
which
is
specifically
that
this
actor
is
the
front
proxy
and
can
set
any
user
information
on
the
incoming
request
right,
so
it
because
it
can
set
any
user
information.
A
It
is
by
definition,
root
because
it
can
just
ask
to
be
rude
if
it
wants
to
right,
you
can
always
do
that,
so
that
that
path
is
kind
of
well
defined
already,
but
it's
sort
of
outside
of
really
both
authentication
and
authorization
right.
It's
basically,
this
actor
is
trusted
right,
so
that
that
mechanism
kind
of
exists
in
this
fine.
If
we
wanted
to
make
it
generic
it's
a
little
bit
harder
because
that
that
tunnel
isn't
really
authenticated
right.
A
It's
it's
like
fully
trusted,
and
you
know
the
thing
doing
that
work
is
the
front
proxy,
but
from
the
kubernetes
api
perspective,
it's
the
original
authenticated
user
that
flows
through
that
tunnel
through
those
headers.
So
it
gets
a
little
murky
right
like
you
can't
like
it's,
not
that
user
you're
authorizing
it's
the
fact
that
it
was
the
front
proxy
that
told
you
that
you're
authorizing
so
it
gets
a
little
weird.
I
mean
we
could
totally
make
all
this
work
the
proxy
itself,
but
then
yeah,
but
it
doesn't
have
an
identity.
D
A
But
you
know
we
don't
necessarily
need
to
get
too
far
into
this
one
particular
thing.
What
I
wanted
to
make
sure
is
so
I
had
you
know
I
had
volunteered
for
the
audit
id
thing.
You
know
I
reached
out
to
tim
and
then
you
know
I
can
continue
doing
more
iv
stuff,
but.
C
So
sorry,
good
question:
remember:
I
think
there
was
a
concern
about
how
like,
during
bootstrap
time,
the
the
api
server
wouldn't
be
generating
audit
id,
because
it's
not
being
written
to
audit
logs.
So
in
those
cases
we
can't
use
the
audit
id
at
all.
A
Yeah,
so
it
I,
I
think,
no
matter
what
the
use
of
an
audit
id.
Let's
just
pretend
audit
id
was
always
trusted,
no
matter
what
it's
it's
an
optimistic
use,
it
has
to
be
so
both
for
internal
calls
for
the
api
server
as
it's
filling
its
cache.
A
You
know
those
don't
really
have
audio
ids
associated
with
them.
I
think,
but
more.
C
A
C
I
guess
I
I
I
I'm
all
for
the
improvement
of
a
gen
who
gets
to
generate
audit
id.
I
just
want
to
make
sure
it's
a
separate
effort
from
like
making
kms
observability
better
right,
because
I
feel,
like
the
scope,
is
increasing
quite
a
bit
here,
just
because
I
think
having
the
uid
generated
is
already
serving
the
purpose
that
we
came
here
for
and
be
also
because
the
improvement,
the
kms
improvement
doc,
has
a
bunch
of
other
capabilities
that
I
think
a
lot
of
people
really
want
to
see
specifically
around
the
performance
aspect.
C
I
so
I'm
questioning.
Are
we
using
our
time
correctly
here?
If
the
uid
serves
a
purpose
that
tell
us
hey,
you
requested
the
encryption
decryption
hey,
you
actually
did
the
encryption
decryption.
I
can
correlate
the
the
request,
I'm
good.
So
then
can
we
again
I'm
wondering
if
our
scope
is
increasing
a
bit
beyond
kms.
At
this
point,.
A
Yeah,
that's
totally
fair,
that's
totally
fair!
So
I
I
guess
the
way
I
had
framed
it
in
my
head,
the
the
like
the
cap.
That
initial
is
you
know,
working
on
writing
and
stuff
at
this.
Current
stage
does
not
need
to
say
anything
about
audio
id
like
it.
Just
it
just
doesn't
need
to
care
about
it.
It's
just
just
say
I'm
going
to
do
uid
and
that's
what
I'm
doing
yep.
C
A
A
I
mean
I
didn't
I
didn't
I
didn't
know
if
there
was
you
know
thinking
through
this.
I
don't
know
of
other
tracks
of
work
related
to
kms.
That
can
be
done
in
parallel
with
the
observability
stuff
and
the
reference
implementation.
If
they
are,
then
we
should
talk
about
those
in
there
and
you
know
prioritize
them.
I,
I
think
you're
just
asking
a
priority
question
right,
like
yeah.
C
Yeah
yeah
yeah
yeah,
I'm
not
saying
we
should
drop
anything,
I'm
just
saying,
yeah
and
also
january
20th,
right
we're
on
the
11th.
So
how
do
we
get
the
cup
like
actually
open
and
reviewed,
and
then
you
know
and
the
performance
one
there's
a
lot
of
opinions
there.
So
I
I
want
to
make
sure
we
have
our
time
spent
on
the
things
that
that
that
really
make
the
biggest
impact
today.
A
Yeah,
so
I
mean
within
the
next
week
right.
The
most
critical
thing
is
the
cap
right.
Everything
else
can
work
and-
and
it's
totally
fine
if
it
weighs
right-
it's
not
really
a
big
deal
right.
C
Yeah
and
the
uid
is
it
works
and
it
it's
not
something
that
causes
backward
compatibility
issues.
So
I
I
don't
I
I
guess
I
I
don't
feel
like
we're.
Gonna
get
a
lot
of
pushbacks
right
to
get
that
kept.
Merge,
yeah.
A
I
mean
I
I
the
only
real
pushback
I
have
on
the
uid.
Is
I'm
unclear?
It's
not
even
pushback.
It's
just
unclear
to
me
exactly
how
we
guarantee
that
the
uid
gets
logged
like
always
to
make
sure
that,
when
there
are
failures
that
it
is
guaranteed
to
be
there
in
some
way,
that's
it.
That's
literally
my
only
question.
I'm
sure
this
is
totally
technically
possible.
We
could
totally
make
sure
that
this
always
occurs.
A
Otherwise,
I
don't
have
any
concerns
from
it
seems
totally
fine,
it
seems
to
serve
the
purpose.
It
seems
to
be
a
very
small.
Iterative
change
seems
totally
fine.
C
Okay,
so,
let's
make
sure
that's
addressed
in
the
cap,
you
know
how
do
we
make
sure
the
uid
is
written
to
the
autolog.
A
C
A
Yeah,
so
that
that's
the
so
that's
the
thing
we
can
focus
on,
so
I
don't
know
like
when
an
issue
you
plan
on
trying
to
have
the
kept
open.
You
know,
certainly
you
know
we
can
just
prioritize.
You
know
whenever
you
have
the
kep
open,
you
know
read
out
myself
and
whoever
else
cares
to
review
it
can
review
it
and
get
it
in
a
state,
and
I
think
I
think
wednesday
is
the
cigar
meeting
right.
A
Was
it
last
week,
sorry,
so
we
wanted
yeah.
I
couldn't
remember
if
when
yeah
wednesday,
so
we
don't
have
another
sick
odds
meeting
before
kept
freeze
it
kept
freezes
before
the
next
cigar
meeting
great
job.
E
A
Be
right,
yes,.
A
A
All
right,
I
can
go
ahead
and
put
this
on
there
as
like.
We
will
talk
about
it,
make
sure
we
use
the
beginning
of
the
meeting
to
look
at
any
open
questions
and
comments,
get
those
addressed
and
basically
be
like
all
right
cool.
This
is
this
is
within.
This
is
implementable
within
124,
whatever
we're
saying.
A
A
C
I
would
love
to
talk
about
the
performance
part
of
the
thought,
and
also
we
have
a
little
bit
of
time
left
on
this
call
right.
I
also
want
to
make
sure:
is
there
anything
that
we're
adding
in
as
part
of
this
observability
cap?
How
does
that
relate
to
the
performance
stuff
right,
anything
that
we
need
for
performance?
That
could
be
impacted
by
this?
C
A
A
Right
so
so
we
we
haven't,
really
talked
about
performance
at
all.
Right
so
I
mean
so.
We
have
what
20
minutes
right.
So
we
we
can.
We
can
just
switch
right.
We
can
just
talk
about
performance,
stuff
and
it's
okay.
You
know,
if
we're
not
going
to
do
any
of
it
in
124,
it'd
be
good
to
have
the
discussion
started.
A
B
Also,
I
think,
last
time
we
talked
about
recovery.
Maybe
we
can
also
close
on
that.
One
too,
like
I
think,
in
this
proposal
we
said
we
could
have
an
option
where
the
user
deletes
it,
and
then
we
also
talked
about
the
cluster
admin
going
and
deleting
the
key
in
xcd.
So
maybe
we
can
also
close
on
that
one
to
say
what
we
intend
to
do.
Do
we
want
to
leave
it
the
way
it
is
today
where
list
calls
will
fail,
and
then
the
user
knows,
and
then
they
can
go
and
delete
it.
A
So
well,
it's
been
months
now,
but
the
last
time
we
talked
about
this
in
cigar.
The
feeling
I
got
from
folks
on
the
call
was
that
it
is
not
the
api
server's
responsibility
to
handle
storage
errors
like
it's
just
not
in
a
position
to
do
anything
good
about
that.
Just
as
you
cannot
work
around
your
database
being
broken
within
your
app
right.
A
Is
the
api
server
supposed
to
do
anything
about
that,
and
I
think
the
answer
is
well
no
you're
supposed
to
go
fix
that
cd?
Is
this
different?
I
don't
know
like
what
I
dislike
about
the
kms
api
and
all
the
encryption
apis
is.
They
have
put
us
in
a
place
where
we
don't
control
the
world
for
our
storage
right.
A
We
rely
on
an
external
service
that
can
arbitrarily
fail
for
an
operation
that
previously
could
effectively
never
fail,
minus
fcd
being
on
fire
right
and
so,
and
this
this
same
exact
concern
is
the
same
concern
I
had
brought
up
for
the
service.
The
external
service
account
signer
kept.
That
is
basically
like
this
right.
It
will
turn
what
is
currently
an
api
that
can
never
fail,
because
the
private
key
is
directly
provided
to
the
api
server
and
uses
it
for
signing.
A
D
Okay,
yeah
one
question
from
my
side:
what's
the
issue
that
they
like
the
thing
that
they
were
against
the
fact
that
we
want
to
automate
the
deletion
of
the
key
that
are
corrupted
like
of
the
data
that
cannot
be
decrypted
anymore
or
was
it
the
addition
of
dash
dash
force,
delete
to
keep
code.
A
A
Those
deletes
and
force
deletes.
All
of
those
basically
are
just
like
additions
that
are
like
saying:
hey
it's
okay,
to
violate
the
kubernetes
api
and
it's
like,
but
it's
not
like
the
kubernetes
api
is,
is
authoritative
here
right
and
if
you
wish
to
violate
those
things,
you
basically
need
to
use
the
back
door
of
directly
hitting
the
database
and
basically
perform
the
violation
right.
We
should
not
give
you
a
front
door
approach
to
bypassing
finalizes,
because
you
have
basically
broken
your
cluster
right
like
okay,
yeah.
It.
D
Introduction
to
one
id
that
I
had,
which
was
to
handle
recovery
via
observability,
because
by
observability
we
should
be
able
to
know
whether
any
kind
of
data
can
be
decrypted
or
not
anymore,
straight
from
the
provider
itself.
So
if
we
can
bring
back
this
information
to
the
users
to
say
that
hey
this
secret
or
whatever
cannot
be
decrypted,
because
you
don't
have
access
to
the
key
anymore
or
whatever
we
just
add
an
alert.
C
D
C
D
The
energy
would
be
like,
instead
of
using
quebecois
to
do
that,
they
would
use,
for
example,
lcd
control
to
directly
direct
delete
the
object
in
the
database
so.
A
So,
but
let's
step
back
right
and
say
that
this
is
an
eks
cluster,
an
aks
cluster
or
gke
cluster
right,
where
you
only
have
access
to
the
kubernetes
api
right.
Are
we
okay
with
basically
staying
in
these
environments,
where
you
know
you're
you're,
working
with
a
provider
that
is
managing
the
infrastructure?
For
you
and
is
handing
you?
You
know
api
server
credentials
in
that
case,
there's
only
two
options:
you
make
a
support
ticket
with
your
provider
and
ask
them
to
go
press
buttons
or
every
provider
implements
something
right.
A
A
That's
not
free
right!
That's
a
lot
of
work
right
and
at
the
end
of
the
day,
anyone
writing
that
code
is
going
to
be
scared
as
hell,
because
they're
giving
you
a
button
that
lets
you
is
guaranteed
to
break
something
somewhere
at
some
point
in
time.
Right
like
it
is
effectively
like
a
support
ticket
so.
D
They
have
to
do
a
manual
action,
and
this
is
like
a
bit
safer
but
more
tedious,
but
the
idea
would
be
that
we
still
have
a
safe
way
to
recover,
even
though
it's
tedious,
but
at
least
we
have
something
because
currently
like
we
have
nothing
and
yeah,
I'm
not
sure
if
we
want
to
keep
that
to
keep
it.
That
way.
C
A
The
what
I,
what
I'm
curious
about,
I
think
rita
this
was
something
you
had
brought
up
a
while
back
is
so
yes,
bad
things
will
happen
and
we
want
to
be
able
to
recover
from
them.
Is
there
a
way
within
like
the
kms
api
itself,
where
we
can
make
it
very
difficult
for
this
to
ever
occur?
So
the
particular
case
I'm
thinking
of
is,
can
we,
I
think
the
word
you
had
to
use
reader
was
a
lease
I'm
naive
to
how
kms
apis
work
in
the
cloud
kms's.
A
If
we
consider
an
expansion
of
that
interface,
can
we
specifically
hold
a
lease
against
the
encryption
key
that
we
are
using
and
basically
make
it
incredibly
hard
for
you
to
get
into
this
case
right,
like
you,
have
to
like
explicitly
go
out
of
your
way
to
break
something
right,
because
today
it's
not
within
that
contract.
Right,
like
like,
like
basically,
I
feel
like
the
kms
api
should
be
like
hey
I'm
using
this.
A
Right
so,
but
again,
I
am
naive
to
the
cloud
kms
apis
and
I
don't
know
what
that
ends
up,
looking
right
like
because
we
we
have
to
whatever
we
build
as
part
of
our
grpc
api.
Really,
yes,
this.
C
Is
part
of
the
dock
like
we
can
write
this
as
a
here's?
What
you
should
be
doing
right
if
you're
using
this
feature,
you
should
turn
on
the
the
lease
lock
feature
in
your
kms
and
maybe
point
them
to
a
in
the
reference
implementation
to
how
to
actually
ensure
that
leases.
A
Right,
but
I
what
I'm
saying
is:
is
that
part
of
like
how
does
that
manifest
within
our
grpc
api
and
then
how
do
we
then
manifest
that,
within
our
stores,
layer
to
allow
for
like
a
high
fi,
fidelity
connection
right
so
that
way
without
like
without
like
a
live
cluster,
you
can
still
validate
right
like
if
the
cluster
is
completely
turned
off
or
you
have
the
lcd
data,
you
should
still
be
able
to
validate
how
that
data
was
encrypted
right.
What,
basically,
just
that
data
set,
should
tell
you
what
leases
you're
currently
holding
right.
B
So
I
think
that
is
where
we
thought
about
passing
the
kick
back
right
so
that
it's
actually
stored
in
hcd
and
then
that
metadata
is
available
so
that
the
kms
plugin
knows.
So
these
are
the
kicks
that
is
currently
used
in
hcd.
So
I
I
cannot
delete
it
or
no
action
should
be
taken
on
those
right,
but
also
like
we
could
land
up
in
this
particular
problem.
B
In
like
one
of
three
scenarios,
one
is,
the
key
is
completely
outside
in
kms
gets
deleted
for
some
reason
like
if
it's
azure
keyword,
let's
say
someone
just
deletes
azure
keyboard
by
mistake.
So
basically
we
are
in
a
state
where
we
cannot
recover.
Then
the
second
one
is
is
also
related
to
performance.
Right,
like
we
hit
the
rate
limit
because
of
some
reason.
B
So
we
have
guidance
for
those,
but
again
that
can
also
lead
into
errors.
So
we
can
get
into
this
error
state
in
one
of
those
three
scenarios,
so
the
least
probably
covers
one
thing.
But
again
that
is
something
we
can
tell
users
that,
like
be
really
careful
that
you
don't
delete
this
key
and
then
for
each
particular
kms
external
store.
We
can
tell
them
that,
like
hey,
if
this
option
is
available,
then
this
is
how
you
enable
and
make
sure
it
doesn't
get
accidentally
deleted.
A
A
I
I'm
curious.
If
we
have
ideas
and
like
the
procedure
you
described
is
you
know
totally
technically
correct,
but
it's
also
relatively
complex
to
orchestrate
on,
like
an
aja
environment,
especially
a
cube
adm
self-hosted
or
an
openshift
self-hosted
like
it's
a
lot
of
work
like
it's
easier
when
you're,
not
self-hosted,
but
it's
not
easy,
never
gets
easy.
A
Is
there?
Can
we
do
something
within
the
grpc
api
for
it
to
do
rotation
itself
right?
Because
if
we
are
tracking
within
the
kubernetes
api
server,
what
keys
we're
using?
I
could
imagine
the
way
of
the
grpc
api
saying
hey.
I
would
like
a
new
key
now
on
some
interval
right.
It
gets
a
new
key.
It
then
initiates.
A
You
know
it
then
initiates
the
rotation
on
its
own.
Based
on
you
know,
however,
it's
gonna,
you
know
orchestrate
within
its
net
right
and
then
you
don't
have
to
worry
as
much
right.
So
it's
like.
Basically,
this
is
hard
code
to
write,
it'd
be
cool.
If
we
wrote
it
ourselves
once
tested
it
really
well
and
just
told
people,
don't
don't
write
your
own,
we
already
did
it
right.
D
Yeah,
I
I
think
that's
a
good
idea,
but
the
the
issue
that
I
would
see
is
that
what
we
want
with
kms,
like
the
the
core
id
behind
this
kms
feature,
is
to
have
like
the
key
handled
somewhere
else
like
by
a
trusted
entity
such
as
a
kms
in
the
cloud
provider,
for
example,
and
every
operation
related
to
this
key,
such
as
the
generation
and
all
that
stuff
should
be
handled
by
the
gamer.
D
So,
for
example,
if
any
user
said
that
they
want
their
key
to
be
rotated
every
30
days
in
the
kms,
I
don't
think
the
cluster
itself
should
have
its
says
to
tell
the
kms
hey.
I
want
a
new
key
now,
it's
not
the
policy
that
was
defined
in
the
canvas
itself,
so
I
don't
think
we
should
force
rotation
ourselves
from
the
inside
and
yeah
yeah
go
ahead.
A
As
I
said
to
be
specific,
what
I'm
more
saying
that
we
do
is
so
there's
a
set
of
configuration
within
the
encryption
configuration
option
I
want
to.
I
I
want
to
keep
it
as
is
or
smaller.
I
don't
want
to
add
any
knobs
there.
What
I
would
like
is
for
the
grpc
api
to
allow
us
to
ask
the
plugin.
A
How
often
do
you
want
to
do
rotation
right?
So
in
this
case
you
know
it
would
be
like
hey
tell
me
what
your
rotation
policy
is
and
if
it
returns
like
zero
you're,
just
saying
no
rotation,
but
you
know
if
you
return
30
days
that
would
help
initiate
that,
but
once
that's
there,
then
we
can
orchestrate
it
for
you
right.
That's
what
I
want
like.
I
want
that
difficulty
of
orchestration
and
the
specifics
of
how
the
rotation
needs
to
be
done
to
be
handled
by
us
once
correctly,
but
yeah
you're
right.
D
Just
a
proposition
that
I
would
have,
because
I
was
thinking
about
that
also
for
our
downstream
use
case,
and
basically
I
think
I
mentioned
in
the
first
meeting.
We
discussed
that
a
bit
and
you
told
me
that
the
kms
itself
tells
the
api
which
key
it
has
used
to
encrypt
the
data
like
which
one
of
the
cac.
So
if
we
keep
in
the
api
server,
for
example,
a
list
of
the
the
last
kk,
then
we
will
know
if
a
new
one
has
been
created
by
the
kms.
D
A
So
that
is
that
is
problematic
today,
because
we
don't
have
any
mechanism
that
looks
across
keys
right,
so
we
only
ever
act
on
a
single
key.
So
if
that
action
happened,
when
all
the
api
servers
were
not
running
and
they
came
back
up,
they
would
be
unaware
of
any
change
because
they
would
start
with
the
new
key
right
and
also
it's
problematic.
When
you
have
many
api
servers,
because
they
all
might
not
see
that
event
right,
only
one
of
them
would
so
then
how
would
the
others
know
that
it
occurred?
A
That
kind
of
thing,
I'm
not
saying
this
is
technically
insurmountable.
I
just
think
we
we
haven't,
we
haven't.
We
don't
have
a
storage
format
that
allows
for
that
yet,
but
yeah.
If
we
had
a
better
storage
format
that
tracked
the
key
encryption
key
id
somewhere
in
there
and
then
also
we
built
in
a
mechanism
that
basically
was
like
an
observability
controller.
That's
like
I'm
gonna,
look
at
everything,
that's
being
encrypted
and
be
like.
Well,
how
is
it
being
encrypted
and
then
it
then
it
sees
that
yeah
you're.
A
I
I've
observed
you
know
like
if
there's
a
leader
or
whatever
I
have
observed
that
a
the
key
encryption
id
being
returned
by
the
kms
is
different
than
the
ones
that
was
the
one
that
was
being
previously
used.
I
can
automate
a
rotation
now
totally
that
the.
D
Total
work
yeah.
If
I'm
not
wrong,
this
could
be
even
simpler,
because
if
you
create
an
object
currently
via
the
kms,
it
will
be
encrypted,
like
the
the
data.
Queue
will
be
encrypted
via
the
latest
key
encryption
key.
So
at
the
very
least
like
if
your
first
creation
gives
you
a
specific
key
id,
you
will
take
that
one
as
the
latest
key
encryption
key
that
exists,
and
then
it's
pretty
easy
to
notice.
D
Like
some
updates,
you
just
have
to
compare
like
the
the
key
of
the
last
encryption
with
the
one
that
you
add
in
your
storage,
like
the
the
one
that
you
cached
previously
and
if
the
id
is
different,
then
you
can
say:
hey
a
new
key
has
been
created,
so
you
need
to
start
like
iterating
over
how
to
rotate
all
the
data.
C
Hey
we
have
two
minutes
left
I
I
thought
it
was
kind
of
funny
that
we
went
from
observability
to
recovery
to
rotation,
but
not
we
didn't
really
talk
about
performance.
C
It's
it's
okay,
and
this
is
all
really
helpful
discussion
nonetheless,
but
I
do
feel
like
we're,
probably
at
the
point
where
I
think,
if
specific
people
have
ideas-
and
it
sounds
like
there
are
lots
of
really
great
great
ideas,
it
might
be
good
to
maybe
start
writing
cups
against
these
different
topics
or
or
like
a
within
the
existing
docs,
so
that
we
can
continuously
iterate
asynchronously
and
then
we
can
use
this
time
to
discuss
what
what
we
are
you
know
proposing
and
then
of
course
I
I
personally.
C
I
also
want
to
understand,
like
of
the
few
of
us
other
than
the
observability
one,
which
one
should
we
tackle
next
right,
because
there's
a
bunch
of
topics
as
everybody
mentioned:
recovery
rotation,
there's,
there's
performance
and
then
there's
also
encryption,
config
hot
load
of
encryption,
config
and
then
there's
also
different
encryption
protocols.
E
C
I
mean
these
are
really
great,
though,
and
and
and
we
are
documenting
the
discussion
as
we
go
so
but.
A
I
think
the
most
likely
failure
mode
in
production
today
would
be
performance.
B
A
C
D
C
I
think
long
term.
We
want
to
reference
implementation
that
has
every
one
of
these
topics
in
there,
but
initially
right
but
initially,
as
you
said,
once
we
get
the
cap
approved
for
observability,
we
can
always
start
there
first,
so
whatever
gets
approved,
we
can
add
them.
Incrementally,
okay,.
A
D
E
Christoph
did
you
have
something?
So
what
are
the
exact
exchanges,
because
we
need
to
be
more
or
less
done
by
next
tuesday
right
so
that
we
can
kind
of
finish
everything
up.
First,.
C
E
C
Once
a
niche
pushes
the
pr
for
cap,
let's
all
go
review
and
then
ping
segoth
approvers,
and
then
we
will
also
present
it
at
the
next
wednesday's
psych
call
just
to
close
any
open
items
and
then,
in
the
meantime,
if
you
have
time
think
about
all
the
questions
and
edge
cases
that
we
want
to
discuss,
asyncly
or
synchronously
on
this
call
for
performance.