►
Description
How can we maintain trust in the cloud and ensure we are building secure systems? Red Hat's Lucy Kerner, Michael Peters, and Lily Sturmann join this episode to discuss Keylime, the CNCF hosted project that provides a highly scalable remote boot attestation and runtime integrity measurement system. Keylime enables users to monitor remote nodes using a hardware-based cryptographic root of trust. Keylime originated out of the security research team in MIT's Lincoln Laboratory, and we will cover the background and goals behind the project. We will also delve into how we can ensure we don't sacrifice security while adopting new technology.
Learn more about Keylime:
https://keylime.dev/
Follow us:
Lucy Kerner https://www.linkedin.com/in/lucyhuhkerner
Randy Russell https://www.linkedin.com/in/randy-russell-0312243
Scott McBrien https://twitter.com/stabby_mc
What is the Level Up Hour?
Every Wednesday at 9am ET we stream live demos on how Red Hat Enterprise Linux (RHEL) users and admins can benefit their organizations and improve their careers by learning how to use containers, Kubernetes, and Red Hat OpenShift.
Learn more at:
https://red.ht/leveluphour
http://red.ht/3ayHlHI
How to participate in the The Level Up Hour:
Watch on YouTube every Wednesday at 9amET
Chat with us on Discord https://discord.gg/5VMVGJt
Check out our GitHub repo https://github.com/level-up-hour/
About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
https://www.openshift.com/
https://github.com/openshift/
#RedHat #OpenShift #tluh
A
A
A
A
A
C
Right
and
it's
it's
not
a
simple
answer
right,
it's
not
like!
I
can
say
you
can
you
know
security's
not
like
go.
I
buy
this
one
product
or
I
do
this
one
thing
or
you
know,
and
I
have
the
silver
bullet
solution
kind
of
thing
right.
It's
about
tackling
it
across
people,
process
technology,
it's
about
shared
responsibility
with
the
public
cloud
providers
and
yourself
it's
using
a
layered
security
defense
strategy.
Again,
there's
no
such
thing
as
100
security.
There's
always
going
to
be
a
trade-off.
C
You
can't
fix
everything
you're
not
going
to
have
enough
people
time,
resources
right!
That's
always
going
to
be
a
challenge.
You
know,
for
example,
you
know
you
have
a
strategy
for
data
data
security
across
the
hybrid
cloud,
whether
you're
talking
about
data
in
the
cloud
or
on
premise,
the
cia
tria.
You
know
confidentiality,
integrity,
availability,
it's
it's
still
true
right!
You
still
need
to
know
you
know.
Is
the
data
appropriately
accessible
when
needed?
C
Is
it
who
has
access
to
the
data
you
know
making
and
then
making
sure
that
you
tackle
the
security
data
security
from
not
only
the
cloud
platform,
but
the
cloud
image?
The
cloud
hosts
right-
you
know,
the
growth
of
workloads
in
the
cloud
is,
is:
is
there
it's
growing
all
the
time
right,
so
in
developers,
you're
seeing
them
have
much
more
control
over
the
application
like
pipeline
and
life
cycle,
they're
downloading
things
directly
off
the
internet,
whether
it's
open
source
packages
or
libraries,
that
kind
of
thing?
C
You
want
to
make
sure
that
you
have
that
consistent
repeatable
framework
like
an
sdlc
in-house
to
develop
applications
not
only
quickly
but
also
securely
and
built
in
early
in
that
pipeline,
when
you're
consuming
open
source
technologies,
you
wanna,
if
you're,
not,
if
you're
doing
it
yourself
right
if
you're
consuming
it
directly
yourself
and
not
using
enterprise
open
source
from
a
vendor
such
as
ourselves,
you
wanna,
have
you
wanna,
create
your
own
product
security
team.
You
know
look
for
find
out
new
vulnerabilities
that
are
all
in
all
the
open
source
packages.
C
You're
using
you
know.
First
of
all,
you
know,
but
first
you
have
to
know
all
the
open
source
technologies
and
packages
you're
using
assess
the
security
impact
of
the
software
that
you
have.
You
know,
of
course,
creating
or
getting
the
security
fixes
and
not
only
getting
it,
but
also
doing
the
isolating
and
doing
back
ported
fixes
as
well
as
I
mentioned
earlier.
It's
not
just
a
technology
problem,
there's
a
huge
people
and
process
aspects
to
it
as
well.
C
You
know
you
see
a
lot
of
research
and
survey
out
there
around
there's,
just
not
enough
security
resources
out
there
and
people,
people
there's
not
enough
security
people
out
there
right.
So
you
can't
just
you
know,
throw
bodies
at
the
problem.
That's
not
that's
not
going
to
be
sufficient
right.
You
want
to
especially.
C
There's
not,
and
also
you
know
the
good
ones
they
usually
you
know,
there's
a
very
short
lifespan
of
you
know
sees
those
like
they
don't
stick
around
forever
right,
like
a
lot
of
them
leave
within
a
year,
maybe
two
years
right
so
and
goods
really
security.
People
are
high
high
demand,
especially
you
know
anyone
who's
doing
cloud
security.
So
that's
another
challenge
right
so
and
especially
with
containers
and
kubernetes,
and
these
newer
cloud
technologies.
C
You
really
need
to
have
that
cross-collaboration
that
no
site,
you
can't
do
silos
right
and
think
you
can't
have
silos
and
one
of
that's
one
of
the
huge
challenges
and
why
that
these
technologies
are
not
getting
adopted
very
quickly
right
because
you
have,
you
need
to
have
app
dev
operations,
security
and
compliance
all
of
those
teams,
working
understanding
how
you
do.
How
do
you
tackle
security
differently
in
this
new
world?
C
You
know
the
tools
may
be
different.
The
way
you
think
about
security
is
different,
and
so
you,
you
wanna,
make
sure
all
of
these
teams
are
educated.
You
don't
wanna,
be
thinking
about
during
security
by
you
can't
just
like
throw
it
over
the
wall
anymore.
They're
doing
last-minute
11th
hour,
panic
like
pen,
testing
and
you
know,
security
scans,
those
kinds
of
things.
You
know
those
are
highly
prone
to
human
error
right.
B
B
C
Yeah,
so
I
think
one
of
them
is,
as
I
said,
the
this
a
lot
of
teams
are
used
to
this
new
way
of
thinking.
So
I'll
give
you
an
example,
so
I
was
at
a
customer
about
two
years.
You
know
before
covert
right
and
I
was
sitting
with
the
security
team
and
they
were
trying
to
manage
their
containers
in
kubernetes
environment
and
they
were
trying
to
use
some
of
their
legacy
tooling,
because
that's
what
they're
familiar
with
right,
they
were
tracking
ip
addresses.
C
They
were
trying
to
track
ip
addresses
of
all
of
these
containers
and
they
were
like.
Why
are
they?
I
can't
keep
track
of
them.
It's
crashing
all
over
the
place,
my
tool
right,
so
they
didn't
realize
that
you
know
this
is
a
immutable
environment.
This
is
an
environment
where
containers
are
coming
up
and
down
and
coming
up
and
down.
C
You
can't
be
tracking
it
with
things
like
you
know,
nagios
or
you
know
things
like
the
standard
monitoring
tools
that
they're
used
to
so,
but
that's
what
they're
familiar
with,
and
not
only
are
they
familiar
with
they're
comfortable
with
it
they've
built
their
career
around
these
tools.
So
it's
very
hard
for
them
to
kind
of
say:
oh
well.
No,
you
have
to
use
this
like
tool.
You've,
never
heard
of
you
know.
C
There's
you
know
that
kind
of
thing,
and
also
it
is
the
this
different
way
of
thinking
right
and
then,
and
it
also
like
they'll,
be
saying
things
like
well,
I
need
to
be
installing
my
antivirus
or
my
third-party
scanner
on
you
know
core
os
like
in
immutable
os
they're
not
used
to
well,
that's
not
what
you're
supposed
to
be
doing
right,
so
this
kind
of
education
session
educate
and
then
not
only
that,
but
the
compliance
teams
as
well.
They
are.
If
you
look
at
the
auditor,
you
know
a
lot
of
times.
It's
about.
C
The
auditors
are
very
used
to
this
defend
the
castle
perimeter
security,
type
of
approach.
Where
you
know
it's
not
you're
not
used
to
this
kind
of
immutable
infrastructure.
So
a
lot
of
the
compliance
rules
out
there,
like
the
security
controls
and
the
security
policies
are
built,
are
written
for
this
per
counseling,
a
perimeter
security
type
of
approach,
where
you
know
you're
not
dealing
with
this
kind
of
immutable
infrastructure.
So
that's
that's
another
challenge
that
the
that
it's
a
lot
of
this.
C
You
know
security
teams
talking
to
the
compliance
teams
and
the
auditors
to
try
to
make
them
understand
that
you
know
yes,
third
party,
doing
antivirus
or
agents
doesn't
make
sense
in
this
environment
or
you
know
you.
Those
kinds
of
those
kinds
of
things
is
another
big,
big
challenge
as
well,
and
also
for
the
containers
and
kubernetes
side
that
that's
it's
another.
C
You
know
layer
of
defense
that
teams
have
to
take.
You
know
everything
from
using
trusted
content.
That's
another!
That's
a
big
challenge,
like
you
know,
developers
just
downloading
stuff
from
the
internet,
not
using
things
like
what
we
offer.
For
example,
like
ubi,
you
know
container
images
that
we've
put
in
red
hat
container
catalog.
C
You
know
those
kind
of
operator
hub
right,
they're,
just
also
using
an
enterprise
registry.
Those
kinds
of
practices
should
be
in
place
to
you
know,
control
the
application
security.
You
know
protect
the
platform
itself
and
then
also
detecting
and
responding
at
the
run
time,
and
then
so
you
know
protecting
the
platform
itself.
You
know
if
you're
not
going
to
use,
let's
say
open
shift,
then
you're
going
to
have
to
figure
out
a
strategy
to
you
know.
C
How
am
I
going
to
do
all
of
the
deployment
things
right
like
you
know,
if
I
don't
have
I'm
not
using
operators
and
and
enabling
are
back
by
default,
or
you
know,
protecting
the
platform
data
which
we
do
by
default
and
openshift.
You
know
all
the
policy-based
deployments
we're
doing
with
like
acs,
advanced
cluster
manager,
advanced
cluster
security
that
we
provide
and
then
all
the
runtimes
related
security
that
we
provide
right.
Also
with
things
like
acs.
C
B
C
B
C
Right
right,
red
hat,
advanced
cluster
security
right
so
for
kubernetes
yeah.
So
that's
that's!
That's
our
product!
Where
we're
you
know,
it
is
focused
on
securing
that
the
workloads
that's
running
on
top
of
kubernetes
right
so
and
and
at
cluster
scale.
So
this
is
this.
Is
you
know,
with
a
lot
of
built
dot
of
the
technologies
that
are
built
in
that
allows
you
to.
A
One
of
the
premises
here
is
that
it's
not
something
that's
locked
in
for
20
years
is
that
it's
you
know
it's
something
that
is
going
to
evolve
and
improve
and
and
change
a
lot.
I
think
that's
part
of
the
issue,
but
stepping
back
even
further
you
know,
open
source
is
is
sometimes
believed
to
not
be
secure,
but
in
fact
it
is
actually
the
more
secure
choice
because
it's
transparent,
but
you
touched
upon
lucy
one
of
the
you
know
sort
of
a
fundamental
issue.
A
There
is
that,
okay,
on
the
one
hand,
it's
open,
you
can
see,
you
can
look
in
and
you
can
see
where
the
vulnerabilities
lie,
but
you
you
have
to
do
one
of
two
things
right.
You
have
to
either
go
and
look
yourself
or
you
have
to
have
somebody
that
you
trust
to
to
do
that,
and
presumably
we
are
one
of
those
parties
that
you
can
look
look
to
to
trust.
Can
you
talk
a
little
bit
about?
C
C
C
You
know,
as
I
mentioned,
you
know,
make
sure
that
if
you're
not
transferring
that
risk
to
a
vendor
to
provide
enterprise
open
source
such
as
ourselves,
that
you
yourselves
make
sure
that
those
open
source
packages
that
you're
using
are
consumed
securely
right,
make
sure
that
you're,
you
know
you
find
out
where
all
those
vulnerabilities
are-
and
you
know
find
out.
You
know
where,
where
how
all
those
open
source
packages
that
you're
using
we,
we
also
don't
just
take
the
upstream
linux
and
make
rel
right.
C
We
did
we,
we
put
it
through
a
it's
own,
our
own
software
supply
chain
process.
You
know
we're
very
picky
about
what
packages
make
it
into
rel.
You
know
we
do
security
scanning
on
the
source
code
itself.
You
know
we
do
things
at
the
compile
time
or
we.
We
also
have
extensive
quality
engineering
quality
assurance
testing.
We
do
on
all
of
our
software.
You
know
we
have
a
secure
way
of
distributing
our
packages
and,
of
course
we
have
our
own
global
product
security
team
right.
A
A
You
know:
how
do
you
secure
the
platform?
You
know
how
do
you?
How
do
you
get
to
that
level
of
comfort
that
you
might
have
with
you
know
the
old
defend
the
castle
procedures
and
routines
and
so
on?
But
when
you
are
in
this
world
that
it's
someone
else's
castle,
that
you're
you're
running
your
mission,
critical
apps
in
and
so.
C
E
E
It
also
comes
up
in
edge
and
iot
cases
where
it
may
not
be
running
in
a
vm,
but
it's
running
on
some
hardware
that
may
not
be
under
your
physical
control
and
so
key
lime
tries
to
come
in
there
and
give
you
this
root
of
trust
when
it
can
hardware
based
when
it
can't
software
based,
but
it's
using
something
called
a
tpm.
A
tpm
is
a
hardware
chip
or
can
be
simulated
in
software.
That
has
a
lot
of
really
nice
cryptographic
properties
such
as
built-in
random
number
generators,
cryptographic
hash
functions.
E
It
can
do
encryption
concealed
data,
it
can
store
chronographic
keys,
but
also
one
of
the
cool
things
about
it.
They
have
these
some
registries
kind
of
like
a
normal
cpu,
but
specifically
for
security
that
can't
be
written
to
only
extended.
So
when
an
application
is
writing
to
the
tpm,
they
don't
overwrite.
What's
there,
they
instead
extend
the
hash,
so
they
the
hash
of
over
the
first
value,
might
be
in
there
and
then
they
add
a
new
value
and
it
gets
hashed
and
added
to
that
other
hash.
E
So
every
thing
that's
measured
and
stored
into
this
tpm
or
what
we
call
extended
into
the
tpm
tpm
registers
is,
is
a
sort
of
a
history
of
every
action.
That's
come
before
it,
and
this
gives
them
some
really
nice
properties
that
you
can
do
for
or
use
for
a
lot
of
things
like
measured
boot
and
remote
attestation,
which
that's
key
lime's.
Core
focus
is
on
on
those
two
things:
measure
building
and
remote
attestation.
E
D
A
D
Sure
so
michael
did
a
good
job
of
explaining
what
the
tpm
does.
So
it's
it
has
more
of
a
specific
functionality,
for
example
in
measured
boot,
it's
very
useful
and
it
gives
this
property
of
integrity.
So
you
know
that
the
system
is
still
in
the
state
that
you
expect
t's
are
similar
in
that
they
have
this
hardware
route
of
trust,
which
is
important,
because
hardware
is
basically
much
more
difficult
to
manipulate
than
software.
You
know
you
have
a
lot
of
different
software
attacks.
D
We
want
to
have
a
hardware
route
of
trust
for
maximum
security,
because
it
is
a
much
more
tamper,
evident
and
so
the
tee
uses
this
hardware
root
of
trust
in
a
very
different
way.
It
is
not
a
a
specific
functionality.
It
more
gives
you
the
ability
to
have
general
computation
in
a
secure
area
of
memory.
On
a,
I
would
say,
regular
cpu,
only
some
cpus
are
manufactured
with
tes,
but
they're
generally
available
now,
and
so
the
way
that
you
would
use,
that
is,
let's
say
that
you
have
some
code.
D
You
want
to
run
that
is
very
sensitive
or
some
data,
that's
very
sensitive.
You
can
run
just
that
application
in
this
secure
area
of
memory
and
that's
the
property
that
the
tee
gives
you
and
the
key
word.
There
also
is
confidentiality.
It
has
confidentiality
properties
and
many
of
them
also
have
integrity.
D
D
E
Yeah
one
way
I
like
to
think
about
it
is
that
the
tpm
helps
protect
the
files
on
the
system
and
and
things
like
your
your
firmware,
your
your
kernel,
those
kinds
of
things
while
a
tee
is
a
runtime
protection.
So,
as
your
programs
are
running,
they
can't
be
altered
by
the
underlying
hypervisor
or
altered
by
somebody.
B
E
E
They
don't
do
that
currently,
so
in
some
situations
we
have
to
use
software
emulation
for
the
tpm.
So,
even
though
you
don't
get
a
literal
hardware,
route
of
trust
when
you're
not
using
a
hardware
tpm,
there
are
scenarios
in
the
cloud
and
other
places
like
that
where
software
tpm
can
still
give
you
some
good
guarantees
about
your
system.
B
E
Yes,
kind
of
in
the
future.
Actually,
that's
that's
it's
interesting!
You
bring
that
up,
like
microsoft.
Has
a
patch
out
currently
for
the
linux
kernel
in
the
part,
called
the
integrity
measurement
architecture
that
would
do
that
with
the
device
manager.
So
when
new
devices
are
added
or
removed,
it
would
cause
triggers
to
the
integrity
measurement
architecture,
which
then
extends
those
attributes
into
the
tpm
which
could
be
verified
by
something
like
keyline,
so
that
that
brings
me
to
another
point.
E
The
this
linux
integrity
measurement
architecture
or
we
call
it
ima
for
short-
is
a
key
part
of
this
that
linux
has
so
it's
a
part
of
the
kernel
that
can
intercept
certain
system
calls
like
I
mentioned
microsoft,
trying
to
extend
it
to
do
device
management,
but
currently
it
does
anytime.
A
file
is
read,
it
could
potentially
be
intercepted
by
imi
and
measured.
A
So
you
used
a
phrase
that
I
think
is
actually
very
important
for
what
keyline
brings
to
the
table
here
and
about
you
know
this
broader
issue
of
securing
the
platform,
and
that
is
remote
attestation.
You
know
lilly
or
mike.
Can
you
maybe
give
us
a
little
bit
of
clarity
about
what
remote
attestation
refers
to,
and
you
know
maybe
draw
some
parallels
to
to
an
earlier
world.
You
know
what
it
would
be,
the
parallel
to
right.
E
Sure
yeah,
so
I
think
a
lot
of
people
are
familiar
with
something
what
we
call
an
ids
or
an
intrusion
detection
system.
So
it's
like
aid
on
linux.
It's
an
open
source
system,
it
it.
You
set
it
up
to
monitor
the
files
on
your
system
or
certain
files,
and
if
they
get
changed
without
you
knowing
it,
it
can
send
alerts
about
that.
So
that's
sort
of
the
old
way
of
doing
it.
Basically,
software
monitoring
software
remote
app
station
takes
it
a
step
further
and
puts
that
hardware
route
of
trust
in
there.
E
So
it's
not
a
separate
program,
monitoring
things.
It's
the
linux
kernel
itself
when
it
accesses
a
file,
it
measures
that
file
and
then
records
that
entry
into
the
tpm
and
so
each
new
file.
That's
added
gets
that
new
measurement
added
to
the
tpm
and
so
the
agents,
the
key
lime
agent
running
on
the
host
can
when,
when
the
verified,
the
healing
variable
we'll
talk
about
the
architecture
a
little
bit
later.
E
And
then
we
can
walk
that
list
of
measurements
and
validate
that
we
get
the
same,
resulting
hash
at
the
end
that
the
tpm
quote
comes
back
with.
So
the
totally
separate
system,
I'm
running
separately,
can
verify
that
the
integrity
of
that
machine
and
which
is
nice,
but
it
gets
even
cooler
when
you
think
about.
I
can
do
this
across
my
fleet.
E
I
can
have
a
keyline
server
monitoring,
hundreds
and
thousands
of
servers
or
of
other
nodes
and
protecting
their
integrity
rather
than
having
just
ideas
on
each
individual
one
and
not
having
like
a
centralized
view
of
that.
So
remote
attestation
gives
you
that
centralized
view
of
a
large
cluster
and
also
that
hardware
route
of
trust
so
like
lily
was
saying
like
hardware
attacks
are
much
harder
than
software
attacks.
E
If
I,
if
so
many
root,
kits
a
machine,
how
can
I
guarantee
that
my
intrusion
detection
system
is
still
going
to
work,
because
it's
software
monitoring
software
that
could
be
compromised,
but
they
can't
do
that
without
physically
getting
that
chip
altering
that
chip
and
when
you
alter
or
or
tpm's
are
designed
so
that
they're
tampered
with
they're
broken
so
it
just
it.
It
provides
some
nice
security
all
the
way
down.
The
stack.
C
E
E
This
was
back
in
like
2016
so
a
couple
years
ago,
but
he
came
across
this
paper
from
these
mit
researchers
that
talked
about
this
theoretical
way
to
to
do
this
trust
in
the
cloud
with
tpms
and
then
started
working
with
them
to
build
a
prototype
keylime
launched
as
a
community
project
back
in
2018
last
year
we
were
part
we
became
part
of
the
cloud
natives,
the
cncf,
the
cloud
native
computing
foundation
as
a
sandbox
project
and
yeah.
There's,
not
I
don't
know
of
anything
equivalent
in
the
commercial
space.
E
D
Just
quickly
that
is
really
cool
is
that
it
gives
people
the
option,
because
there
is
such
a
chain
of
trust,
that's
cryptographically,
verifiable.
It
gives
people
the
option
to
actually
audit
that
and
check
over
it
themselves
if
they
want
to,
they
don't
have
to,
but
it's
not
necessarily
offloaded
to
a
third
party,
so
you're
not
just
moving
the
problem
around.
It
actually
gives
you
this
property,
where
you
can
check
it
and
verify
it.
D
C
A
E
Exactly
I
think,
when
tpm's
first
came
on
the
scene,
where
one
of
their
their
first
users
was
to
do
what's
called
trusted
boot,
which
is
different
than
what
we
call
measured
so
trusted
boot,
basically
allowed
the
vendor
to
lock
down
the
machine
or
the
the
piece
of
hardware
and
not
let
the
user
tamper
with
it
or
alter
it
in
ways,
whereas
measured
boot
is
the
opposite.
So
trusted
boot
restricts
the
freedom
of
the
owner
of
the
device.
E
Measured
boot
increases
the
freedom,
because
you
can
control
what
software
runs
on
there
and
you
can
measure
it
and
verify
it
yourself.
So
I
think
those
terms
can
be
a
little
confusing
and
people
can
conflate
them,
but
measure
boot
gives
the
user
and
the
owner
of
the
hardware
the
freedom
and
control
versus
trusted
boot
which
restricted
it
from
the
defender.
E
Well,
I
mean
you
could
have
it
doesn't
prescribe
the
policies
you
can
have
policies
that
do
prohibit
things
as
part
of
measured
boot
right.
If
you
don't
want
somebody
tampering
with
your
kernel,
you
can
prevent
that
from
happening
right.
That's
a
good
example,
but
you
as
the
owner
of
the
system,
get
to
decide
what
that
policy
is.
A
E
So,
just
just
to
get
some
little
visuals
on
this,
the
key
lime
architecture
we
mentioned
this
before,
but
we
have
the
keyline
agent
running
on
some
untrusted
hardware
somewhere
talking
to
a
tpm
or
a
virtual
tpm,
and
then
over
the
network.
It
can
be
trusted
or
untrusted
doesn't
matter.
It
talks
to
the
keyline
server
side
and
the
keylab
server
side
is
basically
split
into
two
parts,
which
is
the
verifier
and
the
registrar.
E
E
And
then
the
verifier
is
going
to
kick
off
her
verification
action
to
remove
that
node
from
the
load
balancer.
So
this
is
a
real
world
scenario
where
you
have
a
normal
web
application
behind
a
load
balancer.
If
any
of
those
nodes
gets
compromised,
you
would
like
to
remove
it
from
the
load
balancer.
So
this
will
do
this
automatically.
E
E
This
is
that
the
the
output
from
the
key
lime
agent
running
in
that
application-
and
this
is
the
second
one-
so
I'm
going
to
start
the
keylime
agent
here
on
the
third
application
and
we
can
see
it
starting
up,
and
so
it
talks
to
the
the
register
gets
everything
recurrent
returned,
and
so
now
it's
just
waiting.
So
you
can
see
here
waiting
for
revocation
messages.
E
And
so
I
use
that
I
do
that
with
this
keyline
tenant
command,
and
so
I
walk
through
these
options,
real
quick
just
so
you
can
understand,
what's
happening
so
the
keyline
tenant,
we're
saying
that
the
verifiers
at
this
ip
address
the
tenant
or
the
new
post
that
we're
monitoring
is
at
this
ip
address.
This
is
its
id,
which
is
corresponds
to
its
ec2
id
and
which
could
be
anything
like
if
it's
an
ec2,
it's
easy
to
id.
E
If
it's
just
random
hardware,
you
can
use
hardware,
specific
keys
or
or
ids
on
that,
whatever
you
want,
it
could
be
even
randomly
generated
use
uids.
It
doesn't
really
matter,
but
it's
just
to
track
this
machine
over
time,
and
then
we
give
it
an
allow
list
and
allow
list
is
a
list
of
all
the
files
and
hashes
that
we
are
allowing
to
run
on
this
system.
E
Right
right,
but
you
can't,
you
can't
know
ahead
of
time
what
they're
going
to
be.
This
is
just
the
ca
certificates
that
are
used
to
communicate
with
that.
This
is
a
payload
that's
sent
over
when
access
station
passes,
and
this
payload
will
also
include
the
revocation
actions,
and
so
we'll
look
at
what
all
of
these
look
like.
E
So
at
the
top
here.
This
is
what
a
sample
allow
list
looks
like
it's,
basically,
just
a
list
of
hashes
and
what
those
files
should
be
so
etsy
password
should
be
this
or
anything
else.
So
it
just
it's
a
list
of
every
file
and
they're
expected
hashes,
and
then
our
excludes
are
here
and
so
we're.
This
is
a
pattern
matching,
so
we
can
exclude
anything
in
fact
see
we
have
etsy
in
our
allow
list.
We
have
a
lot
of
other
things
too.
The
allow
list.
E
Is
pretty
long,
you
know
64
000
entries,
but
we're
just
going
to
ignore
everything
in
etsy
right
in
this
scenario,
it
can
change
we're
not
guaranteeing
that
there
something
might
be
pushing
out
in
a
more
mature,
evolved
situation.
We,
you
might
actually
want
to
monitor
the
files
in
etsy
and
say
they
can
only
change
after
going
through
our
ci
cd
pipeline
and
our
ci
cicd
pipeline
also
updates
the
allow
list.
D
E
A
E
B
So,
michael,
if
I
can
interrupt
you
for
just
a
second
yeah
so
being
familiar
with
aid,
they
it
tracks
a
whole
bunch
of
properties
about
files,
and
then
the
downside
is
that
you
actually
have
to
run
a
check
against
your
database
to
know
that
anything
has
changed.
So
here
we're
looking
at
less
metadata
about
files
just
looking
at
the
content
but
dynamically
monitoring
those
files
so
that
as
soon
as
the
content
changes,
we
can
take
action
right.
E
Yes
or
yes,
or
no
so
in
this
scenario,
I'm
showing
how
to
use
keylime
to
just
do
the
measurements
of
the
file.
So
the
contents
of
the
file,
like
you,
said,
there's
another
way
to
use
it
and
to
use
ima
as
well,
so
that
it
you
have,
each
file
is
signed
and
by
a
key
so,
for
instance,
in
in
upcoming
versions
of
red
hat.
E
Each
file,
that's
installed
by
an
rt
rpm
through
red
hat,
like
an
official
red
hat
file,
will
also
have
signatures
of
that
file
from
signed
by
redhead
keys,
and
that
signature
includes
the
contents
of
the
file
and
also
the
attributes
of
that
file,
so
yeah.
So
the
attributes
can
also
be
measured.
It's
just
you
get
two
different
things
like
and
you
can
use
either
or
with
t-line
and
ima,
or
you
can
use
both
together.
So
they
just
have
different
reasons.
B
So
there
are
certain,
let's
call
them
environments
where
maybe
the
data
integrity
is
one
concern
which
we
solved
here,
but
then
also
access
to
that
data
may
also
be
a
supplementary
concern
so
like
if
someone
was
able
to
get
read
access
on
a
file
that
they
did
not
have
read
access
on
before
that.
That
could
be
a
problem.
E
Yeah,
if
you're
using
the
ima
signatures
versus
the
I'm
a
measurements,
then
the
signature
when
you,
when
it's
signed
by
the
vendor,
will
have,
or
or
in
this
case,
like
when
you're
signing
the
file
in
it,
could
be
red
hat
specific
files.
Or
if
you
want
to
do
your
own
files,
then
that
signature
would
be
you'd
have
to
manage
the
key
and
do
the
signing
and
all
that
yourself.
But
you
sign
it
with
the
contents
of
the
file
and
the
attributes
of
that
file.
So
yeah.
E
E
Yeah,
but
I
think
this
is
the
simpler
way
to
to
visualize.
What's
happening
is
just
to
look
at
the
file
contents
and
then,
as
you
can
see
from
here,
I
have
a
an
elastic
load
balancer
with
has
these
three
nodes
app
two
f3
and
f1-
that
are
these
three
nodes
that
are
being
monitored
so
now,
let's,
let's
go
back
and
add
so
we
added
this
node
and
now
waiting
on
the
replication
messages.
So
now,
let's
do
the
keylime
tenant
command
to
tell.
E
E
And
so
now
we
see
here
not
this
file
was
not
found
in
the
allow
list.
Some
entries
couldn't
be
validated
and
then
sending
revocation
to
listening
notes
right
so
that
triggered
the
event
on
this
remote
attestation
system.
So
I
just
ran
this
thing
that
shouldn't
be
allowed
to
run,
and
now
I
get
this
so
now
we
come
back
here
and
I've
noticed
there's
a
little
bit
of
a
lag,
but
now
we
can
see
that
the
node
is
draining.
E
B
E
Well,
actually,
the
script
was
there
from
before,
but
still
wasn't
in
the
list.
What
triggered
the
the
accession
value
was
the
script
being
executed
so
because
I
have
ima
set
up
to
only
monitor
files
that
are
executed
by
root,
but
you
can
have
it
set
up
to
do
lots
of
different
things
to
monitor
a
lot
every
file,
that's
accessed,
that's
read
or
write
written
to
or
any
file.
That's
that's
executed.
In
this
case
it
was
executed
as
root,
and
so
that's
what
triggered
the
attestation
value
and
the
nice
thing
is
like.
E
If
I
go
and
remove
that
file
and
then
try
to
restart
out
to
station,
it
will
fail
again
because
the
the
history
or
the
the
trace
that
that
file
was
run
is
now
in
that
tpm
and
can't
be
removed,
so
even
just
removing
the
file
won't
restart
at
the
station.
So
I
can't
like
just
compromise
the
system
run
a
script,
real,
quick
and
then
get
delete
the
script
and
get
out
the
trace
of
that
ran
is
still
there.
B
So
being
being
an
operations
person
and
may
or
may
not
have
made
a
mistake,
one
or
two
times
in
my
history
of
doing
things,
let's
say
that
I
accidentally
did
something
caused
a
recoveration
to
occur,
and
it's
not
as
simple
as
just
reattaching
the
the
node
like
approving
it
and
reattaching
it.
How
do
we
do
that.
E
E
That
something
like
key
lime
when
you're,
when
you
don't
have
a
very
mature
environment
or
or
a
very
disciplined
environment,
where
any
change
that
goes
through
has
to
be
automated
and
go
through
your
pipeline.
If
it's
changing
or
altering
files
that
are
being
monitored,
you
have
to
create
a
new
allow
list
to
go
with
that
and
update
key
lime
is
the
same
way
you're
going
through
it,
and
then
you
know
so.
E
If
you're
mainly
ssh
into
these
systems
and
manually
changing
files,
you
have
to
be
very
careful
with
something
like
keyline,
because
you
probably
want
to
put
it
more
in
an
advisory
state.
So
the
way
keyland
does
revocation
right
now,
there's
different
ratification
actions
they
can
send
to
be
executed.
It
can
also
do
in
an
upcoming
release.
We'll
have
web
web
hook
support
so
that
it
can
call
out
to
a
web
hook.
So
what,
when
you're?
First
starting
out,
you
might
want
it
to
just
alert.
E
You
send
a
notification
to
slack
this
node
failed
attestation.
It
needs
to
be
restarted
or
something
like
that,
so
that
you
can
make
sure
that
your
process
of
how
you
update
these
machines
is
very
mature
and
you
don't
want
to
fat
figure,
dos
yourself
or
or
running
yeah.
So
it
does
take
a
lot
of
discipline,
and
but
it
is
very
it's
for
these
very
specific
cases
where
you
do
not
control
the
hardware
and
you
want
to
control
what's
running
on
them.
B
Yeah-
and
I
think
discipline
is
a
great
description
over
the
years,
we've
moved
from
kind
of
the
gilded
age
of
this
administration,
where
I
lovingly
handcrafted
four
systems
and
that's
what
my
job
was
to
you
know.
I
have
to
manage
hundreds
or
thousands
which
brings
that
need
for
greater
discipline
and
not
just
like
ssh
and
running
commands
by
hand,
because
it
would
be
impossible
to
manage
it
that
way.
E
E
Right-
and
this
brings
up,
I
think,
leads
into
another
scenario
like
how
do
you
manage
these?
Allow
lists
like
how
do
you
craft
them?
How
do
you
know
what
software
is
approved
and
not,
and
that's
where
a
vendor
like
red
hat
comes
in?
We
have
future
plans
to
make
this
an
integral
part
of
our
systems
to
allow
to
give
you
pre-approved
pre-signed
allow
lists
from
our
red
hat
products
that
you
can
then
incorporate
into
key
lime
or
any
other
system,
and
that
you
can
combine
that
with
your
own.
E
B
E
Yeah,
so
there
is
a
very
simple
web
app
that
comes
with
keylime,
although
it's
it's
very
simple,
but
there's
rest
apis
to
all
the
different
pieces
of
key
line.
So
when
you're
integrating
key
limit
into
something
else,
because,
specifically,
I
think
most
organizations
want
a
single
payment
class.
They
don't
want
something
else.
They
want
to
look
into
to
see
what
the
status
of
the
nodes
are.
E
So
the
rest
apis
for
the
register
on
the
verifier
can
let
another
system
know
the
status
of
individual
nodes
at
any
time,
and
these
revocation
actions
can
either
be
like.
I
said
these
python
scripts
that
are
executed
either
on
the
nodes
themselves,
like
if
a
web
app
node
one
node,
fails.
All
the
rest
of
the
nodes
can
execute
some
script
to
coordinate
off
that
node
or
or
in
some
way,
but
you
can
do
a
lot
of
like
custom
actions.
E
That
way,
like
I
said,
showed
here,
we
did
an
elastic
load,
balancer,
removing
deregistering
a
node.
You
can
do
like
a
kubernetes,
coordinate
and
drain
where
the
node
is
the
worker.
Node
is
actually
pulled
out
and
all
the
the
work
on
it
is
migrated
to
another
node
and
with
the
web
hook,
you
can
integrate
with
things
like
slack
or
other
messaging
systems
or
or
you
can
just
send
it-
have
a
single
web
hook
that
sends
back
the
notification
and
then
you
do
a
whole
bunch
of
chain
stuff
with
it.
A
Very
interesting
stuff
indeed,
so
let
me
just
kind
of
redirect
us
a
little
bit
out
of
key
lime
which
we'll
come
back
to
in
a
moment,
but
you
know:
are
there
some
other
projects
that
are
sort
of
in
the
same
space
and
trying
to
accomplish
some
of
the
other
kinds
of
some
of
the
kinds
of
things
that
we're
trying
to
accomplish
with
key
lime?
I
know:
there's
there's
sig
store,
for
example,.
D
Yeah,
so
so
we
have
some
exciting
future
we're
coming
up
with
key
lime.
I
can
definitely
talk
about
here,
the
first
one,
the
the
key
lime
rust
agent,
which
is
what
I've
mostly
been
working
on.
There
is
a
small
team
of
us
to
port
the
the
agent,
which
is
the
part
of
key
lime
running
on
the
node.
That's
proving
its
state
over
to
rust,
which
is
a
safe
systems.
D
Programming,
language
and
rust
is
a
very
cool
language,
but
the
important
thing
here
is
that
this
makes
it
more
appropriate
for
minimal
and
immutable
operating
systems,
so,
like
cases
on
the
edge
or
cases
like
red
hat,
core
os
or
fedora
core
os,
we
don't
necessarily
want
to
pull
a
bunch
of
dependencies
in
python
into
that.
So
this
rust
agent
will
let
us
bundle
like
a
self-contained
binary
and
easily
like
place
it
in
those
environments
which
will
make
it
more
widely
accessible.
D
So
that's
pretty
cool
and
then
michael,
I
don't
know
if
you
want
to
comment
on
some
of
these
other
ones.
I
could
I
could
read
them,
but
we're
moving
towards
some
containerization
is
kind
of
the
the
exciting
stuff
we
were
able
to
run
this
vtpm
the
software
tpm
in
vms
right
now,
but
we
want
to
definitely
also
be
able
to
do
it
in
containers.
So
some
of
the
future
work
is
moving
in
that
direction.
D
E
So,
as
the
demo
showed
yeah,
we
were
monitoring
a
vm
because
that's
where
I
am
is
focused
like
the
linux
name,
space
or
linux
imo
subsystem.
So
it's
not
namespace
which,
if
you're
talking
about
containers
like
file
system
and
the
network
and
the
pid
namespace,
all
those
things
are
what
allow
the
container
to
exist
kind
of
separately.
So
there's
work
being
done
to
create
an
ioma
name
space
that
will
let
containers
have
their
own
view
of
the
ima
measurements.
E
And
then
you
could
run
a
key
lime
agent
inside
of
there
and
verify
the
contents
of
the
containers
at
runtime.
Other
things
with
the
like
the
run
time
of
containers
with
run
cd
tpm's
being
allowed.
So
the
container
can
talk
to
the
tpm,
kubernetes
operators
and
all
kinds
of
stuff
that
are
in
the
future
for
coming
up.
E
Now
you
mentioned
this
like
six
store.
Six
store
is
an
interesting
project.
That's
also
coming
out
of
our
group
here
at
emerging
technologies
in
red
hat,
and
it
allows
the
software
supply
chain
to
publicly
sign
artifacts
in
a
way
that
wasn't
available
before
and
not
only
just
publicly
signed,
but
publicly
verified
from
everybody
in
in
what's
called
a
transparency
log.
E
So
this
is
very
similar
to
the
transparency
or
the
certificate
transparency
project
out
of
google,
which
tracks
every
ssl
certificate
that's
ever
issued,
and
that
browsers
can
then
verify
and
look
at
certificates
as
they're
issued
or
revoked,
and
so
it
takes
those
concepts.
Without
that
idea,
kind
of
combined
with
the
let's
encrypt
I
idea
of
making
democratizing
ssl
certs.
So
the
same
store
is
democratizing
artifact
signing.
Essentially,
so
you
can
prove
the
provenance
of
any
artifact.
E
All
these
signatures
can
then
be
verified
all
the
way
down,
and
then
the
integrations
with
key
lime
would
be
really
nice,
because
your
allow
list
could
also
be
a
signed
artifact
where
you're
pulling
your
allowance
from.
For
instance,
if
red
hat
starts
publishing
a
lot
lists
for
a
certain
version
of
our
operating
system,
then
we
can
publish
that
allow
list
in
the
signature
to
sync
store.
E
Keyline
can
automatically
pull
it.
You
know
find
out
where
it
is
through
sig
storm,
pull
it
down,
verify
it,
because
it's
publicly
signed
in
this
public
transparency
log-
and
you
get
all
these
just
nice
guarantees
about
your
whole
system
and
everything
moving
through
the
system.
So
there's
a
lot
of
really
cool
opportunities
there.
So
here
in
emerging
tech,
we're
looking
at
like
one
to
two
years
out-
and
this
is
going
to
be
an
exciting
space
to
keep
watching
for
integration.
A
Well,
it
sounds
like
it
there's
a
lot
there
and
you
know
just
to
sort
of
wrap
things
up
here
before
we
get
to
our
sweet,
sweet
internet
points.
You
know
somebody's
interested
in
learning
more
about
key
lime.
I
guess
they
can
go
to
keylime.dev
and
I
think
we
posted
that
in
the
chat,
but
I
know
that
we've
actually
also
got
some
security
workshops
coming
up.
Can
anybody
tell
us
a
little
bit
about
some
of
those.
C
Yeah,
so
if
you
go
on,
I
don't
know
if
you,
if
we
put
it
in
the
chat,
but
it's
red.edu,
okay,
rad.ht
security
workshops.
So
we
have
the
virtual
security
workshop
coming
up
in
august
october
21st
and
it
is
focused
it's
the
various
hands-on
exercises.
We
have
that's
focused
on
openshift
security
and
advanced
cluster
security.
So
you
know
if
you
want
to
look
at
different
use
cases
around
around
these
technologies.
I
I
highly
encourage
you
to
sign
up,
but
it's
virtual
and
it's
also
free
to
attend.
C
Yeah,
so
for
a
security,
symposium
yeah,
we-
it
was
back
in
july,
but
it
is
it
is
on
demand,
so
you
can
go
and
listen
to
all
the
different
sessions
we
had
around.
You
know:
containers
security,
kubernetes
security,
data
security,
compliance.
We
had
a
various
different
sessions
around
this,
so
you
know
encourage
you
to
go.
Listen
to
those
as
well.
E
Also
coming
up
at
cubecon,
we
should
have
a
booth
for
key
lime
and
sig
store
as
well
so
feel
free
if
you're
at
kubecon
in
person,
and
there
might
actually
be
some
virtual
q
a
booth
times
as
well.
I
don't
know
the
details
of
those
but
be
on
the
lookout
if
you're
curious
about
those
and
you're
attending
kubecon,
virtually
or
in
person
and
come
come
by
and
say,
hi
to
us,.
A
Yeah
good
mention
there
will
be
a
a
solid
red
hat
presence
there,
including
myself
and
my
colleague
henry
mayne,
to
talk
about
some
of
the
certification
things
that
we
do
around
kubernetes
and
open
shift,
and
so
you
know
with
that.
You
know.
Thank
you
very
much
all
of
you
for
joining
us
today.
This
has
been
incredibly
informative.
E
A
It'll
pass.
I
think
this
is
a
fascinating
subject
and
it's
also
a
very
fast
moving
one,
so
we
might
have
to
have
you
back
to
give
us
an
update
on
where
things
are
in.
You
know
a
matter
of
six
months,
maybe
you
know
maybe
less,
because
that's
how
fast
things
move
sometimes
so
anyway.
Thank
you
again.
It
is
that
time
now
for
us
to
talk
about
sweet,
sweet
internet
points,
and
you
know
we
actually
have
a
bit
of
drama
this
week.
Scott
look!
A
Right
so
keep
coming,
keep
collecting
the
sweet,
sweet
internet
points.
You
do
that
by
going
to
one
of
the
urls
here
in
the
bottom
of
of
this
slide,
which
we
will
also
post
to
the
chat,
you
know
enter
your
code,
collect
your
points,
remember
narendev
looked
you
know
beyond
reach
and
yet
now
neck
and
neck
with
an
lhacm,
so
collect
your
sweet,
sweet
internet
points
and
you
know,
keep
watching
and
again
please
like
subscribe
and
share,
and
thank
you
everyone
for
joining
the
level
up
hour
this
week.