►
From YouTube: TGI Kubernetes 058: Amazon Firecracker
Description
Come hang out with Joe Beda as he does a bit of hands on hacking of Kubernetes and related topics. Some of this will be Joe talking about the things he knows. Some of this will be Joe exploring something new with the audience. Come join the fun, ask questions, comment, and participate in the live chat!
This week we will be covering the new minimal VMM from AWS called Firecracker. I'll explain what a VMM is, how it relates to containers and Kubernetes and why this is a cool new building block. We'll go through the getting started instructions to get it up and running (on GCP with nested virtualization).
A
All
right
welcome,
welcome,
welcome
everybody
to
the
addition
of
TGI
kubernetes.
This
is
episode
58
and
today
is
the
30th
of
November.
The
year
is
flying
by
and
I
just
got
back
from,
Las
Vegas,
where
I
was
at
reinvent
and
so
we're
gonna
cover
one
of
the
topics
that
we
some
one
of
the
new
projects
that
Amazon
announced
there.
So
I'm
really
excited
about
this
thing
called
firecracker
today.
A
Just
for
those
who
don't
know
so,
TGI
kubernetes
is
a
weekly
livestream
that
I
your
host,
Joe,
Beda
or
sometimes
Chris
Nova
she's
been
doing
more
and
more
of
them.
Lately
we
spend
our
hour-and-a-half
digging
into
keys
on
hands
on
keyboard
topics
in
the
kubernetes
space,
and
so
sometimes
we'll
explain
it
stuff.
We
know
well
sometimes
we'll
be
explaining
new
projects
other
times
we'll
be
looking
at
new
technologies.
One
does
not
relate
to
cooperate.
A
We're
not
going
to
do
any
kubernetes
at
all
for
this
one,
but
but
it
definitely
relates,
and
so
we'll
definitely
get
into
that.
So
first
off
I
like
to
say
oh
and
I'm,
the
CTO
of
hep
geo,
which
is
a
small
startup
that
is
helping
enterprises
be
successful
with
kubernetes
a
couple
weeks
ago
we
announced
that
we're
in
the
process
of
being
acquired
by
VMware,
but
don't
worry,
nothing's
gonna
change,
we're
gonna
keep
doing
TG
at
kubernetes.
So
that's
gonna.
That's
going
to
keep
going
so.
B
A
So,
first
of
all,
let
me
say
hi
to
everybody,
I
love
this
there's
folks
from
all
over
the
place
all
over
the
world
dialing
in
and
so
Olaf
was
here
earlier
saying:
hi
from
Copenhagen
Manolo
from
Spain,
the
Canary
Islands
I
just
listened
to
this
audio
book
on
Magellan's
voyage.
We,
you
know
he's
Portuguese,
but
he
he
sailed
for
the
Spanish.
So
yet
in
the
Canary,
Islands
obviously
play
a
role
in
any
any
maritime
history
that
was
really
interesting.
Okay,.
A
Good
to
see
y'all
ally
from
Sweden
Michael
is
in
Vegas.
All
right
was
just
there
Rory
from
Scotland
Norman
from
London
Roy
from
Toronto
Diego
from
Norway.
Now
I
didn't
from
Bosnia
awesome
Michael
from
Johannesburg
BK
from
Boston,
so
George
is
going
to
be
joining
us
he's
a
he
works
for
hefty
Oh.
He
had
a
link
in
the
in
the
in
the
comments.
If
you
want
to
check
it
out
where
we
have
sort
of
crowd-sourced
notes.
A
A
The
expert
on
this
stuff
I'll
tell
you
sort
of
my
experience
here
and,
and
you
can
apply
your
own
filter
there,
Ben
from
Casey
Jason,
saying
hi,
he's
he's
a
hep,
Tony
and
I'm
na
Alex,
also
VMware,
who
does
open
Foss
good
to
see
Alex
joy,
a
rude
also
from
AWS
helping
us
out,
Oh
Nova's
down
the
street.
Oh
no!
You
didn't
come
into
the
office
today.
No,
but
I
don't
see
her
right
there,
so
yeah,
so,
okay,
so
I,
so
so
Tim,
oh
man,
there's
just
so
much
to
go
through
here.
A
I'm
gonna
skip
ahead
a
little
bit.
I
wish
I
could
call
it
everybody,
but
I
don't
want
to
take
the
entire
thing.
Oh
mark
also
mark
peak
there.
Also
a
vmware
person
that
I've
been
talking
to
quite
a
bit
got
Gustavo
from
Wheaton
from
Illinois.
I
grew
up
in
Glen
Ellyn
right
next
to
Wheaton,
so,
alright,
okay,
so
here's
the
plan
good
to
see
y'all,
I'm,
always
blown
away
that
you
guys
are
gonna,
spend
your
Friday
afternoon
Friday
evening
Saturday
morning
with
me:
that's
really
really
cool!
A
So
usually
I
start
out
with
going
through
some
interesting
stuff.
That's
happening
the
kubernetes
world.
Now
it
turns
out
that
it's
actually
a
little
bit
of
a
quiet
week
with
with
kubernetes
I.
Think
people
tend
to
not
announce
stuff
when
reinvents
going
on,
because
that's
just
a
sort
of
a
you
know
attention
denial
of
service
attack
in
a
good
way
and
so
as
it
and
we're
leading
up
to
cube
con,
which
is
going
to
be
in
two
weeks,
which
is
obviously
going
to
be
super
exciting
for
the
kubernetes
community.
A
A
couple
of
things,
though,
that
I
do
want
to
go
through
so
first
off
and
George
pointed
me
out
to
this-
is
that
and
all
these
notes
are
going
to
be
in
that
hack
MD
we're
going
to
put
those
things
into
a
github
repo
and
link
to
them
from
the
video
recording.
So
if
you
don't
see
stuff
here,
if
you
can't
find
it,
you
can
always
find
it
there
later.
A
So
the
first
thing
is
that
the
CNCs
Linux
Foundation
they
do
certification
exams
for
kubernetes,
the
the
CK
a
and
the
CK
ad
kubernetes
administrator
and
application
developer
exams.
Those
things
are
on
sale
right
now
and
I
think
that
sale
ends
today.
George
was
saying
so
be
aware
of
that,
and
then,
in
that
same
vein,
we're
doing
some
training.
Hefty
o
is
as
part
of
Q
cons.
So
if
you're
going
to
be
in
town,
let's
see
yeah
so
George
put
the
notes
there
again.
A
If
you're
going
to
be
in
town,
the
Monday
before
cube
Cod
we're
going
to
be
doing
some
training
and
intro
kubernetes.
Our
training
team
is
is
really
excellent,
and
this
is
some
good
training.
I,
unfortunately,
won't
be
able
to
be
there
because
I'm
gonna
just
be
spending
my
time
at
the
contributor
summit.
So
if
those
who
are
deeper
the
kubernetes
world,
we're
gonna
be
having
the
contributor
summit
on
that
same
Monday.
Also
and
George.
Do
you
want
to
put
a
do?
A
You
have
a
link
to
the
schedule
that
you
can
drop
in
there
and
we
can
definitely
put
that
in
there
because
I
think
they
contribute
or
something
for
those
who
who
are
on
the
list.
Now
that
was
folks
who
are
contributors
to
kubernetes,
who
signed
up
earlier,
are
going
to
have
access
to
that,
but
I
think
we're
going
to
be.
You
know
at
least
I
think.
Last
time
we
tried
to
record
some
of
the
sort
of
new
contributor
getting
started
stuff,
so
hopefully
we'll
be
able
to
create
some
good
good
assets.
A
Out
of
that,
let's
see
other
news
envoy
graduates,
the
CNCs,
so
our
our
front
here,
another
another
Seattle
light.
Okay,
there's
the
the
contributor
summit
there.
Let
me
go
ahead
and
grab
that.
So
this
is
the
2018
contributor
summit.
If
you
know,
if
you're
involved
in
the
community-
and
you
can
make
it
to
these
things-
that
are
usually
really
really
interesting-
a
lot
of
stuff
going
on
and
I
know,
George
in
Paris,
both
working
through
the
cig
contributor
experience
has
been
ok.
The
sessions
will
be
on
YouTube
afterwards.
A
There,
as
I've,
been
spending
a
lot
of
time
putting
this
stuff
together.
So
we
got
a
great
schedule
planned
there,
and
this
is
we're
looking
for
both
organizing
current
contributors
mapping
out
what
we
want
to
do,
how
we
want
to
work
but
then
also
having
a
workshop
for
new
contributors
to
actually
get
going
with
it.
A
So
that's
the
contributor
summit:
that's
gonna
be
happening
on
on
the
Monday
before
cube
con
alright,
so
we
have
on
voice
graduates,
so
so
the
CNC
F
has
different
levels
of
projects,
the
the
sort
of
the
beginning
level
level
for
stuff.
That's
just
getting
started.
It's
called
a
sandbox
project
from
there
I
believe
it's
called
a
incubation
project
and
then
it
graduates
to
a
fully
graduated
project,
and
we
you
know
kubernetes-
was
the
first
to
do
that
and
there's
a
few
other
projects,
but
envoy
is
the
latest
to
graduate
envoy
is
a
really
great
project.
A
It's
what
I
love
about
envoy
and
I.
Think
you
know
Matt
who's,
the
the
engineer
at
lift
who
started
it
said
envoy,
doesn't
try
to
to
try
to
do
too
much.
It's
clear:
it's
it's
solely
a
data
plane,
load
balancer,
which
is
really
really
cool.
So
that
means
that
folks
can
do
all
sorts
of
interesting
things
by
programming
it
in
dynamic
ways.
We've
released
this
a
ingress
controller
for
kubernetes,
called
contour
built
on
envoy
SDO,
builds
on
envoy
and
Amazon.
Yet
reinvent
just
announced
this
new
hosted
surface
mesh
project
called
app
mesh
I.
A
Think
the
project
that
we're
looking
at
today,
which
is
a
firecracker,
a
virtual
machine,
is
actually
I
think
a
similar
type
of
project,
in
that
it's
a
building
block
that
can
be
reused
and
remixed
in
a
bunch
of
different
ways
and
probably
in
ways
that
the
you
know
if
it's
successful,
that
the
authors
weren't
weren't,
really
anticipating
when
they
when
they
wrote
it.
Let's
see
so.
Okay.
A
A
We've,
it's
moved
to
the
point
where
1.13
is
in
a
fork,
so
only
critical
changes
are
going
to
be
making
it
in
and
then
the
the
main
line
in
kubernetes
has
been
opened
up
for
changes
for
1.14,
so
that's
moving
along
also
and
yeah,
so
lots
of
good
stuff
coming
there
I'm
not
gonna
like
like
one
of
the
big
projects
and
it's
a
real
pain
in
the
butt.
So
the
people
who
do
it
the
cig
release
folks
are
are
amazing.
A
A
Admin
are
going
to
be
going
to
GA
and
that's
that's
been
a
long
time
coming.
I
think
we've
been
probably
a
little
bit
too
too
timid
to
actually
declare
a
GA,
because
a
lot
of
folks
are
using
it
in
production
already.
But
it's
exciting
to
see
that
be
happening
so
see
it
says
like
yeah,
so
we're
gonna
have
a
lot
of
talks
at
cube.
Con
I
know
a
lot
of
folks
have
been
working
on
their
talks.
A
A
Well,
we're
there,
where
we're
gonna
play
with
some
of
the
new
stuff
that
folks
are
announcing
the
new
I,
don't
think
we
were
able
to
get
that
set
up
so
that
we
could
broadcast
at
the
same
time
and
where
it's
going
to
be
a
little
bit
tricky
to
sort
of
work
through
the
acoustics
and
everything.
But
I'm
really
excited
to
try
that
out
for
the
first
time
yeah.
A
So
that's
gonna
be
super
cool
all
right,
so
that
is
all
the
pre-announce
stuff.
If
there's
other
stuff.
That
folks
want
to
call
attention
to
put
it
in
the
notes
and
and
I'll,
be
happy
to
try
and
look
it
up.
One
of
the
things
that
we've
been
doing
the
last
couple
times
we
knew
it
again
is
that
we're
gonna
give
away
some
hefty
o
swag
it's
limited
edition.
For
now.
You
know
post-acquisition,
I'm,
not
sure.
A
What's
going
to
happen
with
that,
with
the
hefty
owe
name,
that's
one
of
the
things
we're
figuring
out,
but
if
about
halfway
through
what
I'm
going
to
do,
is
I'm
going
to
share
a
link
to
a
forum
along
with
a
codeword
and
you
all
can
go
there
and
we're
gonna
submit
seven
folks
who
can
go
ahead
and
you
know
we're
going
to
have
seven
folks,
pick
them
and
then
give
them
the
send
them
some
swag.
So
that's
something
that
we've
been
doing.
It's
been
a
lot
of
fun,
so
stay
tuned
for
that
I'll.
A
Try
and
do
that
about
halfway
through
so
a
root
says
the
pencil
yeah.
So
this
is
one
of
the
things
that
I
saw
at
at
reinvent.
Everybody
is
doing
enameled
pins
and
we
had
a
bunch
there,
also
and
well,
I.
Think
we'll.
Try
include
that
in
the
swag
pack
and
and
so
we've
been,
we
produce
a
hefty
open
and
a
kubernetes
pin,
but
the
the
the
enamel
pins
are
like
the
new
hotness.
A
It's
like
the
new
sticker,
so
they
were
pretty
much
everywhere
and
reinvent
and
I
expect
we're
gonna
see
a
lot
of
those
at
cube
con.
Also,
okay,
so
a
couple
things
here
so
Priyanka
saying:
could
you
use
the
new
stack
camera
contraption
to
stream
live
from
Q
con?
That
would
be
pretty
cool
I'd
like
to
try
that
out.
I
don't
know
if
I'm
gonna
have
enough
time
to
sort
of
get
one
of
those
and
make
it
work,
and
you
know-
and
usually
you
know
any
sort
of
live
streaming
from
a
conference.
A
It's
the
sort
of
you
know:
Wi-Fi
hostile
environment.
So
that's
one
of
the
things
where
we
didn't
want
to
go
through
and
and
make
a
bet
on
that
without
without
being
sure
that
it
would
work
and
then
so,
how
do
you
think
VMware
is
going
to
utilize
the
amazing
engineering
team
at
hefty,
oh
they're,
going
to
utilize
us
well,
but
I
think
also
we're
going
to
utilize
them.
I
think
one
of
the
reasons
why
we
did
this
acquisition
is
that
it
makes
sense
from
both
directions.
A
We're
gonna
be
getting
a
lot
of
uplift
by
having
VMware's
resources
with
us
and
they're
already
doing
a
lot
in
the
kubernetes
space.
So
one
of
the
release-
cig
release
chairs,
for
instance,
is
Tim
pepper,
who
works
at
VMware,
so
they're
already
doing
a
lot
of
work.
Open
source
Alex
who's
on
here
is
leading
up
the
open,
fast
project
he
works
for
VMware,
so
I
think
you
know
they're
already
doing
a
lot
we're
in
joint
forces.
A
Hopefully
we
can,
you
know,
help
each
other
out
and
do
more,
but
you
know
I'm
really
looking
forward
to
it.
So
it's
it's
going
to
be
a
it's
really,
a
two-way
street.
How
that
how
that's
actually
going
to
work
all
right,
so
I'm
gonna
keep
moving
here.
Let's
see
I'm
just
to
go
through
here
so
everybody's
into
the
pins
yeah
arun
says
the
only
non
amazon
non
kubernetes
and
ml
pin
is
hefty
sweet.
Thank
you
Arun
and
then
and
then
get
lab
pins.
Okay,
yeah,
so
get
labs
been
doing
some
good
stuff
there.
A
A
A
If
you
all
want
to
get
involved,
the
link
should
be
further
up
in
the
comments
and
what
we're
going
to
be
covering
today
is
this
project
that
Amazon
announced
and
amazon
has
so
many
announcements
that
reinvent
they
announced
some
before
reinvent
they
have
these
like
night
keynotes,
where
they
started
announcing
sup.
This
was
one
of
the
early
things
announced
at
reinvent
I.
Believe
was
like
Monday
night,
because
I
thought
to
dinner
with
my
wife
and
this
started
happening.
She
was
giving
me
the
side
iron,
because
I
started
like
commenting
on
it.
A
This
is
firecrackers.
This
is
a
micro
VM,
so
it's
a
virtual
machine
manager
and
and
I'll
go
into
some
details
on
what
this
is.
But
what
you
can
think
about
is
that
this
is
some
of
the
sort
of
hypervisor
mechanisms.
That's
highly
tuned
for
a
set
of
types
of
VM
workloads
that
are
also
largely
aligned
with
containers
in
some
ways.
So
the
use
case
here
for
like
a
micro,
VM
or
a
container
are
actually
very,
very
similar.
Specifically
it's
something
that
you
want
to
start
up
fast.
A
Is
there
some
sort
good
TLDR,
the
most
important
announcement,
a
tree
event
was
not
fun
at
all,
so
Arun.
If
you
want
to
you,
know
I
folks
that
are
commenting.
They
can't
post
links
but
a
ruin.
If
you
want
to
launder
a
link
through
through
George,
he
can
go
ahead
and
get
that
in
the
notes
and
and
and
here
so
that
folks
can,
if
you
have
a
good
summary
of
like
the
TLDR
for
reinvent
I'm,
sure
folks
would
love
to
see
that
alright,
so
it
so.
C
A
Stands
for
virtual
machine
monitor
it's
essentially
a
hypervisor
system
for
being
able
to
manage
a
small
virtual
machine
and
in
this
case,
the
with
respect
to
firecracker
that
virtual
machine
is
optimized
for
being
very
small
and
resource
efficient
loading
up,
very
fast
and
being
very
secure.
And
so,
let's
see
ok,
so
George
is
putting
a
ton.
The
tunnel
link
in
the
notes.
So
if
you
want
to
go
the
hack
empty,
you'll,
see
them
all
there
and
that's
where
other
folks
can
can
start
to
share
them
a
little
bit
and
then
and
then.
A
But
you
can
also
go
to
this
page
here,
which
is
like
has
a
lot
of
the
the
the
reasons
and
why
AWS
actually
did
this
stuff
and
there's
some?
These
are
a
little
bit
better
than
market
expertise.
I
think
these
are
actually
kind
of
some
useful
diagrams
on
how
this
stuff
works.
Now
firecracker
builds
on
top
of
KVM
and
so
a
lot
of
times
people
when
they
talk
about
the
hypervisor.
They
talk
about
KVM,
but
there's
actually
KTM
is
a
kernel
capability,
but
it
doesn't
do
anything
in
and
of
itself.
A
It
actually
needs
a
user
space
via
mem
to
actually
drive
the
capabilities
that
are
there
Layton
in
the
kernel,
and
now
it's
the
saying
the
Kennecott.
It
has
a
nice
lab
for
running
this
stuff.
I'm
gonna
be
doing
it
from
scratch,
just
because
I
like
the
catechol,
it's
a
good
way
to
get
your
hands
dirty
with
stuff,
but
I
like
to
see
you
know
personally
I
like
to
run
on
stuff.
That
is
that
I
can
really
sort
of
you
know,
feel
and
and
get
my
hands
around
directly,
so
yeah.
A
So
KVM
is
a
virtual
machine
capability.
That's
built
into
the
kernel
and
then
the
vmm
is
the
thing
that
essentially
runs
k,
vm
and
sets
it
up
so
that
it
can
actually
run
in
so
there's
actually
a
carrying
between
a
user
mode
component,
which
is
via
the
vmm
and
the
kernel
components
which
are
KVM
and
so
and
typically,
when
folks
say,
hey
I'm,
running
a
KVM
hypervisor,
it's
actually
queue
yet
QEMU,
which
is
this
project.
That's
been
around
forever.
A
That's
actually
driving
KVM
and
so
QEMU
can
both
do
emulation
with
no
hardware
or
kernel
support,
but
it
also
can
be
used
to
drive
the
k
vm
directly,
and
so
what
you'll
find
is
that
a
lot
of
times
people
when
they
talk
about
hypervisors,
they'll,
use,
k,
vm
and
QEMU,
or
you
know,
QT
k,
vm
driven
by
QEMU,
almost
interchangeably.
So
let
me
go
ahead
and
draw
some
diagrams.
A
Hopefully
this
will
actually
help
folks
out
a
little
bit
and
I
know
that
this
goes
ahead
and
kiemce
you,
q,
yeah,
ok,
ki
mu
is
what
Chris
says
it
will
come
up
with
our
own
fancy
names
here.
Whoo.
This
thing
is
out
of
focus,
come
on
focus
there.
We
go
all
right,
so
so,
generally,
what
you
have
is
you
have
a
machine
and
then
you
know
here's
your
kernel
and
here's
user
mode.
A
Okay,
and
this
is
a
machine
Linux
machine
and
inside
the
kernel,
there's
all
sorts
of
different
devices
and
stuff.
But
one
of
these
things
is
KVM
and
KVM
actually
gets
exposed
through
slash,
dev,
slash,
k,
vm
and
so
that's
a
device
like
a
block
device.
To
some
degree,
I
mean
it's
a
character,
mode
device,
there's
different
types
but
like
a
serial
there's
all
so
this
is
how
essentially
the
the
kernel
presents.
Interfaces
or
API
is
up
into
user
mode
and
then
generally,
what
you
have
is
you
have
something
like
Q.
A
This
will
actually
go
through
and
talk
to
KVM
and
all
the
stuff
that
interacts
with
the
underlying
hardware.
So,
like
let's
say
here,
we
have
the
processor
and
typically
this
is
x86,
but
KVM
can
actually
work
with
other
processors
also,
but
essentially
takes
some
of
these
special
instructions
and
special
capabilities
of
the
processor
to
enable
hardware
assisted
virtualization
and
it
packages
it
up
in
a
useful
way
for
the
for
the
OS
to
be
able
to
be
shared.
A
A
So
that's
that's
a
relationship
between
the
VM
m
in
k,
vm,
and
it
turns
out
that
this
vm
m
logically,
is
pretty
simple.
But
in
reality,
because
of
a
lot
of
the
legacy
that
exists
for
being
able
to
boot,
a
machine
QEMU
ends
up
being
fairly
heavyweight.
Now.
The
way
that
this
works
is
typically
is
is
that
each
of
these
threads
in
here
actually
makes
a
call
into
k.
A
It
goes
through
and
and
deals
with
that
exceptional
thing
right,
and
so
a
lot
of
this
is
dealing
with
typical
devices
like
IO,
like
Network
block
devices
keyboards
that
type
of
thing,
and
so
what
happens
is
that
when
the
the
guest
says
hey
I
want
to
write
a
packet
out
to
the
network,
it
either
through
an
interrupt
or
a
protected
memory.
It
pops
back
out
into
host
mode
and
then
up
to
the
host
mode
to
actually
take
that
information,
depending
on
how
it
interfaces
for
that
device
and
do
whatever
it
needs
to
do.
A
We
mentioned,
run
it
depending
on
how
compatible
you
want
to
be
the
interesting
things
can
be
very,
very
interesting.
Now,
one
of
the
things
that
one
of
the
things
that
firecracker
did
is
it's
a
very,
very
targeted
VMM,
and
so
specifically,
it
only
works
with
Linux
and
it
does
it
in
a
way
where
it
skips
past
all
of
the
lizard
brain
stuff
for
booting
the
machine
and
goes
directly
to
executing
the
kernel.
And
then
the
second
thing
is
that
the
types
of
devices
the
number
and
the
type
of
devices
that
firecracker
implements
is
incredibly
limited.
A
Only
the
stuff,
that's
is
really
really
really
necessary
and
specifically
its
networking
and
storage
are
and
and
I
believe,
I
read
somewhere.
It
was
a
keyboard
with
a
single
key
in
it
right
because
you
need
to
be
able
to
to
send
that
signal
in
so
super
super
limited
interfaces
into
it.
I'm
sure
there's
a
handful
of
their
stuff
like
there's,
always
interesting.
Things
are
on
random
number
generation
and
stuff
like
that,
but
in
terms
of
the
complicated
devices
they
keep
those
things
very
limited,
not
only
that,
but
a
lot
of
times
for
legacy
purposes.
A
The
vmm
will
will
emulate
hardware
for
doing
something
like
a
networking.
So
it's
very
common
for
a
VM
m
to
support
the
e
1000
networking
card
right,
which
is
ancient,
and
it
turns
out
that
those
things
were
built
for
hardware.
They
interface
between
the
the
the
kernel
and
that
hardware
was
built
for
a
hardware
interface,
not
built
for
a
virtual
machine
and,
and
so
it
ends
up
being
relatively
inefficient,
and
it's
easy
to
actually
write
buggy
software
for
doing
that
stuff.
A
It's
easy
to
write
stuff,
that
is
a
security
nightmare
when
you're
trying
to
emulate
a
whole
bunch
of
different
types
of
devices,
and
so
fire
cracker
made
a
couple
of
decisions.
So
the
first
decision
they
made
was
to
limit
the
number
of
devices,
and
the
second
thing
was
that
they
actually
decided
to
use
an
interface
to
those
devices
called
Verdejo
and
so
Verdi.
Oh,
is
a
interface
pattern,
library,
sort
of
a
protocol
between
the
guest
OS
and
the
VM,
that's
optimized
for
virtualization
oftentimes.
When
when
the
guest
is
optimized
for
virtualization,
we
call
that
paravirtualized.
A
Now
it
turns
out
that
Verdi,
oh,
is
common
enough
that
it's
in
upstream
kernels.
Now
you
don't
need
to
do
much
special
with
a
modern
linux
kernel
but-but-but
jettison
the
legacy
types
of
of
devices
is
one
of
those
things
that
makes
fire
cracker
fast,
lightweight
and
more
secure,
okay,
reducing
here
a
single
pin
to
be
more
specific
for
the
keyboard
for
resetting.
So
there's
a
serial
console
and
they're
thinking
about
getting
rid
of
it.
A
So
right
now
it's
Verdi
own
edvard,
I/o
block
the
one
pin
you
know
I
eighty
42
and
then
the
uart,
and
so
the
you
are,
is
useful
for
the
serial
console
and
so
serial
and
you
art
are
the
same
thing:
I,
don't
what
is
you
art
stand
for
us?
I,
don't
know
if
Tama,
so
the
serial
console
is
is
useful
for
getting
debug
logs
out
of
the
kernel.
So
if
your
kernel
crashes,
like
you
know,
it's
it's
figuring
out
what
happened?
A
You
need
something
like
like
the
serial
console
so
that
you
can
actually
get
some
clue
to
actually
what
went
wrong
there.
So
super
super
stripped
down
to
be
able
to
do
this
stuff,
and
so
you
know
Universal
asynchronous
receiver,
transmitter
Chris.
Did
you
know
that
off
the
top
of
your
head,
or
did
you
have
to
look
that
up
I'm
guessing
that
that
she
knew
that
off
the
top
of
her
head?
A
A
Now
you
may
wonder
how
do
I
know
all
this
stuff,
and
the
reason
is:
is
that
I
built
something
very
similar,
at
least
a
team
that
I
was
working
with
built,
something
very
similar
when
I
was
at
Google
and
we
were
starting
up
Google
compute
engine,
it
turns
out
that
the
we
are
also
used
KVM
we
started
out
using
QEMU.
We
decided
not
to
move
forward
with
qmu
as
the
vmm
for
GCE,
for
both
performance
and
security
reasons.
A
It
turns
out
that
qmu,
because
it's
been
around
for
a
long
time,
has
and
has
support
for
all
sorts
of
things
from
a
sort
of
you
know,
threat
surface
area
perspective.
It
really
really
scared
me
as
the
lead
of
that
project,
and
so
we
ended
up
building
out
our
own
vm
m
and
it's
not
quite
as
minimal
as
what
what
firecracker
does,
but
it
does.
A
The
the
other,
interesting
things
here
is
that
the
backend
for
the
network
and
the
block
device
in
firecracker
are
very,
very,
very
similar,
so
it's
like
so
so
essentially
inside
a
firecracker,
and
we
can
try
and
dig
through
the
code
and
find
this
there's
a
loop
that
says:
hey.
Is
there
a
packet?
Okay,
there's
a
packet.
Let
me
deal
with
that.
Oh
do.
I
have
a
packet
like
injecting
like
something
that
I
was
received.
Okay,
let
me
do
with
that
same
thing
with
block
devices.
A
A
So
when
a
packet
comes
from
the
guest,
a
firecracker
turns
around
and
injects
it
into
this
type
of
this
networking
concept
in
the
the
Linux
kernel
called
a
tap
right
which
lets
you
inject
and
it
receive
network
packets
instead
of
dealing
with
sockets
okay,
so
it
doesn't
it's
just
injecting
and
getting
package
directly.
So
that's
a
tap
device
and
similarly
with
storage,
it's
just
has
a
disk
image
backed
by
a
file.
So
it's
essentially
a
file
containing
an
exe
for
image
is,
is
what
they
typically
use.
Okay,
so.
A
Time
the
the
the
GC
vmm
grew
a
lot
of
features,
and
it
wasn't
quite
as
simple.
You
know
these
things
don't
survive
the
simplicity
when
you
have
to
add
features
everything
from
supporting
Windows
to
supporting
live
migration
to
supporting
all
sorts
of.
You
know
esoteric
ways
of
doing
disk
and
networking
and
stuff
like
that,
so
those
things
tend
to
tend
to
grow
over
time.
A
One
of
the
things
that
that's
great
about
firecracker
is
that
is
pared
down
to
everything
as
possible,
all
right,
so
that
is
oh
and
then
there's
other
features
that
are
really
interesting
and
I.
Think
you
know
looking
at
this
diagram
here,
is
that
the
way
that
you
deal
with
it
is
via
a
RESTful
API,
so
it's
meant
to
be
programmable
meant
to
be
a
building
block
from
the
get-go
I
think
this
is
one
of
those
interesting
things
early
on
with
containers.
A
The
fact
that
that
that
LXC,
which
was
the
original
sort
of
container
underpinning,
was
not
meant
to
be
programmed
created
a
lot
of
headaches
as
you
started,
building
things
like,
say
docker
on
top
of
it.
So
the
fact
that
firecracker
was
built
to
be
programmed
by
whatever
from
the
outside
is
really
great
from
the
get-go
there's
built
in
rate,
limiting,
because
one
of
the
things
you
want
to
do
is
make
sure
that
none
of
your
guests
can
actually
take
things
over.
That's
really
interesting
and
then
there's
a
metadata
service
which
is
really
fascinating.
A
So
when
you
put
up
an
ec2
VM
or
a
GCE
VM
or
a
lot
of
other
VMs,
you
get
access
to
this
link
local
address,
which
is
like
192
dot,
168
I
want
I,
probably
got
it
wrong.
I
always
forget
it.
But
and
then
you
can
just
talk
HDTV
this
thing
and
you
get
a
bunch
of
context,
and
this
is
how
you
can
we
use
this
in
GCE
to
get
your
SSH
keys
in.
We
use
this
to
get
all
sorts
of
startup
scripts
and
stuff
like
that
in
there
similar
things
are
done
with
with
ec2.
A
So
the
fact
that
that
metadata
service
so
that
you
can
get
context
into
the
VM
when
it
boots
up
is
there
from
the
get-go
is
really
really
interesting
and
that's
actually
probably
one
of
the
more
complicated
parts
of
this
entire
stack
is
because,
what's
coming
out
of
the
guest,
our
packets,
but
to
actually
deal
with
the
metadata
service,
you
need
TCP
streams,
and
so
there
is
code.
That's
part
of
this
that
converts
packets
to
TCP
to
a
stream
right,
which
means
it's
a
there's.
A
user
mode.
Tcp
stack.
A
That's
part
of
this
metadata
metadata
service,
which
is
it
which
is
which
is
actually
a
fairly
tricky
things
to
do
when
we
were
doing
it
with
GCE.
We
started
looking
at
purpose-built,
TCP
stacks
that
were
built
for
embedded
systems
because,
to
some
degree
this
this
is
kind
of
an
embedded
system
and
I
know
that
the
one
we
picked
caused
all
sorts
of
headaches
for
folks
and
they
had
to
rip
it
out
and
rewrite
over
time.
So
yeah
all
right.
So,
let's
see
so
so
a
couple
things
here.
A
So
firecracker
is
40
thousand
lines
of
code,
including
the
auto-generated
system.
Call
bindings
and
I'm
sure
that
that
qmu
is
is
a
lot
larger
than
that.
George
is
asking
the
underlying
VM
for
AWS
is
n,
it
is
n,
but
I
also
think
there's
some
KVM
in
there
now
I
think
they've
been
talking
about
that
they've
been
switching
things
up
over
time
and
then
there's
going
to
be
small
changes
over
the
coming
months,
with
with
more
feedback
in
terms
of
a
firecracker
for
sure.
A
Okay,
so
I've
been
talking
a
lot
here
and
okay,
so
so,
let's
see
I'm
seeing
down
here,
so
it's
160
9.2
54.1
69.2
54,
which
is
link
local
address.
That's
what
ECD
ec2
uses
and
I
copied
that
for
GCE,
because
well
I
change
it
and
then
yeah
and
I
assume
that
the
same
thing
happens
with
with
firecracker
I
am
one
of
things
I'd
like
to
see
I
didn't
see
in
the
documentation
is
how
do
you
actually
program
that
metadata
server?
How
do
you
give
it
access
there
and
how
dynamic
is
that?
A
Because
that
I
think
is
really
really
interesting?
Okay,
a
couple
other
notes
here,
interesting
stuff,
to
keep
in
mind:
firecracker
is
written
in
rust,
which
is
a
really
interesting
choice.
It's
a
great
choice
for
this.
We
wrote
the
GCD
and
C++
just
because
that
was
the
thing
to
do
at
Google
and
there
was
so
much
support
around
it.
Rust
is
a
great
choice
here,
because
it's
it's
a
much
safer
memory,
safe
language,
which
is
one
of
the
the
scariest
things
when
you're
doing
this
type
of
stuff
and.
A
But
it
doesn't
use
a
traditional
GC,
and
so
you
can
do
things
that
are
more
sort
of
real-time
II.
So
it's
it's!
It's
it's
more
system
Z
in
some
ways
than
go
in
terms
of
dealing
with
low-level
stuff.
It
also
has
a
lot
of
sort
of
a
much
more
interesting
type
system
than
go-go
is
like
dead.
Simple
right
go
is
a
sort
of
you
know.
A
A
A
They
didn't
know
that,
but
that's
the
case
all
right,
all
right
solo
and
then
the
other
thing
is
that
firecracker
is
actually
a
it's
an
I
hesitate
to
say
fork,
but
it's
built
upon
the
Chrome
OS
virtual
machine
monitor.
So
this
was
a
project
that
was
built
for
Chrome
OS
so
that
you
can
run
VMs
on
Chrome
and
written
in
rust.
A
The
purposes
of
these
projects
are
pretty
different,
chrome,
there's
stuff
down
here
like
okay,
you
know
how
do
you
you
know:
here's
the
emulated,
there's
like
they,
they
emulate
Wayland,
which
is
the
sort
of
UI
Thanks.
So,
like
the
set,
you
know
the
the
use
case
here,
for
these
things
are
relatively
different
and
so
I
think
it
makes
sense
to
to
specialize
something
for
exactly
what
what
fire
cracker
needs.
But
I
also
think
that
that
it's
it's
interesting.
That
this
was
forked
from
or
derived
from
the
LaCrosse
VM
stuff
yeah.
A
A
A
So
there's
a
it's
a
building
block,
there's
a
lot
of
gaps
to
fill
yet,
but
from
a
sort
of
you
know,
you
know
boxes
on
a
white
board
capability
type
of
thing,
I
think
there's
a
lot
that
can
be
done
here.
Okay,
so,
let's
see
what's
the
killer
use
case
for
a
prior
cracker
so
for
for
AWS,
it's
things
like
lambda
or
their.
A
You
know
what
is
it
called
the
container
individual
container
instances
that
names
escaping
me
right
now,
where
they
essentially
want
to
be
able
to
do
a
really
fast,
really
lightweight,
relatively
short-lived,
a
piece
of
guest
code
that
you
don't
trust
right
because
you're
inviting
anybody
to
run
there.
So
you
want
high
degrees,
isolation
of
high
security,
stuff,
a
small
surface
area,
and
you
want
to
be
able
to
contain
that
thing
really
well
with
respect
to
the
impact
on
the
rest
of
the
system.
A
Far
gate
far
gates,
the
name
of
I
was
look
yeah
and
so
far
gate
is
a
sort
of
you
know,
container
at
a
time
type
of
thing
for
ECS
and
eventually
for
eks.
Also,
okay,
so,
let's
see
folks
are
firecracker
may
someday,
be
able
to
integrate
with
kata-kata
is
a
layer
on
top
yeah.
Exactly
so,
Noah's
got
the
details
there,
alright,
alright!
So
that's!
Let's,
let's
just
let's
start
doing
stuff!
Ok
I've
been
talking
for
a
while
here,
oh
let's,
before
we
go.
So
what
I'm
going
to
do
here
is
if
we
J
dot
Pepto.
B
A
Can
register
to
get
some
hefty
o
swag?
The
code
word
is
boom
Bo
om.
So
if
you
want
to
put
that
in
there
and
we'll
pick
seven
folks
at
random
and
send
you
off
some
swag
boom,
because
a
firecracker
get
it
alright,
so
go
ahead.
Put
your
name
in
there.
We
we
try
not
to
spam,
you
too
much
with
the
email
and
you
can
always
unsubscribe
and
for
every
good
email
that
we
get.
We
actually
donate
some
money
to
black
girls
code,
so
that's
actually
kind
of
cool,
alright,
so
boom.
Alright.
A
A
A
In
fact,
with
GCE
we
started,
we
started
building
GCE
on
AMD
processors,
but
by
the
time
we
shipped
we
moved
to
to
Intel,
mostly
so
that
we
could
get
the
hardware
encryption
instructions,
okay,
so
nested
virtualization,
here,
there's
instructions
on
you
got
to
do
this,
like
fancy
dance
on
GCE,
to
create
your
own
disk
image,
to
be
able
to
run
something
with
an
estimate.
Realization
I
did
this
yesterday
and
got
it
work
and
so
we're
just
gonna
launch
something.
A
So,
let's
see
so
there's
a
three
metal
and
other
metal
instances,
comings,
okay,
so
there's
smaller
or
more
targeted
instances
coming
okay,
so
just
until
now,
but
AMD
an
arm
is
coming
up
in
terms
of
being
able
to
do
to
do
virtualization
there.
Also,
okay,
alright,
so
I'm
gonna
go
through
them
and
launch
a
GC
instance
and
the
Amazon
folks,
you
can
your
eyes
wear
a
blindfold
here.
We're
gonna
call
this
thing
called
firecracker
we're
gonna,
give
a
name
to
our
instance.
We're
gonna
run
it
in
US
West.
One
B
will
do
one
CPU.
A
We
have
to
do
a
customized
here.
You
have
to
go
through
and
say
that
I
think
we
need,
like
you
can
run
on
broad
Weller
later
I
have
to
change
the
image
because
it
does
Debbie
and
by
default,
I
have
an
abundance
of
virtualization
image
and
I'm
going
to
do
a
200,
gig
standard,
persistent
disk
for
that,
and
then,
let's
see
any
other
things
that
we
want
to
do
here,
I,
don't
think
we
need
any
of
the
other
Voodoo
going
on
here.
A
A
This
nested
virtualization
thing
also
rears
its
head
when,
when
you're
running
Windows-
and
you
want
to
run,
say
docker
inside
of
Windows
right,
because
the
way
that
Windows
and
Mac
OS
for
that
matter,
but
nobody
runs
Mac
OS
and
a
VM
typically
the
way
that
Windows
docker
for
Windows
works.
For
you
know,
Linux
containers
is
to
actually
run
a
VM
behind
the
scenes,
and
so
so
that's
one
of
the
other
places.
Okay,
so
my
VM
here
is
launched,
see
that
quick
startup.
Do
you
see
how
fast
that
launched
Amazon
folks,
that
was
awesome?
A
A
A
I
I
think
the
image
for
firecracker
is
just
a
disk
image,
probably
without
a
partition
table
I
believe
if
I'm
looking
at
it
right-
and
we
can
maybe
play
around
a
little
bit
and
see
if
we
can
mount
that
on
a
loop,
loopback
device
we'll
have
to
see
if
we
need
to
use
Capehart
axe
or
not,
let's
see
so
Oracle
cloud
Ravello
now
supports.
You
can
also
use
something
like
packet,
which
is
a
bare
bare
metal,
stuff,
Senna's
KVM
generally
faster
than
Xen
I
mean
it's
out.
A
You
know,
there's
pluses
and
minuses
to
both
the
nice
thing
about.
Kbm
is
because
it
uses
that
user
mode
process.
You
can
mix
both
regular
lentes
proxy
processes
and
VMs
in
the
same
way,
and
you
can
reuse
a
lot
of
the
Linux
capabilities
for
limiting
usage
like
C
groups,
so
you
can
run
a
firecracker
VM
in
a
container.
If
you
give
it
access,
if
you
give
it
the
right
permissions
to
be
able
to
do
it,
which
is
essentially
how
we
ran
GCE.
So
GCE
is
a
container
based.
A
Orchestrator
I
mean
borg
is
a
container
based
Orchestrator.
We
would
have
VM
scheduled
on
the
Borg
and
then
they
would
actually
use
the
GCE
vmm
talking
to
the
KVM
device
to
be
able
to
do
this
stuff.
Okay,
there's
a
default
kernel
and
and
root
FS
images,
and
that's
what
we're
gonna
play
with
here
to
be
able
to
go
okay,
so
the
GCE
SSH
firecracker.
A
A
A
A
A
Now
it
does
get
tricky
with
the
networking
and
stuff,
and
so
that's
something
that
we're
gonna
have
to
figure
out,
but
I
think
firecracker
could
be
integrated
with
kubernetes
and
multiple
levels,
and
arun
is
saying
here
that,
with
respect
to
container
d,
there's
a
container
d
firecracker
bridge
that's
being
built
out,
okay
cool.
So
we
have
that
up
and
running.
A
So
we
have
that
going
I'm
gonna
log
in
GCE,
SSH
firecracker
down
here
also,
so
we
have
two
of
these
things
going
on
and
then
I'm
going
to
start
following
the
getting
started
guide
here
for
firecracker,
now,
I'm
not
going
to
build
it
myself.
What
we're
gonna
do
is
we're
gonna
get
the
firecracker
binary
on
this
change
a
little
bit
because
last
time
I
went
through
this.
It
talked
about
the
the
jailer
stuff,
and
now
we
don't
see
jail
or
here
and
anywhere
at
all.
Okay.
A
The
firecracker
runs
and
it
runs,
but
a
lot
of
times.
You
want
to
wrap
your
vmm
with
extra
layers
of
production,
because
when
you're,
when
you're
as
paranoid
as
a
public
cloud
provider
needs
to
be
it's
all
about
defense
and
depth,
and
so
with
GCE.
We
had
this
rapper
process
that
we
call
the
vmm
nanny
and
it.
C
A
I
left
and,
and
then
that
was
also
rapped
with
the
Borg
lit
managing
the
container
stuff,
so
you're
gonna
have
like
layers
or
protection.
There
is
support
and
it's
not
part
of
the
getting
started
anymore.
For
this
thing
called
jailer
that
works
with
with
firecracker
for
that
extra
set
of
yeah.
So
jailer
is
a
separate
doc,
which
is
great
to
actually
see
those
things.
A
Layered
I
really
really
like
that
they
could
have
kind
of
built
it
into
one
thing
and
had
it
be
like
one
of
those
things
that
reacts
Acutes
with
a
different
parameter,
I
like
that,
these
things
are
really
separate.
Alright,
so
George
here
is
gonna
have
to
leave
us
pretty
soon
because
he
has
a
hard
cutoff
to
go
pick
up
a
kid
he's,
East
Coast.
So
thanks
for
joining
us
George
other
folks
are
gonna
have
to
pick
up
the
slack
on
the
on
the
hack
MD.
A
Let's
see
it,
okay,
so
there's
a
link
there
for
the
firecracker
integration,
stuff,
okay
and
then,
if
I
just
rub
for
the
container
D
stuff
or
if
I
just
run
firecracker,
is
there
a
help?
Look
at
that?
Okay,
there's
not
a
lot
to
actually
be
sent
here,
which
is
kind
of
interesting.
So
the
way
that
it
works
is
I.
Think
we
don't
have
this
socket
yet
here
is
we
just
run
fire
croc
firecracker
and
we
pointed
to
a
socket
okay.
So
how
about
shot
some?
A
A
This
is,
what's
called
a
UNIX
domain,
socket
it's
essentially
something
that
sits
in
the
UNIX
file
system
that
that
you
can
connect
to,
and
it's
a
way
for
two
programs
to
essentially
have
a
data
stream
that
they
can
exchange
between
them
and
so,
instead
of
actually
opening
up
something
on
HTTP
localhost
for
being
able
to
talk
to
firecracker
firecracker.
Actually,
the
API
sort
of
the
the
HTTP
endpoint
is
hosted
over
this
UNIX
domain.
A
Socket
and
docker
does
something
very
very
similar,
and
the
advantage
for
using
a
UNIX
domain
socket
here
is
that
you
can
go
through
and
you
can
use
units
at
UNIX
Ackles
here,
it's
it's
it's
it's
hard
to
reexpose
this
over
the
internet.
You
know
over
the
network
accidentally,
and
so
it's
a
more
secure
option.
A
The
other
thing
that
you
can
do
here
and
I'm
firecracker
is
probably
not
taking
care
of
taking
advantage
of
this
is
that
you
can
there's
some
extra
sort
of
ways
that
you
can
look
at
the
UNIX
domain
sock
and
ask
about
the
identity
of
the
process.
On
the
other
end
of
that,
so
that's
other
ways
that
you
can
build
security
into
this
stuff.
Okay,
so.
A
Any
plans
to
clean
up
the
UNIX
domain
socket
on
and
then
jailer
eventually
I,
don't
know
about
that
limb
adding
maybe
they
know.
Okay
firecracker
can't
clean
that
up
once
it's
jail
because
it
gets
probably
you
know
two
rooted
and
all
that
so
I
can't
even
see
it.
So
it's
it's
probably
something
along
the
lines
of
the
the
jail
or
opens
that
thing
up
and
then
passes
an
FD
down
to
the
to
the
exact
one,
yeah
and
yeah.
Let's
see.
A
All
right
so
I
think
we're.
So
now
it's
it's
it's
running,
and
it's
not
very
impressive
at
this
particular
instant
and
the
reason
why
is
it's
sitting
there
waiting
for
us
to
ask
you
some
questions
and
tell
it
to
do
stuff,
and
so
this
is
one
of
those
things
where
it's
really
interesting,
that
this
stuff
is
built
to
be
programmed
from
the
outside
from
the
get-go.
So
what
we're
gonna
do?
Oh
and
I,
don't
want
to
do
it
in
that
window,
I'm
gonna!
Do
it
down
here.
A
Is
that
we're
gonna
download
two
things
hello,
VM
Linux
dive
in
this
is
a
kernel.
It's
an
uncompressed
kernel
actually
and
then
hello,
root,
F
as
X,
4
and
and
then
the
first
thing
we
do
here
is
we
set
the
guest
kernel
and
it
says:
couldn't
connect
oh
I
accidentally
control
seed,
it!
Oh
now
it's
upset
about
that.
Okay,
so
now,
I
have
to
do
the
RM,
I'll,
restart
it
again.
Okay,
boom,
alright.
A
So
so
what
I
did
here
is
that
and
you
guys
really
need
a
firecracker
cuddle
to
actually
like
using
curl
is
not
not
a
lot
of
fun
here.
So
this
is
like.
If
somebody
out
there
wants
to
go
and
build
firecracker
cuddle,
you
know,
go
for
it.
There's
a
go
Lang
SDK
to
go
with
it,
I'm
sure
somebody's
already
working
on
it.
A
So
the
first
thing
we're
that
we
see
here
is
that
we're
talking
to
this
UNIX
domain
socket
it's
expecting
to
see
it
as
local
host
here
and
we're
passing
JSON
document.
That
says
the
boot
source
is
a
binary
and
here's
the
boot
arguments,
and
so
these
are
a
lot
of
folks.
Don't
know
usually
you'll
see
this
in
grub
or
something
like
that,
but
there's
actually
command-line
arguments
to
the
kernel
itself.
Fire
fire,
cuddle
cattle
cattle
fire
cuddle
does.
A
C
A
This
essentially
said
use
this
kernel
to
boot.
Up
now,
most
of
the
time
when
you
boot
a
machine,
it
reads
the
kernel
that
it's
going
to
boot
from
disk,
but
because
we're
skipping
the
BIOS
as
this
you
need
to
actually
provide
the
kernel
separate
from
the
disk,
and
so
essentially,
what's
going
to
happen
is
firecracker
is
gonna,
read
that
kernel
off
of
disk
write
it
into
memory
and
then
set
the
the
the
the
instruction
pointer
to
be
at
the
start
of
the
kernel.
It's.
A
Really
all
there
is
it's
essentially
you
just
like
slam
that
thing
into
guest
memory,
and
then
you
say:
okay,
when
you
start
executing
start
executing
at
the
start
of
the
at
the
start
of
the
the
kernel
and
so
that
skips
all
the
BIOS
boot
stuff.
Now,
when
we
did
this
with
GCE,
this
was
an
extra
security
mechanism.
We
actually
didn't
let
customers
originally
provide
their
own
kernel
as
a
level
of
paranoia
and
the
kernel
that
we
provided
didn't.
A
Let
you
load
dynamic
modules,
and
so
we
tried
to
actually
control
who
was
able
to
actually
get
access
to
ring
zero
in
the
kernel,
because
when
we
were
looking
at
some
of
the
history
of
guest
escapes
with
hardware
capabilities,
some
of
those
were
only
available
from
ring
zero
and
so,
depending
on
how
paranoid
you
are
and
how
much
you
trust
your
hardware
you
may
want
to
lock
down
further
than
then
by
turning
off.
You
know,
loadable
modules
in
your
kernel,
all
right,
so.
A
C
A
There
any
plans
to
allow
an
in
ardian
in
ram
FS
yeah
I
saw
there
was
a
PR
at
least
a
proposal
on
that.
So
that's
actually
coming.
Ok,
so
the
next
thing
we
need
to
tell
it
is
that
hey.
We
also
need
a
get
guest
block
device
here,
and
so
this
is
a
similar
type
of
thing
here
and
we're
well
we're
gonna
load,
the
drive,
ideas,
route,
FS,
here's
the
path
on
the
host
and
it's
the
route
device,
and
it's
read-only,
that's
as
simple
as
it
is
now.
A
My
guess
here
is
that
there's
no
partition
table
here,
but
we'll
take
a
look
at
that
when
we
put
the
thing
up
and
then
we
actually
tell
the
thing
to
start,
and
so
this
says
go
ahead
and
start-
and
you
saw
here
that
we
saw
Linux
booting
and
here
we
are
and
we
have
a
guest
login
and
the
password
for
this
image
is
root
and
root.
So
woohoo
we
are
there
now
this
is
you
see.
This
is
root
log
on
a
ttyS
0.
This
is
actually
going
over
the
UART.
A
A
So
ok,
so
the
first
thing
you
see
is
that,
like
it'll,
once
in
a
while
spam,
you
saying
like
failed
to
log
metrics,
so
every
60
seconds
this
thing
tries
to
log
metrics.
If
you
don't
initialize
a
logger,
it
gets
mad
at
you
and
it'll
keep
doing
that.
So
you'll
start
seeing
this
stuff
every
60
seconds.
So
here
we
are
so
we
are
up
and
running.
A
A
So
this
is
the
VM.
You
know
not
much
going
on
yet,
but
it's
running
alpine,
which
is
really
small,
and
then,
if
we
look
from
the
outside,
we'll
actually
see
that
it's
just
like
a
regular
old
process
like
everything
else,
so
I
can
go
through
and
like
kill
this
one
and
boom
I
am
terminated
and
and
because
of
the
way
that
KVM
works
is
that
when
you
actually
kill
the
user-mode
process,
the
vmm,
if
that
crash
is
for
whatever
reason
the
kernel
tries
to
clean
up
everything
else
around
it.
A
C
A
A
All
right
and
I
can
do
touch
hi
firecracker,
and
what
you'll
see
is,
if
I
do
another
way
that
you
get
out
of
this
is
like
I.
Don't
know
you
hit
control-c.
Well,
it
sends
the
control-c
in
I.
Don't
think,
is
there
an
escape
key
when
you're
at
the
console
here
you
know
I
could
do
the
e
like
the
SSH
escape
key,
but
that
would
exit
me
out.
So
the
only
way
to
get
out
of
this
is
I
think
you
can
do
a
halt
which
will
actually
cause
it
to
shut
down
or
power.
C
C
A
Actually
cause
it
to
shut
down,
so
there's
no
power
management
device,
so
there
we
go
so
it
doesn't
actually
do
a
reboot
when
you
tell
to
reboot
that
actually
ends
up
being
a
shutdown,
but
what
we
can
do
here
is
you
know
that
got
persisted
into
this
into
this
this
this
device.
So
when
we
boot
this
thing
up
again
and
we'll
just
go
through
and
make
this
easier
like
this,
oh.
A
A
Simple
layered
file
system,
where
you
have
a
root
filesystem,
then
you
keep
track
of
deltas
on
top
of
it.
That
may
improve
startup
time
when
you're
trying
to
initialize
these
things,
depending
on
on
what
you're
doing,
usually
you
need
to
keep
track
of
two
open
files
and
some
sort
of
like
bit
mask
based
on
which
blocks
have
been
written.
Where
and
when.
A
So
you
can
have
a
file
system
on
the
instance
and
then
mount
parts
of
it
into
different
Mike
and
you
aruna
last
I
saw
like
there
was
no
9p
support
and
you
can
overlay
a
ram
disk
on
top
of
the
read-only
root
layer,
yeah,
so
I,
don't
think.
I
saw
that
right
there.
So
one
of
the
things
that
G
visor
does
and
there's
this
protocol.
That's
like
left
over.
It's
ancient
it's
from
from
this
old
OS
called
called
plan
9
that
was
written
by
the
same
folks.
A
That
wrote
go
so
you
know,
there's
your
your
your
history.
There
was
this
thing
called
the
9p
file
system
and
it's
essentially
a
protocol
for
file
events,
it's
kind
of
like
fuse,
but
instead
of
being
used
to
implement
a
file
system
in
user
mode,
it's
used
to
implement
a
file
system
across
something
like
a
Verdi
o-linked,
and
so
that
is
a
common
way
to
essentially
host
part
of
the
host
file
system
into
the
VM.
That's
not
something
that
firecracker
supports.
A
A
So
you
can
mount
a
read-only
root,
filesystem
in
lots
of
micro
VMs,
but
then
they
have
to
keep
all
their
mutations
in
RAM
or
on
another
block
device,
and
that
can
all
happen
in
the
guest
but
yeah
no
support
for
like
a
queue
cow.
There
are
tools
in
the
queue
mu
to
be
able
to
convert
back
and
forth
from
queue
cow
to
2
to
the
X
stuff.
But
again,
there's
no
there's
no
partition
table
here
either.
A
So
that's
something
to
keep
in
mind
all
right,
so
that
is
cool
now,
the
other
stuff
that
I
would
love
to
play
with
and
I
think
you
know
we're
gonna
run
out
of
time
here
and
I'm,
not
sure
where
the
documentation,
so
you
can
build
it.
I
built
it
myself.
It
builds
in
a
docker
container,
so
you
can
build
on
on
a
Mac,
but
you
can
run
it
on
a
Mac.
So
it's
kind
of
like
a
little
useful
useless.
A
The
docker
data
kit
supports
an
IP
failure,
interesting
just
not
sure
about
the
it's
okay.
Society
says:
I
missed
the
beginning
conversation.
Where
do
we
employ
a
firecracker
on
ec2
or
launch
an
ec2
using
fire
car
so
I'm
doing
it
site
on
GCE
because
of
Nestor
virtualization
can
rewind
and
watch
that
you
can
also
do
it
on
any
of
the
the
metal
instances
on
on
on
ec2,
also,
okay,
so
yeah
I'm
just
doing
on
GCE
just
to
poke
the
AWS
guys
a
little
bit.
A
That's
just
how
I
roll
okay,
let's
see
the
other
okay,
so
the
things
that
I
would
love
to
do
that
I'd
love
to
dig
into
if
we
have
time,
there's
jailor,
which
essentially
is
a
thing
that
sets
up
a
bunch
of
other
sort
of
protections
around
it.
So
this
is
sort
of
a
defense-in-depth
creating
a
bunch
of
layered
stuff
around
it.
A
I'd
love
to
figure
out
how
to
get
V
host-based
V
sock
support,
which
allows
one
or
multiple
visa
I
haven't
heard
a
V
sock.
That's
interesting!
All
right!
So
I'm
gonna
have
to
learn
about
that.
I
assume
that
V
sock
is
a
virtualized,
socket
remote,
able
socket
type
of
thing,
and
then,
let's
see,
there's
a
design
doc
here.
This
looks
really
interesting.
This
goes
into
more
detail.
A
lot
of
the
stuff
that
I
that
I
talked
through,
but
probably
more
correct
than
me.
A
B
A
This
is
the
thing
is
that
when
you
do
like
CNI
in
kubernetes,
it
essentially
sets
up
a
network
namespace,
and
you
get
like
a
traditional
network
device
being
able
to
remote
that
and
bridge
that
into
a
VM
ends
up
being
a
little
bit
tricky,
and
so,
if
I'm
guessing
what
V
sock
is,
is
its
paravirtualized
at
a
higher
layer.
So,
instead
of
being
paravirtualized
at
the
at
the
at
the
the
packet
level,
it's
paravirtualized
at
the
at
the
the
the
socket
level.
Now,
there's
pros
and
cons
to
something
like
that.
So
I
wanted
to.
A
Things,
why
would
you
want
to
run
a
VM
instead
of
just
a
plain
old
container?
It
is
heavier
weight
than
a
container
and,
and
it
can't
take
longer
to
boot.
Up
I
mean
this
thing
is
really
fast,
but
it
had
to
go
through
the
entire
guest
boot
process
to
be
able
to
get
there.
And
if
you
look
here,
it
was
somewhere
on
the
order
of
one
and
a
half
seconds
to
go
from
zero
to
a
prompt
right
to
be
able
to
boot.
A
A
What
drove
this
is
you
use
it
over?
Containers
for
lambda,
I
think
yeah.
So
there
is
a
security
consideration
there.
Also
the
attack
surface
with
the
Linux
Cisco
interface
is
a
heck
of
a
lot
bigger
than
the
attack
surface,
with
the
VM
and
they're
different
and
to
some
degree
one
of
the
things
is
like
you
know
why
not
both
right,
so
you
can
run.
A
You
can
run
something
like
firecracker
inside
of
container
a
container
or
container
technologies
like
C
groups
and
namespaces
and
set
common.
That's
a
lot
of
the
stuff
that
the
jailer
process
does
so
to
some
degree.
There's
like
hey
hey.
Why
not
both
type
of
thing
so
Michael
is
asking
about
divergence
with
this
in
the
in
the
the
the
chromium
OS
V
mm.
I.
A
Think
a
lot
of
it
if
you
go
back
and
watch
the
beginning,
I
was
talking
about
a
lot
of
the
the
paring
down
of
the
devices,
so
that
was
really
focused
to
the
bare
minimum
necessary
for
a
server
workload.
I'd
love
to
see
these
things
actually
converge
in
some
way
back
together
again
at
some
point.
I
think
that
would
be.
That
would
be
really
really
interesting
in
my
mind,
okay,
so
the
next
thing,
let's
see
if
we
can
dig
into
one
more
thing
here,
I
would
love
to
look
at
the
rate.
A
A
So
if
you
want
to
learn
about
this,
the
API
I
think
probably
the
canonical
thing
would
be
the
the
open,
API
specification,
which
is
here
so,
if
there's
better
documentation,
that
I
should
be
looking
at
AWS
folks,
I'd
love
to
see
it
and
I'm
sure
this
is
the
type
of
thing
just
announced
to
us.
It's
breaking
out
of
containers,
our
real
concern
or
more
academic
I
mean
yes
and
no
I.
Think
so.
Jesse
Frisell
has
a
sort
of
like
a
capture.
The
flag
container
thing
that
super
lockdown.
A
That's
worth
tape,
checking
out
I
last
I
heard
nobody's
broken
out
of
that,
but
I
think
there
are
reasons
why
running
your
own
kernel
per
workload
is
actually
a
good
thing.
There
are
certain
things
say
in
the
Linux
networking
staff
stack
that
are
not
contained
where
there's
a
there's,
a
global
number
of
X
that
can
be
there
and
that
ends
up
being
shared
across
a
whole
bunch
of
of
containers
and
so
running.
Multiple
containers
in
that
are
totally
antagonistic
and
antagonistic
to
each
other.
A
You
know
it's
much
easier
to
sort
of,
say:
yeah
contained
out.
Af
is
our
thing
yeah.
It's
a
host
in
Afghanistan
as
far
as
I
can
tell,
as
you
know,
there's
you
know,
there
are
ways
to
do
s
stuff,
cross
container
and
so
a
lot
of
times.
Containers
work
well,
in
my
mind
in
a
multi
team
scenario,
where
generally
people
aren't
trying
to
to
to
kill
other
people.
If
somebody
does
something
bad,
you
call
them
up
and
you're
like
hey,
stop
doing
that
like
so,
the
containers
can
work
really
well
in
that.
A
You
start
getting
some
noisy
neighbor
types
of
things.
Typically,
you
know
with
something
like
this
and
I
couldn't
see
it
there.
It
looks
like
the
the
the
the
sink
coming
from
vireio
block
was
being
actually
passed
onto
the
filesystem.
I
was
surprised
that
that
wasn't
being
neutered,
but
that
is
one
of
the
things
that
actually
is
being
written
out
in
the
in
the
the
metrics
so
yeah,
okay,
so
I
keep
referencing
the
metadata
servers.
These
ec2
metadata
servers.
A
Isn't
it
missing
because
you're
running
this
on
GC
P
or
am
I
missing
something
so
so
GC
p
GC
also
has
a
metadata
service
with
a
lot
of
cool
features
that
don't
exist
inside
of
ec2
like
it's
dynamic
and
we
can
update
it
on
the
fly
and
you
can
actually
have
processes
that
can
do
a
hanging
get
against
it.
So
you
can
update
it
right
away.
A
It's
the
way
that
we
inject
SSH
keys
dynamically
into
GC
things
so
I'm
just
talking
about
GCE
there,
but
also
there's
a
metadata
service
exposed
into
firecracker,
hey
Wellington
from
Berlin
how's.
It
going.
Can
you
boot
these
things
into
a
C
group?
You
can?
You
can
run
firecracker
inside
of
a
C
group
and
I
believe
that
that's
something
that
Beeler
does
and
then
there's
no
reason
why
you
couldn't
use
cgroups
further
inside
of
this.
It
depends
on
whether
those
things
are
compiled
in
your
Gaston
carnal
or
not.
A
Okay,
yeah
all
right.
So,
let's
see
so,
let's
go
through
meta
data.
Okay,
so
the
creates
a
microvia
metadata
service
data
store
and
let's
say
you
can
patch
this.
You
can
get
it
and
then
the
patch
you
can
put
it
the
body
is
schema,
object.
So
I,
don't
know
what
that
body
is
supposed
to
look
like
all
right.
So
there's
no
documentation
of
the
body
of
this
stuff.
A
Does
that
mean
I,
just
whatever
I
put,
there
is
what
can
I
and
kind
of
update
this
after
it's
booted
I,
wonder
let's
actually
play
with
this
okay,
so
here
we
went
through
and
we
actually
we
have.
This
thing
up
and
running
can
I
go
through
and
I'm
gonna
do
like
a.
Let
me
let
me
go
to
here
and
I'm.
Gonna.
Sorry
give
me
a
second
here.
We
have
to.
A
Yolo,
it
will
make
it
work
so
so
I'm
constructing
sort
of
a
curl
line
that
I
can
actually
cut
and
paste,
and
this
is
the
type
of
thing
where
you
know
this
is
kind
of
painful
so
having
this
stuff
having
this
stuff
in
behind
a
some
sort
of
fire,
cuddle
actually
would
be
nice
and,
let's
see
it
so
I
do
I
need
okay,
I,
don't
need
it
inside
of
the
quotes
here.
So
what
we're
gonna
do
mmds
right
is
is
the
thing
that
we
need
there
and
we're
gonna,
say
hello.
A
A
If
you
want
to
run,
do
if
you
send
it
through
through
Chris
Nova,
she
can
probably
post
it
here.
No
Mandy's
super
curious.
How
you
get
arbitrary
code
uploaded
to
lamda
validated
and
running
inside
a
firecracker
VM
yeah.
You
could!
Probably
you
know
you
could
host
that
on
another
disk
device.
You
could
host
that
in
a
you
can
make
it
available
through
the
metadata
server
stuff
like
that,
you
need
to
create
the
immense
before
you
start.
Okay,
but
then
you
can
update
it.
B
A
A
Okay
Julian
says:
maybe
it
stupid
question,
but
it
seems
like
a
lot
of
work,
is
being
put
to
isolate
secure
software
workload
with
software
any
way
to
change
the
hardware
it
might
help.
Yet
so
Julian
I
mean
like
that's,
not
a
stupid
question.
The
reality
is,
is
that
the
hardware
is
helping
right.
There
is
hardware
under
the
covers.
That's
really
accelerating
being
able
to
do
this
type
of
virtualization.
A
Okay,
so
there
is,
you
can
put
this
so
something
so
somehow
it
didn't
like.
Well,
there's
a
lot
of
good
detail
here.
Okay,
for
some
reason,
it
didn't
like
that.
My
hello
world
thing
I
got
a
couldn't
connect
to
server,
so
so
the
errors
aren't
here
are
great
here,
because
this
thing
gave
me
a
201
created
and
it
didn't
validate
this
body
at
all,
but
somehow
something
didn't
work
as
I
did
it.
A
A
A
A
A
A
C
A
A
Firecracker
itself,
that's
interesting,
so
jailer
copies
firecracker
so
that
you
can
upgrade
firecracker
without
actually
screwing
up
stuff.
That's
already
there
so
listen
to
mileage.
What
am
I
trying
you
don't
trying
to
get
the
metadata
sort
of
working
inside
the
VM,
so
the
jailer
doesn't
do
any
configuration?
Ok!
Well,
that's!
Ok!
I
think
you
know
we're
running
out
of
time,
so
so
a
couple
other
things
that
I
think
are
worth
exploring
is
and
I'm
sure
the
docs
will
be
coming
over
time.
It's
just
tough
to
sort
of
figure.
A
A
Take
a
look
at
the
the
metrics
that
are
coming
out
because
I
think
that's
super
interesting
and
then
the
rate-limiting
I
think
I
saw
some
reference
in
there
when
I
was
looking
at
the
code
that
the
rate-limiting
uses
a
token
bucket
type
of
mechanism.
That's
really
really
interesting
and
I
think
understanding
how
that
works
will
help.
You
understand
how
cloud
services
typically
rate
limit
types
of
these
types
of
things
and
I'd
love
to
see
sort
of
what
things
can
you
rate
limit,
which
would
be
kind
of
fun?
A
A
Needed,
like
you
know,
like
a
lot
of
open
source
projects
that
it's
there,
but
I'm
and
I'm
sure
inside
of
Amazon,
there's
a
whole
bunch
of
documentation
how
this
stuff
works,
but
I
think
it's
probably
pretty
entwined
with.
How
does
firecracker
relate
to
other
systems
at
Amazon,
so
cleaning
that
up
getting
it
to
the
point
so
that
it's
generic
enough
for
other
people
to
use
is
always
going
to
be
going
to
be
a
lot
of
fun.
So
alright,
so
lawson
topology
is
asking.
Will
there
be
another
podcast
about
this?
A
A
All
the
docs
okay,
so
inside
of
amazon-
if
you
want
about
this
start
hitting
these
guys
up
directly
outside
of
Amazon,
you
know
maybe
volunteer
to
help
write
the
docs
and
they
can.
They
can
maybe
pass
you.
The
information
I
think
Doc's
are
always
a
great
way
to
get
started
in
a
community
for
sure,
and
you
can
read
the
code
right.
You
know
it's
self-documenting
code,
loss
and
apology.
Will
there
be
another
podcast
about
this?
A
I
may
do
another
TGI
Kay
on
this
it'll
probably
be
a
while
from
now
we
have
a
pretty
big
backlog
of
stuff
that
we
want
to
do.
This
will
be
recorded
on
on
YouTube,
and
so
you
know,
as
things
continue,
obviously
the
docs
will
get
better
and
and
there'll
be
more
to
more
to
explore,
but
yeah
any
last
questions
before
I
go
ahead
and
sign
off
here.
I'm,
you
know
really
interesting
stuff,
I
I
think
it's
it's
great
to
see.
A
Building
blocks
like
this
and
I
think
this
is
for
me,
I
think
I,
look
at
this
I
and
then
I.
Look
at
other
things,
like
you
know,
container
D,
I,
look
at
things
like
envoy
and
for
me,
I
love,
seeing
things
that
are
built
to
be
built
upon
I
mean
that's
actually
one
of
the
one
of
the
ideas
with
kubernetes
itself
now.
A
Obviously,
kubernetes
is
a
much
bigger
system
than
this,
but
we
built
kubernetes
from
the
get-go,
assuming
that
people
are
going
to
build
more
layers
on
top
of
it
and
that's
something
that
I
really
really
think
is
cool,
so
yeah,
so
I
believe
that
the
these.
So
the
question
is:
is
there
the
Firecracker
slack
public?
If
you
go
to
the
bottom
here,
there
is.
A
Let's
see
the
slack
workspace
and
then
most
of
the
maintainer
czar
on
a
european
time
zone
all
right,
so
it's
a
global
project
or
at
least
global
as
in
not
in
the
US,
and
so
you
can
go
to
the
slack
workspace
there.
Let's
see,
where
does
that?
Take
you,
oh
yeah,
so
these
guys,
you
guys
are
so
I'm
just
advice
to
the
slack
team.
The
link
that
you're
using
here
will
wear
out
over
time
because
those
things
can't
last
forever,
and
so
you
need
to
set
up
something
like,
like
slack
gates,
do.
A
A
Let's
see
all
right,
hopefully
so,
how
does
firecracker
compare
to
Linux
KITT
I'm?
Not
an
expert
but
Linux
KITT
is
essentially
a
way
for
building
and
packaging
sort
of
minimal
Linux
distributions
that
are
purpose-built,
so
I
think
it's
probably
you
know
it
might
be
possible
to
actually
run
Linux
KITT
VMs
with
fire
cracker
over
time.
They
may
not
create
images
in
the
right
format.
That's
something
that
I
think
is
probably
worth
looking
at.
A
Your
fire
cracker
might
be
something
that's
appropriate
for
running
micro,
kernels
and
in
that
or
you
know,
kernels,
and
that
it's
like
there
is
nothing.
You
know
you
don't
need
a
there's.
Nothing
beyond
that
that
that
the
workload
is
the
kernel
that
you're
running
so
there's
just
one
thing
that
gets
loaded,
I.
Think
that's
going
to
be
something
that's
going
to
be
interesting
over
time.
A
A
Yeah
people
have
run
unit
kernels,
so
I
think
that's
going
to
be
interesting
over
time
and
I.
Think
it's
going
to
be
interesting
to
see
again.
You
know
a
lot
of
this,
so
much
of
this
stuff
is
defense-in-depth,
so
we
start
seeing
things
like
like
web
assembly
running
inside,
of
something
like
like
firecracker.
Oh.
A
Talk
about
is
the
comparison
between
firecracker
and
gee
visor.
So
about
a
year
ago,
at
the
last
cubic
on
Google
announced
this
project
called
G,
visor
and
geo.
Visor
is
something
that's
kind
of
in
between
a
container
and
a
VM.
It
can
reuse
the
KVM
to
actually
do
its
thing,
but
it
doesn't
run
a
kernel
in
the
traditional
way.
A
Instead,
what
it
does
is
it
it
sort
of
para
virtualizes
everything
all
up
the
stack
by
reimplemented,
a
lot
of
the
syscall
interface,
and
so
in
some
ways
it's
even
lighter
weight
because
it
doesn't
have
a
traditional
Kuril
running,
but
it,
but
in
doing
so
it's
actually
reusing
some
more
of
the
the
host
kernel
in
some
way.
So
it's
it's
another
sort
of
click
stop
in
the
spectrum
here.
G
visor
is
super
super
interesting
from
a
technology
point
of
view,
but
you
know
from
a
compatibility
point
of
view.
A
A
There
we
go
check
it
out
there
and
so
check
that
out
the
code
word
there
is
boom,
as
in
what
of
the
noise
a
firecracker
makes
and
we'll
see
you
all
next
week.
Thank
you
so
much
for
joining
in
and
it's
going
to
be
a
lot
of
fun,
alright
and
then
next
week
and
then
there's
Q,
Khan
and
so
Q
Khan
is
going
to
be
super
exciting.