►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Toolchains Behind Successful Kubernetes Development Workflows - L Körbes, Tilt
Kubernetes solved a lot of problems, but it created a clumsy development workflow: Every code change requires fiddling with containers, registries, and manifests. Managing config files isn't trivial. Distributed debugging; a mystery. Dev clusters are tricky to set up, and sharing cluster state among team-members is mostly fiction. L Körbes, an expert in Kubernetes development tooling, outlines successful development workflows in three different settings: a very large enterprise, a small and agile startup, and a popular open source project. L will share how they set up dev clusters, manage configs, automate the development feedback loop, share context across teams, debug, and, finally, deploy to production. Learn how these teams made their Kubernetes dev workflows not only seamless, but amazing to use!
https://sched.co/Zet3
A
Hello,
everyone
welcome
to
successful
kubernetes
development,
workflows,
I'm
al
I'll
stay
here
in
the
corner
before
we
start
talking
about
kubernetes
development,
workflows,
I'd
like
us
to
take
a
quick
detour
and
talk
about
ids
just
for
a
second,
so
I
assume
most
of
you
are
familiar
with
ids
you
learned
to
code
using
one
or
you
didn't,
but
you
learned
about
them
afterwards.
Either
way
you
know
kind
of
what
ids
are
now.
A
What
ideas
do
is
that
they
address
a
bunch
of
problems
that
developers
wanted
solutions
for
so
they
provide
you
with
syntax
highlighting
code
completion,
a
bunch
of
stuff
like
that-
and
these
things
are
basically
so.
The
I
and
ide
stands
for
integrated
and
an
id
is
basically
a
collection
of
solutions
for
a
problem
set
that
is
common
among
developers
when
they're
trying
to
code
now
me
particularly,
I
have
an
interesting
relationship
with
ides
and
that's
because
I
didn't
know
they
existed
for
the
longest
time.
A
So
for
many
many
years
my
ide
was
this
and
I
wrote
rootkits
and
c
using
notepad,
I
wrote
my
own
wordpress
before
wordpress
came
along
in
php,
also
using
notepad,
and
I
didn't
know
because
I
learned
to
code
basically
from
books
and
I
lived
in
a
small
city.
I
didn't
have
any
friends,
so
it
took
many
many
years
for
me
to
actually
learn
that
ides
existed,
but
I
mean
it
was
fine.
I
could
get
the
job
done.
It
was
just
a
lot
less
convenient
than
it
could
have
been
otherwise.
A
So
let's
tie
this
back
into
kubernetes,
so
lots
of
us
here
watching
this
talk
at
kubecon.
I
would
bet
that
the
good
bunch
of
you
watching
here
I
would
bet
that
a
good
amount
of
you
can
write
kubernetes
ammo
by
hand
from
memory
and
it
would
run.
I
bet
a
good
chunk
of
you
know
everything
there
is
to
know
that
is
relevant
about
containers
and
kubernetes,
and
so
many
other
technologies
like
that.
Basically
we
are
the
people
coming
to
kubecon.
We
know
all
the
stuff.
A
Some
of
you
even
know
what
this
is
and
if
you
don't
don't
google
it
because
you're
gonna
have
nightmares,
but
if
you
do
know
this
stuff,
you
know
what
I'm
talking
about
and
the
thing
is
we
learned
to
code
using
the
equivalent
of
notepad.
We
learned
to
do
things
by
hand,
because
when
we
started
we
had
no
choice,
but
that's
not
the
case
anymore.
A
So
what
I'm
going
to
do
here
is
I'm
going
to
tell
you
I'm
going
to
tell
you
the
most
common
problems,
so
I'm
going
to
give
you
the
problem
set
and
I'm
going
to
tell
you
how
many
different
companies,
teams
and
projects
solve
those
problems.
But
the
thing
is
we're
not.
We
don't
have
the
eye
in
ide,
yet
it's
not
all
integrated.
So
I'm
going
to
give
you
a
bunch
of
solutions
and
you're
going
to
pick
them
yourself.
A
So,
let's,
let's
get
to
it,
I'm
al.
If
you
want
to
see
that
picture
there,
you
go.
I
work
at
tilt
previously.
I
worked
at
garden
they're.
Both
companies
focused
on
the
development
experience
side
of
kubernetes,
and
that
means
that
half
my
job
is
listening
to
developers
complaining
about
their
development,
workflows,
working
to
solve
them
and
the
other
half
of
my
job
is
basically
coming
here
and
sharing
those
findings
with
you.
A
I'm
going
to
repeat
this
slide
again
at
the
end,
if
you
want
to
get
all
the
contact
information
now,
what
are
the
problems?
The
problems
are
one
development
clusters,
two
managing
both
load
of
config
files,
three
getting
feedback
after
you
write
your
code,
four
sharing
cluster
context
and
I'm
gonna
explain
what
this
is.
When
you
get
there
and
extras.
You
know
this
word.
It
means
extras:
okay,
protagonists,
I'm
going
to
tell
you
about
how
these
groups
fixed
those
problems.
A
They
are.
One
is
datadog
cloud
monitoring,
sas
provider,
I'm
sure
most
of
you
know,
datadog
their
engineering
team
has
about
800
developers,
I'm
going
to
talk
about
uno,
which
is
a
smaller
but
not
tiny
company,
so
they're,
a
manufacturer
of
electric
scooters
in
berlin,
their
scooters
have
internet.
I
don't
know
how
that
works,
but
yeah
their
team
is
about
25.,
I'm
going
to
talk
about
mindspace,
their
creative
learning
and
gamification
agency.
Very
tiny.
Their
development
team
is
far
people.
A
So
here
we
have
examples
of
three
different
magnitudes
of
problems
and
solutions,
and
we
also
have
cluster
api
and
we
have
probably
a
bunch
of
friends
from
cluster
api,
presenting
at
kubecon
right
now
and
with
cluster
api.
Basically,
you
can
have
a
management
cluster
that
you
use
to
create
and
manage
other
clusters.
So
you
have
your
management
clusters
that
you
use
to
create
a
cluster
on
google
and
amazon
and
microsoft,
and
it's
like
you
manage
clusters.
The
way
you
manage
kubernetes
resources,
that's
basically
the
idea.
A
A
This
is
a
funny
joke
my
friend
made
when
working
in
cluster
api,
and
the
point
is
that
you
can
run
your
app.
You
can
run
your
app
in
a
container.
You
can
run
your
app
in
a
container
in
a
vm
and
then
you
can
run
your
app
in
a
container
in
a
vm
and
that
app
is
going
to
create
other
containers
and
vms
that
runs.
Apps
and
yeah.
Cluster
api
sometimes
takes
some
efforts
to
understand,
but
anyway,
let's
move
on
development
clusters.
A
So
if
you're
not
using
a
development,
cluster
you're,
basically
developing
a
notepad,
because
you're
writing
code,
that's
supposed
to
run
in
a
cluster
if
you're
developing
it
outside
the
cluster
you're,
going
to
end
up
with
discrepancies
and
that's
annoying
and
that's
hard
to
track
down,
and
you
don't
want
your
developers
to
have
to
deal
with
it,
so
you
should
just
use
a
development
cluster.
Now
there
are
two
types
you
can
use.
You
can
use
a
local
cluster.
You
can
use
a
remote
cluster.
A
A
Now
there
are
many
types
of
local
clusters,
so
here's
just
some
examples,
mini
cube
kind
micro
case
they
all
have
their
pros
and
cons.
So
you
need
to
know
what
you're
doing
and
what
they
are
good.
At
so
I'm
gonna
when
I
talk
about
cluster
api,
I'm
to
tell
you
why
they're
using
the
specific
type
of
local
cluster
that
they're
using
the
good
thing
about
local
clusters,
is
you
just
double
click
to
install?
You
don't
have
to
pay
a
monthly
fee
as
long
as
you
can.
A
You
know,
as
long
as
you
can
pay
electricity
to
run
your
laptop
and
the
main
thing
to
be
concerned
about
is
compute,
so
your
laptop
is
going
to
run
out
of
memory
or
it's
not
going
to
have
enough
processing
power.
That's
the
thing
that
might
be
annoying
network
is
not
so
much
of
a
concern
because
you
don't
need
to
be
pushing
and
pulling
things
through
the
internet.
So
that's
not
so
much
of
a
problem.
A
It
is
a
problem,
though,
if
you're
using
a
remote
cluster,
because
if
pulling
images
and
pushing
images,
it
takes
a
lot
of
bandwidth,
and
that
means
a
lot
of
time
depending
on
how
good
your
connection
is,
so
you
can
with
a
remote
cluster,
you
can
have
like
infinite
memory
and
cpu,
but
network
is
going
to
be
annoying
an
annoying
issue
on
your
development
workflow.
So
that's
something
to
pay
attention
to
now.
A
remote
cluster.
A
A
You
have
to
have
someone
with
experience
in
these
things
to
provision
all
the
infrastructure
that
you
need
to
maintain
it
and
basically
to
do
better
than
just
fiddling
around,
because
if
you're,
if
you
don't
know
what
you're
doing
you're
gonna
end
up
wasting
a
lot
of
money,
so
you
don't
wanna
do
that
right,
so
datadog.
How
did
they
deal
with
this
issue?
They
have
300
plus
services,
so
it
won't
fit
a
laptop.
A
They
use
one
huge
development
cluster
in
the
public
cloud
that
is
self-managed,
so
people
always
ask.
Should
every
developer
have
a
cluster
should
every
develop?
Should
everyone
work
on
one
big
cluster?
What
I
always
see
is
everyone
develops
on
one
big
cluster,
you
split
either
by
teams
or
by
developers
where
everyone
has
their
own
namespace
and
usually
that's
how
it's
done.
A
Datadog
has
wrapper
tools
for
provisioning,
so
developers
don't
have
to
do
any
of
it
by
hand,
and
something
interesting
is
that
they
have
like
a
staging
where
every
service
is
running
in,
like
a
basically
the
stable
form
of
it,
the
current
form
of
it
when
you're
working
on
one
of
them,
you
basically
clone
it
into
your
namespace.
You
modify
the
version
that
is
in
your
namespace,
but
you
don't
need
to
run
every
surface
again
in
your
namespace.
A
Only
the
one
you're
working
on
the
other
ones
that
you're
not
working
on
you
can
just
use
the
ones
that
are
in
staging.
So
that's
really
cool
with
something
I
hadn't
seen
before
talking
to
datadog
and
it's
an
interesting
pattern.
They
also
have
options
where
you
can
inject
debugging
tools
into
the
things
that
you're
working
on
and
so
on,
and
this
is
a
more
common
thing
in
big
companies
where
you
basically
have
a
push
button
and
suddenly
that
container
has
a
ton
of
debugging
tooling
and
that
kind
of
thing
mine
space.
A
They
use
docker
compose
for
tests
because
just
because
running,
kubernetes
and
ci
can
be
a
bit
fiddly,
so
they
use
compose
for
that,
and
their
local
clusters
are
basically
exactly
the
same
as
in
production
as
what
they
run
in
production,
except
for
you
know,
number
of
replicas,
and
that
kind
of
a
thing-
and
this
brings
me
to
an
important
topic-
and
that
is
kubernetes
on
the
laptop-
is
fine
when
you
do
it
right.
So
everyone's
always
joking
around
oh
kubernetes,
on
the
laptop
it's
going
to
set
your
laptop
on
fire,
that's
silly!
A
No,
if
you
know
what
you're
doing
and
you
you're
doing
things
at
the
right
scale,
kubernetes
on
the
laptop
works,
just
fine.
So,
let's
stop
with
you
know
the
silly
thing
of
bashing,
kubernetes
and
laptop
because
that's
pointless.
I
would
never
tell
mine
space
this
company
to
use
a
remote
cluster
when
they
are
100
happy
with
the
local
cluster
and
they
don't
want
or
have
to
deal
with
the
extra
complexity
of
managing
a
remote
cluster.
A
Moving
on
uno
also
works
also
uses
docker
for
mac.
Mostly-
and
here
we
have
an
interesting
case-
and
I
just
told
you
that,
if
you're
doing
things
at
the
right
scale,
a
local
cluster
is
fine.
When
you're
past
a
certain
scale,
you
need
to
move
on
to
a
remote
cluster,
sometimes
you're
past
that
scale
and
you
can
still
handle
things
locally
because
it's
convenient.
A
So
how
do
you
do
that?
What
uno
did
is
that
they
split
their
app
into
car
services
and
optional
services.
The
car
services
are
always
running.
The
optional
services,
don't
depend
on
each
other,
they
depend
only
on
the
car
services
and
they
are
basically
flip
switches
that
you
can
turn
on
and
off.
So
you
run
all
the
car
services
plus
that
one
that
you're
working
on
then
you
turn
that
one
off
and
you
turn
the
other
one
on
to
work
on
that
one
and
basically,
as
long
as
you're,
not
running
everything.
A
I
assume
at
some
point
they're
going
to
grow
past
it
and
they're
going
to
have
to
start
using
remote
development
clusters,
but
right
now
this
has
been
working
for
them,
even
though
it's
been
a
long
time
that
they
can't
run
everything
on
the
laptop
at
once.
They
still
have
a
workflow
where
that
can
work.
So
that's
interesting
and
they
are
very
amazing
at
automation
and
that
kind
of
stuff
we're
gonna
get
to
that
in
a
second
cluster
api
they
use
kind
as
the
local
development
cluster.
A
The
reason
for
that
is
because,
when
your
business
is
creating
the
leading
clusters,
you
want
a
solution
that
can
create
and
delete
clusters.
Fast
and
kind
is
very
good
at
that,
so
with
minicube
they
would
have
trouble
with
microcase.
They
would
have
trouble
they
use
kind,
because
it's
the
perfect
solution
to
that
issue,
which
once
again
brings
us
to
the
point
that
you
should
know
the
tools
that
you're
going
to
use
and
you
should
use
the
right
tool
for
the
right
job
and
usually
when
people
are
like,
you
know
screaming
on
twitter.
A
A
Take
away
if
you're
a
small
company
use
a
local
cluster.
Once
you
can't
fit
your
app
on
your
laptop
anymore,
then
you
move
on
to
a
remote
cluster.
But
you
need
to
know
that
that
move
is
going
to
be
costly
because
suddenly
now
you're
paying
a
monthly
fee
and
you're
going
to
need
a
professional
you're
going
to
need
the
expertise
to
create
to
maintain
to
modify
that
remote
cluster
and,
of
course,
that's
also
consideration.
A
A
So
what
should
we
do?
Instead,
datadog
uses
helm,
so
they
use
helm
templates,
different
values
for
different
environments.
Of
course
you
don't
need
100
replicas
in
development.
You
just
need
one,
and
basically
you
write
yaml
the
first
time
when
you
create
a
service,
you
do
the
template
and
all
of
that
on
helm
and
everyone
else.
Basically,
it
integrates
with
tilt.
No
one
else
has
to
think
about
it,
except
the
person
that
writes
it.
A
The
first
time
I'm
going
to
talk
about
tilt
in
a
minute,
mind,
space
same
thing:
helm
templates,
integrated
with
tilt
what
the
what
they
do.
That's
different
at
minespace
is
that
datadog
is
huge,
so
there's
a
very
broad
range
of
services
running
if
you're
on,
if
you're
doing
something
smaller,
you
can
make
services
very
uniform,
so
a
silly
example
of
that
would
be
take
the
same
ammo
same
docker
file,
same
everything,
just
change
the
name
and
the
binary
that
it's
gonna
run.
Basically,
of
course
it's
not
that
simple,
but
you
get
my
point.
A
A
A
Once
again,
everything
is
extremely
uniform,
so
my
my
I
think
the
big
takeaway
here
is,
if
you
can
make
things
our
uniform,
do
it
because
it's
gonna
save
you
trouble
if
you
don't
know
how
the
templating
stuff
works.
Basically,
here
you
have
the
template
on
one
side,
the
generated
yammer.
On
the
other
side,
you
change
the
template
and
it
propagates
into
all
the
different
config
files.
So
you
don't
have
the
risk
of
updating
one
config
file
forgetting
to
update
the
other,
and
then
they
don't
mesh
well
together
anymore,
take
away.
A
Everyone
uses
a
templating
solution,
big
companies,
sometimes
they
create
their
own
templating
solutions.
I
would
tell
you
to
not
do
that
unless
you
absolutely
know
you
have
to
most
people
use
helm,
I
think
from
my
experience
of
from
the
companies
I've
been
in
touch
with
and
there's
a
lot
of
integration
with
tools
like
tilt,
which
is
the
next
category.
A
We're
gonna,
discuss
feedback,
loop,
automation
so
well,
let
me
tell
you
a
story,
so
if
every
time
I
make
a
change
to
my
code,
I
need
to
docker
build
docker
push
and
keep
cuddle
apply
to
see
my
change
running
live.
If
I
need
to
do
all
of
this,
I
am
closing
with
notepad.
That's
silly.
We
don't
want
to
do
that,
and
sometimes
people
say
okay
yeah,
but
I'm
saving
30
seconds
at
a
time.
At
the
end
of
the
day,
that's
maybe
10
minutes.
It's
10
minutes
of
developer
time.
A
That
important
and
the
question
is
you're
asking
the
wrong
question.
I
don't
care
about
the
fact
that
it
takes
10
minutes.
I
care
about
the
fact
that
every
time
a
developer
has
to
do
that,
they're
moving
their
focus
away
from
what
they
actually
want
to
be
doing.
So
you
have
a
very
high
cognitive
cost
to
being
interrupted
all
the
time
if
a
developer
can
simply
save
a
file
blink
their
eyes
and
their
new
code
is
up
and
running
instantly.
A
So
how?
What
can
we
do?
There
are
plenty
of
tools
I'm
going
to
talk
about
tilt,
but
there
are
others
like
garden
scaffold
draft.
Basically,
these
tools,
they
watch
your
source
code.
Every
time
you
make
a
change
to
your
source
code,
the
conventional
workflow.
Is
you
rebuild
that
docker
image
you
ship
it
to
registry?
You
update
your
development
cluster
with
that,
and
this
is
all
behind
the
scenes.
The
developer
basically
just
changes
the
file
and
waits
a
couple
seconds.
They
don't
have
to
do
anything
by
hand.
A
A
Now
I
posted
this
on
twitter.
Basically,
how
long
does
it
take
between
changing
a
line
of
code
and
that
code
being
live
and
running
in
your
development
cluster?
A
lot
of
people
said
30
minutes,
which
I
find
really
sad.
I
always
aim
for
two
seconds
you
can
get
to
two
seconds.
I
have
a
whole
talk
about
tips
and
tricks
and
hacks
to
achieve
this,
the
smallest
the
time,
the
better.
A
A
Currently,
40
percent
of
developers
use
totes,
they're
rolling
it
out.
They
have
so
tote
is
one
of
those
tools
that
allows
for
tons
of
customizations
and
datadog.
They
basically
pull
their
ci
image
locally
and
then
their
ci
images
builds
things
locally,
which
are
then
synced
into
the
the
build
cluster
into
that
dive
cluster.
So
that's
cool.
Of
course
they
use
helm
from
tilt.
So
basically,
when
you
change
a
helm
template
you
don't
have
to
run
helm
tilt's
going
to
do
that
for
you.
So
for
a
developer.
It's
basically.
I
change
something.
A
I
save
a
file
magic,
it's
up
and
running
and
they
create
a
lot
of
lots
of
buttons
until
to
help
with
discovery.
So
that's
really
cool
what
this
means
is.
If
we
look
at
the
green
things
here,
those
are
all
my
services,
except
for
the
last
one,
that
one's
a
custom
button.
Of
course
you
have
services
on
one
side
and
logs
and
the
other
in
this
dashboard.
This
is
what
tilt
looks
like
and
when
you
click
that
that
button
that
says
flush
database,
it's
going
to,
of
course,
flush
your
database.
A
Sometimes
you
have
a
lot
of
automations
in
your
make
file
and
people
don't
know
about
them,
and
thus
they
don't
use
them
and
datadog
uses
all
of
these
buttons
to
facilitate
developers
knowing
things
that
they
can
do
so,
for
example,
there's
a
button
to
get
dependencies.
There's
a
button
to
do.
Database
migration,
so
uno
is
also
very
interesting
in
this
regard.
It's
one
of
my
favorite
companies.
They
automate
basically
everything
that
is
humanly
possible.
A
The
tilt
extensions
feature
was
somewhat
inspired
by
what
they
were
doing
and
it's
really
cool,
so
they
automate
setting
up
internal
proxies
tls
management,
vault
integration
using
tracing
as
a
developer
tool,
a
bunch
of
crazy
stuff.
With
that,
I
don't
know
enough
to
explain
to
you
basically
everything
that
every
possible
thing
that
can
be
automated:
they
do
they
automate
things
with
tilt.
So
it's
a
very
smooth
workflow,
where
basically,
a
developer
doesn't
need
to
worry
about
anything.
They
just
get
things
up
and
running
with
one
command.
A
They
write
their
code,
they
change
their
code
and
everything
else
in
the
whole
world
is
automatic
and
they
don't
have
to
think
about
it.
So
a
big
shout
out
to
david
rubin
for
all
his
amazing
work
there
and
for
writing
the
first
third
party
tilt
extension
now
mindspace
mindspace
was
one
of
the
first
users
of
tilt.
Their
usage
of
it
was
similar
to
uno,
but
basically
so
many
of
the
customizations
that
they
wrote
for
tilt
were
then
turned
into
native
features
that
now
they
basically
use
vanilla
tote.
A
But
there's
their
setup
is
somewhat
similar
to
owner
now
cluster
api.
They
have
lots
of
interesting
things,
so
they
automate
a
bunch
of
stuff
like
certificate
management
like
what
I
talk
about
uno.
They
have
a
bunch
of
overlays
so
that
there's
like
this
custom
automations
and
then
you
can
overlay
certain
things
on
top
of
the
custom
automations,
so
developers
don't
have
to
all
have
to
do
the
same
thing.
They
can
customize
some
things
they
used
to
build
binaries
inside
containers.
A
Now
they
build
things
locally
and
they
live
reload
binaries
into
basically
a
scratch
container.
It's
not
scratch,
but
it's
kind
of
like
scratch
so
that
they
don't
need
all
the
go
tool
chain
inside
containers.
So
that
is
cool
as
well.
Takeaways,
when
you
have
a
very
uniform
structure
of
services
which
we
talked
about
in
the
previous
section,
you
can
do
lots
of
cool
things
with
automation
and
that
is
really
cool.
So
that's
the
stuff
that
uno
does
the
stuff
that
mindspace
used
to
do
the
stuff
that
cluster
api
does
with
datadog.
A
It's
a
bit
more
complicated
because
they're
very
big,
and
they
have
things
that
are
very
different,
but
they
also
do
tons
of
automation
as
well.
So
takeaway
is
use
a
tool
like
tilt
or
garden
or
scaffold
or
draft
automate,
the
bejesus
out
of
your
system,
so
that
developers
can
really
focus
on
developing
and
not
on
boilerplate
cluster
context.
Sharing.
A
A
So
if
you're
copying
and
pasting
logs
here,
there's
some
give
and
take
I'm
not
going
to
sure
I'm
not
going
to
say
that
copy
and
pasting
logs
is
always
bad,
but
it's
kind
of
getting
into
notepad
notepad
territory,
so
for
datadog,
because
they
have
the
big
huge
development
cluster.
When
someone
needs
to
look
into
what
their
colleague
is
doing,
they
just
switch
to
their
namespace.
So
it's
pretty
easy.
A
I've
seen
also
this
pattern
in
other
other
companies
of
the
scale
of
datadog.
So
this
is
a
very
common
thing
to
do
so.
Use
a
wrapper
tool.
Don't
use
cubecode,
make
it
easy
for
your
developers
and
they
can
just
switch
around
namespaces
and
see
what
their
colleagues
are
doing
pretty
easy
problem
solved
for
mindspace,
and
I
did
this
research
before
corona,
so
I
don't
know
what
people
are
doing
now,
but
they
were
doing
things
the
analog
way.
A
So
here's
my
laptop,
take
a
look
at
it
and
you
know
what,
at
the
certain
scale,
that
is
better
than
high-tech
solutions.
Uno
was
in
the
same
space
except
that
uno
uses
so
much
automation.
So
if
anyone
has
ever
had
the
same
problem
twice,
they
just
automate
it
out
of
existence
so
congrats
to
them,
and
it
makes
it
so
that
things
are
easy
to
use
in
this
regard.
A
Cluster
api
uses
snapshots,
that's
a
feature
of
tilt,
which
is
basically
you
have
the
dashboard
that
shows
all
of
your
logs.
All
of
your
alerts.
Everything
when
you
take
a
snapshot
till
it
basically
packages
up
all
the
current
state
of
your
application.
You
can
send
that
to
your
friend.
They
just
click
a
link.
They
open
it
in
the
browser.
They
suddenly
can
see
all
the
logs
all
the
alerts,
everything
that
your
application
is
currently
doing,
the
current
state
of
it,
and
then
that
makes
it
easy
for
your
colleagues
to
help
debug.
A
What's
going
on
in
your
cluster
takeaways,
if
you're
a
big
company,
everyone,
you
have
a
big
cluster
for
everyone,
a
different
name
space
for
everyone,
just
look
into
your
colleagues
namespace
and
then
you
can
see
what's
going
on
a
middle
ground
is
you
can
use
a
feature
like
snapshots
and
if
you're,
a
very
small
company,
just
use
the
analog
solutions?
Don't
bother
with
extra
complexity,
though
I
think
like
if
you're
already
using
tilt,
you
should
just
use
tilt
snapshots,
but
that's
a
separate
discussion
now,
a
couple
of
extras.
A
We
are
getting
close
to
the
end
here,
so
datadog
and
uhnu.
They
don't
use
debuggers,
like
you
know,
a
debugger
like
gdb.
They
don't
use
those,
they
always
output
things
to
logs
and
then
they
use
tracing
as
a
as
a
debugging
tool.
So
unu
uses
jaeger
datadog,
of
course
uses
it
uses
datadog,
so
datadog
dog
feeds
datadog.
A
That
was
hard
to
pronounce
and
a
common
complaint
about
tracing
is
that
you
have
to
adjust
your
code
to
include
tracing.
It
doesn't
just
work
on
top
of.
What's
already
there,
you
have
to
put
the
code
there.
I
asked
datadog
about
this
and
they
said.
Oh,
you
know
what
we
have
this
auto
instrumentation
thing
here,
that
for
a
good
chunk
of
people,
it
solves
the
problem
so
kudos
to
them
and
that's
great,
and
I
look
forward
to
being
able
to
play
around
with
that
myself.
A
This
is
what
tracing
looks
like
so
here
you
have
basically
one
request
getting
traced
through
a
bunch
of
services.
You
can
see
the
blue
thing.
There
is
basically
things
moving
inside
one
service
once
it
changes
to
orange.
That
means
that
that
request
is
now
on
another
service,
and
then
you
can
just
track,
as
one
request
goes
through
many
different
parts
of
your
application.
You
can
see
where
it
goes
and
what
happened
along
the
way,
and
that
is
a
cool
thing
too
mindspace.
A
They
use
regular
normal
debuggers,
so
they
use
tilt
to
do
the
exposing
the
parts
thing
and
then
basically,
their
ide
connects
to
the
node
remote
debugging
thing,
and
that
does
the
job
and
cluster
api
is
probably
the
most
like
analog
thing
here,
where
basically,
they
don't
have
a
use
case
where
they
really
want
to
use
an
actual
debugger.
So
their
debugging
method
is
a
ton
of
print
lines
that
then
get
output
to
the
console.
A
They
have
a
very
fast
feedback
loop,
so
that
works
for
them
and
if
you're
gonna
use
this,
if
you're
gonna
do
a
print
line,
debugging
style,
I
would
suggest
using
a
tool
like
stern
to
aggregate
all
logs.
Of
course,
if
you're
using
something
like
tilt,
does
that
for
you?
But
if
you're
not
stern,
is
a
great
way
to
to
do
this
as
well.
A
Take
away
integration
with
tracing
and
debugging
tools
is
kind
of
rare
but
growing,
especially
tracing.
I
see
a
lot
of
people
using
it
more
and
more.
It's
easier,
I
think,
than
debugging
debugging,
it's
like
when
you're
once
you're.
In
that
mind,
state
it's
great,
but
until
you
get
there,
it
can
look
a
bit
daunting
so
take
away.
Notepad.Exe
is
not
a
development
environment.
So
let's
not
do
that.
Let's
give
our
developers
proper
tools
to
make
their
lives
easier,
so
that
they're
happier-
and
you
know
what
so
that
we
make
more
money
yeah.
A
So,
thanks
for
coming
to
this
talk
and
one
more
thing,
we're
creating
the
my
team
at
tilt,
we're
creating
this
website
called
dax.dev.
We're
going
to
have
videos
articles
in
live
stream
about
developer
experience,
we're
basically
going
to
talk
about
all
the
things
that
I
talked
here
like.
Should
I
use
helm
or
customize,
should
I
use
kind
mini
minicube
or
microcades,
so
we're
gonna
have
a
ton
of
content
about
all
of
this
stuff
by
the
time
you're
watching
this
talk,
we're
probably
gonna
already
have
something
up
there
right
now.
A
It's
just
a
sign
up
page.
So
do
sign
up
if
this
is
if
this
floats
your
boat,
so
that
you
can
learn
more
as
we
as
we
learn
more
about
the
subject.
So
here's
how
you
can
get
in
touch
here's
links
for
all
of
those
companies.
Please
do
reach
out
tweet
message,
email
and
talk
to
you
soon.
So,
thanks
for
watching
this
talk
and
bye,
bye
have
a
great
rest
of
the
conference.