►
Description
Panel discussion between Christian Posta (Global Field CTO at Solo.io) and Adam Zimman (VP of Platform at LaunchDarkly) at the Test in Production Meetup.
If you’re interested in joining us at a future Meetup, you can sign up here: https://www.meetup.com/Test-in-Production/
#launchdarkly #solo #testinproduction
Subscribe to LaunchDarkly:
https://www.youtube.com/c/LaunchDarkly?sub_confirmation=1
Learn more about LaunchDarkly:
Website: https://launchdarkly.com/
LinkedIn: https://www.linkedin.com/company/launchdarkly
Twitter: https://twitter.com/launchdarkly
About LaunchDarkly:
Feature management allows development teams to innovate faster by fundamentally transforming how software is delivered to customers. With the ability to gradually release new software features to any segment of users on any platform, DevOps teams can standardize safe releases at scale, accelerate their journey to the cloud and collaborate more effectively with business teams.
Today, LaunchDarkly deploys peaks of 20 trillion feature flags each day, and that number continues to grow. Founded in 2014 in Oakland, California by Edith Harbaugh and John Kodumal, LaunchDarkly has been named on the Forbes Cloud 100 list, InfoWorld’s 2021 Technology of the Year list, and the Enterprise Tech 30 list.
A
A
B
So,
interestingly
enough
that
actually
a
exact
name
for
me
holds
a
lot
of
meaning,
because
when
I
was
running
a
large-scale
service
at
VMware,
we
were.
My
team
was
responsible
for
hands-on
labs
at
vmworld
and
one
of
my
developers
as
a
joke
actually
put
that
as
the
screen
saver
for
500
user
terminals
in
Moscone
Center
that
were
set
up
for
lat
individual
lab
stations,
and
we
actually
had
a
bug
in
our
user
interface
because
it
wasn't
adequately
tested.
B
Yeah.
It
wasn't
funny
at
the
time,
but
you
know
I,
think
for
me.
Then
it
had
very
different
connotation
in
that
it
was
this
notion
of
kind
of
cowboy
development
of
you
know,
kind
of
individuals
that
didn't
take
testing
seriously
or
hadn't,
been
in
situations
where
the
production
product
mattered,
and
that
made
it
very
real
for
the
individual
of
my
team
and
as
I
started
working
more
and
more
in
the
dev
tools.
Space
started
to
realize
that
teams
that
were
running
large-scale
production
environments.
C
We
said
from
Friday
afternoon
till
Sunday
morning,
once
a
quarter,
you
took
off
the
whole
weekend
from
your
family
and
you
sat
on
a
phone
call
and
chat
rooms,
and
you
did
your
deployment
and
invariably
doing
that
deployment
would
break
something
and
it
would
be
emergencies
and
fires
and
all
of
this
stuff,
because
they
were
trying
to
deliver
three
months
worth
of
functionality
and
software
that
we
had
all
been
working
on
and
trying
to
coordinate
with
each
other
and
all
Ram
it
into
production
at
exactly
the
same
time
and
from
there.
So
I
thought.
C
Obviously,
I
was
like
this
sucks,
but
it
was
just
how
it's
done
from
there.
I
went
to
go
work
at
Zappos
calm
and
we
worked
very
closely
with
the
folks
at
Amazon
and
there
and
used
their
tooling
and
the
release
process.
That
Amazon
was
180
degrees
different
and
what
we
were
doing
at
a
big
bank
and
I
thought.
Oh,
there
is
some
hope
there
is.
C
Maybe
an
answer
to
some
of
the
pain
that
we
felt
every
three
months
on
the
weekends
and
then
so
what
but
and
I
dug
into
what
Amazon
was
doing,
but
it
was
so
common.
It
was
very
Amazon
specific.
So
once
once
some
of
the
technology
like
containers
and
communities
and
some
of
these
cloud
native
tooling
things
came
out
started
to
come
out,
it
started
to
resemble
is
like.
Oh,
my
god,
that's
exactly!
C
That's
what
Amazon
had
eight
years
ago
and
once
I
saw
Envoy
proxy
when
I
saw
Matt
Klein
give
a
talk
on
my
proxy
about
three
years
ago.
I
thought
that
is.
That
is
something
that
gives
you
a
level
of
control
over
the
network
traffic
so
that
you
can
separate
out
deployment
and
release
like
you
mentioned,
and
that
is
incredibly
incredibly
powerful
and
so
from
there
for
the
last
three
years
or
so,
I've
been
working
with
enterprise
customers
to
help
them
understand
exactly
what
that
means,
because
I
I
don't
know.
C
A
C
So
some
of
this
technology
is
coming
out
as
being
open
sourced
and
is
becoming
more
available
and
coupled
with
their
desire
to
figure
out
okay.
How
do
we
take
this
hundred-year-old
company
and
become
more
agile
and
and
all
of
this
stuff?
That's
there's.
There's
there's
a
legitimate
market
pressure
for
them
to
do
that,
because
they
sing
like
borders
and
Blockbuster,
and
you
know
you
can
go
on
and
on
and
on
about
these,
these
institutions
that
are
being
disrupted
by
technology
companies
and
they
don't
want
to
be
the
next
one.
Yeah.
B
B
A
B
So
I
think
that
you
know
for
me
as
I've
spoken
with
a
lot
of
our
customers
and
prospects
companies
that
are
looking
to
move
in
the
direction
of
either
feature
management
or
just
more
generally.
In
this
you
know
kind
of
the
enterprise
term
of
digital
transformation,
but
looking
to
basically
modernize
their
developer
tool
chain,
their
workflows.
B
There
are
a
lot
of
constraints
that
aren't
even
that
that
are
applicable
to
that
same
problem
statement
that
are
not
necessarily,
you
know,
kind
of
generational
as
a
you
know,
kind
of
I
made
it
personal
right.
You
know
one
great
example
is
regulation,
and
you
know
you
think
about
organizations
that
work
with
unions
have
worked
with
a
number
of
government
organizations
that
have
union
employees
or
union
workers
and
in
their
union
contracts.
Now
they
will.
B
It
is
not
uncommon
for
them
to
actually
have
language
about
the
number
of
times
in
which
they
can
their
the
software
that
they're
required
to
use
can
be
updated,
as
well
as
the
training
requirements
around
updates
to
software,
and
so,
if
the
organization
is
using
some
type
of
tool
or
platform
that
updates
at
a
cadence
that
is
beyond
those
restrictions,
they
say.
I
can't
use
this
right
or
they
put
unreasonable
restrictions
on
that
vendor
that
provider
that
are
very
difficult
for
them
to
accommodate.
C
And
I
think
for
solo,
we
sell
to
large
organizations
that
for
them
to
adopt
some
of
our
technology
they
have
to,
they
have
to
be
at
a
certain
level.
Let's
just
say
like
in
terms
of
I'll,
give
a
simple
example:
they
need
to
have
a
particular
version
of
kubernetes
or
a
minimum
version
of
communities,
for
example,
and
sometimes
you
know
they
want
to
adopt
the
later
technologies
and
the
newer
technologies
and
they
want
to
operationalize
them,
but
for
them
to
do
that.
C
Obviously
that
gives
us
the
ability
to
deliver
features
more
quickly
and
leverage,
some
of
the
techniques
and
methodologies
around
progressive
delivery
and
and
reducing
the
risk
of
making
changes
and
so
on,
because
enterprise
users,
enterprise
organizations
they're
also
very
resistant
to
moving
too
fast.
So
yeah,
that's
that's
something
that
we
definitely
deal
with
very
much.
Do.
A
B
C
C
If
you
need
to
write,
if
you
need
to,
and
in
my
opinion
I
would
say,
if
you
guys
are
small
team
starting
on
with
the
monolith,
that's
exactly
the
right
way
to
do
it,
don't
over
complicate
it
by
doing
the
micro
service
and
bringing
all
this
other
other
stuff
and
go
go
at
the
speed
that
that
makes
sense
and
is
doesn't
take
on
any
unnecessary
complexity.
For
that
yeah.
B
I
think
it,
for
you
know
us
at
launch.
Darkly.
We
have
tried
to
create
some.
You
know
entry
level
offerings
for
smaller
organizations
that
are,
you
know,
kind
of
modestly
priced
and
give
you
the
ability
to
start
adopting
the
methodologies
around
building
with
feature
flags
and
building
with
feature
management.
The
you
know
kind
of
main
thing
that
we've
done
for
like
startup
your
size
is
that
we
have
entry
level
plan.
That
literally
has
just
a
shared
login
account
right.
B
If
you're
there
just
two
or
three
engineers,
you
don't
need
the
kind
of
granular
access,
control
or
audit
logging
that
you'd
have
you
no
need
for
when
you
get
to
an
engineering
team
of
20
or
30
or
hundreds
or
thousands
right,
and
you
know
also
from
a
volume
perspective.
You
know
like
trying
to
think
about.
Like
you
know,
you're
not
gonna,
be
doing
a
whole
lot
of
flagging
you're
not
going
to
be
doing.
You
know.
You've
got
three
people
that
are
going
to
be
pushing
to
production.
B
You
know
you're
don't
have
that
as
nearly
as
much
to
keep
track
of,
but
getting
in
the
practice
of
doing
it
is
something
that
we've
seen
a
lot
of
teams
that
are
represented
by
you
know,
individuals
that
are
coming
out
of
organizations
that
have
strong
feature
flagging
practices.
They've
said
look,
I
could
build
this
myself,
but
is
that?
Where
I
want
to
be
spending
my
time
or
do
I
want
to
spend
my
time
on
the
product
that
I'm
actually
going
to
be?
You
know
kind
of
extracting
value
or
delivering
value
for
yeah?
B
It's
something
that
we
have
talked
a
lot
about
and
I
think
that
it
is
going
to
be
somewhat
dependent
upon
the
types
of
services
or
applications
that
you
know
someone's
offering.
You
know
you
brought
up
Gmail
or
you
know,
kind
of
the
Google
work.
Suite
is
a
great
example
where
you
can
go
into
the
Google
Labs
interface
and
you
can
effectively
it's
user
facing
feature
flex
and
that's
actually
how
they're
implemented,
where
you're
turning
feature
functionality
on
and
off
for
your
personal
account.
B
You
know,
I,
think
that
that
is
actually
a
great
way
to
build
products
and
I
think
that
you
know
being
able
to
use.
You
know,
launch
Darkly
in
that
capacity.
It's
not
something
that
we
see
a
lot
of
at
this
point
in
time,
but
it's
definitely
something
that
you
know
kind
of.
We
want
to
get
to
a
place
where
we
can
scale
ibly
support
that
right
now,
what
we
encounter
is
more
often
than
not.
B
People
are
really
looking
for
the
kind
of
core
infrastructure
use
cases
or
you
know,
being
able
to
do
the
management
from
the
company
perspective
as
opposed
to
pushing
out
that
delegation
all
the
way
to
the
end
users,
but
ultimately,
I
do
agree
with
you
that
I
think
there
is
a
future
where
that
kind
of
end-user
delegation
is
perfectly
appropriate.
The
other
way
that
you
can
do
that
is
you
can
actually
use
user
attributes
that
people
can
change
the
configuration
on
and
then
those
attributes
are
just
detected
by
the
feature
flag.
B
So
that's
the
way
in
which
we
encourage
our
users
to
do
it
today,
so
that
gives
them
one
kind
of
layer
of
removal
or
comfort
for
them
to
make
sure
that
the
change
of
like
the
actual
code
path
is
being
kind
of
held
back.
You
know
to
the
company,
whereas
the
user
can
change
kind
of
just
an
attribute
or
key
value.
Yeah.
C
That's
a
good
question,
so
one
one
of
one
of
the
things
I
mentioned
in
the
slides
is
that
there
are
these.
These
dark,
unknown
areas
that
exist
in
our
in
our
systems,
whether
we
care
to
admit
it
or
not,
right
security
is
one
of
those
areas
and
to
go
through
and
and
assume
that
different
parts
of
the
application,
let's
say,
have
different,
or
they
trust
each
other
at
levels
that
you
expect,
but
then
going
in
and
proving
that
they
do
or
finding
ways
that
they
they
may
not,
or
the
reverse
right.
C
So
I
think
chaos,
experimentation
or
more.
Just
that's
why
I
don't
like
to
turn
it's
it's
more
exploratory
experimentation
and
that
absolutely
includes
security
and
penetration
testing
and
validating
trust
across
the
network
and
and
those
types
of
things
so
yeah
absolutely
I
was
hoping.
Somebody
would
ask
that
question
so
I'll
give
my
boy.
B
C
Once
you
get
some
traffic
going
in
there,
you
might
then
want
to
flip
the
feature,
fog
and
say:
okay,
let's,
let's
actually
turn
this
new
feature
on
I
mean
we
made
the
code
changes,
we
have
the
flag
off,
we
put
it
out
into
production.
It
looks
good
now,
let's
actually
turn
the
feature
on
and
some
of
those
that
are
then
going
to
see.
C
You
know,
let's
say
version
2
of
the
schema,
and
some
of
them
will
see
version,
1
and
so
on,
and
so
there's
probably
a
lot
more
handful
of
these
types
of
examples
where
it
makes
sense
to
both
control
the
traffic
as
well
as
progressively.
You
know,
open
up
that
functionality
to
the
to
the
users
yeah.
B
The
analogy
that
I've
found
successful
in
kind
of
visualizing
this
is
you
use
a
service
mesh
to
be
able
to
kind
of
build
a
new
building
and
deploy
the
new
version
of
the
building,
and
then
you
use
the
feature
flags.
Those
are
your
lights
which
is
inside
and
you
can
kind
of
turn
on
and
off,
based
on
wherever
you
place,
those
switches
so
per
floor
per
room.
B
Whatever
you
need
to
do,
and
it
gives
you
varying
levels
of
granularity
right,
you
can
have
the
ability
to
be
able
to
say:
hey,
I
want
to
test
out
this
new
structure
that
I've
created,
as
opposed
to
this
new
feature
that
I
want
to
be
able
to
control
III.
Don't
think
that
I
think
that
the
two
experiences
that
you've
had
I
think
both
are
you
know,
kind
of
you
know,
exists
in
the
industry,
obviously,
but
I
think
that
the
you
know
notion
of
having
a
few
folks
with
the
keys
to
the
kingdom.
B
As
you
put,
it
is
very
common
for
older
companies
and
companies
that
have
not
yet
built
the
kind
of
stability
in
their
infrastructure
and
the
practices
around
continuous
delivery
and
that
type
of
methodology
a
lot
of
times,
there's
a
fear
aspect
of
things.
Breaking
I
think
that
this
is
something
that
Christian
talked
about
of.
You
know
like
his
experience
at
the
bank,
or
you
know
even
just
the
idea
like
you,
you
don't
know.
B
Majors
is
all
infrastructure.
Fails.
It's
just
it's
true.
It's
at
some
point
in
time.
There's
going
to
be
a
stress
that
you
didn't
account
for,
and
so
the
question
is
is:
are
you
going
to
build
it
so
that
it
fails
gracefully
and
it
has
you
know
some
notion
of
what
those
failure
modes
are
going
to
be?
Have
you
actually
planned
for
them?
Have
you
tested
them?
B
Have
you
validated
those
tests
that
it
does
what
you
expected
to
when
it
fails,
or
are
you
just
going
to
cross
your
fingers
and
say
no
I'm
just
going
to
make
sure
it
never
fails
right
and
I?
Think
more
and
more
companies,
especially
with
you,
know,
more
and
more
companies
putting
their
products
and
services
out
on
the
Internet,
where
there
is
so
much
that
goes
on
that
we
can't
account
for
that.
B
They
are
confronted
with
this
requirement
from
a
user
perspective
of
you
know,
clean
failure,
modes
of
understood
failure,
modes
and
being
able
to
actually
have
services
and
products
that
fail,
gracefully,
so
I
think
the
experience
that
you've
had
is
this
transition
where
you've
moved
to
a
company
that
Coursera
is
obviously
they're
built
around.
You
know
internet
delivery
right,
that's
the
only
way
they
function
and
so
they've
had
to
make
sure
that
their
failure
modes
are
thought
through
and
that
they,
you
know,
have
the
ability
to
accommodate.
For
those
now.
B
A
byproduct
of
that
is
that
they
feel
a
lot
more
confident
failing
because
it's
not
about
whether
or
not
you
fail.
That's
eventually
going
to
happen
it's.
How
long
does
it
take
you
to
recover
right?
The
the
metric
that's
been
increasing
in
popularity
from
operations,
team
perspective
or
from
a
DevOps
methodology
perspective
is
mean
time
to
recovery
or
MTTR
right.
B
This
is
like
how
long
does
it
take
me
to
recover
from
some
type
of
failure
or
outage,
as
opposed
to
you
know,
kind
of
this
notion
of
five
nines
of
you
know,
service,
availability
or
things
like
that.
You
know.
Ultimately,
you
know
if
your
ability
to
recover
from
an
incident
is
not
well
thought
through
and
orchestrated,
then
your
the
number
of
nines
you
have
isn't
going
to
matter.
C
C
They
figured
out
how
to
make
money
and
make
a
lot
of
money
right,
and
so
they
build
their
organization
and
in
such
a
way,
to
continue
to
turn
that
back
crank
to
make
money,
and
you
do
that
by
optimizing
for
efficiency
and
you
segment
people
into
their
one
little
spot
and
where
they
don't
have
to
see
the
big
picture.
They
work
on
their
one
little
thing
and
you
and
they
build
an
engine.
C
That's
able
to
turn
out
money
right,
but
some
of
these
other
companies
they're,
you
know
they're,
they
might
be
turning
out
money
but
they're
they're
less.
You
know
a
machine
and
more
they're,
more
organic
and
they're.
You
know
a
ability
to
adopt
a
change
and
they're
the
market.
Around
them
is
fundamentally
built
to
be
not
a
machine,
it's
to
be
adaptive
and
reactive
and
and
change,
and
so
from
the
old
enterprise
machine
model.
Where
you
don't.
You
have
very
low
trust
with
anybody
else
in
your
organization.
C
Everything
isn't
completely
siloed
and
you
handoff
over
walls
and
stuff,
and
you
separate
out
people's
responsibility
for
the
actions
that
they
take
versus
the
you
know.
Let's,
let's,
let's
learn:
let's
try
something
new
and
if
it,
if
it
fails,
that's
fine,
we're
learning
right
and
then
we're
exploring
new
ways
to
deliver
value
to
our
customers
and
make
money
so
organizationally.
C
B
I
mean
I
think
that
you
know
you
raise
an
interesting
point
with
the
kind
of
delegation
model
and
you
know
the
kind
of
number
of
options
you
know
that
are
there
and
I
think
those
are
actually
two
separate
challenges.
I
think
that
the
the
delegation
model
does
not
necessarily
require
that
you
have,
you
know
an
infinite
or
you
know,
even
a
large
number
of
options
that
you
are
allowing
access
to.
B
It
could
be,
you
know,
just
a
standard,
boolean,
true
or
false
of
is
a
feature
on
or
off,
and
that
could
vary
for
you
know
various
end
users
or
customers
or
in
your
case
universities,
but
in
from
a
code
perspective,
that's
literally
you've
just
got
to
code
paths
and
then
you've
just
defined.
You
know
users
going
to
encode
pathway
or
code
path,
B,
and
so
it's
fairly
simple.
The
other
aspect
that
you're
talking
about
of
you
know
creating
a
multitude
of
variants,
a
multivariate
option,
for
you
know
all
of
those
different.
B
You
know
end
users
or
universities.
While
it's
something
you
could
do,
it's
not
necessarily
something
that
I
would
encourage
unless
you
had
a
really
well-defined
need
for
it.
This
is
kind
of
like
the
problem
of
you
know.
You
know
the
kind
of
n
matrix
problem.
That's
so
many
you
know
packaged
software
vendors
used
to
face
of
you
know
what
revisions
did
they
support
and
for
how
long
right
and
what
you
don't
want
to
do
is
as
a
SAS
service.
B
You
don't
want
to
create
that
same
combinatorial,
matrix
like
that
would
be
horrible
right
and
so
I
would
never
advocate.
For
that.
I
think
that
the
idea
of
supporting
two
versions,
or
maybe
even
three
versions-
to
give
people
some
more
flexibility
or
you
know
one
thing
that
we've
seen
a
lot
of
our
customers
do
is
they'll.
Have
we
allow
for
hierarchy
in
our
Flags?
B
So
you
have
the
ability
to
do
kind
of
like
roll-up
features
where
you
say
either
you're
getting
the
new
stuff
when
it's
available
or
you're
not,
and
you
are
getting
that
at
some
type
of
regular
cadence,
so
it
could
be
quarterly
could
be
annually
and
then
you're
getting
that
kind
of
here's
all
the
new
stuff.
That's
come
out
and
the
nice
part
about
that
is
it's
kind
of
like
being
on.
B
You
know
if
you've
ever
joined
a
like
a
beta
program
from
like
slack
or
Apple,
or
something
like
that,
where
there
are
people
who
are
just
like
yeah
I
blow
up
my
machine
I,
don't
care,
you
know,
but
I
want
to
see
the
latest
and
greatest
features
as
soon
as
they're
coming
out,
and
then
there
are
other
people
who
are
just
like
no
I
just
wanna
be
on
the
stable
release
right,
and
so
you
can
think
of
that
as
the
kind
of
stable
release
model.
Where
you're
saying.
B
Whatever
it
is,
or
you
say,
I
want
to
see
the
new
stuff,
but
I
want
to
accept
it
when
I'm
ready,
for
you
know,
updating
my
documentation
for
doing
training
for
making
sure
that
I'm,
not
in
the
middle
of
a
sales
cycle
or
I'm,
not
in
the
middle
of
my
you
know,
peak
tax
season,
you
know
whatever
your
requirements
are
that
make
it
so
that
you
want
a
predictable
user
experience.
You
can
account
for
those.
Does
that
make
sense?
Yes,.
C
Thank
you,
of
course,
and
and
having
those
control
points
you
know,
from
the
edge
of
ingress
all
the
way
down
into
the
depths
of
your
code
and
all
the
different
combinations,
and
so
on
that
you
can
get
presents
a
pretty
interesting
opportunity
for
being
able
to
explore
what
people
really
want,
because
you
give
them
the
choice.
To
name
all
these
feature.
C
B
It's
that
feedback
loop
notion
right
where
you're
using
these
control
points
as
a
mechanism
to
be
able
to
guide
your
iterative
development
right.
You
think
about
it.
You
know
continuous
delivery
model
or
even
agile
to
a
degree
where
you
may
have
large-scale
epics
that
are
split
up
into
multiple
Sprint's,
where
you
have
an
end
state.
But
the
idea
of
agile
was
that
you
still
have
the
ability
to
be
able
to
alter
based
on
feedback
of
those
individual
Sprint's.