►
From YouTube: Getting To, and Through, Our First Black Friday with Critical Apps [I] - Arnaldo Pereira
Description
Getting To, and Through, Our First Black Friday with Critical Apps on Kubernetes [I] - Arnaldo Pereira, Luiza Labs
I'll talk about how we adopted kubernetes in Luiza Labs, where we develop and operate the sales channels of Magazine Luiza, a Brazilian retail company of 4+ USD billion in annual revenue.
How we deploy the apps, and ended up writing an open source deployment tool; how we scale our clusters; contingency plans; cultural impact on application development; ultimately how kubernetes helped us going through a black Friday smoothly
About Arnaldo Pereira
18 years in technology, half in startups and the others in mid-big companies - most of them managing development and SRE teams
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello
guys,
my
name
is
Arnaldo
I'm
here
to
talk
about
our
foot
first
black
friday,
with
kubernetes
on
production,
running
critical
apps
on
it
really
on
production,
not
kidding
so
really
a
little
bit
about
me.
I
started
working
at
eleven
on
street.
Fair
I
was
fired
on
the
first
day,
really
because
I
couldn't
lift
the
boxes.
The
boxes
were
very
heavy
for
me,
so
I
learned
very
early
on
life
that
life
could
be
tough,
so
it's
a
very.
It
was
very
good
for
me
to
learn
that
earlier
life.
A
A
One
of
them
was
with
Wi-Fi
and
the
other,
with
telecom
voice
over
IP
and
professionally
I
ie
I
went
to
from
the
startups
through
the
big
companies.
So
my
path
was
upside
down.
The
people
who
usually
I
don't
know,
works
for
Google
stuff,
like
that.
The
big
companies
then
great
startups
I
did
the
opposite,
not
that
I
work
it
for
Google,
but
anyway,
I
have
four
kids
I
like
skateboarding
are
like
saxophone,
jazz
and
right
now.
I
am
a
sorry
manager
at
Reza
labs.
A
So
what
is
Luiza
labs?
Luiza
labs
was
created
at
2013
to
do
only
research
and
development
for
a
company
called
magazine
who
is
actually
Raisa
Labs
has
it
is
a
company,
it
is
a
child
company
of
another
company
called
magazine.
Luiza
I
will
talk
about
magazine
Luiza,
so
laser
lips
was
created
in
2013.
A
It
brought
a
whole
new
culture,
Church
magazine
Luiza,
and
we
we
were
creating
products
very
in
a
very
fast
way,
faster
than
magazine
movies,
IT.
So
eventually
we
took
over
the
whole
IT
from
magazine,
who
is
the
whole
operation
so
right
now
we
we
develop
and
operate
everything
for
every
IT
from
from
magazine
Luiza
and
we
don't
believe
in
innovation
departments
isolated
from
from
other
departments.
We
believe
in
that
innovation
happens
together
with
corporation.
So
that's
what
we
do
why
this
magazine
Luiza.
It
is
not
a
magazine.
A
It's
a
retail
company
that
has
60
years
old
800
starts
over
the
country
over
Brazil,
it
was
founded
in
franca,
some
paulo
brazil
franca,
it's
a
small
city
in
the
countryside
of
brazil.
The
company
itself
has
a
very,
very
interesting
story
that
are
not
take
our
time
talking
about
it,
but
you
you
can
google
about
it.
A
We
make
a
little
more
than
forbidden
in
animal
revenue.
We
have
several
digital
standing
channels
and
we
in
2016
crisis
year
special
in
Brazil
we
had
the
most
value,
is
stock
worldwide,
not
only
in
retail
companies,
but
every
every
market
now
Black
Friday,
which
you
guys
are
here
for
that
right.
So
I'll
talk
a
little
about
that.
A
Operations,
we
had
two
guys
two
really
technical
guys
receiving
the
man's
from
everyone
who's
everyone,
their
developers,
mostly,
we
had
around
40
developers,
creating
features
crazy
features
all
the
time
and
we
had
these
two
guys
and
stream
to
them.
So
we
had
this
crazy
environments
and
this
big
bill,
big
AWS
view
we
run
most
of
our
things
in
AWS,
like
92,
5%,
I,
think,
and
we
have.
A
We
had
an
architecture
area
that
we
called
architecture,
but
it's
not
a
it
wasn't
an
a
classic
architecture
area,
as
in
we
didn't
design
the
architecture
of
the
applications
then
give
to
the
developers.
So
the
developers
just
write
the
code.
We
reacted
like
on
the
developer
code,
culture
and
the
automation
of
infrastructure
and
stuff,
like
that,
so
the
developers
when
they
write
code,
they
know
that
the
code
they
write
is
not
their
child,
as
in,
if
other
developers
criticize
their
work,
they
are
criticizing
their
code,
not
them.
So
it's
we.
A
We
try
to
increase
the
the
quality
of
the
code
without
getting
to
the
people's
ego
and
stuff
like
that
and
how
he
deployed
the
code
in
2014
manually.
We
have
chef
and
fabric
and
mostly
that
and
on
Black
Friday
right
on
from
Thursday
11:15
9.
We
went
down
and
we
stayed
down
for
a
whole
hour,
maybe
a
little
more
than
that,
maybe
a
little
less
than
that,
but
rather
now,
which
was
ashamed.
A
This
26
this
26
minutes,
was
around
midnight
and
30.
Something
like
that.
So
we
lost
plenty
of
money
as
well,
but
not
as
much
money
as
2014,
so
it
was
way
better,
but
very
bad
as
well
and
this
year.
Actually
they
see
we
were
that
the
CEO
was
very
happy
with
us,
which
is
crazy
right.
We
went
down
hard
to
2026
minutes,
so
I
can't
say
how
much
money
we
lost,
because
it's
something
that
I
am
NOT
allowed
to
say,
unfortunately,
but
because
I
would
like
to
say
it's
it's
a
very
interesting
number.
A
A
The
thing
is,
the
problem
was,
with
once
per
take
a
specific
route
of
that
API.
That
was,
that
was
calculating
the
delivery
cost
of
the
products
which
was
critical
for
selling
the
products
so
that
that
service
went
down
and
we
switch
it.
We
quickly
switch
it
to
another
new
API
which
wasn't
tested
very
well.
That
was
that
was
the
reason
that
new
VP
I
wasn't
on
production.
So
we
had
just
we
didn't.
We
didn't
have
a
choice
we
had
to
just
switch
to
that.
To
that
new
API.
A
We
did
that
and
the
DI
key
on
the
past
the
teams
on
the
past.
They
were
afraid
to
bring
those
problems
to
the
light.
So
the
way
we
work
is
we
bring
the
problems
to
the
light
and
stuff
like
that.
So
the
CEO
was
very
happy
and
okay,
we
we
stayed
just
or
I,
don't
know,
26
minutes
done
so
breaking
up
a
little
the
timeline
and
not
jumping
straight
to
2016.
A
Kubernetes,
we
decided
that,
of
course,
we
we
needed
something
a
little
bit.
We
needed
a
little
more
than
ansible
our
shack
fabric
and
stuff
like
that
to
deploy
our
stuff
and
to
run
our
stuff
in
production.
Because
up
to
that
point
we
we
just
had
ec2
and
stuff
like
that
running
in
production,
and
we
we
rented
some
stuff
on
the
last
Beanstalk
as
well.
A
We
run,
we
are
running
production
apps
on
kubernetes,
since
May
2016
those
production
apps
were
deployed
by
hand
using
coop
control
by
my
team
by
the
architecture
team,
and
we
were
running
those
production
labs
on
AWS
and
we
took
one
specific,
specific
application.
That
was
the
mobile
site
that
received
lots
of
traffic
and
we
ran
it
on
Google
Cloud,
on
Jiki
for
I.
Think
three
months
and
the
mobile
book
mobile
site
used
children
on
South
America
on
NATO
blasts
and
it's
very
important
that
the
latency
is
very
important
for
us.
A
As
the
latest
increases,
our
conversion
rate
decreases
its
e-commerce
route
right,
so
Google
Cloud
isn't
on
South
America
yeah,
it's
it
will
become
available
in
the
next
few
weeks,
I
think.
But
at
that
time
it
wasn't.
So
when
we
we
put
the
mobile
site
on
North
America
to
run
on
Jiki,
we
were
worried
that
conversion
rate
would
decrease
or
something
like
that.
It
didn't
so
magically
it
didn't.
So
it's
something
that
to
have
in
my
Google
Network
is
awesome
and
I.
A
Don't
know
how
they
do
that,
but
so
we
needed
a
quick
way
to
quick
and
easy
way
to
deploy
applications
to
production
and
to
any
environment
that
we
wanted,
because
our
experience
on
at
that
point
to
deploy
applications
use
group
control
wasn't
very
easy
because
I
mean
we
can
learn
the
jános
and
could
control
and
stuff
like
that
very
fast
I
think.
But
we
can't
train
100
developers
to
do
that
and
if
you
do
that,
you
can
expect
to
have
many
mistakes
and
it
doesn't
scale
at
all
doesn't
scale.
A
That's
called
reason:
why
is
that?
Because
we
wanted
to,
if
we
had
a
kubernetes
cluster,
we
wanted
to
just
drop
the
tool
there
and
deploy
the
application
without
changing
the
team
workflow
without
touching
the
team
workflow.
So
it's
it
kind
of
works
as
Heroku
and
it
builds
on
tops
on
top
of
build
backs.
It's
a
simple,
simple
as
possible,
and
it
was
written
by
one
full-time
engineer,
which
is
not
really
a
good
thing,
but
anyway
he
was
actually
learning
go
and
coronaries
at
that
point
and
which
something
to
keep
in
mind.
A
A
This
is
this
is
crazy
because
those
700
deployments
were
made
by
mostly
by
a
single
team
that
was
working,
crazy,
developing
features
on
the
didn't
knew
the
delivery
cost
API
that
broke
the
last
year
in
2015,
so
which
was
actually
a
rewrite
of
that
that
new
API.
So
it
was
crazy
and
a
sign
of
success,
both
both
of
kubernetes
entries.
So
it
really
would
have
really
a
good
thing
with
the
reason
developers
don't
write,
docker
files
or
jános
it's
simpler.
A
They
use
the
just
write
prop
file
and
that's
it
operations
quickly,
understood
kubernetes
architecture
of
orchid
with
corporations
operations
understood
quickly
developers
friction
with
thoughts,
reduce
it
yeah,
that's
it
with
how
we
scale
the
the
clusters,
the
kubernetes
clusters.
It's
we
do
that
in
a
very
simple
way,
and
we
do
that
in
a
very
manual
way
as
well
skinning
knob.
We
just
increase
the
instance
cow
on
auto
scaling
groups
scaling
down
it's
a
little
more
tricky.
A
A
That's
it
what
issues
we
had
with
the
cluster
with
the
first
cluster
that
we
had
in
production,
some
load
balancers
were
edited
directly
on
the
DWS
dashboard,
those
those
load
balancers,
those
last
load
balancers.
They
were
provision
it
by
kubernetes
through
tourism.
So
when
the
developer,
when,
when
the
developer
deploy
an
application,
there
is
uploads
the
application
to
Terry's
API
there
is
a
API,
creates
the
deployment
on
kubernetes
and
also
scales
the
the
application
inside
kubernetes
and
provision
an
elastic
load
balancer
through
a
search
on
kubernetes.
A
A
Edging
SSL
certificate
for,
for
example,
so
when
kubernetes
decided
to
reload
the
service,
we
lost
the
certificate.
So
the
service
went
down.
We
had
those
issues
we
had
to
to
many
key
I
caused
or
made
the
bless
we
had.
We
had
to
many
routes
on
a
blast
table
because
we
were,
we
were
using
kuba
net.
As
the
network
driver,
we
were
weren't
using
an
overlay
driver
because
we
didn't
know
about
it.
We
discover
it
when
we
had
to
scale
the
cluster
so
very
bad.
The
notes
were
on
out
scanning
group,
but
the
we
didn't.
A
A
If
you
use
Google
Cloud,
it's
just
son
so
which
brings
us
to
a
point,
kubernetes
works
so
well
that
we
first
install
it
it
with
could
up
which
I
hate
by
the
way-
and
we
forgot
about
it-
we
just
focus
it
on
trees
and
we've,
giving
Teresa
CLI
to
developers
and
stuff
like
that.
Then
we
forgot
about
the
cluster
itself.
So
that's
not
something
we
do
you
do.
You
must
do
but
anyway,
so
right
before
Black
Friday.
A
We
had
that
tension,
don't
touch
the
environment
because
it
was
very,
it
seemed
to
be
bulletproof,
because
we
we
made
several
stress
tests
and
versus
I,
have
a
better
version
of
it
that
we
will
avoid
possible
issues.
Then
I'm
not
sure
that
that
will
happen.
I
have
no
idea,
but
if
they
happen
I'm
down,
so
my
position
was
I
will
bring
the
new
cluster
with
cops
with
a
chi
with
all
the
things.
But
my
director
gave
me
a
straight
order.
A
A
A
So
what
I
did
I
brought
a
new
cluster
of
cops
but
I
configure
it
as
a
fallback
cluster,
with
much
more
ta
z,
h,
al
everything
and
I
copied
it
all
the
apps
by
copying
the
it's
it's
cd3,
so
the
developers
didn't
had
to
didn't
have
to
deploy
your
applications
again
and
I
asked
it
for
another
run
of
stress
testing
on
the
new
cluster
and
the
metrics
were
fine
that
new
cluster
had
another
late
driver.
Of
course,
all
those
issues
were
solved
on
a
new
cluster.
A
We
configured
DNS
weighted
records
so
with
zero
weight
on
the
new
cluster
and
100%
on
the
on
the
the
old
Western.
So,
if
anything
started
to
happen,
we
could
switch
to
it
to
the
fallback
cluster.
So
my
director
was
ok,
I'm
happy
with
that
I
kind
of
didn't
touch
the
environment
and
the
application
owners
were
happy
too
because
they
from
their
perspective,
they
were
using
the
old
cluster
because
they
were.
A
So
2016
Black
Friday
developers
around
100
training
architecture,
focus
on
kubernetes
tourism,
on
an
automation.
How
deploy
and
run
code
we
had
ansible
run
deck
kubernetes
and
tourism
within
the
network
meant
not
whatsoever,
no
incidence
at
all
and
we
had
our
best
year.
I'm
not
allowed
to
say
how
much
he
did
on
the
Friday
and
on
our
first
hour
or
anything
like
that.
But
what
I
said
on
the
beginning,
our
company
had
the
most
valid
stock
stock
on
the
on
the
year
so
and
also
one
developer.
A
A
So
what
happened
in
three
years,
2014
forty
developers
sixty
minutes
down
ops,
was
manly.
Touching
the
environment,
2015
100
developers,
26
minutes
down
some
more
information
and
way
more
freedom
to
developers
to
tell
us,
60,
120
developers,
0
minutes
dot.
Even
more
automation
developers
could
deploy
a
critical
extra
production
whenever
they
want
without
notes
we
are,
we
would
create
openly.
A
The
features
have,
the
truth,
is
they
don't
know
and
they
will
feel
farm
just
to
feel
the
farm
and
they
will
they'll
get
bored
by
that
and
that
doesn't
work
by
our
experience,
so
what
we
are
doing,
and
that
proves
that
is
diva
more
control
to
them,
by
focusing
focusing
in
small
things
that
taking
care
of
building
a
pipeline
and
getting
the
right
developers
and
let
them
release
so
the
right
people.
Right
tools
are
important,
but
the
tools
are
secondary.
A
More
mistakes
on
tourism
we
chose
to
use,
go
swagger,
it's
painful,
it's
very,
very,
very
painful
single
developer
for
a
while.
This
is
this
is
something
that,
if
you
don't
have
a
choice,
do
that
put
one
developer
work
it
for
us.
I
knew
that,
but
it's
something
bad!
No
tests
on
tourism
take
aways,
always
educate
people
and
always
care
about
that.
Doesn't
don't
forget
about
it.
Don't
think
that
you
have
to
run
all
the
time
and
can't
educate
people.
A
Educated
education
must
be
in
your
framework
and
power
people
and
get
rid
of
anyone
doing
anything
close
to
the
opposite.
If
you
have
anyone
in
your
company
making
others
feeling
bad
get
rid
of
them
right
away,
if
you
rely
on
strict
and
slow
processes
really
softer,
you
might
have
the
wrong
people
so
I'm
talking
about
the
people
right
again
so-
and
this
is
a
what
kubernetes
talk
black
friday
talk
but
kubernetes
Appa
is
about
people.
A
Kubernetes
is
too
hard,
but
only
a
few
people
need
to
understand
it
so
that
you
can
use
it
effectively,
move
to
the
cloud
native
apps
as
soon
as
possible,
even
even
if
you're
running
on
premise,
because
that
doesn't
matter.
Actually,
if
you
have
a
cloud
native
architecture,
even
if
you're
running
on
premise,
when
you
need
are
you
want
to
move
to
the
cloud
you
might
be
able
to
do
it
without
changing
anything
without
without
stopping
anything.
A
A
A
A
A
Updates
in
production
on
the
cluster
on
the
application
to
either
deploy
ruling
updates
on
kubernetes.
What
is
this?
It
creates
another
deployment
entity
on
kubernetes,
so
it's
it
is
a
whole
lot
of
water
things
for
the
it
abstracts
that
for
the
for
the
developers,
but
basically
it
creates
another
at
the
end
of
the
process.
It
creates
another
docker
image
and
creates
another
deployment
antigen
on
organelles
and
running
updates.
It.
A
Right
right
we
have
one
cluster
per
environment,
so
we
have
one
environment
called
stage
and
we
have
another
called
production.
And
up
to
this
point
we
I
mean
in
the
past.
We
use
it
to
chew
a
trick
up
great
clusters
by
creating
any
one
and
switching
all
that
things
to
that.
And
that
was
a
painful
process
because
for
for
obvious
reasons
and
right
now,
we
are
upgrading
the
clustering
place.