►
From YouTube: Scaling Edge Operations at Onefootball with Ambassador: From 0 to 6000 rps - Jonathan Juares Beber
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Scaling Edge Operations at Onefootball with Ambassador: From 0 to 6000 rps - Jonathan Juares Beber & Rodrigo Del Monte, Onefootball GmbH
Onefootball is a media company with more than 10M monthly active users delivering more than 10 TB daily content. We needed a Kubernetes-based API Gateway and Ingress solution that could handle our 6,000 rps workload reliably and efficiently. In this session, we'll talk about why we chose the open source Ambassador API gateway, and how we made the migration to it. We will cover the challenges identified and the benefits we've seen, like: Cost reduction: Reduced the number of cloud-based load balancers from >100 to 4. Observability: The combination of Ambassador and Prometheus' capabilities to empower our small SRE team. Maintainability: We took advantage of Ambassador's Kubernetes declarative configuration, and we were able to decouple the cluster settings and applications' delivery process allowing more velocity on the shipment of new features.
https://sched.co/MPYF
B
Our
main
focus
today
is
not
to
create
how-to
or
to
bring.
You
is
tensive
technical
details.
It's
more
related.
We
even
with
a
high
traffic
environment
and
I,
is
now
a
straight
team.
We
were
able
to
deploy
ambassador
and
bring
some
money.
Reduce
money
safe
to
our
company,
reduces
operational
workload
and
deliver
new
API
gate
features
to
our
services.
So.
B
A
B
B
Now
you
just
give
a
quick
overview
of
our
environment
to
make
more
sense
for
you
when
we
start
talking
about
immigration,
so
we
run
on
mr.chu
instance
and
kubernetes
over
ec2
instance.
We
are
running
today
over
hops,
but
we
are
also
trying
a
key
s.
We
have
more
than
50
micro
serves
today
in
production.
B
It
says
MAOIs
engineering
team,
around
30
to
40
engineers,
and
we
have
applications
right
in
meinen,
golang
and
but
we
also
have
applications
in
HP,
nodejs
fight
on
among
other
languages,
and
we
are
a
media
company,
so
mainly
our
trafficking,
our
images
and
static
content.
So
we
use
a
CDN
solution
to
help
us
to
handle
and
did
not
receive
more
hits
that
we
already
received
and.
A
Now
we
are
using
Ambassador
as
ap
I
get
an
ingress
solution.
Some
part
of
our
content
is
also
cash
in
our
side,
so
we
are
have
used
of
rats
on
the
last
cache
and
the
kind
of
traffic.
It's
like
sometimes
pike.
We
have
some
huge
spikes.
For
example,
the
last
Cristiano
Ronaldo
transferred
to
events
generate
the
huge
spike
there,
so
we
can't
cash
everything
so
for
that
we
use
our
art
to
also
scale
our
widget
rapid
and
we
start
providing
some
context
about
where
we
were
by
the
end
of
the
World
Cup
2018.
B
Three
months
before
the
World
Cup,
we
start
to
migrate
everything
from
more
than
350
to
two
instance.
A
lot
of
elph
are
proxies
nodules
to
the
lovely
kubernetes
word
and
supported
by
how
Prometheus
and
this
cloud
native
environment,
Fred
Vinick
is
a
team
leader
at
one
football
and
told
them
a
little
bit
about
this.
This
process
in
the
last
EDA
blood
Summit.
B
So
you
can
search
for
how
one
football
won
the
World
Cup
to
have
more
details,
but
our
mission
that
time
was
to
keep
everything
as
simple
as
possible,
and
we
also
wanted
to
say
thank
you
to
the
company
and
specially
to
the
product
team
that,
on
that
time,
freeze
it
out.
The
software
delivers
software
features
just
to
pure
eyes
to
focusing
on
stability
and
performance.
So
that
was
where
we
were
at
that
time.
A
And,
of
course,
we
were
happy
about
our
recipes,
but
we
had
to
move
forward
and
when
we
were
migrating
to
the
Cuban,
actually
we
decided
to
not
change
too
much
using
the
rule.
First
make
it
work,
they
make
tried
to
make
it
fast,
and
at
that
time
we
had,
for
example,
one
euro
be
worth
serves.
We
had
like
40
service
running
and
also
for
login
regime'
Q
clusters
and
some
AJ
products
as
well,
and
we
were
surrounded
by
all
these
obesity
and
configurations
and
DNS
entries,
and
that
did
not
our
lives
is.
B
So
and
the
business
stream
was
pushing
forward
like
we
had
the
freezing
so
now
there
was
time
to
again
start
delivering
new
features
and
with
new
features
were
coming
up
new
services
forever
new
service.
We
had
a
new
LP,
a
new
CDN
config.
If
it
was
a
user
facing
servicing,
we
had
overhead
of
monitoring
because
every
new
server
was
not
centralized.
We
have
to
take
care
where
this,
how
this
traffic
is
coming
to
the
service,
how
we
would
monitoring
it
and
so,
for
example,
inside
the
ends
in
lbs.
B
A
And
so,
as
is
now
as
every
team,
one
of
our
requirements
was
find
some
simple
solution.
We
previous
tried
some
other
solutions
that
required
some
state
database
config,
which
could
add
some
overhead
to
the
team
and
Vasudev
keeps
everything
inside
kubernetes
take
advantage
of
the
annotations
inside
the
serbs
object
and
that
for
us,
for
us
was
a
killer
feature.
B
At
first,
we
intend
choose
ambassador
just
as
an
ingress
solution
so,
and
you
probably
know
that
we
had
another
ingress
solutions
at
kubernetes,
but
the
difference
is
once
we
needed.
Api
gated
features
and
they
I
get
a
capped
abilities.
Ambassador
was
there
red
choose
and
really
you
really
easy
under
the
hoods
buses
are
using?
B
Why
there
is
a
well-known
in
stable
solution
and
what
each
facilitate
us
to
integrate
with
another
serve
smash
solutions
that
also
use
invoice,
for
example,
is
true
that
one
thing
that
we
are
trying
now
but
and
also
apply
different
invoice
futures
or
and
the
performance
is
always
the
same-
that
if
I
would
provide
for
us,
because
I'm
ambassador
just
generate
configs
and
pass
it
to
my
other
traffic.
Go
just
through
invoice
so
far
as
ambassador
of
itself
should
be
a
open
source,
kubernetes
native
API
gate,
as
that
of
our
safe
and.
A
You
might
be
asking
yourself
why
not
use
just
the
Cuban
Edison
grass,
but
ambassador
is
more
than
HTTP
router.
We
used
to
say
that
has
some
batteries
include.
For
example,
we
use
the
traffic
shadowing,
so
we
are
delivering
new
machine
learning
service
in
production
and
we
use
this
traffic
shadow
to
test
the
service.
We
will
provide
more
details
in
the
further
slides.
We
also
use
housing
based
on
headers.
A
B
So
we
had
the
up
the
API
gate
solution,
the
final
that
we
had
to
start
thinking
about
how
to
migrate
everything.
At
that
time
we
had
just
around
40
micro
services,
and
but
it
was
production
in
stations,
so
it
was
more
than
80
migrations
to
plan
and
apply
and
it
was
a
small
team.
So
we
had
this
challenge
up
front
so.
A
In
doubt,
time
was
not
an
option
for
answer
were
receiving
production
traffic,
so
basically
the
DNS
setup
was
completely
manual
in
our
side
and
it
was
hard
to
maintain
and
other
mistakes
that
sometimes
happen,
for
example,
could
fail
and,
for
example,
we
could
create
new
service
and
forgot
to
point
the
to
the
right
side
in
the
DNS
sites
to
the
right
service.
Of
course,
it's
possible
to
automate,
but
it's
human,
her
prom.
We.
B
Always
use
at
the
helm
to
the
player
application,
so
each
application
inside
its
on
the
repository.
It
had
a
kubernetes
path
and
it
was
nice
because
it
keep
it
the
it
kept
the
application
code
really
near
to
its
to
the
its
configurations.
But
at
the
same
time
it
was
hard
for
us
through
tracing
change
across
40
different
repositories,
so
for
sure
it
didn't
help
in
our
productivity.
B
A
The
searching
is
not
actually,
as
every
team
is
more
than
a
competency
area
across
multiple
cross-functional
teams,
in
at
one
football,
and
sometimes
projects
like
that
can
get
lost.
So
you
have
a
project
across
multiple
teams.
So
you
had,
we
didn't
have
any
leader
our
product
manager,
pushing
its
to
be
to
production
and.
B
Since
we
had
multiple
load
bars,
we
start
to
fuse
thread
with
like
with
too
many
points
of
configuration
and
even
ours.
Access
logs
of
the
load
bands
were
spread
across
multiple
load
balancer.
We
could
like
centralize
everything
in
just
one
bucket
out
loud,
but
it
was
not
easily
integrated
with
our
lord
centralized
the
logs
solution,
so
it
was
the
same
way:
father
configs,
like
timeouts
or
cell
phone
figs
or
health
check
endpoints.
It
was
hard
to
find
what
was
being
applied
to
each
application
and.
B
So
the
first
thing
that
we
came
up
was
to
create
our
own
helm
shark
to
upholster.
It
was
really
important
for
us,
because
the
team
could
focus
the
Essery
competence.
Harry
could
focus
in
just
one
get
through
poster
and
we
start
to
create
an
important
separation
between
the
applications
code
version
and
the
applications
configuration
ration.
So
with
these
two
important
artifacts
start
to
be
generated,
the
code
application
itself,
so
it
deprecations
code
itself.
So
it
was
a
docker
image
artifact
without
the
dependencies
and
a
ham
chart
virtual
artifact
as
well.
B
A
The
ham
shark
poster
that
we
created
it's
simple
as
possible.
The
ham
configurations
that
live
with
inside
the
application
was
moved
to
decentralize
it
reporter
and
introduced
some
simple
version
in
this
helm.
Sharks
like
a
semantic
version
and
each
time
that
a
new
commit
is
married
to
the
branch
master,
how
the
change
it
charted
is
builded
and
pushed
to
the
repository
and.
B
When
the
application
is
butan
deployed,
also
our
CD
solution
applies
the
last
time
shard
fashion
to
each
application,
so
with
this
Martin
will
be
able
to
trace
it
easily.
We
could
change
Martin
on
application
and
add
the
Ambassador
config
from
our
general
application
/
to
request
even
out
of
them.
If
you
wanted
and
again
keep
tracing
of
that
very
easily,
which
application
is
using
Ambassador
or
not,.
A
And
now
how
the
configuration
about
the
load
balancer,
it's
centralized
in
just
one
cuban
and
service
the
ambassador,
and
we
have
to
run
it
now.
It
was
time
it
wasn't
start
to
create
out
applications
mappings
for
each
service
and
ambassador
works
like
my
control
plane
and
it
tries
to
simplify
invoice
configuration
so
to
create
these
maps.
I
will
need
to
create
a
notation
inside
the
Cuban
Air
Service,
for
example.
Then
this
image
represents
bisoder
definitions,
so
we
have
the
kind
mapping
with
the
host
and
points
to
the
search
that
you
use.
B
Now
it's
a
little
bit
outdated
because
that,
while
just
releases
this
year,
it
is
but
yes
here
you,
you
can
focus
on
the
host.
The
first
host
configuration
it's
a
regular
expression.
We
you
can
use
regular
expressions
or
not.
We
mainly
our
service,
receive
answer
a
lot
of
names,
so
we
use
it
as
X
in
meaning
out
deserves.
But
what
we
saying
is,
if
one
request
comes
to
ambassador
in
matches
this
host
hijacks,
the
traffic
will
be
sent
to
the
disservice
and
it's
not
mandatory
to
send
to
the
same
service
of
where
the
annotation
is.
A
When
we
are
going
to
production,
we
are
confident,
but
not
enough
to
switch
the
DNS
around
and
even
for
non-critical
service.
So
the
in
the
odd
but
goat
Technic
that
we
use
in
Dar
Jimmy's
when
we
were
migrations
migrate
in
some
service,
is
to
use
the
hot
53
waves
so
and
it
works
it
work
at
previa.
Well
for
this
case,
so
our
immigration
process
was
like
that.
We
first
deployed
the
Ambassador
mapping.
A
Then
we
just
testing
the
mapping
shacking
in
rule
or
hit
redirecting
like
these
shacks
to
the
host
and
it's
a
simple
curve
can't
asset,
and
then
we
start
to
increase
the
traffic
in
the
hot
53
wave.
So
we
started
with
1
percent
or
10
percent
depend
of
the
in
the
begin
dependent
of
the
service
and
start
to
monitor
it,
gained
more
confidence
and
to
increase
into
the
100
percent
of
the
traffic
to
ambassador
and.
B
This
kind
of
process
that
it's
not
so
easy
when
you
have
to
sync
between
a
lot
of
people
when
the
big
race
was
done,
we
removed
the
audio
out
well,
the
balancers
from
Amazon,
and
just
with
that,
with
we
saved
Martin
twenty
thousand
dollars
per
year.
So
it
was
not
so
much,
but
in
the
end,
it
also
removed
its
complexity
in
our
DNS
configuration
and
since
applications
now
pointed
to
the
always
to
the
same
address
to
the
load
bar.
So
there
was
a
responsible
for
ambassador.
A
Another
thing
that
we
did
at
the
same
time
we
also
create
a
new
domain.
That's
API
dog
food,
calm!
It's
this
second
map,
so
we
can
have
more
than
one
mapping
in
your
configuration.
So
basically,
this
mapping,
what
it's?
Okay,
everything,
that's
API,
dot,
room,
food,
food,
calm,
/,
the
shark
name
or
the
surfs
name,
please
sign
to
the
specific
service
and
at
the
same
time
we
could
keep
the
compatibility
with
the
old
clients
that
use
the
old
URL
ow,
for
example,
serves
dot
from
food.com.
A
B
Since
we
started
using
Buster
as
I
told
you,
we
could
focus
in
just
one
point
of
corporation
but
Martin
that
ambassador
pod
was
integrated
with
our
solutions
for
our
applications,
so
the
same
pipeline
did
deliver
our
applications.
The
same
is
Crapper
metric
solution
that
we
use
for
our
applications.
The
same
map,
the
same
CD
pipeline
was
the
same
that
we
are
using
for
our
API
gated
solution.
So
it
will
attack
us
directly.
B
A
And
before
ambassador,
we
had
some
lack
of
metrics
when
talk
about
the
load
balancer.
All
the
throughput
was
measured
now
why
it
works,
use
DPM
tool
and
of
course,
there
was
way
to
to
collect
metrics
from
him
from
cloud
and
generate
some
reports
automatically,
but
first
it
was
not
done.
Second,
we
we
imported.
We
could
import
this
matrix
to
existing
solution
or
would
have
a
third
place
to
start.
This
solution.
A
B
It
was
pretty
much
the
same
thing
for
us.
We
just
have
like
every
why
everyone
basilar
pod
has
the
employee
pod
void
as
well,
and
it
has
a
site
container
that
was.
Is
that
stay
exported?
This
is
touch
the
exporter.
Project
is
responsible
for
converting
stats.
D
metrics
form
a
true
parameters
from
it.
B
B
A
As
you
said
before,
we
provide
account
for
more
than
10
million
users
around
the
world
in
13
different
languages.
You
know
for
sure
we
have
all
these
test
suite
that
you
might
know
like
unit
test
in
Turkish
integration
tests,
uy
tests
and
so
on,
but
sometimes
it's
difficult
to
avoid
some
odd
behaviors
in
production.
B
B
So
and
we
try
to
avoid
this
bad
experience
for
the
users
we
try
to
create
controlled
explosions,
so,
for
example,
releases
software,
/,
language,
new
features,
/
language
or
with
increasing
percentage
of
the
traffic,
so
it's
basically
I
kind
of
redeployed.
So
the
problem
was
before
Ambassador.
We
were
creating
this
kind
of
release,
logic
on
the
application
code
level
or
we
are
trying
to
create
it
on
the
CDN
level
and
it
was
hard
to
maintain
error-prone
and
the
the
worst
part
was
hard
to
hold
back.
A
And
then
comes
ambassador
if
the
batteries
included
and
if
some
fails
in
this
list,
we
can
quickly
hold
back
the
helm,
sharp
version
and
we
start
everything
here
is
two
examples
of
kinds
of
release
that
we
do
so.
The
first
one
is
the
shadowy.
So
we
have
in
order
to
target
the
counter
to
the
right
audience.
We
have
to
tag
this
content
and
we
are
improved.
These
serves
use
some
machine
learning
magic
to
do,
but
we
don't
want
to
switch
everything
at
all.
A
B
Second
example:
here
we
have
like
choose
two
services,
and
this
new
API
green
just
received
an
important
software
feature.
It
was
changing
a
feature
that
was
really
important
for
us
and
the
user
facing.
So
what
we
did,
if
you
see
in
the
prefix
in
the
first
one,
we
tagged
just
the
NL
and
a
rube,
so
it's
the
Dutch
language
in
the
Russian
language.
So
we
can
deliver
this.
This
feature
just
to
this
true
language,
and
it
was
really
important
for
the
product
team
and
as
well.
B
B
We
want
to
talk
with
you
a
little
bit
about
what
we
see
being
our
next
steps
with
combustor
now,
so
the
first
finger
Buster
service
as
an
ingress
solution,
but
more
than
that,
they
brought
us
something
that
we
were
not
looking
for
at
that
time.
But
now
we
know
the
power.
So
we
want
to
have
the
same
capability
now
of
tracing
of
cannery,
release,
of
increasing,
centralized
solutions
and
declarative
solutions
to
our
cluster
inside
to
the
traffic
inside
our
cluster.
So
the
everything
that
is
inside
the
cluster
have
to
have
the
same
capabilities
and.
A
Today,
our
TLS
just
being
terminated
by
the
load
balancer
and
one
of
our
concerns,
of
course,
the
Muto
to
Alice
between
applications
and
support,
to
have
the
track
secured
in
features
like
secret
braking
and
to
increase
the
reliability
of
a
certain
another
point
of
the
observability
that
basa
der
allowed
helped
us
is
the
it
brings.
The
metric
is
in
the
proper
locks,
but
we
still
missing
some
distributed
recent
solution
so.
B
Ambassador
for
us
is
really
the
north
to
south
traffic
goalkeeper
so
out
it
traffic
that
is
coming
that
is
enters.
Our
cluster
is
passing
through
ambassador,
it's
great
for
us
and
we
have
an
active
way
to
handle
this
traffic,
but
inside
the
cluster
we
have
the
west
to
east
traffic.
So
we
are
looking
for
is
true
and.
A
Yet,
as
a
part
of
the
new
roadmap
who
we
are
going,
we
opted
to
start
to
use
to
test
is
to
do
because
these
two
is
also
using
my
under
the
hood,
and
she
has
a
good
integration
with
Ambassador.
So
we
are
able
to
have
this
distributed,
trace
mo
totally
as
retries
and
so
on
and
is
to
can
be
use
it,
as
well
as
the
ingress
solution.
But
the
time
that
we
were
evaluating
this
solution,
we
opted
to
go
with
Ambassador,
to
which
the
simplest
to
get
something
where
in
production
and
delivering
value.
So.
B
Ambassador,
for
example,
together
official,
very
well
because
one
it
allows
us
to
define
the
last
context
for
each
application
for
each
mapping,
so
we
just
have
to
map
the
is
true
certificate
and
then
the
MU
total,
as
we
start
working
ambassador.
Have
this
that's
D
that
we
told
you
about,
and
so
we
have
easily
goes
to
issue
primitives
and
says:
please
grab
this
data
is
crap
this
data
from
ambassador's.
That's
the
exporter
side
container
yeah!
That's
just
one
thing,
and
we
have
also
the.
A
A
Integrates
Bureau
of
zip
key
API
take
advantage
of
the
vitro's
capabilities.
It
generates
the
headers
to
be
used
across
the
applications
to
dipti
two
different
applications
and
each
also
defining
a
tracing
serves.
So
you
can
send
it
to
external
stars
like
Zipkin
wagger,
our
light
step,
who
we
receive
all
this
information
and
save
to
yes,.
B
We
have
an
example
here
with
some
tasks,
for
example,
should
settle
distributed
tracing
service,
it's
just
a
few
lines
of
code.
In
this
example,
for
example,
ambassador
we
start
sending
out
distributed
tracing.
The
tails
should
use
examples
if
picking
on
the
part
94
and
with
just
one
tracing
serves
money
fast.
Other
requests,
we
start
to
being
reported.
So
it's
pretty
easy
and.