►
From YouTube: Bridging the gap between on-prem and cloud: a story about... - Tristan J Blease + Michael Chang
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Bridging the gap between on-prem and cloud: a story about Envoy + a hybrid boundary - Tristan J Blease, Groupon + Michael Chang, Groupon
Groupon operates regional data centers where our product and platform engineering teams deploy and manage over 600 microservices on-premise. This year marks the start of our journey to migrate these microservices to the cloud where we expect them to run alongside our existing on-premise deployments.
Like many others on similar journeys, we have an important problem to solve: how do we enable traffic to cross our new hybrid boundaries?
From the previous usage, we know that Envoy can play a pivotal role in solving this problem, but what other components would be needed? How can we create a fully automated system that deeply integrates with our on-premise environments, which currently lack automation in key places? For this project to be successful, we have to answer these questions and more.
Join us to learn how Groupon is answering these questions and to learn how we are rolling out a fully automated system of edge proxies leveraging Envoy and other technologies for traffic management.
A
Good
afternoon
my
name
is
Tristan
pleased
and
I'm
joined
here
by
my
colleague,
Michael
Chang,
and
we're
from
Groupon
and
we're
here
to
talk
to
you
folks,
a
little
bit
about
the
hybrid
boundary,
which
is
a
project
that
we've
been
involved
with
at
Groupon
and
we're
pretty
excited
to
share
with
you
folks
kind
of
a
journey
that
we're
on
our
way
through
right
now.
So
Groupon
as
it
is
today,
has
largely
existed
in
on-prem
data
centers.
Now
today
we
are
moving
our
workloads
to
the
cloud.
This
is
a
pretty
exciting
time
for
us.
I.
A
Think
a
lot
of
folks
in
the
audience
have
probably
already
been
there
done
that,
but
it's
been
a
really
great
chance
for
us
to
kind
of
re-evaluate
the
things
that
we're
doing
and
see
if
those
things
make
sense
or
not
as
we
migrate
and
move
forward.
So
now
Groupon,
if
you're
not
familiar
with
us,
you
know
Wikipedia
sums
it
up
pretty
well
we're
an
American
world
wide
ecommerce
market.
A
We
connect
subscribers
with
merchants
and
we
have
all
kinds
of
activities,
travel,
goods
and
services,
and
we
operate
in
more
than
just
the
US
we're
actually
in
15
different
countries.
Now
we're
also
the
third
largest
e-commerce
app
and
mobile.
From
what
I
understand
so
I've
been
been
to
Groupon
comm.
Do
yourself
a
favor
check,
it
out
see,
what's
see,
what's
changed
since
you've
been
there
last
now
before
we
could
talk
about
moving
to
the
cloud.
Let's
take
a
look
at
how
we
operate
today.
So
Groupon
is
largely
a
service-oriented
architecture.
A
We
went
from
a
monolithic
Ruby
on
Rails
application
to
over
600
different
micro
services
in
a
pretty
small
amount
of
time.
Now
most
of
these
services
are
HTTP,
HTTP
based
REST
API
is
that
serve
up
JSON
and
consume
JSON
and
they're
implemented
with
a
pretty
wide
array
of
different
languages
and
frameworks.
Well,
most
of
them,
though,
today
are
now
being
written
in
Java
and
nodejs.
A
Now
these
services
are
served
up
from
data
centers
that
are
spanning
multiple
geographic
regions,
and
we
have
get-ups
based
infrastructure
management,
tooling
in
system
in
place
that
helps
our
teams
manage
it.
Now.
All
of
our
service
owners
are
responsible
for
managing
their
own
infrastructure,
and
so
they
do
so
with
the
help
of
different
platform
teams
that
are
there
to
provide
support,
whether
it's
through
load,
balancing
firewall
configuration
or
just
host
management
in
general,
and
we
do
have
a
pretty
healthy
mix
of
both
bare
metal,
as
well
as
KVM
based
virtual
machine
infrastructure
to.
B
B
So
the
second
initiative
here
is
the
communities
we
do
want
to
use
that,
and
you
guys
probably
know
better
than
me
and
there
are
tons
of
benefit
in
using
communities
like
improve
the
utilization
and
the
orchestration
of
all
the
resources
and
I
hate.
More
is
cube.
Car
you
guys
are
definitely
learn
more
about
that
and
by
the
way,
this
presentation
will
focus
on
the
first
one,
which
is
the
go
hybrid.
B
Okay.
Now
we
decided
to
move
in
the
hybrid
model.
So
let's
talk
about
what
features
or
Inquirer
Minh
we
need
here.
So
we
know
in
the
hybrid
model,
there
are
two
some
clusters
in
the
on
premises
and
some
others
in
the
cloud.
So
this
question
is:
how
can
we
distribute
traffic
between
the
pram
on
purim
and
cloud
and,
of
course,
we're
not
gonna
roll
out
this
100%
at
day
one?
So
we
definitely
need
some
weighting
feature
to
allows
gradually
ramp
up
traffic's.
B
Additionally,
we
want
to
detect
service
failure
very
quickly
and
a
fallback
to
the
next
available
environment.
If
that's
available,
it
should
be
automatically
along
the
road.
We
want
to
avoid
manual
process
as
first
dimension
before
our
service
team
on
their
infrastructure,
but
they
will
get
some
support
from
the
support
team
like
provisioning,
VMs
or
configure
like
load
balancer,
Dinah's
VARs
and
the
many
of
those
processes
are
done
manually
today
and,
of
course,
well
slow
us
down.
B
So
we're
trying
to
utilize
the
csadd
pipeline
to
automate
the
provisioning
and
the
deployment
of
the
configurations,
and
we
do
want
some
dashboards
and
alerts
in
place
automatically
to
make
sure
the
system
is
running
correctly
and
the
service
are
meeting
device
allows
it.
The
last
one
is
the
security
you
know.
Many
group
answer
is
required:
end-to-end
encryption
to
ensure
the
security
and
the
customer
privacy.
So
we
want
that
in
this
requirement
and
access
control
currently
is
done
pretty
much
by
the
static
IP
address.
B
A
So,
given
we
know
what
our
core
requirements
are
for
operating
in
a
hybrid
model,
let's
take
a
look
at
the
different
components
that
we
decided
that
we
will
pull
in
to
build
up
the
system.
I
suspect
you're,
probably
going
to
know
at
least
one
of
them
all
right.
So
our
key
actor
here
is
Envoy
and
we're
really
just
leaning
on
Envoy
for
its
core
load.
Balancing
and
proxying
capability
envoy
has
been
battle,
tells
tested
elsewhere
at
Groupon.
A
Now
we
also
have
with
it
too,
as
well
a
support
agent.
Our
control
plane
is
distributed.
It
runs
inside
cars
alongside
of
each
of
the
Envoy
instances,
and
so
this
support
agent,
it's
written
in
golang.
It
is
using
the
go
control
plane
to
import
the
XDS
API,
so
that
envoi
requires-
and
it's
also
doing
a
few
other
things
too,
as
well.
So
when
this
agent
comes
online,
the
first
thing
that
does
is
it
goes
ahead
and
it
registers
the
edge
proxy
note
with
a
no
sequel
database
up
in
the
cloud.
A
So
we
use
that
registry
then
to
find
other
Envoy
instances
that
are
deployed
across
different
availability
zones,
so
we
can
mesh
them
together
and
send
traffic
back
and
forth.
Now
this,
as
I
mentioned
also,
the
support
agent
contains
the
configuration
for
on
voice.
So
it
is
also
pulling
that
same
no
sequel
database
a
different
table.
A
They
are
to
put
it
back
the
different
service
configurations
that
are
in
place,
so
it
can
present
that
to
envoy,
as
well
as
to
the
other
components
that
we
have
deployed
on
Prem
for
automating
away
some
of
the
manual
process
that
we're
trying
to
to
fight
as
hard
as
we
can
to
get
away
from
all
right.
So
I
mentioned
we're
using
cloud,
no
sequel
database.
The
agent
will
pull
this
in
a
regular
interval.
We
expect
everything
will
be
eventually
consistent
and
wonderful.
So
far
in
our
testing,
it's
worked
out
pretty
well.
A
All
right
now,
additionally,
we
do
use
bird,
the
bird
internet,
router
daemon,
which
is
used
for
connecting
to
our
top-of-rack
switches
and
we'll
talk
a
little
bit
about
bgp
in
a
second
here,
and
we
also
use
quarantine
s2
as
well.
One
of
the
things
that
is
kind
of
challenging
to
us
in
our
own
Prem
environments
is
that
the
load
balancers
are
hardware
cool
manually,
configured,
not
cool
same
goes
with
DNS,
and
so
we've
kind
of
put
together.
A
So,
let's
take
a
look
at
this
ideal
architecture
from
the
standpoint
of
you
know,
looking
at
it
from
the
perspective
of
a
client
committing
to
a
service
that
is
on-premise
one.
Second,
here,
okay
step,
one
client
resolves
the
DNS
all
right.
This
is
fairly
straightforward.
Now,
what
we've
done
is
we've
introduced
a
new
global
top-level
domain
within
Groupon
that
is
owned
entirely
by
the
hybrid
boundary.
A
So
in
this
model
we
are
using
split
DNS
and
so
the
hostname
when
the
client
resolves
it
is
always
going
to
resolve
to
a
virtual
IP
in
front
of
the
edge
proxy
nodes
that
are
running
within
the
data
centers.
So
an
example
of
split
DNS
if
you're
not
familiar
with
that
is
in
the
NA
one
data
center,
my
app
dot
production
dot
service
may
resolve
to
1000,
but
then
in
the
EU
data
center
that
may
resolve
to
ten
0.51,
so
same
name,
different
IP
addresses
all
right,
so
client
has
got
its
Ford.
A
It's
connected
to
the
on-prem
DNS
server.
Now,
with
that
top-level
domain,
it
knows
to
forward
that
look
up
then
to
core
DNS
that's
running
on
our
edge
proxy
infrastructure.
So
it's
going
to
go
ahead
and
return
the
virtual
IP
back
and
then
the
clients
going
to
go
ahead
and
establish
a
connection
to
that
now.
A
This
is
fairly
straightforward
and
then
it'll
complete
the
connection
to
the
remote
service
running
in
that
cloud
availability
zone,
all
right
cool.
So
we
mentioned
earlier
that
we've
got
the
support
agent,
and
so
this
support
agent
is
configured
to
read
directly
out
of
this
table
are
the
set
of
tables,
I
should
say,
and
it's
again
checking
for
updates
every
30
seconds
just
to
see,
what's
changed
and
refreshing
the
position
of
any
edge
proxy
deployments
in
addition
to
the
services
that
are
configured
to
talk
across
our
hybrid
boundary.
A
So
in
this
model,
with
our
support
agent
powering
both
the
Envoy
config,
as
well
as
the
bird
and
core
DNS
config,
we're
now
in
a
state
where
we
don't
have
to
make
any
manual
changes,
traffic
will
come
in
and
it'll
be
served
awesome.
Now,
if
we
take
a
look
at
this
on
the
inverse,
then-
and
we
look
at
the
cloud
to
unprepped
flow,
fairly
straightforward
and
somewhat
similar
now
in
this
model,
we
don't
need
to
run
core
DNS.
A
We
don't
need
bgp
because
we're
able
to
automate
these
primitives
very
simply
just
by
using
SDKs
and
api's
and
available
in
the
cloud.
So
if
we
look
at
the
connection,
we
see
that
the
clients
gonna
make
that
look
up
against
the
cloud
DNS.
It's
gonna
connect
to
that
cloud,
load,
balancer
back
to
the
edge
pro
and
edge
proxy
instance
to
another
edge
proxy
instance
and
back
to
the
service.
So
again,
the
support
agent
is
configured
identically.
B
Okay,
that's
a
pretty
good
design.
Let's
talk
about
gears.
Gears
is
the
corporate
engineering
architectural
review
system.
This
is
a
kind
of
internal
review.
We
have
to
do
for
every
brand-new
project,
so
a
bunch
of
our
principal
engineer
will
review
the
design
to
make
sure
it
makes
sense,
annual
Fedora
environment.
There's
one
very
important
question
asked
the
question
was:
do
we
really
want
to
support
TCP
services?
I
will
give
you
a
little
bit
background
here.
So,
as
you
may
see,
this
is
a
very
phenomenal
change.
Tour
infrastructure
and
we
created
created
a
migration
plan.
B
B
We
can
use
the
Envoy
HTTP
connection
manager
to
load
advanced
traffic
and
erode
by
the
host
name,
which
is
the
new
top-level
domain
which
is
introduced
and
without
TCP
we
can
drop
actually
BGP
and,
as
you
may
know,
BGP
is
pretty
complicated
to
configure
like
you
have
to
come
up
with
SS
and
the
mess
up.
We
were
maybe
quad
router,
that's
kind
of
dangerous
to
do,
and
you
may
encounter
some
hotspot
issue.
B
So
that's
complicated.
The
downside
of
dropping
TCP
is
like
we
have
to
find
a
way
to
route
traffic,
which
means
the
existing
hardware
load
balancer.
So
the
downside
of
of
that
is,
you
have
to
introduce
the
manual
process
which
is
non-ideal
and
personally
I.
Think
BGP
is
pretty
fun.
You
probably
don't
get
a
lot
of
chance
to
play
with
it
so
set
ready
to
go
okay.
This
is
the
components
we
don't
need,
BGP.
So
no
more
birds,
Viper's.
B
B
A
All
right
raise
your
hand,
raise
your
hand
if
you
like
scope,
change
all
right,
so
we're
working
on
this
we've
got.
You
know
our
groundwork
laid
we're
working
on
implementing
the
system
as
we've
kind
of
shown
and
shared,
and
a
slight
change
in
direction
appears,
and
this
came
out
of
a
really
good
conversation
that
we
were
having
internally.
You
know
we're
all
focused
on
you
know.
We
need
to
make
sure
that
site
reliability
is,
is,
is
key
right,
uptime
equals
revenue
for
e-commerce,
and
so
we're
realizing
that
okay.
A
Well
we're
going
in
at
this
from
the
approach
of
will.
If
we
run
50/50,
that's
got
to
be,
you
know
safer
than
going
all-in
right.
Well,
maybe
not
so
much.
We
started
thinking
about
kind
of
what
was
there
in
terms
of
operational
overhead
and
we
realized
ya
know
we
probably
should
just
go
all-in
and
that's
what
we
did,
and
so
with
that
we
made
the
decision,
then
that
all
right
we're
no
longer
going
hybrid,
we're
gonna,
go
all
in
and
we're
gonna
migrate,
all
of
our
core
traffic
to
the
cloud.
A
Instead
of
trying
to
operate
in
this
hybrid
model,
where
we
could
definitely
do
it,
how
well
we
could
do
it,
I,
don't
know,
ask
anybody
else,
who's,
doing
hybrid
and
see
how
well
they're
doing
it
all
right
so
with
this
meant,
though,
is
that
mean
that
we
are
now
further
reducing
kind
of
our
investment
in
our
on-prem
processes,
all
right,
it's
a
simpler
model.
After
we
get
done
with
this
transition
for
our
developers
and
operators.
How
does
that
impact
us?
Well,
you
know
the
ability
to
wait.
A
Traffic
and
shift
it
over
now
is
way
more
important,
because
now
you
know
we're
looking
to
distribute
that
traffic
across
more
availability
zones,
because
we
want
to
continue
to
maintain
or
further
improve
our
geo
redundancy.
Also,
you
know
we
want
to
make
sure
that
we're,
as
mentioned
before,
we
want
to
do
this
safely.
We
don't
want
to
jeopardize,
say
reliability
in
any
way.
Now
this
led
to
us
start
thinking
more
critically
about
what
we
had
planned
to
do,
and
we
decided
you
know.
A
We're
no
longer
to
be
leveraging
those
components
and
we
don't
need
envoy
on-prem
either,
but
we
kept
it
in
here
on
the
slide
of
course,
because
we're
using
that
on
our
cloud
environments
in
this
model.
Instead,
we've
decided
that
hey
all
right.
We
have
these
well
known
interfaces
that
services
are
already
behind
already
the
hardware
load
balancer.
What
if
we
just
put
the
edge
proxy
in
instances
that
we
have
in
cloud
for
the
AZ's
that
we
want
to
be
distributing
traffic
to
into
those
those
load
balancing
pools?
Can
we
achieve
the
same
thing?
A
And
the
answer
is
yes:
we
can
now
in
this
model
what
that
means
is
we're
no
longer
changing
DNS
names,
and
so
we
can
see
that
you
know
before
we
had
to
look
up
to
resolve
the
custom
DNS
now
one
of
the
hidden
costs
here
is.
That
means
that
we
also
have
to
go
through
each
of
these
600
microservices
to
update
the
host
names
and
make
configuration
changes
now.
A
Groupon
has
a
lot
of
different
languages
and
frameworks
and
use
it's
the
kind
of
thing
that
can
be
done,
but
it's
something
that
takes
a
lot
of
coordinated
effort
across
teams,
because
you
know
our
team
doesn't
necessarily
own
all
of
the
micro
services
and
we
don't
necessarily
want
to
work
with
every
single
team.
You
know
to
make
this
change
it'd
be
a
huge
undertaking,
so
okay,
we'll
eventually
get
there,
but
for
v1.
Let's
just
use
the
same
name
that
these
load
balancers
had
before
and
so
in
this
model.
A
What
we
do
is,
then
we
are
fronting
our
remote
edge
proxy
nodes
with
a
again
just
a
cloud
load,
balancing
endpoint
and
we
are
now
doing
managing
the
traffic
distribution
to
these
endpoints
at
the
hardware
load
balancer
directly.
Now
does
this
mean
more
manual
process?
Yes,
have
we
failed?
No,
at
least
we
keep
telling
ourselves
that,
but
in
this
model,
though,
you
know
we're
now
in
a
position
where
we
can,
you
know
more
seamlessly
migrate
things
over
as
they're
ready
to
go
on
a
wave
by
wave
basis,
all
right.
A
So,
let's
flip
that
around
and
let's
take
a
look
at
what
this
looks
like
from
the
other
direction
so
before
we
can
see
that
we
had
our
edge
proxy
nodes
meshed
together
across
all
of
the
availability
zones,
and
now
we
have
our
cloud
veil
booty
zone
H
proxies
just
communicating
directly
to
the
service
endpoints,
and
so
with
some
clever
subnetting
and
tricks
to
the
firewall.
It
turns
out.
It's
not
that
hard
to
do,
and
in
this
case
we
managed
to
cut
out
a
pretty
significant
chunk
of
scope
as
part
of
this
change.
B
Currently
we
deployed
the
hyper
boundary
project
into
sandbox
and
a
staging
environment
and
confirmed
the
core
functionalities
working
as
expected,
and
now
we
are
actively
load
testing
it
and
tuning
the
system
to
make
sure
it's
ready
for
the
production.
So
next
step,
as
we
mentioned
before,
we
drop
tcp
on
purpose
for
the
initial
rollout.
But
there
are
some
group
on
services
do
use
TCP.
B
So
we
will
eventually
support
TCP
services
in
the
later,
and
you
know
the
other
initiative
here
is
a
group
on
kubernetes,
but
currently
the
hyper
boundary
projects
is
designed
to
operate
independently
from
communities,
and
we
do
want
to
tighten
that.
And
if
you
take
a
close
look
at
each
proxy
component,
you
know
it
has
all
the
information
or
knowledge
required
to
communicate
it
with
another
service.
B
So
each
processor
can
troll
the
traffic
for
you
and
if
you
deployed
at
prophecy
inside
a
container
application
container
or
a
pod,
you
buddy
essentially
become
a
sidecar
so
that
that
way
you
can
reduce
the
extra
hop
and
it
reduce
the
latency,
and
this
looked
a
little
bit
like
the
service
mesh
architecture.
And
finally,
we
do
want
to
evaluate
the
service
mesh
and
do
you
know
there
are
tons
of
open
source
in
implication
out
there
and
they
will
continuously
evaluate
and
if
some
are
fit,
some
can
fit
our
need.