►
Description
For more great content, visit https://solocon.io
SoloCon 2022:
For My Next Trick- Making my platforms disappear with an Edge Gateway
Speakers:
Joe Searcy
Staff Engineer, Distributed Systems, T-Mobile
Session Abstract:
At T-Mobile we have a diverse application landscape that runs on top of a variety of platforms including Kubernetes, Cloud Foundry, Virtual Machines, and Baremetal servers. As we pushed Development teams to strive for higher levels of resiliency and security, we began to see an ocean of solutions pop up and an increase in complexity due to too many choices and too much overhead for individual Development teams to manage their own gateway solutions. We took a multi-year journey to build out a common Edge Gateway solution that would work for all platforms and remove the non-functional requirements for resiliency, security, and observability from our Development teams. This is the story of that journey including the paths we took, the lessons we learned, and how things look now.
Track:
Gloo Edge and API Gateway
A
A
I
want
to
kick
off
this
story
with
talking
through
how
we
came
to
need
an
edge
gateway.
I
feel
like
the.
Why
is
almost
as
important
as
the
how,
as
most
enterprise
companies,
we
were
continuing
to
strive
for
ever
better
resiliency
amongst
the
services
that
we
offer
at
t-mobile,
and
we
got
to
a
point
where
we
started
to
see
some
patterns
starting
to
emerge,
and
I
don't
mean
the
good
kind
of
patterns
here.
A
But
that
didn't
really
solve
the
problem
of
how
do
I
route
traffic
across
multiple
geographically
dispersed
platform
instances?
You
know
sort
of
in
simplistic
terms?
How
do
I
route
traffic
across
multiple
kubernetes
clusters
in
different
regions,
and
we
started
to
dig
in
and
do
a
bit
of
investigation
that
ultimately
led
us
to
discover
a
number
of
problems
that
our
development
teams
were
really
struggling
with.
A
A
That's
a
lot
of
time
spent
not
writing
code,
but
even
more
disturbing
was
that
we
discovered
75
percent
of
production
issues
were
due
to
networking
and
configuration
problems
let
that
sink
in
for
a
second.
At
this
point
we
were
kind
of
worried
and
we
we
had
to
discover
where
all
this
time
was
being
spent.
A
Overall,
there
were
lots
of
patterns,
lots
of
opinions
and
lots
of
inconsistency
that
all
added
together
to
be
a
lot
of
money
and
time
being
wasted
and
that
didn't
make
us
feel
better
at
all.
There
were
just
too
many
solutions
and
that
added
to
sort
of
the
stress
and
difficulty
that
developers
face
when
starting
out
with
a
new
application,
each
solution
seemed
to
be
managed
by
a
different
group
with
different
levels
of
grassroot
adoption.
A
There
were
tons
of
overlapping
features
between
the
different
solutions
and
the
development
teams
seem
to
be
owning
the
life
cycles
of
these
solutions.
So
yet
again,
more
time
spent
not
focusing
on
code
security
solutions
were
sort
of
all
over
the
place,
but
overall
we
didn't
really
have
any
sort
of
centralized
visibility
into
what
was
out
there.
No
inventory
and
some
of
these
solutions
were
actually
built
on
top
of
other
platforms
without
any
consideration
for
that
dependent,
platform's
availability
model.
A
It
was
sort
of
a
mess
so
before
we
ever
got
into
thinking
of
how
we
would
build
a
solution,
we
sort
of
set
out
to
define
what
we
wanted
in
a
finished
product
without
consideration
for
technology
or
implementation
details.
At
this
point,
we
wanted
something
simple
and
easy
to
use
and
it
had
to
be
able
to
meet
the
demands
and
scale
of
our
environment.
A
With
all
of
that
in
mind,
defining
the
desired
user
experience
was
fairly
easy,
simple,
flexible,
centralized
and
a
managed
service.
We
sort
of
kept
repeating
those
things
to
ourselves
over
and
over
again
and
in
the
end,
a
platform.
That's
easy
to
use,
meets
the
needs
of
our
teams
and
it's
self-serve
serviceable.
A
A
Overall,
we
knew
this
would
be
a
very
complex
project,
with
lots
of
cross-domain
knowledge
required
for
it
to
be
successful.
So
we
pushed
for
an
intersource
model
making
sure
all
the
information
would
be
at
the
fingertips
of
any
expert
across
the
company
and
again
self-service
consumption
was
a
must
continuing
to
operate
with
the
traditional
cross-team
ticket.
Cue
model
was
just
not
scalable
anymore,
so
we
used
all
of
this
as
a
criteria
to
evaluate
several
different
solutions
for
the
core
data
plane
layer.
A
A
Since
we
wanted
to
move
very
quickly
on
building
this
solution
out,
we
partnered
with
solo.io
to
help
us
get
up
to
speed
and
also
to
provide
us
a
sort
of
safety
net.
In
case
we
ran
into
trouble,
which
was
going
to
be
inevitable
with
something
like
this
now
beyond
just
glue
expertise,
solo
really
brought
a
great
understanding
of
kubernetes
api
gateway
solutions
and
envoy
itself.
We
continue
to
lean
on
their
expertise,
expertise
to
augment
our
own
today.
A
This
is
the
high
level
architecture
that
we
decided
on
to
essentially
establish
a
more
holistic
application
delivery
network
for
our
environment.
At
the
top
layer
we
have
perimeter
controls.
These
are
a
lot
of
existing
tools
that
we
wanted
to
reuse,
that
bring
in
features
like
monetization
of
partner,
apis
bot,
detection,
waff,
firewall
and
other
things
that
you
typically
find
at
the
edge.
A
A
A
The
control
plane
is
mainly
to
provide
our
abstraction
layer
and
to
orchestrate
our
end,
users
configuration
and
we'll
dig
into
each
of
these
a
little
bit
more
now
so,
starting
with
the
data
plane
for
our
edge
gateway,
it's
broken
up
into
two
major
components:
the
forwarding
plane,
which
is
responsible
for
attracting
traffic
to
the
data
plane
clusters
where
envoy
is
running
for
the
gtm
layer.
We
use
a
dns
based
solution
that
handles
resolution
of
client
requests
based
on
a
pool
of
virtual
ips,
corresponding
to
each
data
plane.
Cluster.
A
A
A
A
Now
we
also
have
a
custom
api
for
the
data
plane
for
maintenance
purposes.
That
gives
us
the
ability
to
mark
any
data
plane
cluster
as
unhealthy
to
remove
it
from
service
and
we're
in
the
process
of
testing
a
fully
separate
blue
green
forwarding,
plane
setup.
So
we
can
introduce
new
features
and
perform
functional
testing
in
an
out-of-band
manner.
A
A
But
we
made
a
couple
of
decisions
in
our
desired
solution
that
blue
edge
didn't
really
cover
at
least
not
then,
and
not
entirely
yet.
But
a
lot
of
the
work
that's
being
done
with
blue
federation
is
very
interesting
to
us
for
sure
we
wanted
our
edge
gateway
to
work
for
more
than
just
kubernetes,
and
we
didn't
want
to
have
the
first
barrier
to
entry
for
folks
to
be
on
board
to
kubernetes.
A
A
A
We
also
didn't
want
to
overwhelm
our
developers
with
having
to
learn
anything
about
envoy
or
blue
edge,
with
our
own
custom
configuration
on
play
and
a
custom
api
that
sits
in
front
of
blue
edge
and
envoy
we're
able
to
decouple
our
configs
from
the
data
plane,
implementation
details
and
limit
how
much
config
is
needed
for
our
users
to
get
up
and
running
now.
How
does
this
work?
A
A
A
Writing
the
configurations
to
our
kubernetes
clusters
in
the
form
of
custom
resources,
and
before
that
happens,
the
config
is
validated
by
a
custom
kubernetes
validating
web
hook,
and
this
web
hook
handles
validation
of
the
incoming
configs
against
defined
best
practices
and
enforces
some
guard
rails
to
ultimately
protect
the
data
plane.
Since
this
is
a
shared
data
plane,
one
bad
config
can
mess
up
services
for
for
everybody.
A
A
A
Even
small
config
changes
became
fairly
cumbersome
due
to
having
so
many
different
hands
all
trying
to
work
in
the
same
environment.
At
the
same
time,
these
environments
were
pretty
complex.
We've
got
multiple
kubernetes
clusters
for
the
data
plane,
multiple
kubernetes
clusters
for
the
control
plane,
so
we
got
creative
and
leaned
on
kind,
kubernetes
and
docker.
It's
a
open
source
project
in
the
kubernetes
community
that
allows
you
to
build
kubernetes
clusters
using
docker
containers
for
your
kubernetes
nodes.
A
A
The
control
plane
clusters
get
our
custom
api,
our
crds
and
the
kubernetes
operator
installed,
and
from
there
we
can
iterate
and
test
changes
to
the
api,
the
operator
to
glue
so
on
and
so
forth.
On
the
fly
we
can
spin
up
an
entire
test
environment
within
a
couple
minutes,
so
there's
no
real
fear
in
breaking
things.
We
just
tear
it
down
and
create
another
test
environment.
A
A
A
A
The
management
console
acts
as
a
read-only
visualization
of
services
on
the
edge
gateway
developers
can
come
here
to
check
compliance
status,
health
status,
the
activity
that
each
of
their
services
have
and
they
can
filter
per
environment
per
team
or
per
individual
service.
Here
they
have
a
lot
of
flexibility
to
get
sort
of
ad
hoc
reporting
and
quick
visualizations
that
everything's
good
and
healthy.
A
One
of
the
other
great
features
of
the
management
console
is
that
users
can
visualize
what
their
services
look
like
on
the
platform.
What
api
endpoints
are
exposed?
What
authorization
methods
are
in
place?
What
actions
are
exposed,
the
overall
compliance
status?
They
can
even
manually
trigger
health
checks
and
they
can
view
their
configuration.
A
A
So,
let's
run
through
these
numbers,
real
quick,
so
latency
we're
still
maintaining
a
p90
of
under
10
milliseconds,
which
is
great
that's
a
huge
reduction
in
versus
some
of
the
existing
gateway
solutions.
We
had
that
we're
running
up
to
like
50
60
milliseconds,
we're
doing
about
150
million
transactions
a
day
and
we're
seeing
sustained
request
rates
of
around
3
000
requests
per
second
and
during
some
of
our
major
retail
events.
We've
even
seen
that
go
up
to
about
5
000
requests
per.
A
So
now
I
want
to
wrap
things
up
by
talking
through
some
of
our
lessons
learned
throughout
this
project
and
we'll
start
by
process
from
talking
to
some
of
our
peer
companies.
It
sounds
like
some
of
the
issues
we
had
are
not
really
unique
to
us,
but
our
company
was
culturally
built
around
one
team
ownership,
and
what
I
mean
by
that
is
one
platform
is
owned
and
supported
by
one
team.
A
Now
this
project
touched
a
lot
of
separate
teams
and
ownership
quickly
became
from
our
experience.
I
definitely
recommend
assessing
things
around
how
ownership
is
seen
and
handled
in
your
organization
and
plan
for
any
needed
cultural
shifts.
The
time
required
for
this
sort
of
thing
can
be
really
significant
and
continuing
along
those
lines
with
transparency
across
teams.
The
inner
source
model
really
helped
us
to
achieve
full
transparency
during
the
design
and
build
phases
of
this
project,
taking
notes
in
all
meetings
to
track
design
decisions,
architectural
requirements
so
on
and
so
forth
proved
to
be
extremely
useful.
A
A
Moving
on
to
the
area
of
design,
make
sure
you
give
yourself
enough
time
to
plan
for
abstractions.
Abstractions
are
hard
and
you
shouldn't
just
do
them
without
good
reason.
Talk
through
them
over
and
over
again
weigh
the
risk
versus
reward
for
us.
We
were
looking
at
the
existing
tools.
Our
users
had
what
configs
those
exposed,
as
well
as
what
config
new
tools
like
blue
exposed.
A
With
things
like
glue
and
envoy,
we
started
with
some
very
simple
validation
up
front
on
the
config
that
was
essentially
input
to
our
platform
and
quickly
learned
that
it
was
very
easy
to
have
bad
configs
submitted.
That
would
cause
issues
with
the
pieces
underneath
our
abstraction,
and
in
hindsight
we
should
have
planned
for
more
in-depth
validation
of
the
config
on
day
one
now,
as
with
any
abstraction,
one
of
the
main
goals
should
be
to
reduce
the
cognitive
overhead
on
the
consumers
of
the
platform.
A
Moving
on
to
the
area
of
unplanned
the
unknown
unknown,
one
of
the
first
things
we
hit
in
this
category
was
scaling.
Issues.
Scale
is
a
very
weird
thing:
every
environment
is
different
and
every
tool
or
technology
can
be
different
plan.
For
this
sort
of
thing.
In
your
timeline,
we
hit
a
few
different
issues
on
scalability
with
our
custom
orchestration
engine
and
also
with
blue
edge
itself.
A
I
definitely
recommend
spending
some
time
to
consider
scale
up
front
if
you're
consuming
an
open
source
project
as
part
of
your
stack
reach
out
to
the
community
and
ask
about
what
skill
the
project
has
been
tested
against.
This
is
where
our
partnership
with
solo.io
saved
us
a
ton
of
effort
and
heartache,
whether
it
was
leaning
on
their
deep
understanding
of
envoy
internals
in
the
early
days
or
having
helped
to
validate
some
of
these
scalability
issues
or
adding
new
features
upstream
to
solve.
For
some
of
these
problems,.
A
A
Now,
thank
you
all
for
coming
to
my
session
today.
I
really
appreciate
the
opportunity
to
share
our
journey
with
deploying
an
edge
gateway
solution,
a
huge
thanks
to
solo.I
o
for
hosting,
and
thanks
to
all
the
awesome
folks
that
I
get
to
work
with
at
t-mobile
to
build
cool
stuff
like
this
now
provided
I
didn't
talk
too
much.
I
think
we
might
have
time
for
some
questions.