►
From YouTube: Ingress on the Rails: Use Community Tools to Automate Ingress Provisioning - Alastair Firth, Camunda
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Ingress on the Rails: Use Community Tools to Automate Ingress Provisioning - Alastair Firth, Camunda
Routing traffic into your clusters is a key part of many Kubernetes workloads. The tools to manage this are great, but getting them all to work together can be a configuration nightmare for the uninitiated. Alastair Firth will demonstrate a concrete, automated, cloud-provider agnostic setup for provisioning DNS, TLS certs, and monitoring using community tools like ExternalDNS, cert-manager, Prometheus, Grafana, and Helm with the kubernetes/ingress-nginx controller so you can securely serve and monitor HTTPS and HTTP/2 (e.g. GRPC) traffic to many hostnames across multiple Kubernetes clusters.
https://sched.co/ZeqC
A
Hi,
my
name
is
alice
de
firth,
a
ministery
in
berlin
for
commando
today,
we'll
talk
about
deploying
the
ingress
engine
x,
controller
using
helm,
who
are
you
well,
hopefully
you're
figuring
out
how
to
deploy
ingress
controllers
on
kubernetes
or
you
already
have
and
want
to
see
how
other
people
are
doing
it.
Just
a
heads
up,
there's
a
lot
of
code
in
this
deck.
So,
if
you're
on
a
bad
connection
and
it's
blurry,
you
might
want
to
get
slides
from
sked
or
github
and
follow
along
so
first
what
you
want
traffic
comes
into
a
tcp
load.
A
Balancer
then
goes
to
some
highly
available.
Nginx
reverse
proxies,
which
terminate
tls
then
hits
your
back
end.
Okay,
I
said
we're
going
to
do
it
on
the
rails.
If
you
pull
up
the
docks
for
deploying
the
controller,
you
can
just
copy
paste
these
lines,
helm,
install
and
then
the
name
of
your
release
and
then
chart
name.
So
this
part
is
pretty
easy,
yeah
anybody
who
started
here
knows
it's
a
bit
more
complicated
than
that.
I
guess
that's
why
you're
at
my
talk,
so
I
hope
I
can
help
our
use
case.
A
A
We
also
want
to
be
cloud
provider
agnostic.
Frankly,
not
many
providers
support
single,
let
alone
multi-region
grpc
layer,
7
load
balancing
anyway,
I
think
there's
only
one
you
can
buy
some
similar
use
cases
include
customer
created,
cnames,
pointing
to
a
unique
url
hosted
by
you,
like
many
web
hosting
providers,
or
pr
based
environments
for
ci,
cd
or
perhaps
kubernetes
platform
as
a
service
for
your
company,
so
devs
can
deploy
new
services
autonomously
back
to
the
rails.
Obviously,
disclaimer,
I'm
not
endorsing
these.
A
On
behalf
of
anyone
just
saying
they're
in
common
use
and
to
my
knowledge,
the
most
widely
used
open
source
op
solution,
they
also
have
maintained
home
charts,
that's
good
for
us
shout
out
to
all
the
maintainers.
Thank
you
a
thousand
times
over
external
dns
is
a
controller
which
watches
ingresses
and
makes
well
external
dns
records.
A
Cert
manager
is
a
controller
which
watches
its
own
crds
and
creates
tls
certificates.
As
kubernetes
secrets,
these
secrets
can
be
referenced
by
ingresses
optionally.
It
can
also
watch
ingresses
and
solve
http01
acme
challenges
which
it
puts
in
the
already
referenced
secret.
Prometheus
operator
is
a
controller
which
watches
its
own
crds.
A
Nginx
can
route
incoming
requests
there
and
only
pass
them
onto
your
back
end.
If
they're
successful
before
you
dive
in.
I
think
there
are
three
main
pain
points
when
you're
working
on
this.
Firstly
deploying
everything
and
in
particular,
managing
secrets,
for
example,
you
might
have
multiple
clusters
in
multiple
regions
for
production,
but
different
urls
and
credentials
for
test
environments,
keeping
everything
in
sync
and
continuously
delivered
without
copy
pasting.
Everything
can
be
tough,
the
tooling
is
still
stabilizing,
and
I
don't
think
anyone
is
convinced
that
we
found
the
way
to
template
this
stuff.
A
A
Okay,
so
the
solution
we
came
up
with
for
the
first
two
points
is
pretty
straightforward.
Basically,
we
use
sops
to
encrypt
all
the
secrets
and
add
them
to
customize
overlays.
We
call
helm
template
on
each
chart
with
our
standard
values
and
add
this
output
to
a
customized
base.
Then
we
make
overlays
for
different
clusters
and
environments
to
set
things
like
the
domains
and
inject
secrets
like
dns
provider
access
tokens,
then
we
call
customize
build
for
each
cluster
and
pipe
it
to
coupe.
Cuddle
apply
it's
pretty
easy
helm.
Template
coupe
cuddle
apply.
A
Of
course.
It's
not
easy.
That's
why
you're
here,
it's
still
a
hundred
times
easier
than
it
was
a
few
years
ago,
though,
the
hardest
part
is
tweaking
each
component
for
different
clusters.
The
approach
we
take
means
that
we
have
plain
kubernetes
manifests
one
per
cluster
and
git.
The
main
disadvantage
is
the
deployment
pipelines
or
operators
need
to
take
care
of
things
that
home
hooks
usually
do
like
installing
crds
and
waiting
for
validating
web
hooks
to
come
on.
A
A
It
plays
nicely
with
cd
systems
like
argo
and
spinnaker,
and
also
allows
an
engineer
to
run
cube-cuddled
diff
our
app
developers,
typically
write
controllers
or
customize,
manifests
and
non-home
charts.
So
this
keeps
the
pipeline
similar
manifest
diffs
are
valuable
for
seeing
what
will
actually
change.
A
A
So
the
format
of
this
presentation
is
basically
me
walking
you
through
our
production
code
and
calling
out
line
numbers
and
what
they
do,
we're
going
to
smash
through
all
five
helm,
charts
and
the
deployment
pipelines
in
35
minutes
so
buckle
up
so
back
to
how
we
solve
the
deployment
and
secrets
problem
and
follow
up
stream.
Changes
first
part
is
a
make
file
for
each
chart.
It
looks
a
lot
like
what
you
do
by
hand
to
install
a
helm
chart,
except
at
line
22.
A
A
One
thing
is:
this:
is
home
help
three
change
their
exhibit?
We
use
basically
the
same
make
file
for
all
charts.
Although
line
30
changes
a
bit
if
they
have
sub
charts,
don't
stress
if
you
think
this
is
crazy.
It's
just
context
for
the
files
we
have
moving
forward
and
should
be
adaptable
to
whatever
process
you
have.
If
you
don't
have
one,
this
is
a
pretty
complete
solution
that
works
for
all
of
these
use
cases.
A
A
A
In
the
overlay
we
can
layer,
customize
manifests
on
top
of
each
other
and
patch
them
in
a
variety
of
ways
or
combine
multiple
resources.
Finally,
calling
customize
build
on
the
overlay
creates
the
ammo
that
you
can
pipe
to
coupe
cuddle
apply
on
the
right.
We've
got
the
directory
structure
of
our
overlay.
A
A
Generally,
you
can
reference
the
secrets
directly
from
the
overlay
and
leave
the
name
suffix
hash
option
on
which
will
make
everything
that
mounted
the
secret
or
config
map
restart
customize
supports
adding
secrets
in
a
number
of
different
ways.
This
particular
one
is
going
to
pick
up
everything
in
the
file
as
a
complete
kubernetes
secret
and
give
it
a
long
and
complicated
name.
A
Here
you
can
see
a
portion
of
the
alert
manager
secret
at
references
note
that
only
the
sensitive
slack
api
url
on
line
three
is
encrypted
supports
specifying
a
particular
suffix
to
encrypt
one
cool
thing
about
this.
Is
that
although
the
manifest
won't
be
deployable,
they
are
valid
yaml,
even
when
the
secrets
are
encrypted.
That
means
customize
builds
them
just
fine,
and
you
can
see
the
overall
changes
in
pr's
if
you
commit
the
manifest
for
each
cluster
without
exposing
any
secrets.
A
A
A
A
There
are
some
alternatives
for
secret
management,
which
has
historically
been
considered
an
insecure
part
of
kubernetes
installations.
Etcd
is
much
tighter
than
it
used
to
be
supporting
tls
and
encryption
at
rest
by
default
and
if
you're
running
single
tenant
clusters,
it's
up
to
you
to
decide
if
these
alternatives
provide
any
real
benefit.
A
The
main
leak
vector
for
sups
is
in
deploy
pipeline,
logs
or
inadvertent,
commits
of
plain
text,
fault
and
sealed
secrets
both
wait
until
the
secret
is
in
the
cluster
to
decrypt
it
and
bolt
doesn't
put
it
in
ncd
at
all.
That
can
be
pretty
hard
to
inject
into
some
of
the
chart
deployed
stuff,
but
worst
case
you
can
decrypt
it
in
your
pipeline.
A
Most
cloud
providers
also
have
a
secret
management
solution
now,
so
your
mileage
may
vary.
Okay.
So
to
summarize,
we've
solved
all
the
painful
bits
we
generate
plain
kubernetes,
manifest
for
each
target
cluster
which
you're
safe
to
commit
with
helm,
template,
subs
and
customize,
and
we
deploy
them
with
coop
cuddle,
apply
or
whatever
magic
goo
you
like
to
fetch
up
stream
changes.
We
just
run
our
make
file
to
pick
up,
chart
changes
and
check
the
diffs
of
the
manifests,
either
in
git
or
with
coop
cuddle
div.
If
you
think
this
is
ridiculous,
that's
okay!
A
When
I
wrote
this
tilde,
this
helm
was
cutting
edge.
It's
just
background,
but
I
hope
it
helps
some
of
you
now
on
to
the
fun
bits
using
cloud
native
software
to
do
stuff
that
used
to
be
hard,
expensive
or
both
just
by
setting
some
config
options.
How
exciting
is
that
some
of
your
kids
probably
never
even
bought
an
ssl
certificate?
A
Lucky
you!
This
bit
is
going
to
be
a
whirlwind
of
code.
I'm
not
trying
to
impart
any
profound
wisdom
to
show
you
useful
options
and
how
to
set
them.
First:
external
dns,
insert
manager,
then
a
basic
prometheus
operator
deployment
and
an
os2
proxy
to
put
in
front
of
it.
Then.
Finally,
the
ingross
controller
itself.
A
First,
we
need
a
quick
overview
of
an
example
ingress.
This
is
a
service
we
have
called
accounts.
Ingress
v1
is
a
bit
of
a
weird
resource
because
there
are
so
many
implementations
of
controllers.
So
a
lot
of
stuff
is
happening
in
the
annotations
controllers.
Like
ingress
engine
x,
external
dns,
insert
manager
will
use
a
combination
of
the
annotations
and
spec
to
decide
what,
if
anything,
to
do
when
they
see
an
ingress
resource.
A
You
might
notice
we're
missing
the
secret
in
the
tls
section.
That's
because
we're
using
a
wildcard
as
the
default
cert
more
on
that
later.
But
for
now,
let's
jump
right
into
line
six
and
seven
external
dns
is
the
first
controller.
We'll
look
at
it's
going
to
create
some
dns
records
on
cloudflare
for
us
when
it
sees
this
ingress.
A
Here
you
see
it
working.
It's
created
an
a
record
pointing
to
our
load
balancer
and
a
text
that
this
particular
external
dns
owns
it.
If
the
ingress
is
deleted,
it
will
delete
both
another
copy
of
external
dns
won't
override
it.
Unless
the
txt
record
matches
what
it's
looking
for,
just
a
little
side,
note
that
many
providers
limit
the
total
number
of
records
you
have
cloudflare
pro
is
about
2500.
So
don't
go
crazy.
A
Okay,
so
here
are
the
home
values
we
use
as
a
base
for
that
external
dns
targeting
cloudflare.
Firstly,
on
line
two
the
default
is
to
watch
services
as
well.
We
limited
to
ingresses
for
reasons
I
frankly
don't
remember
on
lines
11
and
14.
We
set
the
provider
specific
stuff
and
we
blank
out
the
api
key.
A
So
it's
not
set
we'll
set
that
later
in
a
customized
patch,
17
and
18
vlogging
is
cheap
and
you
really
really
want
to
know
what's
going
on
if
this
screws
up
so
turn
it
up,
we
use
json
logging
everywhere
we
can
for
easy
searching
and
metrics
making
line
19.
We
set
the
policy
to
sync
as
opposed
to
just
upsert.
This
means
we'll
actually
delete
unneeded
records
if
we
own
them.
According
to
the
text
record,
we'll
set
that
our
owner
under
patch
2.
A
A
This
is
an
overlay
patch
for
the
external
dns
deployment
which
we
apply
with
customized
for
a
particular
cluster,
we're
just
overriding
some
environs.
Firstly,
at
12
we
set
the
authentication
info
for
cloudflare.
This
is
required,
but
if
you're
using
cloud
providers
dns,
you
can
also
just
let
it
pick
up
the
default
credentials
for
gk.
You
need
to
grant
the
node
the
scope
to
do
that.
It's
a
similar
process
for
other
providers,
most
dns
providers,
are
supported.
A
Secondly,
on
line
23,
we
need
to
set
an
owner
id.
This
is
a
key
which
should
be
unique
per
external
dns
installation
and
will
be
used
in
that
text
record
at
25
we
set
a
domain
filter
which
corresponds
to
the
zones
you
have.
Creds
for
multiple
domains
can
be
provided,
but-
and
this
is
a
recurring
theme-
domains
are
probably
a
good
system
boundary
and
you
might
want
to
consider
your
splits.
A
We
create
the
secret
reference
done
lines,
14
and
19
pretty
much
the
same
way.
We
made
the
alert
manager
one
and
for
cloudflare,
it's
used
by
slurp
manager
too.
So,
let's
jump
over
there
in
our
setup,
cert
manager's
main
job
is
to
make
sure
we
have
a
valid
tls
certificate
for
each
ingress.
Just
like
the
old
kube
lego,
you
can
configure
it
to
solve
an
http01
acme
challenge
with
let's
encrypt
for
each
ingress
and
set
a
secret
name.
However,
you
can
just
give
nginx
a
wild
card
for
all
your
ingresses.
A
Even
if
you
have
multiple
domains
to
serve,
you
can
have
a
single
certificate.
Assuming
you
solve
all
the
challenges
note
at
line
8,
we
set
the
ingress
class.
Each
controller
can
be
configured
to
watch
a
specific
class,
so
you
can
run
multiple
ingress
engine
x's
or
even
mix
and
match,
with
other
ingress
controllers
back
to
that
magical
certificate.
A
There
are
two
custom
resources
which
you
need
to
provide
for
cert
manager
to
get
this
working.
Firstly,
a
cluster
issuer
on
the
left
tells
it
how
to
solve
challenges
for
wildcards.
We
need
to
use
dns01
and
we're
using
cloudflare
with
the
same
credentials
as
external
dns.
Everything
here
is
required,
but
also
pretty
self-explanatory.
A
A
If
you
change
the
subdomains
later,
you'll
get
a
scary
expiry
email
for
the
old
certificate
90
days
later
so
set
a
calendar
reminder
to
ignore
that
email,
obviously,
search
manager
needs
to
be
installed
before
you
can
create
these
custom
resources.
So,
let's
dig
in
for
cert
manager,
jetstack
actually
releases
a
manifest
created
using
helm
template
themselves,
so
we
just
pulled
out.
I
think
I
had
too
much
coffee
when
I
upgraded
us
from
0.10,
and
I
wrote
this
monstrosity.
A
Sorry
just
use
the
helm
chart
like
all
the
other
things
or
download
the
manifest,
as
it
says
in
the
comments.
Hopefully
customize
2167,
which
is
now
merged,
allows
you
to
reference
this
and
customize
so
skip
right
over
this
over
engineered
mess.
The
point
is
that
you
need
a
customized
base
containing
the
default.
Cert
manager.
Installation
manifests
with
the
ability
to
pull
https
urls
from
customize,
hopefully
by
the
time
you're.
A
Reading
this
you
can
ignore
the
last
sled
and
use
a
simple
base
customization
anyway,
once
we
have
the
static
manifests
somehow
or
another,
we
attach
them
as
usual,
except
there
is
nothing
to
patch,
because
james
is
awesome
and
it's
all
just
custom
resources
deploy.
The
base
manifests
the
cluster
issuer
and
the
appropriate
wildcard
cert,
which
are
the
two
crds.
We
just
showed
easy
thanks.
James
os2
proxy
is
pretty
straightforward
way
to
secure
your
monitoring
services,
which
supports
13,
specific
auth
providers,
plus
all
oidc
implementations
like
octa
and
dex.
A
We've
set
it
up
to
make
sure
people
are
in
our
github
org.
This
is
the
grafana
ingress,
showing
how
the
request
is
routed
to
the
proxy
first
note
that
we
use
best
http
host
and
include
two
exos
headers
in
the
requests
we
passed
to
grafana.
This
lets
grafana
create
a
user
internally
if
it
doesn't
already
exist.
A
A
This
is
the
helm
values
file
for
os2
proxy
line.
Two
we
ensure
it
doesn't
try
to
make
its
own
secrets.
Four
we're
setting
up
the
ingress
placeholders
to
patch
later
12.
We
configure
some
logging
and
make
sure
it
actually
passes
the
alt
headers
we
need
to
send
through
to
grafano
17
is
the
provider
specific
config
and
then
continued
on
the
right
on
26
we've
got
the
resources
and,
on
36
an
example
of
pod
anti-affinity
for
two
replicas,
since
this
is
an
internal
only
service
for
us,
it's
a
perfect
candidate
for
preemptable
nodes.
A
Finally,
we
have
the
secret
for
the
os2
provider,
which
we
add
with
customize
here,
we're
using
the
end
file
format
for
customize
to
create
the
secret
to
access
our
github,
oauth,
app
and
again
disabling
the
suffix
hashing,
which
unfortunately
means
we'd
have
to
roll
these
pods.
If
they
don't
fail
probes,
when
we
roll
a
secret
in
the
middle,
is
the
plain
text
and
at
the
bottom
is
what
looks
like
what
it
looks
like
committed?
A
That's
it
for
os2
proxy.
You
don't
actually
need
one
per
cluster.
You
can
send
requests
to
an
os2
proxy
located
in
a
different
cluster
using
its
dns
name,
which
external
dns
kindly
created
for
you,
and
the
requests
will
be
secured
with
tls
from
your
wild
guard.
Search
I'll
show
you
that
in
the
monitoring
section
you
can
also
catch
the
responses,
if
you
like
using
ingress
annotations,
but
I
won't
go
into
that
here
so
on
the
prometheus
operator.
Frankly,
this
chart
is
amazing.
A
A
What
does
prometheus
have
to
do
with
ingress
nginx?
Well,
with
a
few
lines
of
config,
you
get
two
amazing
dashboards
for
the
ingress
engine
x.
Repo
you've
got
nginx
performance,
latency,
throughput
and
error
rates
and
certificate
validity,
and
they
can
be
broken
down
by
path,
service
and
controller
class.
A
There's
a
fair
bit
to
set
up
here,
and
everyone
is
going
to
have
different
use
cases,
I'm
just
going
to
cover
the
highlights
of
our
config
and
you
can
find
more
at
the
github
link
in
the
corner.
One
caveat
is
this:
config
is
for
version
6
of
the
chart,
and
it's
on
9
now,
so
apologies
for
that.
There's!
No
huge
changes,
though
I'll
just
go
through
the
home
values.
Patches
are
applied
the
same
way
as
other
systems
line.
One
is
a
workaround
to
form
force
helm
to
generate
resources
for
older
cube
versions.
A
In
case
the
api
has
changed,
I'm
not
sure
what
we're
working
around,
but
there
you
go
on
4
we're
setting
json
logging,
5
and
6
retention,
time
and
size
slightly
lower
than
our
volume
size
and
then
resources
on
15
we're
also
running
multiple
replicas
with
anti-affinity,
which
is
especially
easy
to
configure.
With
this
chart.
A
19
and
20
probably
aren't
needed
anymore,
but
told
the
operator
to
look
at
crds,
regardless
of
their
labels.
22,
we
give
it
a
physical
volume
31
we
configure
session
affinity.
So
your
values,
don't
change
as
you
browse.
If
a
replica
has
partial
data
and
finally
on
33,
we
have
the
ingress
placeholder,
which
we
patch,
as
shown
before
alert
manager.
Spec
is
here
for
completeness.
This
isn't
the
same
file
not
much
going
on.
A
Besides
configuring,
the
ingress
logging
in
storage,
most
home
charts,
have
storage
off
by
default,
so
they
work
for
demos
and
ci,
but
you
probably
want
persistence
for
production
note
at
54.
We
again
tell
it
not
to
create
a
secret.
We
showed
the
alert
manager
secret,
which
contains
the
routing
rules
earlier
in
this
presentation,
grifano
is
tricky
at
line
74
we're
setting
a
bunch
of
m
variables
for
all
these
override
the
grafana
ini
file,
which
is
a
painted
template.
A
The
end
result
is
that
anyone
coming
in
through
the
ingress
has
an
account
and
explore
access.
They
can
save
dashboards,
but
these
won't
be
synced.
Instead,
what
we
do
is
coupe
port
forward
to
be
an
admin
when
we
create
dashboards,
then
export
them
to
json
through
the
grafana
ui
and
save
them
in
git.
It's
a
bit
of
a
pain,
but
it
means
the
dashboards
are
in
git,
which
makes
them
easy
to
sync
across
stages.
A
A
A
You
can
also
embed
the
dashboard
completely
as
a
config
map
on
line
7,
you
can
see
the
label
which
the
operator
uses
to
decide
what
config
maps
to
pick
up
and
inject
into
grafana.
Similarly,
there's
a
crt
for
recording
rules
and
alerts.
That's
enough
about
monitoring
on
to
nginx
again
we're
a
bit
behind
the
latest
chart.
So
some
of
the
things
we've
worked
around
might
have
been
fixed
upstream.
A
Here's,
the
helm,
values
for
our
grpc
handling
controller
and
we'll
go
through
the
highlights.
You
can
find
a
fully
annotated
values
file
in
most
of
the
home
charts
complete
with
the
default
values.
All
of
these
just
overlay
on
that
one
line
two
to
four
we're
configuring
this
as
a
deployment
rather
than
daemon
set
because
we'll
make
load
balancers
instead
of
just
publishing
the
node
ips
to
dns,
like
in
many
tutorials,
this
controller
will
watch
any
ingress
with
the
class
annotation
of
grpc
on
8,
we
publish
the
service,
which
is
required
for
external
dns.
A
To
make
a
records
with
the
correct
ip
11
is
the
magic
juice
for
wild
card
certificates.
This
is
the
only
way,
I'm
aware
of
to
refer
to
a
certificate
in
a
different
name
space
with
this
controller,
and
this
sets
the
tls
certificate
for
the
default
route
and
any
other
ingresses
that
don't
set
a
secret
name
in
the
tls
section.
A
Alternatives
are
to
use
something
like
kubdy
to
sync
the
secrets
across
namespaces
or
use
http01
acme
certs.
If
you
do
use
http01,
beware
of
rate
limits
at
let's
encrypt.
Of
course,
you
can
also
have
every
application
include
their
own
certificate.
Crd
14,
we
change
the
status
code
returned
if
the
client
is
raid
limited
by
default,
it's
503.
A
A
Finally,
at
64,
we
make
sure
this
is
exposed
on
a
load
balancer
and
the
client's
ip
is
preserved,
and
at
68
we
enable
metrics
and
deploy
the
included
service
monitor
for
the
prometheus
operator.
To
pick
up,
we
don't
have
any
patches
for
nginx.
We
just
run
the
production
setup
on
all
clusters
and
rely
on
the
autoscaler.
However,
you
could
conceivably
want
to
patch
the
resource
usage
or
even
change
some
controller
global
settings
in
the
config
map.
A
A
Okay,
sorry
about
that.
I
had
some
technical
issues
anyway,
yeah.
This
is
one
tweak
which
will
return
grpc
native
error
codes
for
grpc's
ingresses.
This
is
just
a
section
of
the
config
which
you
could
embed
in
your
controller
by
putting
it
in
the
controller
config
map,
I
highly
advise
keeping
your
http
2
traffic
separate
from
the
http
one
traffic
by
deploying
a
second
ingress
controller.
A
That
also
means
an
extra
30
bucks
in
load
balancers,
but
it's
worth
it
in
production.
I
think
the
downside
of
replacing
the
error
codes
like
this
is
that
the
monitoring
is
going
to
break
because
the
http
2
spec
is
the
http
1
status
code
is
200.
Actually
I
think
the
204
here
is
non-compliant
anyway.
It's
an
option.
A
Some
of
you
might
be
wondering
why
we
use
nginx
to
proxy
grpc
at
all.
Well,
it's
mature,
fast
and
fairly
stable,
you're
unlikely
to
see
a
problem
that
you
can't
search
for
and
most
operations
engineers
have
used
it
before.
There's
lots
of
samples
and
things
like
adding
an
alt
proxy
or
rate
limiting,
are
well
documented
for
us
making
this
choice
in
mid
2019.
It
was
a
choice
of
running
nginx
for
hp
and
a
second
less
mature
controller
or
getting
nginx
to
play
nicely
with
grpc.
A
What
we
have
works
pretty
well
for
our
use
case.
When
we
first
started,
we
tried
to
integrate
linker
d2
to
provide
the
request,
level
load,
balancing
and
even
more
monitoring,
but
we
ran
into
some
nasty
bugs.
As
far
as
I
know,
those
have
all
been
fixed
over
the
last
year.
So
our
next
initiative
on
this
front
is
to
reevaluate
that
and
also
survey
the
state
of
the
other
ingerson
controllers.
A
We've
now
got
automatic
dns
records
being
created
by
external
dns,
based
on
ingress
annotations,
wildcard
certificates
from
cert
manager
accessible
from
all
those
namespaces
prometheus
operators,
scraping
our
ingress
controllers
and
showing
us
metrics
in
grafana
for
each
individual
ingress,
authentication
in
front
of
those
metrics
with
oauth
and
finally
nginx
with
a
basic
grpc
setup
and
rate
limits.
I
hope
you've
enjoyed
the
presentation
again
slide
should
be
on
scad
and
github,
along
with
some
sample
code.
Please
join
me
in
the
chat
for
any
questions
join
me
in
the
chat
for
any
questions.