►
From YouTube: Observing Fastly’s Network at Scale Thanks to K8s and the Stri... Fernando Crespo & Daniel Caballero
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Observing Fastly’s Network at Scale Thanks to K8s and the Strimzi Operator - Fernando Crespo & Daniel Caballero, Fastly
Fastly efficiently delivers many Tbps thanks to an Edge Network that expands across tens of PoP across the globe; operating such a network comes with many challenges, so Fastly keeps investing in tooling and automated systems to make that journey as pleasant as possible. One of these systems is Fastly’s Autopilot: an automated system that performs egress traffic optimizations. This talk will provide a system overview, focusing then on the associated telemetry pipeline and how it leverages from our internal k8s-based platform (elevation), some key operators, like the Strimzi Kafka one, opensource networking libraries, like GoBGP, and tooling like FluxCD. Finally, this talk will also share some challenges and findings associated with this very network-related use case.
A
B
Hi
everyone-
I
hope
you
are
all
having
a
great
conference
so
far,
so
I'm
fernando
I'm
working
at
fastly,
I'm
basically
enabling
coordinates
for
fastly
control
playing
applications.
Kubernetes
is
not
a
new
space
to
me.
I
think
I
started
on
the
early
stages.
Think
thanks
to
companies
like
20,
niagara
and
vietnamese,
I
had
a
great
experience
there
and
I
also
had
the
pleasure
to
contribute
to
the
kubernetes
open
source
community
with
bug,
fixes
features
and
and
even
custom
controllers,
like
20
secrets
manager.
So
this
is
me
then
hand
over
to
danny
again.
A
And
yes,
this
is
daniel.
Just
an
engineer
at
fastly
I
been
in
the
like
doing
lots
of
data
devops
stuff,
mostly
in
the
barcelona
area,
so
also
from
spain.
And,
yes,
I
also
love
open
source
and
I
try
to
contribute
as
much
as
I
can
and
also,
let's
introduce
a
bit
our
company
fastly.
Maybe
you
already
know
about
us.
A
Our
mission
in
the
end
is
to
make
internet
users
happy
and
how
do
we
try
to
do
this?
It's
mostly
by
having
a
distributed
architecture
where
our
customers
can.
Can
you
hear
me,
okay,
where
our
customers
can
execute
code
as
close
as
possible
to
the
end
users
and
deliver
contents
beating
in
mind
performance
and
reliability?
A
A
Some
numbers,
if
you
are
more
used
to
requests
per
second
8
billion
requests
per
day,
means
an
average
of
10
10
million
requests
per
second,
so
yeah
the
size
is
relatively
high,
and
that
was
quite
clear,
probably
almost
one
year
ago,
with
that
we
had
an
out
as
a
one-hour
autism.
We
were
on
the
news,
so
that's
scary,
but
also
interesting
at
the
same
time,
and
yes,
this
is
more
or
less
how
our
network
looks
like
today,
so
we
have
many
points
of
presence
across
the
globe.
A
We
try
to
put
them
close
to
our
end
users,
and
the
main
concept
here
is
that
the
scale
of
the
network
is
quite
high
and,
as
a
consequence,
we
need
more
stuff,
so
things
that
are
not
in
that
data
plane
in
our
fleet.
We
need
the
control
plane
as
well
to
orchestrate
and
to
manage
all
this
large
network
actually
in
the
control
plane
is
where
most
of
the
automation
magic
lives,
and
this
story
is
about
one
of
these
automation
services,
which
is
autopilot.
A
The
pilot
is
probably
not
a
very
original
name.
I
think
I
have
heard
about
like
autopilot
three
times
already
in
this
conference,
but
this
is
our
autopilot
and
in
order
to
introduce
the
problem
that
it
solves,
let's
take
a
look
to
our
how
our
pop
networking
looks
like
at
high
level.
The
idea
is
that
yeah,
we
have
devices
network
devices
compute
devices
and
they
are
connected
to
several
asn's
other
esn's,
so
some
of
them
act
as
transit
providers
for
us.
So
essentially,
when
we
send
traffic,
they
can
route
the
traffic
anywhere
on
internet.
A
We
have
other
relationships
with
other
asn's,
so
we
may
have
direct
links,
building
relationships
with
them,
but
here
the
key
message
is
that,
in
order
to
deliver
traffic
from
our
platform,
there
are
multiple
links
available,
so
many
routes
for
a
given
destination
and
in
the
end
we
need
to
make
decisions
about
what
what
is
the
best
way
to
for
war
traffic
to
our
end
users
and
also
to
the
origin
servers,
and
that
is
what
autopilot
is
trying
to
solve.
It's
about,
building
in
mind
performance,
bearing
in
mind
capacity
how
hot
our
links
are.
A
A
Maybe
you
have
your
own
solution
for
this
and
who
is
behind
this
last
iteration
of
autopilot
is
essentially
what
it's
the
nco
team
that
it's
in
nexus
in
network
systems
fastly
and
initially
we
had
like
six
engineers
focused
in
that
initiative,
and
these
engineers
were
we
had
like
mixed
skills,
so
there
were
software
engineers,
network
engineers,
also
sres,
like
me,
and
here
each
one
also
of
the
key
messages.
In
the
end.
A
A
Let's
actually
focus
in
the
yellow
box,
the
telemetry
pipeline,
and
what
is
that
telemetry
pipeline
doing
is
essentially
consolidating
many
inputs,
telemetry
data
from
many
sources.
We
have
systems
information
so
metrics,
mostly
from
our
prometheus
infrastructure
that
describe
the
status
of
the
links.
What
is
the
capacity
of
the
links
we
have,
then?
A
We
have
flow
data,
and
that
is
super
key
for
this
problem
in
we
are
something
like
the
network
data
we
are
moving
in
our
fleet
in
order
to
know
what
this,
what
are
the
destination
of
every
packet
we
are
sending
to
across
our
network.
We
are
sampling,
of
course,
and
then
we
also
consider
performance
data
even
at
application
level,
and
even
we
are
also
proving
the
different
links
we
have
in
order
to
know.
A
What's
the
the
performance
and
the
status
of
these
links
in
our
network,
so
how
the
pipeline
actually
looks
like
this
is
the
portion
that
cares
about
the
the
the
net
flow,
the
s-flow
data
consumption.
So
essentially
we
have
network
devices
switches
in
our
case
which
are
emitting
as
flow
information
with
an
exflo
agent.
A
So
and
finally,
we
have
an
api
in
top
of
this,
that
is
maintaining
like
an
in-memory
view
of
the
network
state,
so
a
offers
and
grpc
api
other
components
are
consuming,
including
the
controller
that
is
the
one
that
implements
the
magic
in
order
to
enrich
that
telemetry
data.
We
also
need
to
know
the
routing
state,
and
we
do
this
via
that
route
manager
service
and
finally,
as
I
mentioned,
we
are
also
collecting
consolidating
metrics
from
our
promises
infrastructure,
essentially
some
counters
and
other
data
directly
coming
from
our
switches.
A
Just
to
give
you
an
idea,
it's
more
or
less,
given
the
sampling,
we
are
doing
it's
about
processing,
hundreds
of
thousands
of
packets
per
second,
the
we
need
a
runtime
to
manage
these
components,
these
services
to
deploy
new
versions
and
so
on,
and
of
course,
as
everyone
wants.
Probably
we
want
to
run
this
in
many
different
locations
as
close
as
possible
to
our
points
of
presence.
So
we
need
to
put
this
in
several
places
and
we
also
want
to
be
a
dissolution.
A
They
wanted
to
be
also
cloud
agnostic,
so
we
didn't
wanted
to
couple
this
to
another
cloud
provider.
Given
our
control
plane
is
very
diverse.
We
wanted
to
be
able
to
run
this
also
in
some
dedicated
data
centers.
We
have
four
control,
plane,
workloads
and
yes,
surprise
in
order
to
run
a
control,
plane
workloads.
A
A
Fasting
was
already
using
kafka
for
other
use
cases,
but
we
didn't
have
like
a
proper
production
really
set
up,
so
there
was
actually
even
opposition
from
the
team
to
incorporate
kafka
in
the
project
scope.
Even
that
was
the
perception
that
that
was
complicated
was
complex.
We
were
on
a
small
team,
so
managing
something
like
this
could
be
a
challenge,
and
now
I'm
handing
over
to
fernando
we'll
we
who
will
describe
a
bit
more,
the
kubernetes
posture
and,
what's
the
platform
point
of
view,
and
how
the
kubernetes
offering
and
internal
offering
evolve
to
match
these
requirements.
B
So,
thank
you,
danny
kubernetes,
at
fastly,
circa
2020.
It
pretty
much
looks
like
this.
You
want
a
cluster,
you
get
a
cluster,
you
want
a
cluster,
you
get
a
cluster,
everyone
get
a
cluster
teams
were
pretty
autonomous
here,
like
teams,
we
learn
like
cloud
projects
or
cloud
accounts
and
create
the
infrastructure.
B
The
problem
is
that
led
to
a
very
fragmented
underutilized
and
expensive
infrastructure.
Like
danny
mentioned,
we
we
have
several
clusters
over
there,
hosting
only
one
service.
Maybe,
and
since
it
also
came
from
mostly
individual
initiative,
there
was
no
standard
way
of
creating
the
clusters,
configuring
them
or
even
deploying
to
them.
Some
folks
will
use
terraform.
Some
other
folks
will
use
the
cloud
console
directly.
Others
pull
me.
B
B
So
to
accomplish
that
we
did
things
like
implement
authentication
with
our
identity
provider.
Not
time
to
the
cloud.
Am
we
were
using
hardware
we're
using
hardware
as
a
container
register?
You
know
gcr,
not
tcr
or
anything
like
that,
we're
using
bolt
for
secrets
management,
because
you
know
kubernetes
secrets,
you
know
and
also
we're
using
ingress
engine
x
and
are
using
the
cloud
ingress
and
also
share
manager
is
not
the
cloud.
Google
cluster
manager,
aws,
rmi
or
anything
like
that.
B
So
as
soon
as
we
released
the
first
version,
we
started
having
the
first
customers
and
we
tried
to
gather
feedback.
The
most
important
feedback
we
got
is
hey.
We
we
need
more
regions,
we
we
need
more
cloud
providers.
We
would
like
to
have
parameter
clusters
because
we
are
hosting
latency
sensitive
applications,
so
we
we
want
to
be
closer
to
this
pop
or
closer
to
that
pop
or
this
fastly
deployment
or
this
cloud
service
right.
B
So
we
got
it.
The
the
other
thing
was
reducing
the
onboarding
overhead,
because
the
onboarding
at
the
beginning
was
a
bit
convoluted,
so
false
will
register
the
namespace
and
we'll
register
a
predefined
service
account.
So
then
we
will
have
to
configure
bolt
so
the
service
account
can
read
in
a
specific
path
involved,
and
that
was
a
bit.
You
know
convoluted,
like
I
said,
also
the
pilot
team.
They
wanted
a
proper
kafka
kafka
support
in
the
company
without
needing
to
go
and
create
more
vms
peering
with
vpc
or
something
like
that.
B
B
So
also
we
provide
the
service
mesh
in
the
first
version
and
we
were
not
providing
like
a
lot
of
visibility
into
a
service
mesh,
so
that
was
a
reasonable
ask
so
with
it,
and
with
that
come
the
second
version
of
elevation,
which
is
more
or
less
how
kubernetes
looks
at
fastly
today
it
it
became
pretty
clearly
pretty.
Oh
sorry,
it
became
clear
pretty
quickly
that
with
four
team
members
we
won't
be
able
to
scale
properly
and
create
cluster
cross,
read
young
and
cross
clouds
and
whatnot.
B
B
B
This
is
one
of
the
examples
who
so
we
built
a
custom
controller
to
configure
hashicor
bolt.
So
we
we
have
also
a
tool
called
kiverno,
maybe
you're
familiar
with
it,
which
is
a
policy
management
tool.
So
now,
when,
when
people
are
registering
a
namespace
in
elevation
under
the
hood,
kiverno
will
create
a
secret
engine,
kv2
secret
engine
that
will
get
mounted
automatically
in
its
last
namespace.
B
B
B
We
created
a
library
chart
and
also
a
default
chart
that
will
encompass
like
you
know,
all
the
best
practices
like
making
sure
that
you
have
a
both
description
budgets
that
you
have
put
on
the
affinities
and
you
don't
have
to
care
about.
Should
I
put
this
annotation
for
this
or
that
annotation
for
that?
What
ingress
class,
what
server
manager
issuer
and
stuff
like
that?
So
with
this
simple
yaml
file,
people
would
will
be
able
to
deploy
an
http
application
with
tls
and
ingress
routing.
B
We
made
some
observability
improvements,
so
in
elevation.
I
think
this
is
pretty
common
pattern.
Whenever
you
deploy
your
containers
into
elevation,
you
you,
we
provide
with
a
default
dashboard,
which
is
called
the
world
load
overview
right.
So
you
will
get
cpu
memory
and
stuff
like
that,
but
we
were
missing
like
a
lot
of
servicemen's
visibility,
so
we
built
in
a
bunch
of
linker
the
dashboard
into
our
default
dashboard.
B
But
of
course
we
enable
the
linker
the
ui
and
we
enable
teams
to
execute
on
their
namespace
linkedin
tab
link
in
the
top
and
all
that
stuff
to
travel
should
because
it's
freaking
important
for
teams
like
autopilot,
which
are
indeed
it's
a
network
service
right
other
than
that
git
ops
is
cool,
but
people
were
like
okay,
so
this
thing
got
merged.
So
now
what
is
it
deployed?
Or
not?
So
we
created
a
bunch
of
grafana
tables
and
dashboard.
B
So
we
have
an
initial
exploratory
phase
where
we
have
like
our
back
roof
relays,
relaxed
policies
relaxed,
and
these
are
basically
development
clusters.
We
also,
we
also
have
a
playground
project
in
our
container
registry
in
harvard
so
people
don't
need
ci
to
push
an
image
to
it.
They
they
they
just
go
ahead,
develop
their
thing,
iterate
push
their
images
and
then
we
will
allow
hardware,
a
playground
project
in
our
development
cluster.
There's
no
enforcement
either
on
using
flags
githubs
or
anything
like
that.
B
They
can
use
the
default
chart
that
we
provide
their
own
health,
chart
royal
ml
customize,
whatever
they
need
to
iterate
and
get
their
application
running
and
then
being
able
to
move
on
to
the
next
phase.
Obviously
we
have
a
ball
cluster
per
cluster,
so
they
go
and
put
their
secrets
there
if
they
need
to
and
once
they
feel
comfortable,
they
go
on
to
the
next
phase,
which
is
build
and
then
deploy.
So
they
just
push
their
docker
file
under
hem
chart.
B
If
they
have
a
hamstring
and
not
using
the
default
chart,
then
a
jenkins
job
will
kick
off
and
then
jenkins
will
contact
hashgraph
with
the
hardware
plugin
that
we
built
in-house
to
issue
ephemeral.
Hardware
robot
account
tokens.
So
then
jenkins
will
sign
the
images
we'll
package
the
hem
chart.
If
any
and
sign
it
as
well
and
then
push
it
to
hardware's
fast
list
project,
which
is
the
only
one
allowed
in
production
and
staging
clusters,
after
that,
they
can
just
go
ahead
and
deploy
using
flux.
B
B
A
So,
yes,
let's
first
see
the
the
first
buckets
good
things
and
one
is
like
configuration
management
for
this
solution.
So
this
this
piece
of
jammel
hope
the
guys
in
the
back
of
the
room
can
read
something.
But
this
is
just
an
example
that
comes
in
the
stream
documentation.
It's
just
how
to
create
an
example:
kafka
cluster.
You
define
several
properties,
including
the
sizing,
how
which
some
defaults
and
even
some
properties
of
the
zookeeper
cluster,
that
it's
backing,
the
the
at
its
back
in
kafka
and
even
other
operators
that
we
can.
A
You
can
plug
to
this
to
this
setup,
but
we
didn't
stop
there.
As
fernando
mentioned,
we
are
using
helm,
charts
to
push
workloads
to
the
elevation
to
the
kubernetes
clusters,
so
we
also
created
one
for
the
telemetry
kafka
and
that
chart
is
actually
exposing
some
values.
Some
of
them
are
sizing
matters
which
are
not
abstracting
the
cluster
too
much,
but
some
others
and
I
think
the
important
piece
is
the
autopilot
cites,
a
value
that
it's
in
the
bottom.
A
It's
the
list
of
pops
that
are
given
deployment.
A
given
autopilot
deployment
is
expected
to
support.
We
have
equivalent
properties
in
other
autopilot
services,
and
the
idea
in
the
case
of
kafka
is
that
if
you
define
a
specific
pop,
you
want
to
support
in
a
cluster
automatically.
You
are
going
to
get
the
topics
you
need
for
the
telemetry
data.
For
that
top,
you
are
going
to
get
a
right
user
with
the
right
grants.
A
A
Then
some
processes
that
are
emitting
the
data,
like
sftd
from
pmrt,
can
pick
the
secrets
and
directly
start
pushing
the
data
to
the
topics
and
the
same
for
the
reader
in
a
predictable
location.
They
can
pick
the
credentials
and
that's
it
so
everything
is
plugged,
and
here
so
the
first
key
point
in
this
section
is
like
we
don't
have
a
separate
tool
on,
or
configuration
management
system
for,
the
kafka
cluster.
A
It's
just
another
hand
release
in
as
we
have
for
the
other
applications,
and
in
fact,
in
the
past
I
have
been
promoting,
for
instance,
polumi
a
lot
also
externally,
but
not
having
to
require
a
different
tool
to
manage
your
kafka.
Cluster
is
even
better
than
the
best
of
the
tools.
So
more
things
about
the
helm
chart
we
have
in
the
middle.
That
is
also,
as
I
mentioned,
a
good
abstraction
opportunity
and
the
sizing
knobs,
which
are
very
coupled
to
the
kafka
setup,
could
be
even
removed.
A
Now
that
we
are
supporting
the
solution
in
many
pops,
we
can't
predict
how
much,
let's
say
how
big
the
kafka
cluster
needs
to
be.
Given
the
number
of
pops
we
are
supporting,
so
we
could
even
drop
these
values
and,
in
the
end,
just
have
the
list
of
pops.
We
are
going
to
support
regarding
portability.
A
These
are
just
a
few
commands.
This
is
the
helm
releases.
Repo
fernando
was
mentioning
so
we
have
several
hand
releases
as
xml
files
in
a
given
folder
one
folder,
a
per
a
cluster
and
in
the
end,
in
order
to
deploy
autopilot
in
a
different
cluster
with
the
same
configuration
it's
as
simple
as
copying
the
helm
releases
and
in
fact
that
is
already
outdated.
A
I
think
the
also
the
streams
operator
does
a
great
job,
exposing
some
operational
flows
directly
as
kubernetes
primitives.
So
in
this
case,
if
you
want
to
browse
which
kafka
topics
do
we
have
you
can
you
can
just
use
qfctl
and
list
these
objects
or
even
other
maintenance
operations
like
I
want
to
trigger
a
rolling
upgrade
in
my
cluster,
you
just
the
streams
operator
maps
this
to
an
annotation
in
the
stateful
set
for
the
cluster.
So
in
order
to
trigger
it,
you
just
put
that
annotation
and
the
magic
happens.
A
The
streams
community
is
maintaining
like
some
great
grafana
dashboards,
but
it's
not
just
the
fact
that
we
have
grafana
to
display
all
the
prometheus
metrics
is
the
fact
that
all
telemetry
services
can
share
a
common
dashboard
where
you
can
easily
correlate
information
from
the
meter
of
the
of
kafka
with
the
kafka
metrics
themselves,
and
that
is
super
powerful
to
in
the
end,
identify
and
correlate
problems
you
may
have
in
your
setup,
and
the
same
applies
to
the
logs.
We
are
using
splunk.
A
A
Then
things
that
are
not
so
great
in
this
setup
is
the
fact
that
yeah
we
have
an
operator
an
operator
is
an
extra
component.
You
need
to
install
and
maintaining
your
cluster
a
compare
to
some
software
as
a
service
solutions,
and
actually
that
brings
an
interesting
topic
that
is
ownership
of
the
operator
right
now.
The
operators
in
the
elevation
clusters
are
maintained
by
our
platform
team,
so
fernando's
team
and
teams
like
network
systems.
They
are
just
consuming
the
operators.
A
However,
however,
we
feel
like
this
something
like
this
may
not
scale
well,
so
in
the
moment
we
have
other
teams
consuming
or
creating
their
kafka
clusters.
Just
coordinating
the
teams
to
to
perform
a
kafka
upgrade
normally
in
hand
with
an
operator
upgrade
can
easily
become
a
full-time
job
so
yeah.
We
bet
that
at
some
point,
if
this
solution
is
kind
of
consumed
by
other
teams,
we
may
need
to
involve
other
teams
to
support
the
solution
and
fernando
the.
B
B
So
the
the
first
very
interesting
challenge
that
we
found
is
that
you
know
autopilot.
Telemetry
api
receives
like
a
constant
udp
flow
non-stop,
it's
non-stop
from
the
switches,
so
we
noticed
that
pod
that
got
restarted
and
rescheduled
to
the
same
node.
Somehow
the
udp
packets
started
to
be
blackholed.
B
B
This
one
is
also
a
good
one,
very
interesting,
because
it
was
a
whole
team
f4,
and
I
appreciate
that
so
they
the
we
hit
the
hard
limit
in
in
aws
security
group
rule
limit,
and
then
we
can
we
reach
out
to
the
time
to
our
time.
There
was
nothing
they
could
do
and
and
basically
the
problem
is
that
the
aws
load,
balancer
controller,
will
create
an
excessive
amount
of
security
group
rules
per
per
load
balancer.
B
So
one
of
the
recommendations
where
I'll
switch
to
alv-
and
this
is
not
http-
so
not
not
very
helpful
or
switched
to
elv-
it's
deprecated
right-
so
not
an
option
either,
and
we
could
we
we
needed
also
to
preserve
the
source
ip,
because
otherwise
we
couldn't
perform
authentication
based
on
ip
and
stuff
like
that.
So
so
we
we
didn't
wanted
to
implement
something
like
proxy
protocol
or
anything
like
that.
B
A
A
It
represents
actually
the
moment
where
we
enable
the
telemetry
pipeline
in
a
non-production
cluster,
and
it
only
took
like
four
weeks
to
get
the
full
solution
working
in
that
non-production
pop
and
considering
that
the
team
was
quite
small
with
also
maintaining
other
services
and
doing
other
projects.
That
was
pretty
nice.
A
Also
reviewing
the
history
of
the
of
the
repo
with
the
helm
chart
for
the
kafka
setup.
There
are.
There
were
some
changes
in
the
last
months,
but
they
are
mostly
connected
to
kafka,
upgrades
and
operator
upgrades
and
not
to
scale
the
solution.
We
essentially,
we
were
able
to
just
adding
additional
values
to
scale
the
solution
to
all
the
pops
and,
in
fact,
if
you
take
a
look
to
the
helm
releases
repo,
where
we
have
the
actual
worlds,
we
are
running
in
production.
A
You
can
see
that
there
has
been
already
like
10
engineers
contributing
to
the
kafka
configuration
that
silently
means
these
engineers
have
been
creating
kafka
topics,
kafka
users,
even
without
knowing
so.
Yes,
we
have
like
this
impression
that
okay
autopilot
actually
manage
itself
and
given
this
right
now,
we
are
working
to
consolidate
other
telemetry
processing
in
the
same
architecture
and
also
the
net
network
teams
of
network
operations.
They
use
this
type
of
network
data
to
operate
the
network
every
day,
but
there
are
even
other
use
cases
like
capacity
planning
and
implementing
other
solutions.
A
On
top
of
this
information
and
yes,
and
not
only
the
kafka
like
the
telemetry
pipelines
consolidation,
we
are
also
now
migrating
other
workloads.
We
had
in
these
dedicated
gt
clusters
to
elevation,
so
we
can
just
focus
in
other
challenges.
We
have
so,
let's
now
start
closing
the
session
main
points.
A
We
need,
like
a
scalable
platform
to
run
a
network.
Automation
at
fastly
and
kubernetes
is
a
great
fit
for
this,
and,
in
addition,
our
very
specific
setup
with
some
operators,
like
the
stream
c1
and
also
some
other
configurations,
is
helping
us
in
projects
like
like
autopilot
and
its
telemetry
pipeline.
A
Okay,
so
how
we
distribute
the
the
certificates
across
the
diff,
the
different
different
workloads
in
the
cluster,
so
here
I
stream
is
our
friend
it's
when
you
declare
a
kafka
user.
That
is
a
specific
object
stream
c,
it's
a
custom
resource
stream.
She
gives
to
the
cluster
since
he
actually,
there
are
different,
like
user
authentication
types.
Yes,
we
are
using
a
tls
based
authentication,
and
that
means
that
the
stream
c
creates
the
credentials
and
stores
the
credentials
as
kubernetes
secrets.