►
From YouTube: A Peek Inside the Enterprise Cloud at Salesforce - Xiao Zhou & Thomas Hargrove, Salesforce
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
A Peek Inside the Enterprise Cloud at Salesforce - Xiao Zhou & Thomas Hargrove, Salesforce
This talk offers a peek inside the enterprise cloud infrastructure at Salesforce. Kubernetes is open source software which is becoming the de facto standard for running services as scale. Enterprise data centers are aiming to be closely managed and very secure. At Salesforce, we are bringing these two together. We are using Kubernetes to manage 2600+ hosts across 20+ private data centers. In this talk, we’ll be looking at the challenges and our approaches for using Kubernetes as the management software from several perspectives: Multi-tenants and self-serving, Management tooling, Security, Testing, Monitoring/alerting, also Visibility.
https://sched.co/UaVe
A
Thanks
everyone
for
coming,
my
name
is
Xiao
I'm.
A
director
of
software
engineering
at
Salesforce
with
me
is
my
colleague,
Thomas
Heung
growth
is
which
is
a
senior
director
at
Salesforce.
So
today
let's
go
take
a
peek
inside
the
enterprise
cloud
at
Salesforce.
We're
going
to
be
talking
about
how
we
manage
kubernetes
at
work
at
scale,
so
we
also
listed
our
email
address
there.
So,
if
you
have
any
questions
and
in
discussions
we
want
to
talk
about
after
the
talk,
please
feel
free
to
email.
Us.
A
So
here
is
today's
agenda.
First,
let's
talk
about
the
unique
challenges
in
using
kubernetes
at
Salesforce.
Salesforce
is
the
leading
cloud
company.
We
own
many
private
data
centers
as
we
develop
our
a
lot
of
the
internal
management
system
to
manage
our
private
data.
Centers
kubernetes
is
a
so
open
source
software,
which
doesn't
necessarily
fit
into
those
existing
management
tools.
A
Does
give
us
a
lot
of
the
unique
challenges
our
team
started
to
work
on
coronaries
back
in
2015,
we
actually
started
used
kubernetes
when
was
coronaries
version.
1.3
kubernetes
has
a
go
a
long
way
from
there.
We
also
have
gone
a
long
way
from
there
today
in
this
platform.
I
want
to
share
what
we
have
learned
in
this
journey.
What
worked,
what
didn't
work
and
what
can
be
improved
so
today
we're
going
to
be
talking
about
four
aspects
of
solutions,
so
first
one
is
multi-tenancy
and
security.
A
How
do
we
manage
kubernetes
cluster
when
multiple
tenants
using
the
same
cluster
with
necessary
protection?
Second,
one
I
want
to
talk
about
is
deployment
management
system
we
actually
developed
in-house
the
internal
development
management
system
and
the
third
one
is
orchestrated
production
visibility.
A
We
actually
have
open
source
some
of
this
software
I'm
super
excited
about
it.
Tom
here
is
going
to
talk
about
it
in
a
later
section
of
the
presentation,
and
last
but
not
least,
I
want
to
talk
about.
How
do
we
do
testing
monitoring
and
our
learning
and
then
we're
gonna
be
talking
about?
What's
the
future
projects,
we
have
something
in
mind:
what's
gonna
be
next
for
enterprise
cloud?
A
Awesome
I'm
glad,
like
some
people
here,
know
exactly
what
we're
going
through
with
these
challenges
and
the
next
challenge
I
want
to
talk
about
is
network
isolation,
which
means
from
production
data
center
to
production
data
center.
They
don't
talk
to
each
other
and
also
we
have
the
challenge
of
restrictive
privileges
for
the
end-user,
which
means
not.
Everybody
can
just
log
into
production
data
centers.
A
So
let
me
give
you
example
for
this
particular
challenge.
Let's
say
we
wrote
out
a
change
to
multiple
production
data
centers
and
it
has
an
issue.
So
what
do
we
do?
Developers
can
just
log
into
production
data
center.
You
know
anytime,
and
even
then
we
get
some
people
who
has
the
privilege
to
log
into
production
data
center.
To
look
at
these
resources.
They
can't
really
just
log
into
multiple
production
data
center
at
the
same
time.
So
this
actually
makes
our
communities
management
system
extremely
challenging
and
also
from
a
security
perspective.
A
We
need
to
uphold
the
standard
to
the
enterprise
level
container
security.
Ok,
so
next
I
want
to
talk
about
it's
very
interesting
internal
integration.
What
that
means
Suz
force
as
a
enterprise.
We
have
a
lot
of
the
internal
services
and
teams
that
we
need
to
integrate
with,
which
means
we
cannot
just
use
any
software
off
the
shelf.
We
have
to
use
the
Salesforce
ones,
which
means
we
have
to
in
develop
our
own
interface
on
top
of
the
Coronet
clusters
in
order
to
use
the
Salesforce
services.
A
So
next
challenge
yeah.
We
have
a
lot
of
challenges.
Next
challenge
I
want
to
talk
about.
Is
we
have
a
lot
of
the
internal
teams
to
support
we're
not
just
aiming
for
one
team,
we're
aiming
for
develop
the
coronaries
platform,
which
fits
all
teams
in
Salesforce
and
those
team?
Necessarily
they
have
to
be
the
quranic'
cast
experts.
We
develop
tools
for
them.
A
So
the
last
challenge
I
want
to
talk
about
is
the
requirement
for
the
advanced
tools.
When
we
first
started
to
use
coronaries
in
2015,
there
were
not
a
lot
of
advanced
tools,
I'm
sure.
A
lot
of
you
guys
know
that,
which
means
we
have
to
develop
our
internal
tools
in
order
to
satisfy
the
needs
of
like,
for
example,
the
marginal
production
data
centers
and
the
skills
were
running
at.
A
Okay,
so
we
talked
about
all
those
challenges
we
are
facing.
What
do
we
do
so?
First,
one
I
want
to
talk
about.
A
solution
is
Marta,
tenancy
and
security,
so
each
of
the
tenant
in
the
kubernetes
cluster
will
get
its
own
names,
which
gets
a
native
kubernetes
separation
by
default.
But
on
top
of
that
we
actually
put
additional
security
to
satisfy
the
enterprise
security
needs.
For
example,
we
have
an
TOS
communication
with
internal
PCI
services.
The
certificates
are
actually
automated
rotated.
A
Instead
of
using
the
native
coronary
secrets,
we
actually
use
internal
secret
management
system
and,
on
top
of
that,
on
the
shared
cluster,
we
have
automatic
detection
of
malicious
behavior
so
that
our
tenant
will
get
alerted
and
necessary.
Action
will
get
taken
across
the
board.
We
also
have
container
security
for
forensic
analysis
and
security
concerns.
We
actually
scan
the
container
images
before
they
even
go
to
the
production
and
also
across
all
get
ripples.
We
have
co-signing
to
make
sure
that
all
the
checking's
are
properly
signed.
A
Also,
we
have
the
channel
tracking
and
sequencing
on
a
host
level
as
well
to
better
support
multi-tenancy
and
provide
a
consistent
set
up
across
all
teams.
That's
used
in
the
same
coordinate
cluster.
We
integrated
with
service
mesh,
which
means,
if
you
deploy
our
covenant,
cluster
you'll,
get
some
cluding
features
for
free.
For
example,
api
is
throttling
low,
balancing
and
many
other
features.
A
The
second
aspect
I
want
to
talk
about
is
this
internal
deployment
managed
system
that
would
deploy
in-house.
So,
like
I
mentioned,
we
started
to
use
kubernetes
one.
It
was
2015,
so
we
developed
this
as
a
version
1
deployment
system
which
proved
that
actually
works
pretty
well.
So
let's
look
at
this
graph.
We
define
our
own
manifest
formats
and
then
we're
checking
those
at
the
tree
so
net.
A
In
the
kid
report,
if
you're
not
familiar
with
JSON
air,
it's
actually
a
pretty
clever
trees
done
templating
tool
where
your
template
is
a
superset
of
the
JSON
string
tags.
If
you're
interested
just
go
check
it
out,
it's
a
pretty
cool,
then
through
the
CI
system,
it
is
actually
getting
delivered
through
different
deployment
pipelines
into
the
production
data
center.
A
After
getting
to
a
detect
production
data
center,
we
develop
in-house
a
controller
that
will
get
those
JSON
files
and
Chrissie
RDS
on
a
kubernetes
cluster
and
after
that,
an
operator.
Actually
it's
also
running
each
of
the
kubernetes
cluster
will
watch
for
every
change
of
the
zrd.
So
we
have
creation,
updates
and
deletion.
So,
whenever
those
CRE
changes,
the
operator
will
actually
do
the
corresponding
action
to
that
kubernetes
cluster.
So
the
cool
thing
about
the
operator
is
we
actually
inject
Salesforce
integration
features
and
security
info
within
the
operator.
A
So
this
system
is
a
go
sneaking
system
so
that
any
change
does
get.
Checking
in
the
ripple
will
be
quickly
reflected
in
one
or
multiple
productions
and
also
I
think
that's
pretty
cool,
because
sometimes
people
log
in
to
the
Karina's
cluster
in
production
and
modify
something
and
this
operator
will
actually
override
this
change
to
back
the
previous
state,
corresponding
to
the
reason
and
in
the
bid
ripple,
because
we
do
resync
of
the
operator
pretty
fast.
A
So
we
say
that
is
the
system
works,
because
we
already
have
40,000
almost
40,000
PR
checking's
in
this
system,
which
means
more
than
40,000
commonest
deployment
has
happened
in
this
system,
so
we
allow
the
system
to
have
so
many
promise,
because
the
teams
actually
manage
the
deployment
provoked
by
the
PR
approval.
So
it's
pretty
scalable.
A
A
A
Cool
so
next
aspect
I
want
to
talk
about
is
orchestrated
production
visibility.
So
if
you
guys
remember,
I
mentioned
that
there's
network
isolation,
production
data
center
didn't
talk
to
each
other,
there's
also
privileged
restrictive
privilege,
so
that
people
cannot
just
like
using
your
laptop
to
log
into
production
data
center
as
they
wish.
So
this
those
restrictions
have
been
making.
You
know
ad
hoc
analysis
or
lighters.
You
know
I
just
want
to
look
at
the
production
deployment.
Those
things
are
hard
to
address
those
we
actually
deploy
developing
house.
A
This
orchestrated
production
visibility
pipeline,
which
is
pretty
cool.
If
you
look
at
this
graph
in
each
of
the
production
data
centers,
we
deploy
a
component
called
snapshot
producer
which
watches
the
kubernetes
coop
API
server
in
the
kubernetes
cluster
and
gets
all
the
status
of
all
the
components,
resources
in
that
cluster
and
it
actually
ships
all
the
state
of
the
current
kubernetes
resources
from
production
through
Kafka
into
the
Rd
data
center,
where
we
set
up
a
sequel
instance.
A
In
front
of
that,
we
have
the
pseudo
cube
API,
because
pseudo
code
API
because
is
not
a
real
Kupa
api
server,
but
the
cool
thing
is
is
speaks
the
same
protocol
as
the
Kouga
api
server.
This
actually
enables
like
people
can
use
standard,
kubernetes
tour
like
coop,
CTL
and
coop
dashboard
against
this
coop
api
server
actually
curing
the
read-only
copy
Seco
in
our
new
data
center,
instead
of
actually
logging
to
the
production
data
centers.
So
this
model
has
worked
really
well.
A
A
B
You
chef,
so
when
we
built
this
system,
we
quickly
found
that
just
clearing
the
raw
sequel
was
incredibly
powerful
for
a
lot
of
scenarios.
So
let
me
give
a
quick
example:
let's
say
an
alert
goes
off
and
I
investigate
and
I
find
that
the
root
cause
is
a
container
that
has
a
a
pod
within
a
NIC
container.
That's
in
a
crash
loop
and
it's
just
failing
to
come
up
so
I
found
that
it's
happening
this
one
cluster
and
now
I
want
to
know.
Is
this
happening
in
all
the
other
clusters?
B
Well,
I
could
run
a
whole
bunch
of
coupe
cuddle
commands
against
each
cluster
and
I
could
collate
all
the
results
and,
if
I
want
to
share
it
with
my
co-workers,
I
could
send
it
to
them
and
taxed,
but
we
found
works
much
better
as
just
writing
a
little
bit
of
sequel.
This
is
valid,
my
sequel,
syntax.
B
This
query
finds
all
pods
and
all
namespaces
and
all
clusters
with
a
phase
of
pending
and
some
message
on
the
NIC
container,
and
the
only
part
that
might
look
a
little
funny
is
where
it
says
payload
greater
than
greater
than,
and
that
is
my
sequels
shorthand
for
jason
extract.
The
payload
column
is
the
kubernetes
resource
in
json
format.
The
exact
same
thing
you
would
get
from
coop
cuddle,
get
pod
Oh
Jason
and
on
the
right
side,
is
the
Jason
path.
I
want
to
pull
a
value.
B
Out
of
so,
let's
run
this
query,
and
this
tells
me
I've
got
four
pods
with
this
issue
two
and
cluster
seven
and
two
and
cluster
21
and
I
can
see
that
the
ones
in
cluster
seven
have
been
broken
for
quite
a
while,
because
one
of
them
restarted
almost
10,000
times
and
I
can
see
that
it
it's
consistently
this
certain
it
container,
that's
failing.
So
this
is
a
useful
query.
B
I
might
want
to
run
this
in
the
future
after
we've
rolled
out
a
fix,
so
we
have
a
component
called
sequel
reporter,
where
I
can
check
this
query
into
a
config
map.
It
lights.
This
up
on
a
dashboard,
and
now
this
query
has
a
URL,
so
I
can
hit
that
URL
at
any
point
and
get
the
results.
This
query
I
could
share
it
with
my
coworkers,
so
I
go
roll
out
a
fix,
I
go
to
the
dashboard
and
no
results
return,
so
I've
fixed
it.
B
But
now
I
want
to
know
if
this
issue
comes
back,
so
we
have
another
component
called
the
sequel
watchdog
again
a
config
map
I
could
drop
in
the
same
query
and
I
could
set
it
to
page
the
on-call
person.
If
there's
ever
a
new
world
returned,
we
also
have
a
third
component,
that's
very
similar,
where
I
can
compute
a
custom
metric.
So
if
I
don't
have
a
built
in
metric
I
need,
but
I
can
compute
it
using
sequel.
I
can
drop
it
in
there.
B
We
use
that
for
things
like
how
many
nodes
are
on
this
version
of
docker,
this
version
of
kubernetes
or
how
many?
What
is
the
count
of
pods
by
phase
and
that's
been
really
helpful,
and
this
has
actually
been
used
so
heavily
that
we
hit
the
size
limit
on
a
config
map
from
handwritten
sequel
queries
and
had
to
do
a
little
work
to
condense
them
down.
B
On
the
same
note
of
visibility
and
monitoring,
this
is
a
open
source
project
for
your
release
last
month.
That
I'm
very
excited
about
sloop
is
a
kubernetes
history.
You
know
a
tool
for
capturing
kubernetes
history
and
showing
it
to
you
visually,
so
you
deploy
this
into
the
cluster.
It
monitors
keeps
track
of
about
two
weeks
of
history
and
lets
you
explore
that
history
visually.
So
what
I'm
showing
here
is
the
last
hour
of
the
lizard
deployment.
So
this
timeline
at
the
top
green
bar
is
the
deployment
these
orange
bars
are
pods.
B
You
can
see
I
started
with
four
and
a
deployment
happened
right
about
11:20
I
started
bring
on
new
pods
and,
as
those
became
healthy,
I
drained
the
old
pods
and
at
the
bottom
are
replicas
sets,
and
if
you
use
kubernetes,
you
know
that
kubernetes
can
be
a
little
lazy
about
garbage.
Collecting
old
replica
sets
they're,
usually
just
count
zero,
there's.
Actually
a
new
replica
start
set
that
started.
B
11
20
bit
got
clipped
off
the
bottom,
but
in
addition
to
showing
this
visually,
you
can
actually
drill
in
and
get
kubernetes
events
and
payload
changes
like
you
could
see
what
actually
changed
in
this
deployment.
At
this
time
and
one
cool
use
case,
we
have
internally
there's
a
team
that,
for
each
test,
run
spins
up
a
kubernetes
cluster
runs
all
their
tests
and
he
includes
loop
in
the
cluster
and
before
they
delete
the
cluster.
They
download
a
copy
of
this
loop
history.
B
So
if
they
wanted
to
debug
a
failure
after
the
fact
they
can
load
it
into
the
sloop
UI
figure
out
what
the
issue
was
and
that's
them
debugging
a
cluster
that
doesn't
even
exist
anymore,
which
I'm
not
aware
of
any
other
tools.
That
does
this.
Unfortunately,
we
don't
have
time
for
live
demo,
but
the
website
address
is
there
and
we're
giving
demos
at
our
Salesforce
booth
tomorrow
from
1:00
to
3:00
and
Thursday
from
3:00
to
5:00.
B
This
next
slide
talks
about
our
approach
to
quality
or
our
team's
approach
to
quality.
The
first
one
is
unit
tests,
which
is
very
common,
but
the
next
part
is
we
tend
to
under
intentionally
under
invest
in
traditional
integration
tests
and
edie
tests.
These
are
typically
things
that
you
would
run
in
a
VM
and
you'd
bring
up
three
different
components
and
they
would
talk
to
each
other.
The
problem
is
a
lot
of
times
these
testing
environments,
don't
perfectly
match
production,
and
so
you
miss
some
things,
but
also
sometimes
you
get
false
alerts.
B
You
know
maybe
there's
not
as
much
memory
in
the
test
environment
as
production,
so
what
we
invest
in
instead
are
watch
dogs
watch
dogs
for
us
is
a
deployment
or
daemon
set.
That
runs
a
set
of
synthetic
checks
and
a
synthetic
check
is
like
pretending
to
be
the
user,
so
I'll
change
a
deployment
I'll
make
sure
that
I
can
connect
to
the
new
pods
I'll
route.
Some
traffic
through
Dean,
you
know,
make
sure
I'm
using
the
right,
dns
I'll
read
and
write
some
data
to
a
local
volume.
B
Maybe
we
have
one
that
hits
the
API
server
with
intentionally
invalid
requests
to
make
sure
that
our
back
rules
are
catching
it.
So
these
checks,
we
run,
you
know
a
couple
times
a
minute.
We
run
them
in
every
cluster
we
operate
and
when
they
detect
a
failure,
you
know
we
wire
them
up
to
page
the
on-call
person.
We
write
the
results
to
see
our
DS,
which
we
can
again
query
and
sequel
and
view
the
history
of
in
sloop.
B
But
this
saves
us
a
lot
of
time
because
we
write
one
check
and
it
helps
us
prevent
regressions
on
new
code.
We
write.
It
helps
us
monitor
all
of
our
live
production
clusters,
so
we
know
that
every
feature
is
working
at
all
times
and
it
really
just
helps
where,
if
there's
any
sort
of
inconsistencies
across
production
data,
centers
say
this
new
one
has
a
different
chipset,
a
slightly
different
disk
driver.
Well,
maybe
I
can't
get
test
beds
that
perfectly
match
every
production
data
center,
but
at
least
I
know
that
I've
got
these
checks.
B
If
there
is
an
issue,
I'll
know
it
immediately,
as
opposed
to
getting
surprised
later
on.
Next
up
is
how
we
roll
out
our
changes
using
phased
release.
This
is
common
practice.
We
first
rolled
a
test
bed
and
then
an
RD
environment
with
customer
clusters
and
then
a
subset
of
production
and
then
the
rest
of
production.
We
have
a
few
hours
of
make
time
in
each
phase
and
all
we're
doing
is
seeing
if
any
watchdogs
fire
and
page
s
and
if
not,
we
are
confident
that
these
new
bits
are
good
and
we
can
proceed
forward.
B
But
we
also
do
is
for
the
vast
majority
of
new
features.
We
build
them
disabled
by
default,
so
I'm
gonna
go
build
a
new
feature.
I
will
first
put
in
a
boolean
into
the
config
file,
set
it
default
to
false,
and
then
my
code
is
largely
under
an
if
statement.
What
this
means
is
the
next
phase
release
goes
out.
My
code
is
now
in
production,
but
it's
not
doing
anything
because
it's
disabled
me
as
the
subject
matter.
B
Expert
can
go
in
after
the
phase
release
and
go
turn
it
on
monitor
my
feature
that
I
know
well,
because
I
wrote
it
if
everything's
good
I'm
done,
if
not
I
can
quickly
switch
it
off
and
what's
really
great
about
this
is
I
didn't
have
to
roll
back.
All
four
phases,
because
I
dick
has
discovered
a
bug,
I
just
turned
off
the
feature,
and
so
that
greatly
minimizes.
Well,
then,
the
other
thing
is
if
it
was
an
enabled
by
default,
it
would
go
out
in
the
phase
earliest
and
the
person
that's
running.
B
The
phase
release
might
not
be
the
expert,
if
not
than
our
outage
might
last
longer.
Last
is
repair
automation,
because
we
have
this
watchdog
signal
flowing
through
CDs
flowing
to
sequel.
We
just
have
a
simple
automation
system
that
can
detect
unhealthy
hosts
and
trigger
power
cycle.
Based
on
that
signal,
this
last
slide
talks
about
some
future
projects,
so
the
first
one
is
mutating
in
mission
controllers.
We
use
these
we're
starting
to
use
these
much
more
heavily
internally
for
injecting
things
around
certs
or
logging
or
a
whole
bunch
of
different
things.
B
We
found
this
to
be
an
incredibly
powerful
integration
point
in
kubernetes,
but
also
probably
the
most
dangerous
one.
If
you're
not
familiar
with
these,
you
can
basically
say
he
commits
is
when
you
go
to
create
this
resource
before
you
finish,
creating
it
send
it
to
me.
I'm
gonna
make
a
little
bit
of
modification,
send
it
back,
there's
a
lot
of
boilerplate
code
and
it
can
be
tricky
to
validate
these
right,
so
we're
building
a
common
framework
that
makes
that
easier.
B
So
we
can
ship,
high
quality
ones
and
consistently
validate
these
next
up
is
cost
to
serve
reporting
and
alerting.
So
with
bare
metal.
First
party
machines,
underutilizing
Hardware,
wasn't
something
we
were
going
to
page
someone
about
because
we
paid
for
the
hardware.
If
we
don't
have
workload
for
it
right
now,
we
can't
give
it
back
as
we
move
more
into
public
cloud.
This
becomes
much
more
important
because
we
can
auto
scale
up
and
down
VMs,
so
we're
gonna
be
reporting
and
learning
both
for
over
utilized
and
underutilized
hardware
so
like.
B
B
So
if
an
internal
team
wants
to
build
a
new
micro
service
and
they're
kind
of
new
to
this
space,
all
they
need
is
a
repo
and
source
code
and
a
build
scripts
and
a
docker
file
and
a
CI
job
to
publish
the
docker
file
and
a
helm
chair
and
a
spinnaker
pipeline,
and
then
they
get
their
app
deployed
to
test
bed.
But
that's
a
lot
of
new
technologies
all
taken
at
once.
So
building
a
one
step.
B
Onboarding
portal:
you
fill
in
a
few
details
about
your
team
and
it
gives
you
a
hello
world,
app
fully
built
and
deployed,
and
then
you
can
start
piling
in
your
business
logic.
Next
up
is
better
helm,
validations
of
PR
time,
so
our
our
helm,
workflow
today
is
I.
Go
make
a
change
to
a
helmet
art
I
send
out
a
PR.
I
have
a
peer
review.
It
it
gets
merged.
Ci
builds
it
ships,
it
spinnaker,
takes
it
and
applies
it
to
kubernetes,
but
there's
a
bunch
of
internal
requirements.
B
We
have
for
our
helmet
arts
for
security
reasons
or
for
auditing
or
any
number
of
things.
We
rely
on
OPA
to
fail
that
deployment.
If
there's
any
issues,
but
this
is
way
down
the
pipeline
and
we
would
much
rather
just
have
a
status
check
on
the
helm,
PR
saying
hey
if
you
check
this
in,
it's
gonna
fail
because
you
forgot
to
put
the
tag
for
resource
accounting
purposes.
So
that's
something
we'll
work
on
and
the
last
is
an
open-source
version
of
the
visibility
pipeline.
We
talked
about
earlier
with
the
pseudo
API
and
the
sequel.
B
The
card
version
is
a
little
coupled
to
some
internal
systems,
but
for
not
a
whole
lot
more
work.
We're
gonna
build
one,
that's
easy
to
deploy
at
public
cloud
and
open
source
for
other
people
to
use,
because
we've
found
it
to
be
incredibly
valuable,
and
with
that,
that's
all
of
our
prepared
content
and
we're
happy
to
answer
any
questions.
We
have
a
microphone
right
here.
If
do
you
mean?
Do
you
mind
using
the
microphone
so
I
for
the
recording.
B
Is
a
logging
pipeline
right?
We're
logging
messages
to
Kafka,
but
but
this
is
something
where,
on
the
other
end
we're
exposing
it
as
if
it's
a
kubernetes
api,
so
you
can
use
your
standard
tools
as
opposed
most
logging
systems.
You
could
write
a
log
query,
so
we
can
actually
run
coop
cuddle
or
coop
dashboard.
But
sorry,
what's
the
question
yeah.
B
B
Do
have
that
as
well.
We're
just
no
one
uses
it.
So
the
nice
thing
about
exposing
a
cuban
api
is,
as
new
tools
are
developed,
mm-hmm,
any
user
can
just
spin.
It
up
point
it
at
the
api
and
so
like
something
can
be
invented
tomorrow
or
like
slip
came
online.
We
can
point
it
at
the
pseudo
api,
and
now
we
have
a
historic
analysis
tool
where,
as
we
don't
get
that
for
free,
if
we
just
ship
it
to
a
log,
storage
and.
A
Also,
I
can
act
is
so
for
this
visit,
but
it's
a
pipeline
we're
more
like
looking
at
the
current
status
of
all
actual
kubernetes
resource
status
and
for
logging
some
stuff.
We
actually
have
a
building
house
logging
system
as
well
to
ship
the
logging
from
the
production
to
a
visit
like
a
dashboard
and
stuff.
So
it's
actually
their
internal
that
that's
posting
to
get
a
category
that
there's
actually
actually
internal
tools.
So
we
need
to
integrate
with
to
build
use
those
two
on
top
of
coronaries
so
to
apply.
Yes,
we
do
have
those
got.
C
B
That's
one
thing:
I
forgot
to
mention
Oh,
most
of
the
things
on
this
future
project:
we're
gonna,
try
and
open
source
like
that.
We've
had
a
really
good
experience,
open
sourcing
soup
and
some
as
much
as
possible
we're
gonna,
try
and
open
source
future
projects.
It's
actually
been
a
really
great
experience,
so
stay.