►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
You’ve migrated your workloads to Kubernetes in the cloud, but now your client requires their application run somewhere your cloud provider isn’t – what to do? This talk will walk through a real use case of augmenting a cloud managed Kubernetes deployed application with an on prem solution hosted in Russia, detailing all the problems and solutions around providing object storage, CDN, container registry, persistent storage and logging on the Rancher Kubernetes Engine using CNCF projects like Rook and Fluentd.
A
Okay,
so
welcome
to
today's
CN
CF
webinar
to
Russia
With,
Love,
kubernetes
and
exotic
locations.
You
know,
I
saw
this
title
and
the
less
lists
of
webinars
that
are
coming
up
and
I
was
like.
I
must
moderate
that
one
so
cool
anyway,
so
I'm,
Priyanka,
Sharma
and
I'm,
an
ambassador
for
the
CN
CF
I'm.
Also
a
governing
board
member
and
I
am
director
of
cloud
native
Atget
lab
and
I'll
be
moderating
today's
webinar.
So
thank
you
so
much
to
everyone
who
is
joining
today,
I'd
like
to
welcome
our
presenter
for
the
day.
A
Michael
gosh
I'm
gonna
do
my
best
with
the
last
name
here
Michael.
What
a
sorry
sorry,
Michael
Michael
double
there,
who
is
a
container
solutions
architect
at
cloud
ops.
So,
just
before
we
get
going,
though
a
couple
of
housekeeping
items
during
the
webinar,
you
won't
be
able
to
talk
as
an
attendee.
This
is
kind
of
been
going
to
be
like
a
broadcast
feelings,
but
the
good
news
is,
you
can
there's
a
QA
box
at
the
bottom
of
your
screen.
A
When
Michael's
done
with
this
presentation,
look
I'll
go
through
the
list
and
we'll
get
through
as
many
as
we
can,
and
if
there's
some
question
that
we
can
weren't
able
to
answer
because
of
time
or
whatever
reason,
Michaels
contact
will
be
at
the
end
of
his
presentation
and
you're
able
to
get
direct
responses
from
him
as
well.
Just
so
everybody
knows
this
session
is
being
recorded
and
we
will
send
out
a
link
afterwards,
along
with
the
presentation.
So
with
all
of
those
good
things
said
and
done.
A
B
B
If
you
came
here
looking
for
Russia
with
love
and
you're
a
rancher
trying
to
get
into
Russia?
This
is
not
the
talk,
and
this
was
me
doing.
Some
SEO
search
never
had
before
the
talk
and
realized
that
maybe
maybe
a
people
won't
be
able
to
find
a
talk
so
easily,
but
he
definitely
the
right
talk
if
you're
here
to
to
learn
about
how
to
run
kubernetes
on
premises.
B
So
the
agenda
today
I'll
do
a
bit
of
an
intro
about
myself
and
the
company
I
work
for
cloud
ops
and
some
cmts
info
talk
about
the
story
so
far
how
we
got
to
be
where
we
are
with
client
and
and
and
where
we
needed
to
go
to
because
of
international
data
loss,
and
things
like
that.
So
I'll
talk
about
some
of
the
technical
problems.
We
came
up
with
around
that
and
solutions.
We
came
up
with
for
the
problems
and
then
we'll
end
up
with
Q&A.
B
What's
wrong.
Apologize
here
looks
like
flip
forward
about
a
hundred
slides
okay,
so
my
name
is
Michael.
Voytovitch
I've
been
about
20
years
in
MIT,
now
I'm
a
container
solutions
architect,
the
cloud
ops
I
started
off
as
a
lamp
stack
developer.
This
was
when
the
P
was
with
Perl
and
I've
worked
with
extensions.
Python
Java
nodejs
I
worked
at
large
enterprises.
A
B
Startups
throughout
my
career
and
anything
in
between
and
about
five
six
years
ago,
I
moved
into
more
of
a
DevOps
for
all
I
supposed
to
developer,
had
three
years
production
experience,
running
the
Google
kubernetes
engine
and
about
one
year's
production
experience
with
rke,
most
of
Google
Cloud,
architects,
certified
professional.
So
at
cloud
ops
we
recognized
a
DevOps
unicorns
of
special.
Sometimes
they
too
need
help.
Navigating
ecosystems
of
projects
and
tools
meet
the
DevOps,
the
cloud
ops,
DevOps
unicorn
back
in
2016.
This
is
what
the
CNCs
look
like.
It's
pretty
simple!
B
You
can
pick
out
most
technologies.
You
can
actually
read
it
from
the
slide
and
you
could
kind
of
figure
out
where
to
start,
if
you
needed
to
pick
some
technology
but
fast
forward
to
2018
and
how
the
cmcs
community
has
expanded.
So
this
is
awesome,
but
also
adds
adds
many
layers
of
complexity
when
you're
trying
to
navigate
the
open
source
world
too
many
to
even
read
on
the
slide.
So
this
is
where
cloud
apps
can
help.
B
B
B
You
can
check
out
to
see
if
you
read
more
about
this
procedure
and
read
out
if
reach
out
to
us,
if
you
need
any
help
and
we'll
have
some
contact
information
at
the
end
of
the
talk,
so
speaking,
of
which,
let's
take
a
look
at
how
we
help
the
customer
with
their
projects
story.
So
far,
we
we
started
working
with
a
large
European
hospitality
client.
They
had
many
interconnected,
but
independently
developed
and
managed
applications,
so
they
migrated
them
from
an
AWS
installation.
B
We
basically
took
a
single
one
of
their
projects,
migrated
from
AWS,
which
is
kind
of
vanilla,
three-tier
setup
on
virtual
machines
into
containers
on
the
google
kubernetes
engine
on
google
kubernetes
google
cloud
platform.
We
also
were
able
to
leverage
several
other
GCP
specific
resources
except
the
object,
storage
offerings
cloud,
CDN
load
balancers,
the
Google
container
registry,
which
is
their
version
of
docker
registry
and
cloud
SQL,
which
is
their
managed
SQL
offering.
B
So
here's
what
our
architecture
look
like
when
we
migrated
everything
into
Google
cloud
platform,
essentially
as
a
service
entry
point,
which
was
a
hardware
base
load
balancer,
which
can
both
serve
content
from
our
CDN
and
can
serve
content
from
the
actual
application,
which
is
running
in
fact,
kubernetes
I'm
going
to
talk
more
detail
about
the
other
components
here,
as
we
get
to
how
we
migrate
these
into
an
on-premises
solution.
But
I
just
wanted
to
give
a
picture
about
how
this
looks
and
so
just
to
round
it
off.
We
have
Redis
for
a
profession
task.
B
B
So
we
completed
this
architecture
about
a
year
ago.
Now
is
March
2018.
We
migrated
the
first
component.
There,
we
kind
of
fleshed
out
all
the
woods
a
lot
of
work
around
building
automation,
terraform
ansible
things
like
that.
But
then
we
discovered
that
the
actions
the
client
is
European,
it
turned
out
about
three-quarters
of
customers
are
from
the
Russian
Federation
and
as
such
there
are
Russian
data
laws.
Specifically
one
v
2f
said
that
requires
any
customer
data
relating
to
customer
Russian.
Customers
actually
risk
to
the
hide
in
Russia
I.
B
Think
this
well,
so
is
the
guaranteed
kind
of
operation
in
case
they
get
disconnected
from
the
internet
kind
of
thing.
So
we
need
to
find
a
provider
that
actually
offered
these
ships
available
to
us.
So,
unfortunately,
as
of
this
writing
and
oneness
exchange
in
the
last
week,
or
so,
there's
no
hyper
cloud
providers
in
Russia,
so
there's
no
Google
cloud
platform,
no
AWS
is
no
Azure.
So
how
is
I
to
make
this
work?
So
we
need
an
on-premises
solution
and
the
first
thing
we
needed
to
do
was
to
find
a
premises.
B
So
we
spent
a
and
I
wasn't
personally
involved
with
this.
It
was
some
my
team
members.
We
have
some
Russian
members
on
in
cloud
ops
and
we
have
a
lot
of
contacts
internationally,
so
we
kind
of
use
that
to
explore
what
was
out
there.
I,
don't
recall
it.
We
valued
about
five
different
providers,
a
lot
of
them.
They
vary
greatly
in
quality,
and
a
lot
of
them
were
also
difficult
to
communicate
with.
Obviously
the
English
Russian.
It's
an
issue,
but
Gmail
actually
work
very
well
in
that
I
mean
some
of
them.
They
disappeared
too.
B
They
had
been
care
form
plugins,
but
as
we
start
experimenting
with
using
our
automations
again
for
that,
due
to
the
geographic
distance
between
where
I
am
in
Canada
and
the
EPI
servers
in
Russia,
we
would
get
kind
of
timeouts.
You
know
it
wouldn't
be
able
to
determine
the
state.
I
was
very,
very
viable
and,
as
well,
I
think
that
there
was
an
interaction
between
the
version
of
VMware
that
they
were
using.
The
provider
was
using
and
the
version
that
the
plug-in
supported
so
our
contacts.
What
to
provide
that.
B
They
also
pointed
us
to
a
kind
of
a
third-party
app
terraform
provider,
and
this
one
actually
worked
a
lot
better.
But
one
key
thing
that
it
was
missing
was
being
able
to
when
you
terraform
a
server
to
specify
how
many
CPU
cores
you
want
it.
So
all
that
to
say
that
we
weren't
really
reliably
able
to
use
a
terraform
provider
to
automate
a
lot
of
this
work.
So
we
ended
up
having
to
actually
do
it
the
hard
way.
What's
my
apologize
kind
of
sticking
again.
B
B
So
we
decided
okay,
let's
just
create
a
base
image,
let's
run
our
basic
automations
against
it
in
terms
of
securing
the
image
secure,
the
VM
things
like
that
and
we'll
use
that
to
build
out
all
of
the
infrastructure
very
manual
process,
but
the
more
you
got
used
to
using
the
UI.
The
easier
became
to
do
well,
of
course,
the
UI
itself,
which
was
kind
of
a
funny
story.
It's
old
version
of
cloud
director,
it's
based
on
flash
and
best
of
all
they
recommended
to
use
Internet
Explorer.
B
B
We
were
able
to
get
to
the
next
step,
just--how,
which
kubernetes
distribution.
Do
we
actually
put
on
here
and
so,
given
we
couldn't
use
anything
on
premises,
gke,
etc.
You
could
extend
it.
We
had
to
went
to
go
with
something
that
was
completely
a
self-contained,
so
we
looked
at
you
spray
initially.
Thank
you.
Baby,
em
and
I've.
Looked
at
these
tools
over
the
years
as
well.
B
With
these
tools
and
fine
things
like
adding
nodes,
decommissioning,
nodes,
upgrading
components,
kubernetes,
it
was
just
very
poor
experience
and,
in
my
opinion,
until
recently
as
well,
I'm
not
sure
if
cube
idiom
has
this
or
not,
there
was
no
abilities
through
multi
master
setups,
and
I
think
that's
something
that's
come
in
more
recently.
Of
course,
if
you
want
to
do
a
production
ready,
kinetics
installation,
you
definitely
want
to
have
multi
master
set
up
to
be
able
to
deal
with
having
proper
HH
setup,
so
enter
rancher,
kubernetes
engine,
all
right,
sacraments
engine
to
the
rescue.
B
My
opinion
is
the
best
custom
kubernetes
restorer
to
date,
and
I
was
convinced
by
this
over
several
articles.
I
read
one
of
them.
I
linked
here,
which
is
a
trick.
Fillet
wrote
a
little
blog
article,
how
they
ended
up
with
that.
They
kind
of
came
to
the
same
conclusions
about
sheep
spraying.
Baby
m
and
r
ke
and,
and
they
decided
to
use
it
for
the
same
reasons
I
did
so.
B
If
you
look
at
RT
all
it
requires
in
order
to
be
able
to
install
your
kubernetes
cluster
is
essentially
just
a
physical,
vm
or
box
running
docker,
preferably
compatible
with
a
kubernetes
version.
You
can
actually
also
run
non
compatible
version
just
at
your
own
risk
on
thing,
with
an
override
SSH
access
with
the
password
list.
Speed
login,
that's
pretty
much
it
so.
B
How
do
you
actually
operate?
Ranch
kubernetes
engine,
a
single
configuration
file
yam,
will
file,
specify
your
cluster
configuration
a
single
go
binary
that
actually
manages
RTE
and
that's
pretty
much
it
so
you
are
K
itself
has
a
set
of
commands
that
you
run
to
operate,
install
or
decommission
your
cluster.
So
the
basic
one
is
our
ke
up,
which
sets
up
initial.
It
connects
all
the
SSH
tunnels.
It'll
actually
detect
the
state
of
your
cluster.
So
if
you're
filling
a
new
cluster,
it
will
actually
start
bootstrapping
kubernetes.
B
So
by
doing
this
it
actually
runs
kubernetes
all
inside
docker
images.
These
docker
images
come
from
Rancher
itself,
and
so
the
API
server
cubelet
on
all
those
things
they
run
in
docker
itself,
so
it
wraps
a
new
cluster
with
with
your
desired
configuration
in
under
five
minutes.
This
is
probably
dependent
on
how
larger
cluster
is.
The
cluster
I'm
I
was
dealing
with.
Here
was
three
masters
for
worker
nodes,
so
it
installed
in
about
five
minutes,
and
this
multi
master
works
out
of
the
box.
Iki
CD
is
fully
distributed.
I
chose
three
setups.
B
B
This
takes
about
five
minutes,
the
customizing,
the
config
file,
all
that
stuff's
very
easy
to
do
so,
let's
take
a
look
at
a
minimal
rke
configuration
file
and
kind
of
the
highlights
of
what
the
things
you
have
to
set
are
so
at
the
very
least,
you
need
a
single
node
and
you
can
define
the
role
of
the
node.
You
cannot
have
it
be.
B
You
could
define
what
your
IP
ranges
for
pods
and
for
services,
so
what
your
CIDR
is
for
those
tend
to
use
so
since
we're
using
a
canal
CNI
in
this
case
the
add
the
addresses
you
pick
inside
your
cluster
don't
need
to
be
routable
outside
of
the
cluster
because
they're
what
they
call
that
encapsulated
packet.
So
you
can
just
pick
any
range
in
there.
That's
reasonable
I
tend
to
pick
something
in
the
10:43
just
tend
to
pick
the
same
thing
every
time.
I
use
it
every
time.
B
Other
interesting
things
for
notes
so
reason
the
canal
plugin
as
far
as
I
know,
right
now
now
I
think,
is
the
main
supported.
One
I
haven't
actually
tried
different
ones
with
rke,
but
there
may
be.
There
may
be
other
implementations
right
now,
Ord
imagine
from
Farrell's
C&I,
it's
just
a
matter
of
kind
of
trying
it
out
getting
it
to
work.
You
enable
our
back.
You
tell
it
to
ignore
the
docker
version,
so
this
happened
especially
recently,
if
you're,
if
you're
any
kind
of
an
assistant
ministration
space.
B
If
your
installed
dock
went
to
a
team
server,
you
cannot
get
dr.
17
for
it
and
I
believe
dr.
17
was
the
last
last
hydration
with
kubernetes,
so
you
kind
of
have
to
install
dr
18,
and
you
have
to
tell
Artie
to
ignore
that
it's
a
non-certified
version
and
then
finally,
you
give
it
just
a
little
name
so
operating
the
wrench
kubernetes
engine
cluster
is
really
simple:
lifecycle
management,
just
it
with
the
way
they
are
ke
binary
works
is
just
super
straightforward.
B
If
you
want
to
add
a
new
node,
you
simply
go
back
to
that
mo
and
you
add
a
another
entry
into
the
nodes
list,
and
then
you
just
run
our
k
up
again
and
it'll
literally.
Do
what
you
think
it
so
it'll
add
a
new
worker.
Node,
add
a
new
control,
know
add
a
new
succeeding
node.
All
of
those
things
will
just
work
it
does
it
in
a
non-destructive
way,
a
kind
of
a
rolling
restart
decommission,
note
same
thing,
just
remove
them
from
from
the
cluster
llamó
and
run
it
again
and
they
get
decommissioned.
B
If
you
need
to
upgrade
so
one
of
the
things
I
didn't
illustrate
in
the
cluster
yamo
file
here
is
you?
Can
there
are
a
lot
of
other
options
that
are
not
mentioned
here,
we're
just
defining
the
versions
of
each
kubernetes
component,
so
the
version
of
the
cubelet,
the
version
of
the
API
server,
etc.
I
tend
to
use
a
default
version
that
rke
supports.
But
if
you
run
into
situation
where
a
specific
version
is
misbehaving,
you
can
actually
go
in
and
override
just
that
specific
components.
B
Let's
say
for
the
sake
of
discussion,
there's
a
bug
in
the
cube.
Let
you
need
to
grab
the
next
version
of
the
image
you
can
just
go
in
and
add
that
definition
to
cluster
llamó
run
rke
up
and
it'll
go
install
the
new
foreshore,
the
cube,
look
for
you
and
then.
Finally,
if
you
want
to
do
a
complete
upgrade
of
kubernetes
upgrades,
say
from
110
to
1,
11
or
111
to
111
subversion,
you
just
grab
a
new
version
of
rke
and
you
just
run
RKA
up
and
it
does
it
the
way
you
would
expect
it.
B
So
when
you're,
using
a
cloud
provider
like
m
KS
or
GA,
or
a
K
F,
the
Amazon
Google
Microsoft
versions,
the
manage
kubernetes
inflation
is
really
elegantly
they'll.
Do
you
know
the
do
rolling
restarts
rolling
upgrades
of
all
of
them
the
arc?
A
binary
attempts
to
do
the
same
thing
it
basically
marks
a
node
is
tainted
it
drains
it
from
all
work
performs
the
upgrade
on
paint
it,
and
then
it
gets
back
any
workloads
that
it
needs
to
run
and
I
notice.
This.
This
is
from
running
our
case
since
kubernetes
1.9.
B
Up
to
now,
which
is
one
point
eleven
point
six
and
when
you
do
upgrades
of
of
kubernetes
always
make
sure
you
do
the
in
between
updates.
Don't
don't
do
major
updates,
because
this
can
cause
this?
Is
it's
more
unknown
and
you
can
you
can
kind
of
screw
up
your
installation
that
way,
so
it's
much
better
to
go
the
in-between
path,
so
1.9
21.72
one
point
11
and
this
I
took
an
opportunity.
B
Okay,
so
it's
not
quite
as
simple,
so
our
K
is
really
great
for
provisioning
and
operating
an
on-premises
cluster,
but
it's
not
a
complete
solution
in
itself.
Notice.
Nobly,
what's
missing,
is
a
storage
provider,
a
container
storage
interface
to
enable
persistent
volumes
and
persistent
volume
claims
and
load
balancer.
Some
other
missing
features
that
we're
using
from
DCP
like
I
mentioned,
was
a
Google
container
registry,
object,
storage,
cloud,
CDN
and
cloud
sequel.
So
let's
take
a
look
at
how
we
solve
each
of
these
problems
in
order
fix
one.
B
The
quickest
solution
is
is
how
to
pull
images
from
the
Google
container
registry,
so
one
of
the
artifacts.
So
since
the
docker
images
themselves
that
we
run
for
this
workload,
they
don't
contain
any
customer
data.
There
wasn't
a
legal
need
to
store
them
in
Russia,
so
we
decided
to
keep
using
our
PCR
repository
and
the
way
you
can
actually
pull
from
to
CRS.
B
You
had
a
special
docker
secret,
which
contains,
as
you
can
see
there,
which
contains
a
service
account
key
from
GCP,
and
you
simply
tell
your
deployment
to
pull
that
to
use
that
pull
secret
when
it
pulls
it.
As
long
as
that
service
account
has
been
granted,
objects
store,
viewer
permission
inside
your
project
that
hosts
your
images.
This
will
just
work
as
expected.
B
Cloud
sequel
on
Prem:
nothing
really
special
to
do
here.
This
is
just
install.
My
sequel,
I
was
able
to
use
the
same
Debian
images
we
use
for
the
kubernetes
cluster
docker
that
and
we
have
some
cloud
ops
managed
to
ansible,
run
books
for
configuring,
my
sequel
5.7,
so
we
essentially
tend
to
do
a
standard
deployment
of
the
single
master
or
two
slaves.
One
is
a
real-time
slave
for
doing
reporting
and
failover
in
case
there's
an
issue
and
a
delayed
slaves
which
ships
bin
logs
instantly,
but
doesn't
apply
them
for
30
days.
B
This
allows
us
to
basically
restore
at
a
point
in
time,
even
if
there's
replicated
corruption
for
up
to
30
days
so
fairly
straightforward.
This
is
not
really
cloud
native
or
or
anything
else
that
the
object
storage
becomes
more
of
an
interesting
problem.
So
when
I
started
the
solutions
about
a
year
ago,
there
wasn't
really
anything
else.
Those
production-ready
specifically
workers,
f-for
on-premises,
that
I
was
able
to
find
so
we
kind
of
I
spoke
with
a
couple
other
consultants
that
cloud
ops
and
we
decided.
Well.
B
This
was
technology,
came
out
of
Red
Hat
as
well,
I
believe
and
replicas
actually
replicated
NFS
before
you
know
either
don't
know
it,
and
but
the
idea
here
is
that
we
could
set
up
two
ingress
nodes
to
front
all
traffic
coming
into
the
cluster
and
by
the
way,
I'll
have
a
diagram
about
all
this
stuff
a
little
bit
later,
but
you
said
of
ingress,
knows
to
serve
contents
from
the
cluster
itself
and
also
just
serve
content
from
Gluster
FS.
So
it
serves
as
a
reverse
proxy
and
the
fede.
B
Yeah,
so
each
so
each
thing
Gris
note
basically
has
a
dedicated
device
breakfast
tours
by
100
gigs
worth
of
data.
Any
data,
that's
written
to
one
node
gets
replicated
to
the
other
and
vice
versa,
and
it
kind
of
gives
us
a
poor-man's
CBN
to
be
able
to
serve
any
kind
of
CDM
asset.
So
it
doesn't
matter
if
the
person
come
in
zone,
one
ingress,
no,
the
other
ingress,
no
they're,
gonna
they're
gonna
serve
the
same
data.
B
Then
what
we
would
do
is
we
would
take
the
Gloucester
FS
volume
mounted
in
Fatty's
kubernetes
worker
node,
and
this
is
where
the
application
that
we're
running
in
kubernetes
could
actually
read
and
write
asset
for
the
CDM
and
then
finally,
we
just
mount
that
volume
that
was
mounted
on
the
worker
node
into
each
pod
host,
pass
volume
and-
and
then
this
allows
the
the
workload
to
basically
write
directly
to
Gloucester
air
force.
This
works
fairly.
Well,
you
know
we're
not
dealing
with
millions
of
hits
per
day
of
things
like
that.
B
A
B
B
We
temporarily
just
reconfigured
the
application
to
use
SQL
database
with
session
Kasich's,
that
of
Redis,
but
I
decided
in
December
after
having
cleaned
in
action
on
cue,
con
and
rook
kind
of
graduated
to
an
incubating
project,
to
experiment
with
that
and
use
that
for
the
PVC
for
the
storage
driver
for
kubernetes.
So
it's
really
straightforward
to
use.
I
did
spend
a
bunch
of
time
playing
with
you
know
their
Quick
Start,
Guide
and
they're
very
stocking
tation.
Things
like
that
I
find
the
quickest
way
to
get
started.
Reliably
was
simply
to
use
the
helm
repository.
B
It
sees
the
helm
chart.
So
if
you
add
the
helm
repo
to
rook
directly
and
then
you
install
it,
that
was
the
most
reliable
way
to
get
up
to
work
so
literally
one
command.
It
takes
a
couple
of
minutes
kind
of
bootstrap
itself,
and
it's
up
and
running
so
rook
itself
is
is,
is,
is
essentially
a
plug-in
for
kubernetes
such
a
container
storage
interface
driver,
so
it's
loaded
dynamically
by
kubernetes,
specifically
the
cubelet
that
runs
on
each
on
each
worker
node.
B
So
there's
some
extra
configuration
that
was
required
for
rke
to
make
this
work
and
potentially
just
providing
two
extra
arguments
into
the
into
the
cluster
llamó
file
that
points
the
cubelet
to
the
location,
where
rook
stores
its
ESI
binaries
and
treats
them
as
executive.
Well,
so
that's
just
what
these
kind
of
these
couple
of
configuration
parameters
use
once
I
added
that
to
the
cluster
gamal
I
was
able
to
run
rke
up
again
and
then
that
literally
patched
the
live
cluster.
B
When
is
all
done,
everything
just
automatically
restarted
and
it
works
so
up
until
this
point,
I've
seen
all
the
Archy
or
the
component
start
up
and
kind
of
crash,
and
some
of
them
weren't
ready.
Adding
this
everything
just
worked
as
expected,
almost
instant.
So
now
we
have
a
rook
cluster,
so
rook
rook
operator.
Rather
so
the
rook
operator
operates
a
chef
cluster
underneath
the
hook,
so
we
need
underneath
the
hood,
so
we
needed
to
now
configure
SEF
itself.
B
So
what
I
actually
did
was
I
cloned,
the
rook
git
repository
and
there's
some
great
example
configurations
to
kind
of
get.
You
started
there.
So
I
thought
we're
talking
about
all
the
vanilla
examples
and,
and
but
there's
there's
a
lot
more,
obviously
right,
and
so
what
we
decided
to
say
it
was
each
worker
node,
we're
gonna,
have
a
dedicated,
100
gig,
a
speed,
brick
and
we're
just
gonna
tell
zeph
to
only
use
that
brick
for
persistent
data.
So
that's
where
you
see
this
device
filter
SDB
in
there.
B
B
So
it's
kind
of
a
couple
of
levels
of
abstraction
or
you
have
rook
controls.
Sef
SEF,
creates
blocks
and
blocks
type
of
storage
classes
for
the
purpose
of
this
example,
like
I,
said
I'm
using
vanilla
configuration,
so
here
you'll
see
very
simple:
configs
force,
F
block
pool
replicated
files
one
this
just
tells
it
to
use
an
object,
storage
device
replicated
exactly
once,
depending
on
your
workload,
depending
on
your
use
cases,
depending
on
how
resilient
you
want
your
data
to
be
how
perform
it.
B
Now
we
have
a
storage,
classical
rooks
F
block
and
we
can
install
ready
from
just
scallops
use
that
storage
class
so
once
again
I'm
a
fan
of
using
helm.
It's
just
very
simple
to
install
a
lot
of
complicated
applications
and
most
of
them,
if
they're
set
up
properly,
will
actually
have
overrides
for
these
types
of
settings.
So
here
you'll
see
an
example
where
I'm
installing
stable
registration
and
I'm
telling
the
master
and
the
slave
to
use
a
persistent
storage
class
of
rooks
F
blocks.
B
So
when
you
run
this
it'll
actually
tell
request
storage
from
from
staff
and
from
Brooke
and
use
it.
You
can
also
specify
how
large
you
want
that
to
be
I.
Think
the
default
in
this
case
is
5
gigs,
or
something
like
that.
I
guess:
10
C's
register
as
a
session
cache,
or
something
like
that.
You
said
you
don't
need
some
more.
You
don't
generally
need
larger
blocks.
But
if
you
do
it's
all
settle
using
the
help
chart
Oh
once
again
to
teach
you.
B
Okay,
so
so
we
talked
about
the
PVC
now
how
about
logging
so
I
mentioned
before
fluency
and
salinity
is
also
used.
When
you
run
in
google
kubernetes
engine
kubernetes
engine
has
basically
become
the
de-facto
blog
engaging
Tsukuba
dies,
it's
another
CNCs
project,
so
it's
much
needed
in
the
logging
space
currently
I
believe
is
implemented
in
Rudy
and
there's
in
Ruby.
There's
a
non
dynamic
language
agent
in
development
called
fluent
bit.
I
think
is
a
roaster
I,
don't
think
all
the
features
are
there
yet
and
no-one's
releasing
in
production.
B
So
I'll
talk
about
the
Ruby
version
as
many
input
plugins
many
output
plugins,
which
is
why
it's
really
so
great
and
it's
been
under
active
development
for
for
a
while.
What
was
a
bit
frustrating
I
think
in
the
last
year
that
maybe
they
got
a
lot
of
pickup
in
an
adoption
when
they
went
from
the
zero
version
to
the
one
version
to
the
one
point,
one
to
the
one
point.
B
So
this
is
kind
of
a
I
should
have
known
better
and
pinned
it
right
right
from
the
start,
but
these
are
kind
of
things
to
watch
out
for
when
you're
dealing
when
you're
dealing
with
on-premises
installations
as
well.
So
what
do
we
need
from
from
our
actual
logging?
We
need
a
comprehensive
kubernetes
monitoring
configuration
so
pull
out
getting
visibility
and
tells
kubernetes
components
themselves.
We
want
to
have
some
sort
of
same
application.
Log
pardoning
parsing
out
of
the
box,
so
some
some
of
our
applications
were
java
or
no
js'.
B
That
was
the
case.
I
ended
up
kind
of
just
reverting
to
the
last
major
version
was
just
one
one
three,
but
the
out
of
the
box.
Json
and
JSON
parting
didn't
seem
to
work
in
this
version
and
I
actually
I
got
a
chance.
Our
our
booths
at
cube
con
was
right.
Next
to
the
fluent
deep
booth,
so
I
went
and
I
picked
on
a
couple
of
the
engineers
they
might
have
thought.
B
I
came
up
as
a
bit
of
a
lunatic
because
I
wanted
them
to
fix
this
in
their
images,
because
nothing
ever
nothing
ever
seemed
to
have
come
of
it.
But
I
tried
to
I
tried
to
make
them
understand
what
the
problem
was
and
like
it
was
just
it'd,
be
very
nice,
as
they
would
provide
a
bundle,
an
image
that
just
does
everything
out
of
the
box
without
having
to
do
is
kind
of
custom
patching
and
adding
things
to
my
nose.
You
can
check
that
I'm
sure
they're
going
to
get
a
better
with
time.
B
It's
just
I've
just
been
dealing
with
these
version
changes
and
they
caused
frustrations
every
time
there
was
a
major
one
and
finally,
we
need
a
way
to
designate
certain
attributes
in
the
JSON
logging
from
our
application
as
a
numeric.
So
this
is
important
when
you're
sending
your
logs
into
elasticsearch
more
efficient
to
search
for
those
types,
strings
versus
numbers,
it's
more
efficient,
as
it
knows
what
type
of
data
you're
searching
for.
B
So
this
is
essentially
what
we
started
with
so
inside
the
fluent
d--
image.
There's
a
kubernetes
conf.
We
just
pull
that
out
of
the
image
and
we
added
this
a
bit
of
peace
to
it,
which
is
what
I
found
from
extensive
googling
and
kind
of
renaming
the
the
JSON
and
JSON
parsing.
So
essentially,
what
this
configuration
says
for
every
line
that
comes
in
from
the
the
docker
container
of
your
application.
B
And
then
you
simply
take
that
entire
file
put
its
content
into
a
config
map
and
you
can
use
it
from
the
salinity
agent.
I
would
say
it
is
a
bit
of
a
ramp
up
time
learning
how
salinity
works
properly
kind
of
grokking,
how
the
configurations
work,
but
it
just
kind
of
comes
with
time.
They
all
said
it
makes
sense
one
day.
So
you
simply
take
the
contents
of
that
file.
Stick
it
into
a
config
map
and
then
you
can
mount
the
config
map
into
the
into
the
fluent
D
pod
itself.
B
So
some
of
you
may
not
know
this.
When
you
mount
files
into
a
pod,
they
can
come
from
a
config
Matic
and
it
can
come
from
a
physical
volume,
but
you
can
mount
you
can
mount
us
a
specific
file
into
a
path,
whereas
most
people
I
think
the
most
examples.
When
you
look
at
how
to
do
the
secure
entities,
they
only
showed
how
to
mount
an
entire
directory
into
a
pod.
They
don't
you
how
to
do
a
single
file.
So
all
that
to
say
is
the
example.
B
I
have
here
at
the
bottom,
where
you
mentioned
subpaths
kubernetes
accounts
that
tells
kubernetes
to
only
mount
in
that
one
single
file
and
not
anything
else
in
that
directory.
What
would
happen
otherwise,
as
fluent
II
would
start
up,
it
would
see
that
there's
no
other
files
other
than
kubernetes,
confident
crash.
This
way,
we're
providing
a
single
override
for
kubernetes,
conch
and
and
fluent
he
just
works
happily
ever
after.
B
A
B
You
start
cilenti
most
important.
One
I
think
there
is,
is
a
logstash
prefix,
so
when,
when
fluent
descends
data
from
this
particular
agent
into
the
Galactic
search
infrastructure,
it'll
prefix
the
index
with
this
value
here
so
one
example
you
have
and
something
actually
use
for
a
client
is.
We
have
kind
of
a
shared
logging
in
shard
shared
log
in
kubernetes
cluster
that
receives
fluent
B
data
from
other
clusters.
Each
other
cluster
has
its
own
value
for
this.
B
B
Optimizing
performance,
isn't
it
and
indexes,
and
things
like
that,
one
thing
I
probably
won't
talk
about
here
is:
if
any
of
you
have
worked
with
Altis
there's
a
component
called
a
curator
which
can
expire
all
data
I
to
be
on
the
top
of
talk
of
this
conversation,
but
I
just
wanted
to
mention
it
because
one
day
you
know
you're,
if
you're
just
logging
for
a
year,
you
may
very
well
run
out
of
logging
space.
If
you
don't
use
that
kind
of
component.
B
So
I
mention
ELQ
right.
So
initially
we
had
a
separate,
dedicated
l
cluster,
so
we'd
run
three
virtual
machines
for
H
a
running
a
elasticsearch
cluster.
These
were
running
ansible
were
play
books
that
the
cloud
ops
had
developed,
but
we
found
this
three
notes
just
for
H
a
but
is
very
underutilized
over
provisioned
and
I
was
really
wondering.
What
do
we
really
need
to
have
like
kind
of
a
dedicated
cluster
like
this?
B
Could
we
not
just
add
that
compute
that
the
L
cluster
used
to
the
kubernetes
cluster
and
then
run
ELQ
inside
kubernetes
well
turns
out?
We
could
there's
a
helmet
art
stable,
flash
elastic
back
to
let
you
do
just
that,
so
we
wipe
the
L
cluster.
We
put
fresh
w-9
images
to
select
the
other
worker
nodes,
I
added
those
three
worker
nodes
to
the
arc
to
our
case
cluster
ammo
file
and
then
I
ran
our
K
up
and
a
couple
minutes
later
we
had
all
those
former
work
elk,
dedicated
worker
nodes
writing
as
kubernetes
worker
nodes.
B
I
inside
a
kubernetes
so
now
to
actually
install
the
l
card.
We
wanted
to
give
it
a
larger
per
system.
So
in
this
case
was
gave
it
50,
gigs
storage
and
we
tilted
to
use
the
rook
storage
class,
but
we've
created
earlier
I
mentioned
before
you
can.
So
this
chart
is
actually
kind
of
a
meta
chart.
That's
made
up
of
all
the
various
elastic
fact:
charts
with
the
structure
combined,
a
truck
for
elastic
search,
art
for
log
stash,
etc.
B
You
can
set
all
those
configurations
inside
the
helm,
install
command
and
you
can
enable
and
disable
them
so
I
didn't
include
them
here.
I
tend
to
disable
log
stash
locks
is
a
great
front-end
for
something
it
doesn't
talk
directly
to
elasticsearch,
but
in
our
case
our
fluently
agent
sinks
directly
to
elasticsearch.
We
didn't
actually
need
a
front
end
for
lots
of
that.
So
we
don't
actually
install
that
and
then
as
well
the
curator
you
can.
You
can
enable
the
curator
here
and
then
you
have
to
go
and
configure
how
long
it
can
be.
B
How
often
should
expire
old
old
data
from,
and
so
unfortunately,
when
you,
when
you
run
an
install
the
helm
charge
for
elk
and
it's
odd
because
I
think
the
bug
is
still
there
I
had
to
fix
it.
Last
week
as
well-
and
this
is
there's
a
weird
bug
with
kibana
when
you
install
it-
which
it
has
a
hard-coded
value
for
where
it
expects
to
find
the
elasticsearch
endpoint
so
first-time
installed,
you
have
to
fix
this
footprints,
always
be
the
case.
I've
been
telling
myself
every
week
that
won't.
You
know
one
of
these
days.
B
I'll
get
some
time
to
fix
it
and
send
the
PR
I
just
haven't,
had
the
contest
to
it,
but
it's
pretty
easy
fix,
I,
just
kind
of
keep
it
in
mind.
You
just
edit
the
deployment
after
you've
installed
it
with
helm,
and
you
point
it
to
you
could
look
up
the
tube
GPL
service
for
it.
So
in
this
case,
if
you
called
inflation
elk,
the
service
is
called
elk,
elasticsearch,
that's
client,
and
so
that
now
plugs
in
Caban
up
to
the
elasticsearch
backend.
B
And
then
you
have
to
tell
fluently
just
to
point
to
point
the
flu
and
D
agents
to
the
elasticsearch
host,
which
is
also
the
same
thing
as
the
elasticsearch
client.
So
you
literally
put
the
same
value
inside
excellent
D
configuration
now
for
actually
accessing
elk
I
discover
ways
to
do
it.
If
you
don't
really
want
to
deal
with
setting
up
a
separate
ingress,
secure
things
like
that,
the
simplest
one
is
to
use
cube.
Ctl
port
forward
just
set
up
a
local
tunnel
into
into
the
Cabana
front.
End
I.
B
You
could
also
set
up,
expose
it
as
a
node
port
and
a
tour
and
additive
engine
X
front
end
or
what
I'm
going
to
show
quickly
is
just
adding
it
as
a
kubernetes
ingress
resource
with
the
username
and
password
to
secure
it.
Let's
make
that
kind
of
the
most
reliable
installation
so
rke
when
you
install
it,
does
install
the
nginx
ingress
controller
as
the
default
ingress
controller.
Given
it
doesn't
have
any
on
cloud
cloud
controller
features.
B
So
you
have
to
use
annotation
to
tell
kubernetes
to
use
the
nginx
controller
and
you
have
to
secure
the
services
necessary.
So
quick
way
to
do
that,
is
you
just
run
HP
password
and
you
give
it
a
username
and
a
password
there
and
it'll
give
you
a
string
that
you
then
can
tell
you
can
give
to
nginx
which
it
will
use
to
actually
secure
your
site.
So
here's
an
example
as
I
get
that
string.
I
create
a
secret
from
it
and
then
I
I'll
show
you
how
this
actually,
how
actually
use
it.
B
So
you
create
a
secret
which
contains
the
offspring,
and
then
you
finally
have
to
create
a
secret
to
contain
the
ssl
certificate
in
this
case,
something
to
be
aware
of,
and
if
any
of
you
have
worked
within
risks
already
you'll
notice
the
certificate
there
has
to
contain
the
entire
certificate
authority
chain
as
well.
It
has
to
be
in
in
a
certain
order,
and
I
find
certain
cell
providers
they'll,
send
you
those
out
of
order
like
the
fillion,
reverse
or
they'll.
B
So
here's
what
you
would
actually
the
ingress
resource
you
would
create
to
actually
use
this.
So
the
first
thing
is,
you
tell
any
pre
create
annotation
to
sell
to
use
the
nginx
ingress
class.
You
give
some
meta
information
for
nginx
to
when
it
asks
for
authentication
for
someone
actually
using
Cubana,
and
they
basically
tell
it
I'm
using
an
off
type
basic
and
pull
that
off
from
the
secret
that
we
created
in
the
previous
slide
and
there's
basically
straightforward
ingress
rules.
Here
we
say:
okay,
I
have
ELQ
about
my
Apps
domain.
B
It's
simply
forward
to
my
Cabana
back
end
and
it's
secured
by
this
TLS
secret.
One
other
thing
I
want
to
mention,
because
I
think
this
is
pretty
neat
especially
coming
from
3ke.
The
DK
ingress
controller
doesn't
do
automatic,
HTTP
HTTP
redirection,
which
is
really
really
really
frustrating,
but
enging
nginx
controller
does
so
there's
no
extra
annotations.
B
You
need
to
kind
of
disable
the
HTTP
redirect
it'll
just
do
it
automatically
with
this
type
of
configuration,
which
is
nice
so
for
whatever
even
the
rest
cloud
provided,
if
not
have
the
ability
to
assign
a
hardware
load
balancer
to
front
the
infrastructure,
it's
frustrating,
so
we
decided
to
basically
just
do
a
bit
of
a
poor-man's
load
balancer,
which
is
to
nginx
ingress
notes.
Are
the
same
I've
been
talking
about
all
along
round-robin
VMs
that
resolves
each
nodes
IP,
it's
kind
of
a
poor
man's
load
balancer.
B
B
A
A
B
Well,
this
was
just
to
illustrate
this
kind
of
the
end
state
and
see
where
every
single
is
and
how
everything
is
made
up
of.
We
also
use
data
dog
and
in
tanana,
as
our
monitoring
agents,
so
not
represented
here,
there's
actually
data
dog
container
also
running
as
well.
These
are
pretty
neat.
They
offer
extra
insight
into
running
workloads
if
you're
running
JVM
or
something
like
that
apologize
and
that's
the
end
of
that
happy.
Unicorns
and
rainbows.
B
Special
thanks
to
Julia
Simon,
a
senior
property
marketing
manager,
cloud,
ops
and
Emma
DeAngelis
freelance
are
just
artists.
They
made
my
presentation,
look
like
chicken
scratch
that
you
look
professional
and
then
just
some
more
information
cloud,
ops,
key
information.
You
can
reach
double
column,
so
calm
or
send
us
email
info
card,
ops
and
there's
rook
and
Artie.
Absolutely
okay
over
to
Janka,
wonderful.
A
Well,
thank
you
so
much
for
that
great
presentation.
I
appreciated
the
detail,
and
now
we
have
some
time
for
questions.
So
let
me
look
into
our
Q&A
box.
So
there's
one
open
question
here:
well
now,
good
yeah
audience
keep
popping
in
your
questions
the
more
the
merrier
so
I'll
start
with
the
first
one.
Valid
shari
asks
if
my
work
notes
all
have
access
to
NetApp's
ash
NFS
should
I
still
consider
different
storage
such
as
Southfork
or
we
can
just
work
with
host
volumes
over
NFS.
What
do
you
think
Michael
great.
B
Question
if
there's
no
extra
cost
for
you
to
work
in
that
environment,
I
think
that's
a
great
option
for
now.
I
say
this
because
while
I've
been
working
with
the
rook
for
a
couple
of
months
now,
two
or
three
months,
I
feel
it
it's
production-ready,
but
like
not
battle-hardened.
Yet
so
you
might
say
for
very
simple
use
cases
use
rook.
If
you
have
kind
of
complicated
setup,
you
can
use
your
net
file
or
touch
you.
Those
kinds
of
things.
B
Little
bit
it's
going
to
be
very
specific
on
the
type
of
workload
and
the
type
of
secure
the
type
of
reliability
you
need,
but
I
would
say
it
would
probably
be
the
right
time
to
experiment
with
using
real
thing.
Rook
is
a
fine
strategic
choice.
Moving
forward
and
I
know
that
they're
gonna
get
there
this
year
with
being
fully
kind
of
battle-hardened
in
production,
but
for
your
specific
use
case,
I
would
say:
experiment
with
it.
First
before
you
start
replacing
the
next
filer
stuff
thanks
great.
A
So
related
to
violets
question
John,
dilly,
I,
hope,
I
said
that
right
has
a
question.
There
are
some
others,
I
had
of
the
queue,
but
since
it's
related
and
I'm
going
with
this
first
and
we
have
plenty
of
time
so
he
says:
why
did
you
choose
a
rook
for
Redis?
Maybe
see
not
the
Gloucester
FS?
You
already
had.
B
So
there's
a
there's:
a
provisioning
agent
for
Gloucester
access
code,
caddies
and
I
had
experiments
experience
with
that
through
no
other
project,
I
was
working
on
and
it
is
very,
very,
very
fragile
way
of
orchestrating
cluster
facts.
So
essentially
the
problem
with
luster
has
I
think
it's
a
bit
of
a
it's
a
fine
replacement
for
NFS,
if
that's
all
you're
using
it
for
but
free
provisioning,
dynamic
storage
in
kubernetes.
It's
not
so
good,
except
in
my
opinion.
France
is
essentially
like
an
iteration.
B
A
B
I
mean
chef
just
gives
you
the
ability
to
if
your
pod
stops
on
one
worker
node
and
starts
on
another.
It
has
access
to
all
the
data
right.
The
standard,
behavior
you'd
expect
some
persistent
volume
right,
whereas
local
data,
your
you
know,
if
it
moves
across
nodes,
that's
that's
not
going
to
work
right.
A
B
A
B
It
almost
sounds
like
that's
more
of
like
a
comparing
to
say,
cube
spray,
which
is
ansible
based
correlation,
Staller
kind
of
something
similar.
If
that's
the
case,
I
would
I
would
argue
those
types
of
installers
kind
of
function
on
the
wrong
paradigm,
I
think
like
they're
there.
What
I
really
really
like
about
our
case
for
single
binary,
there's
a
single
path
for
everything
it.
It
has
more
integrated
control
over
what
it's
doing,
as
opposed
to
you.
B
A
B
So
maybe
just
add
to
that
again
the
way
that
these
types
of
installers
work
you
know
they're
over
an
SSH
connection,
they're
using
Python
I'm,
assuming
in
this
case
or
what-have-you
right,
but
they
actually
perform
direct
inflation's
on
the
server
which
is
really
really
cool
about
RK
is
that
everything
is
docker
images,
so
there's
a
lot
less.
Everything
is
completely
self-contained
in
the
RK
installation.
B
So
good
question
so
since
I'm
in
DevOps
I
do
store
everything
in
get
the
source
repositories,
but
our
cloud
ops
as
a
company,
you
know
we're
DevOps
shop,
we're
not
actually
doing
the
development
of
the
products
that
I've
talked
about.
So
in
a
previous
life,
when
I
was
working
when
I
was
doing
everything
DevOps
and
development
I
tend
to
keep
the
source
code
together.
B
Now
we
kind
of
have
a
clear,
separate
repository
for
the
application
source
code
and
we
have
a
separate
repository
for
the
DevOps
code,
but
yeah
I
make
an
RT
subdirectory
that
contain
all
the
configurations.
I
have
special
SSH
configurations
because
how
I
ain't
access
the
physical
nodes
things
like
that,
but
yeah
I'm
I,
would
group
them
together
if
I
could
but
I
can't
so
I.
Keep
it
separate
like
that,
but
I
still
keep
it
organized
per
project.
Isn't.
A
B
B
A
B
So
it
runs
like
I
said
it's
literally
running
docker
containers
on
the
physical
host.
So
if
that
host,
if
you
want
to
run
SELinux
on
that
host,
you
can
surely
do
that
right
or
whatever
flavor
I
other
than
that'd
use
a
standard
kubernetes
from
docker
security
like
everything
that
every
kind
of
kubernetes
pack
has
in
isolation,
really
nothing
special
I'm,
not
aware
of
any
kind
of
direct
integration
with
that
see.
Linux
is
what
I
was
trying
to
say:
cool.
A
All
right,
so
those
were
the
question.
Oh
wait,
there's
something
in
chat.
Let's
see
here,
teacher
okay,
so
Alexey
Smirnov
says:
is
it
possible
to
customize
the
fluent
parser
for
specific
pod
/
deployments,
or
is
it
local
config
only
can
it
be
configured
to
properly
parse
multi-line,
/
sack,
dresses,
I,
know.
B
Only
on
on
on
that
particular
parser,
I
believe
also
the
configuration
you
can
tell
you
can
when
you
specify
like
a
filter
or
parse,
so
it
you
can
tell
to
look
for
certain
things
and
if
they're
not
there,
it's
just
not
going
to
apply
it
right.
So
in
your
use
case
it
would
be
specific
because
it
wouldn't
be
parsing
anything
else.
B
A
B
Nice
centralized
solution
for
logging,
it's
just
that
the
ease
of
use
getting
it
configuring
and
getting
it
running.
We
don't
do
a
lot
of
heavy
like
gigabytes
per
day
of
logging.
It's
just.
It
was
a
nice
way
for
us
to
route
all
the
stuff
into
Elkin,
and
everything
like
that,
yeah
I
will
say
having
if
you
do
have
like
heavy
log
writing
requirements.
There's
definitely
some
caveats
of
the
stuff
that
I've
shown
so
running
ELQ
inside
kubernetes
you're
dealing
with
the
performance
of
the
underlying
CSI,
and
things
like
that.
B
A
Wonderful
cool
all
right!
Well
audience.
Do
you
have
any
more
questions?
We
have
a
couple
of
minutes,
so
I
see:
okay,
there's
one
more!
Do
you
from
Alexi?
Do
you
think
it's
a
good
practice
to
have
separate
EFL
or
elk
for
cluster
logging
and
a
separate
one
for
application
logging
on,
or
is
that
unnecessary
complexity
for
single
single
capital
clusters.
B
So
assuming
performance
not
necessary,
miss
you
right,
I
like
having
everything
in
one
spot,
because
you
can
do
queries
that
look
at
or
or
dashboards
that
look
at
multiple.
You
know
you
can
look
at
your
application
performance
and
your
VM
performance
in
the
same
spot
as
opposed
to
separating
and
also
like
the
kubernetes
related
state
I
like
that,
but
depending
again,
depending
on
like
how
much
logging
you're
doing
and
if
it's
have
multiple
sources.
That
may
be
something
that
you
need
to
kind
of
shard
on.
B
In
my
experience,
I've
did
that
way:
10
Artemis
card
environment
so
I'll
have
a
broad
cluster.
I'll
have
a
state
across
my
house.
You,
cluster
etcetera,
but
I
tend
to
just
using
you
know,
using
different
indexes
that
go
into
elasticsearch,
just
to
add
to
keep
keeping
performance
and
and
just
log
it
that
way,
as
opposed
to.
A
B
A
Ok
any
so
we
have
two
minutes
left
so
time
for
one
more
question
or
comment:
I
always
welcome
comments,
because
people
might
have
something
to
add
to
the
story.
So,
let's
see
here,
Q&A
has
something
just
Arun.
Agarra
asks
just
curious:
do
you
see
any
difference
between
Splunk
versus
data
table
for
metrics?
Besides
the
cost?
Oh
good.
B
B
Data
dog,
it
seems
like
so
I
actually
like
I,
said.
Maybe
that's
come
down
in
price,
but
what
I
know
about
data
dog
is
that
they
have
a
lot
of
plugins
a
lot
of
features
that
make
it
that
make
it
very.
It
makes
it
very
intelligent
and
what
it
can
actually
log
detect
in
your
system
that
you
don't
have
to
configure
out
of
the
box
like
it
was
that
kind
of
stuff
I
found
really
really
powerful,
but
if
I
didn't
have
to
spend
too
much
brainpower
to
actually
configure
them.
B
A
Reasonable
well,
alright!
Well,
that's
all
the
time
we
have
today
for
questions
and
comments
and
Thank
You
audience
for
joining
in
sharing
such
good
questions.
Sharing
your
thoughts.
I
think
this
is
the
whole
point
of
the
coup
de
Ciencias
webinar
is
for
us
to
all
exchange
knowledge.
You
did
amazing
Michael.
We
all
learned
a
lot.
Thank
you.
Everyone
for
joining
today,
the
webinar,
recording
and
slides,
will
be
online
later,
and
we
look
forward
to
seeing
you
at
a
future
ciencia
webinar
goodbye
and
have
a
good
day
or
evening
thanks.