►
From YouTube: Breakout Session: Managing 250+ ES & Kafka Clusters using Operator Frameworks - Galuh & Daniel
Description
Gojek, a decacorn with 100 million users in Southeast Asia, used to have developer teams that managed their own Elasticsearch logging clusters. However, they have different knowledge and time for it, resulting in different QoS of each cluster. To solve this issue, the infrastructure team started to manage those clusters by provisioning Elasticsearch and Kafka clusters on LXC containers on behalf of the developer teams.
The initial architecture worked fine for 30+ clusters, but significant growth demanded the team to manage 250+ clusters. Elasticsearch and Kafka require special care to administer, e.g. Elasticsearch cluster status should be green before turning off another Elasticsearch node. This complexity slows down the team to do maintenance operations, e.g. the team needs weeks to upgrade these clusters. This talk presents how Operator Frameworks reduces our daily toil.
A
A
B
A
A
A
A
The
data
that
has
been
sent
by
the
user
and
the
service
consists
of
a
producer
who
is
in
charge
of
the
freshly
injured
otok
as
gunpowder
to
retrieve
data
from
older
siblings,
making
it
unclear
whether
it
is
successful
and
sends
the
data
as
accessories
used
by
the
program
to
see
their
faces,
and,
lastly,
is
the
console
that
binds
a
the
comment
will
end.
They
can
communicate
with
each
other.
A
A
Now.
Users
can
use
Market
news
to
set
the
rap
they
have.
The
market
here
functions
to
change,
new
textures
and
also
regulates
Who
can
have
access
to
their
groups,
and
the
router
serves
as
a
reverse
proxy
for
chibana
chibana
in
evdo
Mito
recently
joined
is
an
example.
Display
of
excited.
Can
catfish
story
can
see
their
group
and
also
say
new,
all
the
components
we
use.
It
runs
in
a
container
that
there
is
suwage
m
gang
and
Steffy
m
can
consist
of
one
container
running
at
the
same
time.
A
The
system
we
use
there
is
container
selection
or
myth
We
use,
ready
and
terraform
or
semaphi
handler,
which
has
Well,
for
example.
If
a
service
issues
a
lock,
felowork,
then
antifrogen
in,
will
take
it
and
send
it
to
routers
you
ask
which
group
do
you
want
to
go
to
that
ocean
egg
will
add
contacts
suitable
for
use
by
worker
news
Well
after
the
router
knows
FB
at
most
i7,
you
press
send
that
look
to
that
fact.
A
Via
producer
Hi,
hello
consumer
will
take
time,
Princess
cancer
and
throw
away
the
outsourcing
that
is
no
longer
needed,,
then
send
it,
te
lasser,
save
dammers
in
the
form
of
oil
of
the
Shari'a,,
which
means
that
the
time
entered
today
will
be
stored
in
the
index
according
to
locks
that
are
old
or
exceed
the
retention
time
limit,.
We
have
a
ureter
that
says
once
it
will
delete
the
data..
A
B
C
B
C
B
New
episodes
keep
on
reporting
the
film,
so
we
have
to
make
another
So
next
is
a
manual
of
archaeological.
So
we
need
to
see
that
in
this
architecture
at
1pm
it
has
a
toward,
and
in
the
afternoon
it
is
shared
by
the
container
content
that
we
were
for
Therefore
to
move
the
container
from
one
prepare
anything
else,.
It
's,
something
that
isn't
physical,.
There
has
to
be
a
transfer
of
data,.
B
We
use
a
tool
called
selection,
move
to
do
that,,
so
the
face
reaction
will
do
the
content
stay
K2
in
frame
1,,
then
the
data
will
be
moved,
then
after
it
arrives,
erect
move
will
blame
the
container
on
pm2.
Well,
before
blaming
the
campaign,
we
will
do
a
thousand
pieces
to
respond
to
the
gurkha
regarding
the
configuration
configuration
changed,
for
example,
Hajj
rice
changed
and
etc.
B
B
Next
is
the
location
of
Elastic
cells
So
before
we
relocate
Kafka.
Sometimes
we
also
need
to
relocate
Allah
vikset,
and
this
is
quite
often
done
for
our
own
Elastic.
We
have
already
configured
our
default
l
so
that
if
one
of
the
clubs
dies
it's
okay,
black
magic,
the
data
has
been
published
in
our
datalock.
It
is
alive
given,
among
others,
2
different
system,
novels.
B
Therefore,.
Before
we
turn
off
elasticsearch,,
we
have
to
turn
off
asset
allocation
first,
so
that
there
is
no
unnecessary
data.
Transfer,
then,
for
example,
we
leave
S2
S2,.
We
turn
it
off,
after
we
turn
it
off,
the
cluster
will
turn
yellow,.
Then
we
can
turn
on
the
new
elasticity,
which
it's
still
empty
around
the
outside,.
So
Elastic,
include
automotive,
moving
the
data
on
the
date
of
the
GALAXY
Tab2
harness
so
that
the
Cluster
will
be
able
to
run
Back
to
normal
after
the
Cluster
becomes
a
gossip
status.
B
B
B
just
now
we
love
it,
takes
40,000
for
all
classes
to
happen.
It
does
take
a
long
time
now,
because
we
have
gold
upgrading
to
our
subchapter.
The
times
are
so
complex
and
also
we
need
to
carry
out
operational
works
because
there
is
a
change
in
the
cement
visor.
Well
at
first,
it's
okay
for
50
factors
only
for
230
cassettes,
it's
already
quite
overweight,,
so
we're.
A
Before
we
start
directly
transferring
data
from
accessual,
so
everything
is
new.
Of
course
we
have
to
try
first,
what
we
are
going
to
use
here.
Experimental
results,
We
are
cutting
the
installation
of
the
ishige
operator.
We
also
blame
the
Insto
price
below.
It
can
be
seen
that
the
operator
immediately
runs
normally
Hi
includes
after
the
share
reaction,.
We
just
need
to
run.
Keep
offline
with
the
following
server
on
the
road.
If
30
is
always
formed,
for
Iban,,
submit
the
same
as
the
credentials
record,.
It
will
be
stored
in
zip,
Secret.
A
Will
do
an
update
then
say
from
exit
Doors
6.
The
seventh
is
running
smoothly,
except
there
is
one
note
that
cannot
be
an
eligible
market
until
it
has
to
work
manually,
and
also
there
is
a
minimum
astronaut
configuration
which
is
actually
not
used
at
LSI
agree
that
there
is
still
left
or
adding
this
no
problem.
In
fact,
there
is
no
pot
restore
at
all.
What
needs
to
be
noted
is
that
when
this
lechter
object
is
deleted,
then
all
PV
that
is
claimed
will
also
be
followed.
A
Hi
for
cartaxi
operator
installation
the
same
as
situ
operator
for
installation
to
make
colleagues
the
same
as
fun
deeds.
Cleaning
up
our
comments,
making
This
cluster
ship
is
so
deep
that
the
sustenance
will
be
made
first
before
the
Kafka,
Cluster
and
then
type
We
are
trying
to
make
a
Cluster
in
the
ristio
class.
A
A
B
A
B
Well,
for
migration,
we
have
a
concept,,
namely
double
typing,,
so
we
asked
for
the
double
sliding
concept
at
the
store
to
send
a
lock
not
only
to
Flex-lok,
Galaxy
and
but
also
the
power
of
the
lock
is
heavy,.
So
the
plus
is
istituto
BKT.
Why
can't
we
think
back
but
Resort
Seda
Noah?
So
we
need
to
run
the
reaction
class
from
churned
to.
B
Al-Shifa
liters
of
hazelnut
oil
For
example.
On
the
first
day
we
will
start
the
process.
There
is
a
website
we
can
see.
The
logs
have
started
to
go
to
Qibla,
Maps
and
the
next
day
we
can
see
that
for
the
index
at
the
end
of
the
preference,
it
is
quite
similar
because
it
starts
at
almost
the
same
time.
B
Now
for
migration
itself.
It
is
divided
into
two,,
namely
double-tap,
which
is
not
passive
or
double
post
practical
and
according
to
the
name
of
our
migration
fascism,
you
don't
need
to
do
anything,.
What
we
need
to
do
is
just
start
double
track,.
Then
we
wait
for
as
many
levels
in
this,
now
again,
wait,.
We
can
see
that
each
galaxy
and
cybernet
cluster
is
almost
the
same,
right?.
B
B
Therefore,
we
have
another
way,,
namely
active
migration.,
Now,
active
migration.,
that's
to
handle
this,.
So,
for
example,
we
start
double
sharp
as
an
example
of
a
sentence
and
we
start
doing
Dubstep
the
next
day.
We
can
see
that
the
index
on
the
5th
is
similar
in
reactions
and
decu
berners,
and
we
can
do
the
process
of
snapshoot
and
restore
lovers
next
and
type.
B
The
way
is
to
configure
KLX
in
the
file
and
in
OLX
elastic
rthdcpl.exe.
In
order
to
join
Now
for
this
tapered
a-class,
we
only
need
to
send
the
Lock
from
the
conflict
where
it's
located,
Tea
to
OLX,
the
location
will
be
automatically
distributed
in
elasticsearch,
which
is
in
react
and
try.
Dollar
or
here
has
a
photo
of
Sounds
from
that
is
maybe
not
double-click
really
on
shoes,
because
it's
elastic
and
it's
one
but
not
officially-
reported
because
we
need
to
detect
Elastic
starseed
on
the
keyboard.
B
From
Spike
for
a
while
and
also
God,
it's
not
easy
to
throwback,
because
when
we
come
on
when
there
is
a
problem,
it
's
not
easy
for
us
to
just
react
violently,
because
one
of
these
is
for
the
kibana
migration.
Save
the
education
object.
Before
we
could
see.
There
was
a
migration
of
Ahok
index.co
ibana.
Now
before
we
do
that,
it's
better
When,
it
comes
to
asking.
We
turn
it
off.
B
First,
we
locker
the
migration
and
when
the
result
is
run
again,
we
can
see
that
the
belief
will
detect
the
infection
and
will
have
Raffi
format
it
from
the
old
format
to
the
format
that
is
currently
running
continued
for
the
security
settings
in
that
area.
There
is
what
we
call
the
name
Syam
as
an
example,
the
user
Elastic
itself
is
stored
in
a
sample
that
is
stored
in
each
respective
club
But.
We,
it
also
has
a
real
native
that
can
be
internet
users
can
be
configured
via
how,.
B
So
for
that
what
we
can
do
to
open
it,,
the
typical
move,
the
result
is
that
we
can
do
a
sketch
and
restore,.
After
doing
a
restore,,
we
can
log
in
with
the
signal
stadium
that
is,
reacted
in
the
Cibenen
Cluster,
right?
We
have
discussed
the
migration
process
that
we
are
doing..
The
ideals
in
the
book
itself
are
giving
things
that
are
helpful
or
something
else
that
doesn't
work.
A
Uh,
yes,
of
course,
in
respiration,
we
experienced
a
few
problems.
The
first
thing
to
remember
is
that
in
the
selection,
each
container
is
shared.
instead
of
using
a
disc
from
the
film,,
while
in
Q
it's
like
hell,,
they
have
their
own
CVC
here,.
Even
though
we
already
have
it,
and
the
composition
planning,,
it
turns
out
that
we
still
get
the
full
plot
of
this.
After.
We
check
the
story,.
There
are
physical
changes.,
There
are
business
differences
between
directors
and
minors.
DC
around
60h.
Please
send
it
can
reach
about
83.
C
B
Okay,,
what
we
are
trying
ourselves
is
Well,.
We
want
to
try
this
to
do.
The
default.
What
we
also
tried,
is
to
use
the
effects
in
ff
ff
process..
This
means
that
we
have
one
file
as
an
example,
measuring
1
gigabyte,
and
this
time
we
format
it
as,
for
example,
at
that
time
we
tried
it
as
a
BTP
because
of
the
superficial
method.
We
can
see
on
the
right
that
we
can
insert
a
1
Giga
file
into
the
btrfs
partition,
which
only
has
a
capacity
of
28
table
status.
B
B
B
A
Cancer
2.
We
tried
compression
from
the
application
side.
What
we
tried
was
the
lingering
caste,,
including
compression,,
so
we
can
reduce
dcdc
by
about
two
to
three
times,.
Keep
in
mind
that
compression
using
the
file
system
is
about
five
phases,
5-6
times,
well,,
that's
not
bad
than
no
compression
at
all
or
social
copying,.
So
we
decided
to
use
a
lower
class
afternoon
than
the.
A
Isep
dialectics
that
followed
due
to
the
index
migration
failure,
how
about
diesel
Hi,?
There
are
several
clusters
that
belong
to
canvas
which
cannot
be
immigration,
so
because
they
have
to
export
because
they
become
jeson.,
We,
often
repeat
the
migration
for
acoustics,,
then
in
the
new
cluster
we
import
the
decorations
to
canvas
again
next
after
the
migration
process
has
been
going
on
for
quite
a
while
around
100.
It
feels
like
immigration
suddenly
needed
Pasar
Baru.
It
took
quite
a
long
time
each
axisers
phone.
A
Which
wasn't
needed,
but
the
cup
elastic
container
er
still
took
about
twelve
minutes
for
it
to
be
ready
to
run
as
we
went.
Deeper.
We
found
a
script
where
the
functionality
was
still
all
like
the
variables
that
were
in
the
pot
and
each
one
of
these
took
about
six
close
runs
after
we
check.
In
this
case,
it
turns
out
that
there
are
around
Rp.
11,000
invariable,.
So
here
we
order
an
idea
to
try
to
turn
off
the
service
link
feature
from
Q,,
get
that
in
the
new
pot,.
A
The
same
thing
also
happens
to
Kafka,,
but
the
problem
is
that
it
often
doesn't
support
the
indoservice
link
parameter
to
turn
off
the
Slank
Virus,.
So
the
sad
news
is
that
the
creative
woe
to
Mega
is
more
of
a
phenomenon,,
not
that
all
the
scripts
to
run
bairi
kafta
used
by
that
much
have
the
same
volume.
So,.
They
had
the
idea
to
merinem
the
original
script,
then
from
its
old
name.
We
make
a
pain
script
where
each
of
its
functions
is
to
delete
all
the
variable
informants.
We
don't
need
and
then
call
every
original
one.
A
To
us
potential
in
the
creators
regret,
this
issue
actually
comes
from
various
Chelsea
coaches
separately,
which
didn't
include
the
potential
film,
so
we
had
to
delete
it
again.
First
Then,
after
that
we
were
able
to
come
back
again.
The
last
morning
was
that
there
was
a
higher
difference
in
the
contents
of
kedasri
in
processing
changes
to
restore
sustenance.
If
and
changes
were
invalid,
then
all
changes
would
be
immediately
rejected,
Meanwhile
Z's,
wife,
because
it
doesn't
use
netbook,
clean
water.
A
B
Okay,
now
the
reality
is
better
yes,,
so
every
Irishman,
laughs,
death,
explodes,,
skeletal
muscles.
We
just
need
to
make
additional
automotive
tips..
The
next
tip
is
nokes
to
deal
with
it
yourself,,
because
it's
quite
easy.,
Previously,
sticking
to
coffee
became
difficult
because
everyone
has
their
own
time
and
can
be
tested
when
we
want
to
migrate,.
All
we
need
is
through
the
davishare
test,
Saturday
chord,
then
we
test
again
after
kVA,
m2
and
41
can
move
to
like
12,
then
for
elastic
management
is
chosen
because
automatically,
for
example,
we
find
an
upgrade.
B
We
just
have
to
change
the
magic
version
set
to
play,
then
euy
when
there
is
an
invalid
conspiracy.
As
an
example,
We
want
to
change
Elastic,
it
happens
36
only.
The
political
configuration
is
to
change
the
scorestat
to
8giga
to
43rd
WC
Khalif.
Now.
That
will
be
rejected
by
the
web.
Admin,
lipstick,
share.
Yes,,
like
Ice.
B
We
change
the
full
name,.
The
process
that
occurs
is
different.
If.
We
change
the
website
name,.
It
means
that
singles
that
occur
will
occur
on
that
website.
But.
If
we
change
the
full
name,,
it
will
create
new
notes
with
a
new
configuration,
so
we
can
have
a-class
Vixion
The
cluster
with
the
production
process
and
the
existing
ones
will
be
owned
for
sure
from
the
old
nothing
new
exhaust
now,
for
what
alone
is
also
the
same
automatically
for
the
version
update,.
B
Also
the
addition
of
notes
also
automatically
we
can
get
subtitles
play,,
it
continues
to
increase
Hi,,
but
when
there
is
a
configure
pass,,
it
is
invalid
due
to
absence
the
mission
of
patting.
The
activity
will
continue
as
before,
but
when
we
see
the
give
up
together,
subtitles,
we
ask
deworming
and.
B
B
Provisioning,
that
is,
Darti
is
helping
the
market,
sending
Ajo
to
septic
and
research,.
The
sufferer
will
experience
running
the
helmet,
installing
the
kvbs
tablet
and
continuing
the
instrument
before
us
to
upgrade
Kafka
for
all
the
Clusters
it
took
four
weeks,
but
now
to
update
all
the
Clusters
to
be
visible.
It
only
takes
one
click
and
about
1-2
hours
has
also
been
completed
because
it
was
assisted
by
Clash
of
Clash
of
site,
Maps
and
also
other
things
from
Nikita.
Here,
no
questions
until
now,.