►
From YouTube: Operator Framework to Manage Production Grade Stateful Workloads - Xin Xu & Xinglang Wang
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Operator Framework to Manage Production Grade Stateful Workloads in eBay - Xin Xu & Xinglang Wang, eBay
Containerize big data applications in production environment is a big challenge since these applications require human operational knowledge to correctly operate while protecting against data loss. In eBay we inspire from the idea of CoreOs Operator pattern, and build a framework to make it easier and agile to manage the multi-component, GEO distributed stateful workloads on Kubernetes. With this Operator Framework, users could get any applications they required in just one yaml file, databases or message queue or streaming processor. The framework leverage our self-designed workflow engine to deploy the applications across multiple Kubernetes clusters and manage the applications with in-depend components automatically. The management functions like self-healing, scaling, upgrading, configuring are provided to achieve production grade reliability and high availability.
To learn more click here: https://sched.co/FuJh
A
B
A
A
A
A
B
B
B
Redo
it
for
each
application,
the
scenarios
needs
to
be
considered
and
needs
to
develop
some
logic
to
process
it.
But
if
you
have
a
framework
that
covers
all
the
scenarios,
the
service
provider
only
needs
to
focus
on
their
own
logic
of
the
applications.
Then
it
will
become
a
lot
easier.
So
for
DevOps
common
scenarios,
we
developed
our
own
workflow
engine,
so
we
process
and
treat
these
problems
at
engine
level
like
a
proven
or
rolling,
upgrade
or
the
great
really
great
release,
etc.
B
B
There's
an
example:
here:
it's
a
Kafka
among
go,
build,
you've
got
to
establish
a
lot
of
kubernetes
resource,
but
for
Kafka
it
can
automatically
transfer
to
cluster
for,
if
got
to
goes
through
a
lot
of
complex
operation
like
replica
set
or
it
need
to
establish
the
system,
users,
etc,
etc.
So,
for
framework,
we
only
need
to
provide
a
common
Cabana
tech
resource
and
just
need
to
test
if
there's
a
common
flow
that's
created
and
for
among
go
on
top
of
the
common
flow,
you
just
need
to
add
up
your
custom
custom
logic.
B
Another
case
is
the
configuration
management
ODF
patient.
Is
this
so
provide
a
mechanism
on
framework
level?
We
have
done
version
control
for
the
configuration,
so
the
users
can
define
different
versions
of
config
and
they
can
also
pick
a
certain
version
of
configuration.
We
will
read
the
content
of
the
configuration
and
can.
A
B
Put
that
into
the
other
parts,
when
you
need
to
modify
this
configuration,
you
just
need
to
modify
the
version.
We
just
will
read
the
new
configuration
and
refresh
the
config
map.
All
the
pilot
files
will
be
automatically
updated
within
certain
time
frame.
If
you
can't
imagine
a
McClay
add
the
new
applications,
then
they
can
be
managed
dynamically.
If
not
in
fact,
you've
got
to
custom
your
dynamic
flow.
So
on
top
of
those
operation,
you
just
need
to
add
some
magic
to
test
if
the
logic
is
updated
and
just
need
to
add
the
rolling
reset
flow.
A
B
The
framework
level
will
provide
a
lot
of
blocks
for
you.
All
you
need
to
do
is
to
build
them
up.
If
you
don't
feel
this
enough
blocks
for
you
also,
you
can
also
custom
customize
those
blocks,
so
that
is
the
lifecycle.
Management
functions
really
provides
to
increase
the
reliability.
First
of
all,
we
have
our
self
check
mechanism
in
the
system,
so
for
all
the
applications
in
the
framework,
including
the
framework
yourself,
we'll
provide
a
health
check
function,
there
will
be
a
regular
execution
for
the
house
check
phloem
for
some
basic
in
checks.
B
A
B
A
A
B
B
This
applications
on
its
operations
there's
a
UI,
so
it
makes
a
lot
easier
for
you
to
check
the
log
of
the
application,
so
it
makes
it
easy
for
you
to
debug
and
also
it
integrates
all
the
metrics
at
the
system,
level
or
application
level,
and
with
a
third
party
systems
like
pre
prometheus,
it
can
integrate
with
it.
It
can
also
custom
some
logics
to
test
to
check
your
applications,
whether
there
are
no
more
nodes,
and
these
metrics
can
also
be
integrated
and
integrate
with
the
third-party
monitoring
system.
A
A
A
A
B
B
B
Some
applications
has
a
sacred
information
in
it.
For
example,
the
database
have
their
own
password.
You've
got
to
regularly
update
it,
so
we
have
a
function
it.
It
will
monitor
the
secret
files
in
kubernetes
by
our
custom
aspects.
When
it
check
some
changes,
it
can
automatically
update
the
password
of
the
database.
A
B
A
B
Users
provide
resources
to
us,
they
provide
us
their
namespace,
so
the
users
can
fantasize,
the
namespace
is
open
to
who
they
grant
the
permission
to
whom
and
we
only
allow
authorized
users
to
operate
in
their
own
namespace.
They
are
only
allowed
to
operate
the
applications
that
were
whose
author
and
authorization
is
opened
and
also
we'll
make
sure
all
the
applications
from
our
framework
are
start
from
the
root
users
and
stop
used
as
sasl
encryption.
A
B
Also
did
some
integration
to
make
sure
some
dangerous
operations
will
not
be
executed
in
some
risky
time
frame.
So
we
actually
use
the
design
concepts
of
operator
and
developed
our
own
system
to
make
it
easy
for
the
data
application
providers
to
manage
their
framework.
Comparing
our
own
solution
and
kubernetes
own
state
set
and
design
concept,
we
provide
more.
B
Because,
in
any
lifecycle
management,
the
users
can
customize
their
own
logic,
so
instead
of
equipment
Attucks,
they
can
only
do
something
before
or
after
the
part
is
created.
So,
comparing
with
the
kubernetes
design
concept,
we
actually
add
higher
level
abstraction.
We
provide
a
workflow
interface
to
make
it
easy
for
the
users
to
define
their
own
logics.
A
A
B
B
Has
its
own
operator
is
fast,
but
there
are
a
lot
of
applications,
but
if
you
do
an
operator
for
each
application,
you
might
not
have
enough
resource
or
power.
If
you
put
the
same
logics
together,
there
might
be
a
lot
of
logics
because
each
application
has
different
logics
like
a
Prometheus
or
a
TC
D.
They
have
very
different
logics.
B
A
B
A
B
A
B
File
what
you
do
is
you
have
your
own
workflow
for
every
file
and
you
can
the
things
you
want
from
a
common
workflow
and
common
work
tasks
to
build
your
own
flow.
So
as
a
user
I
can
customize
my
own
workflow
and
pick
those
from
the
common
workflow.
Yes,
because
you're
not
the
end
user,
for
example,
you're,
the
cascade
provider
or
etcd
provider.
You
will
realize
that
you've
got
to
establish
or
create
all
these
resources,
but
the
resource
creation
process
is
available
in
the
framework
you
just
need
to
take
and
use
it.
B
B
Of
course,
if
you
check
the
code,
you
know
how
it's
written,
but
it
didn't
say
that
when
all
the
operator
by
the
controllers,
how
they
need
to
process
the
logic
in
the
controllers,
you
don't
have
to
see
that
you
just
need
to
use
the
service,
but
you're
talking
about
the
end-users
but
I'm
talking
about
who
is
going
to
advise
the
etcd
operator.
There
there's
got
to
be
someone
to
write
it.
For
example,
if
there
is
no
operator
for
the
production
there
will
be
someone
to
write
it
up.
A
B
B
B
The
data
team
to
provide
service
in
such
a
platform
I
see
you
extract
a
lot
of
infrastructure
logic
in
a
framework,
but
in
a
framework
I
didn't
see
the
management
of
data
for
data
team
data
is
important
to
them.
So
it's
quite
troublesome
to
manage
the
data,
for
example,
managing
their
own
data
or
external
data
or
labeled
data.
So
do
you
have
an
experience.