►
Description
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Re-architecting Data Platform with Spark on Kubernetes - SeungYong Oh, Devsisters
Since last year, many data platform projects including Apache Spark began to support the Kubernetes environment. What differentiates the Kubernetes environment and motivates engineers to use it over the existing environments? Should we move to Kubernetes environment while everything works well even now?
The talk will discuss Devsisters' journey of migrating its internal data platform including Spark to Kubernetes. During the process, we found out that migrating to the Kubernetes environment actually involved re-architecting our data platform, which lead to huge benefits including enhancing user experiences and collaboration workflows. The process and outcome of our journey will be shared in detail, along with an overview of the current technology status and its details.
A
A
A
A
A
A
A
Didn't
use
it
well
because
there
were
some
issues
or
stability
that
we
asked
for,,
but
from
2010,
I,
just
told
you
briefly
before
that
I
started
using
Cube
Jit
for
the
development
environment,
game
infrastructure
in
Coop,
Language,
Cham,
Break.,
From,
the
beginning,.
Our
company
originally
made
a
game
called
Cookie
Run,
and
then
published
it.
Oh,
then,
from
the
point
of
view
of
making
this
not
just
one,,
but
let
everyone
use.
It,
I
wrote
this
in
the
form
of
a
public
app.
A
A
So,,
if
game
developers
take
a
picture
like
this
in
the
form
of
the
front
line
yesterday,,
it
is
a
system
that
automatically
collects
everything.
So,
based
on
these
tools,
for
example,
server
developers
need
to
debug.
We
are
also
providing
things
like
a
search
engine
for
server
logs
or
cs
managers
to
check
what
logs
of
users.
A
A
A
A
Have
any
real
questions,?
You
can
even
look
at
the
pottery
heart,
and
opn
people,
marketers,
and
shoulder
game
designers
want
to
do
a
balance
patch,
but
I'm
curious
about
what
actions
users
have,
and
on
what
stage
users
are
doing
a
lot.
It's
good
that
they
can
analyze
it
by
themselves,
or
it
creates
an
environment
where
our
data
scientists
can
analyze
it,
or
so,
simply,.
We
use
kpi
indicators,
for
example,
tail.
Active
users.
We
provide
these
things,,
but
we
also.
A
A
A
A
There
are
also
Meimei
companies
that
deploy
s3
in
batches
like
this,
and
the
analysis
infrastructure
is
configured
based
on
these
things.
What
I
would
like
you
to
look
at
here
as
a
reference
is
that
we
do
not
operate
a
separate
hardware,
hdfs
or
data
warehouse.
Everything
is
data
stored
in
Amazon,
s3
I'm,
taking
the
league
structure.
A
A
Had
an
idea,
but
I
looked
at
it
for
a
while
from
the
middle
of
this
year,
and
after
some
time,
I
saw
that
a
lot
of
projects
support
diss
in
this
section.
Various
projects
related
to
analytics,,
such
as
big
data,,
have
been
supported
in
many
places,,
especially
in
2019
and
last
year..
I
think
it's
starting
with
,,
but
it
seems
that
the
meaning
of
soft
can
be
very
diverse..
Well,
simply,
well,.
There
is
a
chart.
A
A
Will
be
the
first
Cade
with
STA,
and
the
second
one
will
come
out
on
the
first
floor,
eat
as
an
operator
pattern
and
provide
biosof,
data,
and
iCloud
native
to
be
sent.
I.
Think
it's
good,,
but
it's
good.
That
number
9
supports
kissing,,
but
this
doesn't
necessarily
mean
that
we
have
to
move
to
Cuba.
It.
A
A
A
Many
cases
where
it
is
difficult
to
take
it
without
downtime.
On,
the
other
hand,.
There
are
some
of
these
parts,,
but
there
are
some
costs
like
this,
and
when
it
comes
to
profit
when
moving
like
that,,
the
first
PIM
packing
is
called
team.
Packing,
and
now
various
services
are
provided
in
one
node.,
I'm
sure
you'll
see
that
it's
something
that
makes
the
nodes
to
attract
the
maximum
to
27
now,,
but
in
fact,
there's
a
big
server,,
but
this
sucks,
and
if
you're
like
me,,
you're,
writing
everything,.
So
of
course
it's
in
a
frame,
environment.
A
A
A
A
A
He
didn't
work,,
and
if
there
was
a
problem,,
it
was
floating
in
such
a
structure
that
everything
could
fly:
away.
But
until
now,
Taeyang,
simply
Keller
Mistress,
Table,
Air,
Flow
If.
You
do
this,.
Everything
will
be
automatically
answered.
Going
a
little
further,.
There
is
now
an
application.
Operator.
If
you
use
these
things,.
You
can
use
Air
Flow
much
more
effectively,,
so
it
was
the
first
time.
I
got
the
feeling
that
this
makes
a
lot
of
things
easier
than
I
thought.
It.
A
A
A
A
Finally,
users,
analyze
the
accumulated
data
through
SPARC.
There
are
three
skills
that
make
it
possible
to
do.
It.
Among
these
three
use
cases,
the
first
and
second
cases
are
actually
batch,,
so
we
control
them
all,.
So
there
is
no
problem,,
but
the
third
case
is
a
bit
of
a
problem,.
So
in
the
third
case
we
has
been
using
Spock
since
2015,,
and
many
of
these
people
want
to
have
a
strict,
listen
to
use.
Spock,.
A
A
A
A
A
A
A
A
A
It.
I
have
a
Tissot
plate
in
Acube..
This
is
something
similar
to
the
standalone
mode,
I
just
mentioned,
through
the
operator
pattern,.
If
you
decide
what
to
do
and
upload
it,
run
it
and
do
it
like
this.
The.
Second
one
is
to
destroy
the
apartment.
Currently,.
It
is
basically
a
stand-alone
connection,
and
I
have
not
done
anything
like
this.
After.
Supporting
these
three
schedules
for
a
while,,
a
native
Cuban
scheduler
was
added..
So
far,
I
have
abandoned
the
cpm
enter
field.
A
There
will
be.
Well,.
It
could
be
various
programs.
Here,
Spock,.
There
is
a
standalone
master
just
before,
and
when
using
Hoover
Netis,
the
master
becomes
the
scoop
it
3
2k
server.
It
makes
a
request
to
the
ak
server
and
requests
that
the
towers
be
launched.
Then,.
If
you
look
at
the
necessary
parts,,
you
can.
A
A
A
A
You
turn
it
on,.
There
are
also
ebs
in
the
case
of
us,.
So
then
the
data
continues
to
remain,.
So
if
you
work
on
something
for
a
while,,
take
it
down
for
a
while,
and
then
open
it
again,
the
file
will
be
alive.
Again.
We
should
be
able
to
weave
it.
Together.
After
the
sun
went
down,.
We
were
directly
exposed.
We'll,
show
you
some
demos
when
the
time
comes.
You.
A
A
A
A
A
A
A
A
A
A
A
A
Have
to
do
it
at
the
code
level
and
do
custom
ones,,
so
there
is
a
bit
of
complexity
in
those
parts.
And,
not
only
the
soup,,
but
all
the
other
projects
are
in
the
alpha
stage,
Make
something
to
suit
our
team's
needs.
When
I,
say
I
want
to
write,.
There
are
so
many
songs.
Yes,.
There
are
many
cases
where
code
level
correction
is
required,
and
I
would
like
to
say
that
there
is
not
a
little
overhead,
such
as
shocking
or
upstream
dizziness..
A
A
Nevertheless,,
our
thoughts
For
us,,
at
least
in
our
use
case,.
Why
was
that?
In
fact,
I
did
this
design,,
but
this
actual
work
was
done
by
our
data
engineer
who
joined
in
September.
Even,
though
I
had
no
knowledge
about
it,
and
it
wasn't
a
component
that
had
dealt
with
wet,
cougar,,
netis,,
mochi,
and
Peter
Chu
1st-ness,
so
much,
these
customizations,
these
tasks,
and
these
things
were
finished
in
a
little
over
a
month.
A
A
Possible
to
configure
this
environment
so
that
we
can
focus
more.
So,
in
our
opinion,.
There
is
a
bit
of
a
risk
right
now
in
moving
to
the
9th
action,,
and
it
is
true
that
there
is
actually
a
risk,,
but
in
the
end,,
isn't
it
a
better
direction?
I
have
such
an
idea
that
I
want
to
finish
my
presentation.
I.
Don't
know
if
I'll
have
time
in
the
end.
I
think
the
section
is
over
now,.
So
when
it's
over
later,,
if
you
find
me,,
I
'll
answer
you
again.
Thank,
you.