►
Description
Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Lesson Learned on Running Hadoop on Kubernetes - Chen Qiang, LinkedIn
LinkedIn operates one of the world’s largest Hadoop environments, with ~450PB used data, 2 billion files/blocks, and over 400K jobs/day. However, testing cluster features in an isolated fashion has been traditionally fairly difficult. Infra teams such as HDFS, YARN, and Azkaban often step on top of one another for testing new features in our existing test Hadoop clusters. Setting up a new test cluster requires coordination between hardware, infra, and security teams, usually taking weeks to months. We have recently extended Kubernetes’ usage to test Hadoop(HDFS/YARN) clusters, by deploying production-like Hadoop cluster on Kubernetes. This has reduced infra setup time from weeks down to minutes with no network, hardware dependencies, and enables critical infra/workflow teams to test new features on the fly.
https://sched.co/ZeoG
A
Welcome
everyone.
Thank
you
for
joining
lesson
learned
on
running
hadoop
on
kubernetes
session.
My
name
is
chen
chang,
I'm
part
of
data
sre
team
at
linkedin.
Our
team
manages
entire
offline
data
infrastructure,
including
hadoop
spark
presto
and
many
others
kubernetes
and
linkedin
started
about
a
year
ago.
Our
team
was
one
of
the
first
users
and
our
main
use
case
is
to
launch
ephemeral,
hadoop
cluster
for
development
and
testing
purposes.
A
In
this
session,
I
will
cover
hadoop
basics
for
audiences,
who
are
not
familiar
with
hadoop
and
are
briefly
touching
on
hadoop
footprint
at
linkedin.
I
will
talk
about
the
initiatives
for
this
project
architecture
overview
and
challenges
walkthrough.
In
addition,
I
would
like
to
share
the
insights
of
what
currently
we
are
working
on
and
a
future
role
map.
At
the
end,
it
will
be
a
10
minute
video,
including
spinning
up
a
secure,
hadoop
cluster
on
kubernetes,
doing
some
file
operation
running
a
hadoop
job,
as
well
as
in
place
expansion.
A
Let's
start,
what
is
hadoop
hadoop
is
a
collection
of
open
source
software
designed
to
handle
big
data,
storage
and
computation.
The
main
components
of
hadoop
are
hadoop
distributed
file
system
hdfs,
which
is
a
parsex-like
file
system
in
a
design
for
distributed
computing
framework,
and
it
is
highly
reliable
and
resilient.
A
Hdfs
has
three
types
of
components:
a
h,
a
pair
of
name
nodes,
which
holds
metadata
of
the
entire
file
system.
Quorum
of
general
nodes,
saves
the
incremental
metadata
for
later
compaction
and
many
data
nodes
that
store
the
actual
data
above
hdfs
there's
a
yarn.
It
manages
all
distributed.
Compute
resources,
including
cpu
memory
and
gpu
across
many
worker
nodes.
A
Yarn,
has
four
type
of
components:
resources,
manager,
manages
hardware,
resources,
schedules,
workflows
and
monitor
jobs,
history,
server,
archives,
completed
jobs
and
a
proxy
server
redirect
user's
request
to
either
resources
manager
or
history.
Server
depends
on
the
job
state
last
is
the
node
manager.
That
is
the
one
actually
runs
the
task
on
top
of
those.
There
are
many
distributed
computing
framework,
such
as
map
reduce
spark,
presto
and
many
others.
Last
but
not
the
least,
is
the
client
such
as
gateway
for
cli
and
many
other
open
source
schedulers
such
as
azkaban,
apache,
uzi
and
airflow.
A
A
Our
largest
hadoop
cluster
is
over
7000
physical
servers
with
storage
capacity
of
400,
petabytes,
half
million
cpu
v
cores
and
a
1.6
petabyte
of
memory,
and
we
are
constantly
pushing
the
limit
by
adding
more
nodes.
We
have
multiple
super
large
clusters
with
over
4
000
nodes.
Each
we
serve
thousands
of
users
across
entire
company.
A
A
So,
as
I
mentioned,
linkedin
has
a
large
in-house
hadoop
development
and
sre
teams
that
are
constantly
working
with
open
source
community
contributing
features
and
bug
fixes
having
hadoop
ephemeral
cluster
running
on
kubernetes
allow
every
engineer
to
have
own
cluster
for
testing
and
development.
In
addition,
the
cluster
is
with
running
with
immutable
container
images,
so
the
issue
is
always
reproducible.
A
A
It
builds
on
top
of
kubernetes
overlay
network.
Each
component
requires
a
kubernetes
service
to
have
a
static
and
a
dns
resolvable
hostname
for
communication
between
parts
as
well
as
service
discovery
for
every
admin
notes,
such
as
general
node,
name,
node,
resources
manager.
There
is
a
service
associated
with
that
with
one
two
one
mapping
for
worker
nodes,
which
is
a
data
node
and
node
manager.
It
is
n21
mapping.
All
data,
node
or
node
managers
shares
the
same
service,
so
the
graph
here
shows
all
the
communication
paths
in
between
and
all
the
connections
are
through
service.
Endpoint.
A
Let's
move
to
the
challenges
over
the
course
of
the
project,
we
encountered
many
challenges
and
I
would
like
to
share
some
highlights
with
everyone
here.
First
one
is
dns.
It
took
us
days
to
find
out
a
good
solution
without
compromising
security.
Next
is
network.
By
default.
There
is
a
no
fixed
or
predetermined
hostname
in
parts
or
deployments
over
with
overlay
network
hadoop
admin
nodes
requires
those
fixed
hostname
to
be
listed
in
the
hadoop
configuration
before
before
the
deployment.
A
Third
is
the
identity
in
secure
hadoop
every
part
requires
a
unique
kerberos
credential
for
authentication
due
to
the
unknown
of
ip
addresses
or
host
name
at
launch
time.
We
are
not
able
to
pre-generate
those
last,
but
not
the
least,
is
the
orchestration,
because
there
are
strong
dependencies
between
hadoop
components,
for
example,
hdfs
name
node
must
start.
After
all,
the
general
nodes
are
up
and
running.
A
First,
one
is
a
dns
hadoop
worker
nodes,
which
is
a
yarn
node
manager
or
hdfs
data
node
talks
with
each
other
using
hostnames
by
default.
There
is
a
no
global
resolvable
host
name
associated
with
it.
Also
it
is
impossible
to
have
one
service
per
work
part
because
it
won't
scale
natively
with
kubernetes.
A
Next
is
the
webhdfs,
which
is
a
rest
api
endpoint
for
hdfs,
when
name
node
sends
the
redirect
request
back
to
the
client.
It
contains
a
hostname
of
designated
data
node
that
going
to
be
serving
the
actual
request
and
the
last
is
for
map
reduce
or
spark
jobs.
Application
master
and
spark
driver
are
sharing
their
host
names
with
the
workers.
A
A
We
found.
The
hostname
resolution
in
the
reverse.
Lookup
was
inconsistent,
as
you
can
see
my
example
here.
So
when
I
do
the
hostname
dash
f,
it
come
up
with
the
hostname
with
part
name.
However,
when
I
do
a
dns
resolution
on
that
it
shows
up
as
a
known
record,
then
I
do
a
reverse
lookup
of
the
pod
ip
addresses.
I
get
a
actual
dns
resolvable
hostname.
A
So
the
solution
here
is
we
injecting
the
actual
dns
resolvable
hostname
to
the
pods
etc
hosts
file
at
startup
time.
Then
it
resolve
all
the
dns
issues.
We
have
to
running
actual
workload
on
hadoop.
As
you
can
see,
I
get
the
another
example
here,
underneath
it
is
after
we
injecting
the
actual
resolvable
hostname
into
etc
hosts
file.
Then
all
the
forward
and
reverse
lookup
are
matching.
A
Next
one
is
the
network.
There
are
many
components
in
hadoop
and
they
are
communicating
with
each
other.
While
the
cluster
is
up
and
running
like,
for
example,
data
nodes
initiate
the
connection
to
the
name,
node
node
manager
initiates
the
talk
to
the
resources
manager
and
there
are
many
intercommunication
between
components
as
well
and
hadoop
configuration
must
be
pre-populated
with
those
host
names
of
with
those
hostname
of
admin
notes
in
place.
Before
we
start
the
service.
The
solution
is
to
creating
a
kubernetes
service
for
every
hadoop
admin
parts.
A
It
provides
a
dns,
resolvable
and
and
structured
host
name,
which
can
be
defined
and
populated
in
hadoop
configuration
before
the
actual
deployment.
So
I
include
a
snippet
of
configuration
xml
for
hadoop
name
node.
So,
as
you
can
see
here,
as
we
have
name
node,
ha1
and
ha2,
and
each
of
them
using
the
service
endpoint.
A
It
only
applies
to
the
secure
hadoop
cluster
with
kerberos
enabled
every
hadoop
component
requires
a
key
tab
file
with
a
service
principle,
for
example,
hdfs
slash,
dns,
resolvable
host
name,
hr,
kerberos
rom.
The
host
name
has
ip
addresses
embedded
and
it
is
impossible
to
pre-generate
them
and
pushing
in
the
pod.
Our
solution
is
to
introduce
a
key
tab
delivery
service
running
in
kubernetes.
A
It
authenticates
the
request
from
hadoop
parts
generating
a
qa,
only
key
tab
and
and
ascend
back
to
the
client
in
hadoop
pod.
We
have
an
init
container
to
reconstruct
the
host
name
by
replacing
the
underscore
ip
with
the
actual
dns
resolvable
hostname
and
send
the
request
to
the
keytab
delivery
service.
A
Unfortunately,
we
don't
have
much
time
to
go
through
the
authentication
workflow,
but
luckily
it
uses
the
same
authenticate
authentication
mechanism
at
linkedin,
recently
open
source
project
coupe
to
hadoop,
and
now
it
is
available
on
github.
Please
feel
free
reaching
out
to
me.
If
you
would
like
to
know
more.
A
Orchestration
last
is
the
orchestration,
as
I
mentioned.
Hadoop
components
have
have
hard
dependencies
in
between,
for
example,
name
node
must
star
after
general,
nodes
are
up.
The
obvious
solution
is
to
launching
general
node
part.
First,
then
name
node
pause.
Next
such
approach
usually
results
long
deployment
time
and
a
complicated
external
logic
to
orchestrate
deployment
sequence.
What
we
come
up
with
is
introducing
a
in
a
container
to
constantly
polling
status
of
depending
service
using
service
discovery
with
part
env
in
a
container.
What
exists
successfully
when
the
depending
service
is
up.
A
A
I
include
a
example
of
in
a
container
here
without
holding
the
status
of
hdfs
journal
node,
so
that
is
the
end
of
our
challenges
and
what
is
the
next?
So
it
is
a
great
lesson
we
learned
to
having
hadoop
running
on
kubernetes
as
testing
and
sandbox
environment.
We
are
almost
done
to
have
other
big
data
components
such
as
spark
presto
azkaban
as
part
of
our
ephemeral
hadoop
cluster.
A
In
meanwhile,
we
are
also
working
on
long-running
hadoop
cluster
on
kubernetes
to
replace
integration
environment
that
is
currently
running
on
bare
metals
and
the
number
of
issues
we've
encountered
are
not
less,
but
we
do
see
20
times
faster
deployment
speed.
Our
next
major
goal
will
be
kubernetes,
custom,
custom
resources,
definition
and
hadoop
operator.
A
The
key
takeaway
for
this
presentation
will
be:
it
opens
the
opportunity
to
combine
different
workload
onto
the
same
resources
management
platform.
I
know
a
lot
of
company
has
separate
infrastructure
serving
online
and
offline
and
the
online
infrastructure,
usually
idle
or
less
load
during
off-peak
hours
leveraging.
Those
idling
resources
will
easily
save
company
millions
of
hardware
costs.
A
Second,
it
may
change
the
way
how
the
big
data
infrastructure
are
running.
A
lot
of
hadoop
native
distributed
framework
are
now
running
natively
on
kubernetes,
for
example,
spark
and
a
linkedin
open
source
samsa
and
the
last
and
make
hadoop
to
kubernetes
transition
much
easier,
which
kind
of
resonant
the
very
first
point.
A
A
This
is
a
wrapper
and
it
does
three
things.
First,
it
uses
python
ginger
to
templating
engine
to
render
many
yaml
files,
including
pod
service,
stay
for
set
definitions
and
config
maps.
Second,
it
does
yaml
file.
Validation
by
yaml,
linked
and
last
is
to
apply
those
yaml
files
onto
kubernetes
with
a
specific
order.
It
starts
with
service,
then
config
map
and
the
last
part
state4
set.
Each
part
has
at
least
one
in
a
container
to
fetch
kerberos
key
tab,
and
some
parts
have
one
additional
in
a
container
for
dependency
check.
A
A
A
As
I
mentioned,
we
have
two
name
nodes
here
for
high
availability.
One
is
active
and
the
other
is
standby.
We
are
able
to
fail
over
between
them
and
all
the
client
will
automatically
connect
to
the
active
name
node
for
hdfs
transaction.
During
our
startup
process,
we
have
fell
over
testing
to
making
sure
that
it
works.
A
A
A
A
A
A
Usually
this
job
should
take
no
more
than
30
seconds.
As
you
can
see,
we
have
successfully
submitting
the
job
to
our
resources
manager.
Let's
go
the
resource
manager,
page
and
refresh
here,
as
you
can
see
in
my
job.
Here's
my
job
id
my
username
job
name,
start
time
launch
time
and
the
state
is
running
and
the
stdl
from
the
job
submission
is
constantly
getting
is
constantly
getting
updated
with
the
status,
and
here
you
go
the
job
finish
just
under
27.5
seconds.
A
Let's
refresh
the
page
again,
as
you
can
see,
our
job
now
is
in
finished
state.
Let's
go
to
the
job
history
server
so
because
this
job
is
a
completed
job.
That's
why
it's
showing
up
in
our
job
history
server
in
job
history
server.
I
can
dig
into
more
details
with
the
job
like,
for
example,
I
can
see
the
log
for
each
running
task
right
here
that
can
go
logs.