►
From YouTube: Accelerating Humanitarian Relief with Kubernetes [I] - Erik Schlegel & Christoph Schittko, Microsoft
Description
Accelerating Humanitarian Relief with Kubernetes [I] - Erik Schlegel & Christoph Schittko, Microsoft
How can UN humanitarian aid field experts use social media to gain insight, understand trends and track key humanitarian issues? Through a collaboration with Microsoft and UN OCHA, Project Fortis was created to accelerate the surveillance around humanitarian disasters and health epidemics around the world.
This talk discusses the architecture of a high-available native spark pipeline running across multiple Kubernetes clusters to support Fortis customers.
About Christoph Schittko
Christoph Schittko is an engineer with Microsoft working with customers on innovative solutions in the areas of containerization and AI. He's been working with Microsoft customers on building cloud solutions since Azure was called "Red Dog". He’s recently been a contributor to kubernetes and the Azure Container Service. Prior to joining Microsoft he was an MVP award winner, built a web conferencing system before WebEx and taught machines to recognize things in images. Christoph lives in Dallas where he enjoys plenty of sunshine, great food and very German things like beer and the best place to watch soccer in the US.
About Erik Schlegel
Erik is an open source engineer at Microsoft, and based in the Austin area. He's one of the original contributors to the React Native Universal Windows Platform (UWP). Erik leads the engineering effort of Project Fortis, an open source data gathering / surveillance insight platform running on Kubernetes and built with React, Spark and other machine learning technologies. Fortis was created through a collaboration with Microsoft and the United Nations, and used as an early warning indicator for humanitarian crisis(s) and health related epidemics.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
user
interaction
in
social
data
tell
us
a
great
deal
about
the
real
world
by
understanding
the
time-dependent
nature
around
the
discussions
in
social
media
and
public
web.
We're
able
to
get
being
able
to
identify
and
even
sometimes
predict
important
real-world
trends.
Today,
Christophe
and
I
Eric
are
here
to
talk
about
project
Fortis,
which
is
spirited
towards
accelerating
humanitarian
relief,
all
powered
by
kubernetes.
A
Just
a
brief
introduction
of
myself,
my
name's
Eric
Schlegel
I'm,
a
principal
engineer
for
Microsoft
and
we're
gonna
team
called
the
commercial
software
engineering
team.
My
passion
in
terms
of
the
projects
I'd
like
to
work
on
is
how
we
can
use
emerging
technologies
to
help
accelerate
Microsoft's,
most
innovative
partners
and.
B
I'm
Christoph
said
Kooiman
engineer
on
Eric's
team,
I'm
kind
of,
like
you
would
think
up
these
the
chronic
early
adopter,
there's
just
something
about
working
with
undocumented
software.
That
attracts
me,
I'm,
not
sure
what
it
is,
but
I
keep
doing
and
I've
been
doing
it
for
many
many
years.
You'll
find
most
of
my
my
current
work.
B
Now
our
Charter
is
about
unlocking
and
accelerating
innovation,
with
also
two
we
get
to
work
with
small
startups.
We
get
to
work,
the
larger
enterprises
we
get
like
the
United
Nations,
and
then
this
project
will
showcase
some
of
that
work.
This
talk
was
but
showcase
some
of
that
work,
and
but
we
don't
just
engage
with
those
customers.
A
This
part
of
the
UN,
is
responsible
for
providing
humanitarian
relief
for
those
that
are
in
vulnerable
need
that
are
a
mist
of
humanitarian
crisis
and
trying
to
provide
that
aid
as
quickly
as
possible,
so
to
focus
on
a
targeted
scenario
that
they
had
that
at
the
time
was
a
post,
Gadhafi
situation,
the
refugee
displacement
situation
in
Libya
and
their
main,
their
main
focus
at
that
point,
was
trying
to
get
more
insight
around
refugee
displacement.
Terrorist
attacks,
famine
that
was
occurring
in
that.
B
A
So
the
other
look
obvious
challenge
is
the
amount
of
data
sources
that
had
to
be
manually
curated
and
in
areas
like
Benghazi.
The
availability
is
is
quite
an
accessible
because
of
the
the
tumultuous
situation
that
was
happening
there,
so
that
team
was
operating
out
of
Cairo
because
it
was
the
closest
place
that
they
could
access.
A
A
So
what
we
did
was
we
figured
okay
if
you
were
to
go
out
if
we
were
to
listen
to
all
those
data
streams,
Twitter
public
web
radio
and
being
able
to
collect
all
that
information
and
then
being
able
to
use
a
variety
of
different
machine
learning
and
NLP
technologies
and
approaches
to
be
able
to
do
topical
extraction
off
of
that
data,
as
well
as
to
be
able
to
identify
the
location
that
people
are
talking
about,
extract
the
sentiment
from
the
conversation
being
able
to
extract
what
organizations
being
mentioned.
What
person?
What
what
event?
A
A
Timing
is
a
timing,
as
it
was
absolutely
critical
for
our
use
case,
so
we
wanted
to
build
a
data
processing
pipeline
that
was
as
near
real
time
as
possible
with
Fordyce.
We
were
able
to
do
a
turnaround
time
in
less
than
15
seconds
in
terms
of
when
an
event
is
actually
tweeted
to
the
time
that
we're
able
to
process
it
aggregate
it
and
be
able
to
provide
it
on
the
dashboard.
So.
A
What
we
wanted
to
do
is
build
something:
that's
repeatable,
something
that's
adaptable
to
other
scenarios,
so
we
then
take
the.
We
took
the
same
exact
forward,
a
solution
and
we
repurpose
it
through
a
collaboration
with
amia
University
in
wohl
to
help
identify
and
predict
where
Zika
virus
was
spreading.
So
the
way
we
did
that
was,
we
took
a
look
at
weather,
forecast
data
to
try
to
predict
where
mosquitoes
were
migrating
towards
to
then
inference
where
whether
those
regions,
those
areas
or
high-risk,
sounds
then
layering
in
the
social
media
data
that
we
that
we
currently
have.
A
On
top
of
that
that,
whether
the
weather
pattern,
data
we're
able
to
then
assess
if
people
are
talking
about
fever-
or
you
know,
Zika
like
symptoms
and
there's
a
high
level
of
confidence
that
Zika
is
in
this
place,
and
then
you
could
also
layer
in
things
like
interventions
where
you
have
health
workers
that
are
actually
in
there
trying
to
try
to
remediate
that
that
the
disease,
so
just
to
talk
a
bit
about
the
functional
architecture
of
fortis.
So
we
build
streaming
connectors
all
base
off
spark
where
we
started
collecting.
A
So
our
kubernetes
cluster
is
running
on
Azure
off
of
Azure
container
services
and
we're
using
spark
streaming
so
the
data
stream
will
come
in.
We
would
then
distribute
that
data
through
through
spark
RDD
partitioning,
then,
based
on
the
events
that
people
are
talking
about.
We
would
then
run
some
and
the
extractors
to
trying
to
figure
out
mentioned
places
even
down
to
a
street
level
or
a
corner
or
a
district
or
administrative
boundary,
and
then
we
would
tag
the
event
to
that
place.
A
A
This
way
that
we,
once
we
index
it
by
geo
tiles,
we're
able
to
get
both
a
spatial
and
a
time-dependent
view
on
a
set
of
topics
and
entities
by
using
a
variety
of
different
cognitive
services
for
speech,
analytics
and
text
analytics
for
sentiment.
We
then
used
Lucene
to
do
the
to
do
the
the
entity,
the
entity
recognizer,
so
we
built
a
whole
bunch
of
analyzers
with
inspark.
We
filter
the
content
based
on
a
geofence
that
the
user
defined,
and
then
we
push
the
data
into
Cassandra.
A
A
So
our
solution
was
built
on
on
kubernetes.
We
wanted
to
run
spark
on
kubernetes
and
the
main
driver
for
why
we
wanted
to
do.
That
is
because
we
wanted
to
increase
the
resource
utilization
of
all
of
our
pods.
We
didn't
want
to
have
a
VM
just
stood
up
to
run
a
single
spark
worker.
We
wanted
to
be
able
to
maximize
all
the
nodes
we
have
within
our
cluster.
We
wanted
to
be
able
to
decrease
our
operational
cost
as
much
as
possible.
A
Simplified
deployment
model
as
well,
so
we
are
so
one
one
of
the
great
advancements
that's
happening
within
the
spark
and
kubernetes
space
is
a
project
called
spark
native.
We
have
a
bunch
of
open
source
contributors
that
originated
from
Bloomberg
and
Palantir
and
a
bunch
of
other
companies
as
well,
and
what
this
is
is
it's
a
it's
a
it's
a
framework
that
allows
you
it's
a
plug-in
framework.
A
A
A
The
streamlined
developer
experience
the
coop
CTL,
so
one-to-one
CLI
to
monitor
entire
cluster
and
high
availability
with
with
with
this,
allow.
This
freed
us
from
having
to
use
ooh
keeper
and
minimize
our
our
our
operational
management
stack
and
then
the
ability
to
elastically
scale
out
our
our
workers
via
the
coop
CTL
max
replicas,
and
then
also
simplified
our
DevOps,
with
being
able
to
leverage
tools
like
Prometheus
and
spark
and
spark,
and
the
kubernetes
dashboard,
to
monitor
and
log,
and
to
raise
events
in
certain
scenarios.
A
So
it
just
to
talk
through
a
bit
over
the
12,
the
deployment
model,
all
the
code,
that,
with
four
days,
is
all
open
source.
So
the
way
that
we
deploy
a
spark
job
is
once
you
cut
a
new
release
on
github.
We
that
kicks
off
at
Travis
CI
build.
We
then
use
simple,
build
tool
assembly
to
generate
the
fat
jar.
We
take
that
fat
jar
and
then
we
posted
the
blob,
storage
and
Chrysostom
to
talk
more
about.
B
B
We
started
with
the
helm,
charts
that
you
find
on
on
the
kubernetes
repo,
but
now
you
made
some
of
some
interesting
modifications
to
meet
our
needs,
as
as
Eric
mentioned
density
and
just
resource
utilization
was
one
of
our
goals,
so
we're
deploying
to
two
namespaces
into
into
a
single
cluster.
We
have
one
namespace
for
SPARC.
We
have
one
namespace
for
Cassandra.
We
modified
the
templates
a
little
bit
on
the
SPARC
side.
We
upgraded
to
the
latest
version,
2.2
we're
eyeing
2.3
at
the
moment.
B
So
so
we
have
spark
streaming
and
all
the
latest
features
available,
and
we
also
made
some
some
changes
in
order
to
be
able
to
kick
off
jobs
as
we
launch
the
applications.
So
if
you
go
to
our
our
catalyst
code,
repo,
you
will
find
it
a
chart
that
has
those
modifications,
but
from
a
spark
perspective,
we
run
a
single
master
that
we
kick
off
as
a
deployment.
Then
we
have
a
stateful
set
of
worker
nodes,
work,
yeah,
they're,
executors
and
that
run
the
spike
jobs.
B
We
also
have
the
Cassandra
nodes
running
in
the
cluster
and
with
Cassandra
we
optimized
to
charge
mostly
with
an
eye
on
high
availability
within
within
kubernetes
and
optimizing
that
for
how
things
run
in
Azure.
We're
also
using
stateful
sets
here,
because
the
the
pod
identity
becomes
very
interesting
when
you
try,
try
to
discover
services
and
we'll
talk
about
that
in
a
second
who's
been
the
keynote.
This
morning,
everybody
everybody
so
Kelsey
Kelsey
was
asking
about
spinning
up
kubernetes
jobs,
kubernetes
clusters.
B
He
made
it
sound
like
a
big
deal
because
we
use
the
Azure
container
service
and
for
the
Azure
container
service,
some
spinning
up
a
cluster
is
kind
of
a
one
command
line
kind
of
kind
of
a
deal.
Do
you
type
a
bunch
of
letters
and
you
wait
five
minutes
and
you
have
a
cluster,
so
the
azor
container
service
is
what
we
use.
Then
we
actually
use
a
derivative
of
that
it's
a
way
to
spin
up
a
cluster,
but
that
is
your
cluster.
It's
not
managed
by
Microsoft
you're.
B
You
have
some
control
over
the
configuration,
it
has,
let's
say
reasonable
defaults
for
networking,
for
the
distros
for
and
and
then
so
on.
But
if,
if
you
have
advanced
configuration
needs
like
so
some
of
our
customers
do,
for
example,
they
want
to
run
a
GPU
enabled
VM
or
they
have
a
very
special
distro
that
they
want
to
run
on.
They
have
a
I,
don't
know
a
special
version
of
Korres,
for
example.
We
also
have
what
we
call
the
ACS
engine.
B
That
is
a
way
to
run
a
provision,
very
advanced
configurations
of
of
kubernetes
clusters,
all
the
way
down
to
controlling
the
pod
cider,
so
that
the
service
networks
and
so
on.
So
you
get
to
customize
the
address
space,
which
we
also
did
for
something
we're
going
to
talk
about
in
a
second,
you
may
have
heard
just
a
little
plug
for
our
employer.
We
have
the
AKS
the
azure
container
service,
but
the
cake
not
with
a
see.
B
This
is
a
fully
managed
service,
where
Microsoft
will
maintain
the
operating
system,
the
kubernetes
runtime,
and
they
won't
even
charge
you
for
the
Masters.
So
you
just
pay
for
the
agent
notes.
So
much
for
that
Sokka
says
so.
Kubernetes
for
us
is
a
just
yeah
standing
up.
The
cluster
is
the
smallest
of
all
our
voice.
B
We
did
some
extensive
research
on
how
we,
how
we
configure
the
cluster
for
from
a
networking
and
benchmarking
perspective
and
about
this
its
reformatted
again,
the
benchmark
that
we're
running
is
taking
activity
data,
so
data
about
people
going
on,
runs
and
bicycle
trails
and
stuff
like
that
and
produces
heat
maps
from
that.
So
here
on
the
slide,
you
see
heat
maps
for
people
going
running
or
bicycling
in
Austin.
It's
a
rather
large
data
set
that
we
use
to
benchmark
and
we
have
a
configuration
that
exercises
the
network
and
and
avoids
resource
contention.
B
So
we
have
a
kubernetes
cluster
where
we
run
spark
and
we
have
virtual
machines
where
we
run
Kassandra,
of
course,
they're
connected
via
the
network
and
we,
the
azure,
see
and
I
plug
in
that
was
fairly
new
and
throughout
our
project
it
came
out
sometime
this
summer.
We
wanted
to
compare
that
to
calico,
which,
which
many
of
our
customers
were
using.
B
So
we
still
get
a
good
good
performance
advantage
out
of
the
addressee
and
I
have
about
ten
fifteen
percent,
which
is
kind
of
what
what
people
expect
when,
when
you
compare
native
networking
to
two
calico,
so
we
were
quite
happy
that
this
was
indeed
the
result
that
we
expected
going
into
the
benchmark.
So
our
cluster
is
now
configured
for
running
as
a
C
and
I.
B
Now,
there's
a
little
bit
of
trickery
to
that,
because
if
you
imagine
you
have
three
virtual
machines
and
they
all
run
in
the
same
rack
and
that
one
rack
has
a
problem
like
somebody
trips
over
the
power
cord,
hopefully
not
or
the
top
of
racks,
which
goes
bad,
then
all
three
VMs
would
be
unavailable
in
your
application
would
essentially
be
be
not
there.
The
same
is
true
for
containers.
B
The
way
we're
placing
containers
in
the
cluster
and
I
had
this
conversation
very
recently
with
with
a
an
engineer
over
at
mrs.
fear,
just
because
you're
deploying
more
than
one
replica
into
a
container
doesn't
necessarily
mean
that
you're
highly
available
because
or
into
a
cluster
doesn't
necessarily
mean
you're
highly
available,
because
even
if
you
have
three
replicas
of
let's
say,
Cassandra
running
of
all
of
those
replicas
were
running
in
the
same
rack,
and
that
rack
has
a
problem.
B
Then
you
still
have
an
issue
and
the
high
availability
setup
wasn't
as
good
as
it
should
have
been.
So
what
we
did
Cassandra
has
some
built-in
high
availability
features
that
when
you
tell
Cassandra
where
the
racks
are
and
where,
where
the
data
centers
are,
it
will
automatically
place
data
across
the
ring
so
that,
even
if
a
single
node
has
a
failure,
all
the
data
is
replicated
across
the
ring.
We
also
had
some
extra
setup
in
the
in
the
chart.
B
B
Snitch
I
think
it's
a
great
word
that
we
configured
if
you
running
where
they
they'll
be
configured
in
a
custom
custom
container
image
that
we
have
out
on
on
docker
hub
right
now,
ec2
has
their
own
snitch
and
we
should
probably
write
our
own
at
some
point,
but
for
Azure
this
works
actually
quite
well.
So
this
enabled
us
to
do
a
a
multi
Azure
data
center
set
up
for
for
high
availability.
We
can
now
take
that
container
because
it
has
the
data
center
information
and
the
rack
information
baked
into
each
Cassandra
image.
B
I
can
take
that
and
deploy
that
into
two
data
centers,
so
I
bring
up
two
data:
centers
imma
bring
to
kubernetes
clusters.
I
will
then
provision
a
single
Cassandra
ring
across
those
with
a
single
line
of
script.
So
that's
actually
kind
of.
We
were
quite
proud
to
show
that,
but
not
everybody
wants
to
a
Azure
data
centers.
We
have
some
customers
that
say:
ok,
we
have
actually
our
own
data
center
that
we
would
like
to
do.
B
High
availability,
and
can
you
stand
up
a
cluster
that
is
a
would
be
a
hybrid
Cassandra
ring
over
our
on
Prem
bring
their
Cassandra
that
we
already
have
and
extend
that
into
the
cloud,
and
we
can
certainly
do
that
with
the
networking
connectivity
features
that
we
have
between
a
cloud
provider
and
an
azure.
So
in
this
case
it
would
be
Ashley
Express
route
that
would
connect
the
on-prem
environment
and
a
cloud
environment
to
form
a
single
network,
and
then
we
can
bring
up
Cassandra
on
those
on
the
to
true
clouds.
B
The
last
one
we
thought
we
show
is
our
our
work
in
progress,
and
we
learned
in
the
keynote
this
morning
that
a
multi
cloud.
This
is
a
big
deal,
so
we
were
looking
at
like
yeah.
That's
that's!
Actually,
what
we've
been
working
on,
so
we
have
been
experimenting
with
a
multi
cloud
setup
for
Cassandra,
taking
a
the
half
the
cluster
in
an
and
half
the
cluster
in
another
cloud
and
AWS
and
connected
those
two
via
via
a
VPN
connection
and
then
bootstrapped
our
Cassandra
cluster,
just
to
see
how
it
works.
C
B
Ssh
connections
into
two
different
defend
and
virtual
machines.
One
is
running
in
Azure.
The
left
on
the
right.
One
is
running
in
AWS.
Just
to
make
sure
you
you
believe
me,
I
can
actually
go
to.
I
can
go
to
the
metadata
service,
and
this
looks
very
much
like
an
azure
VM,
that's
running
in
in
our
West
US
region.
I
can
do
with
the
same
over
here
and
you
see
this.
This
is
truly.
B
B
51
31
and
surprisingly
enough,
even
though
we're
connecting
in
an
azure
data
center
to
an
AWS
data
center,
we
have
a
latency
for
in
the
in
the
tens
of
milliseconds.
So
so
that's
that
we
feel
quite
confident
it's
good
enough
to
run
a
distributed
cluster.
So
if
I
look
here,
I
actually
have
Cassandra
deployed.
It's
the
same
help
chart
that
I
have
on
my
repo.
B
I
do
also
have
Cassandra
pods
running
over
here
and
when
I
say
something
like
Cassandra
Burnett
check.
The
status
of
my
cluster
you'll
see
I
do
have
indeed,
I
do
have
indeed
a
data
center,
a
two
data
center
cluster
one
here
in
AWS,
and
one
and
one-half
of
that
in
in
our
Azure
data
center.
So
now
I
have
a
single
distributed.
Cassandra
ring
across
two
data,
centers
see
it
just
comes
back
and.
B
The
answer
is
not
yet
currently,
because
Federation
is
still
a
little
bit
early
in
its
life
cycle
and
we
try
to
do
something
that
that
was
stable
enough
for
customers
to
experiment
with
we
actually
are
looking
for
customers
that
would
like
to
work
with
us
on
setups
like
that.
So
if
you,
if
you're
interested
in
exploring
multi
cluster
configurations
or
multi
data
center
configurations,
multi-cloud
configurations
or
know
as
somebody
that
does,
we
would
absolutely
love
to
work
with
you
and
as
the
as
departing
fighting
slide.
B
So
to
say,
everything
we
do
is
open
and
everything
we
do
is
on
github,
so
the
everything
from
the
ingestion
pipeline
with
SPARC
and
Cassandra.
That
is
yours,
to
stand
up
complete
with
the
graph
QL
based
on
it.
We
have,
of
course,
the
multi
data
center
multi-cloud
configurations
documented.
We
have
to
cross
cloud
networking
documented
and
everything
else.
We
also
expand
on
a
site
on
Microsoft
comm
called
the
developer
blog.
So
if
you
go
to
Microsoft
comm
slash
developer
blog,
you
will
find
lots
of
information
about
other
projects
that
we're
doing
so.
B
C
C
A
Sure
sure
so
we
used
there's
a
speech
SDK
that
comes
with
cognitive
services.
We
built
a
protocol
on
top
of
that,
where
we
were
doing
real-time
speech
to
text.
So
the
the
API
word
speech
text
out
pretty
much
the
the
Cortana
API.
Ultimately,
so
we
integrated
that
in
real
time
with
inspark,
so
we
built
the
whole
sparks
room
connector
around
that
the
other
one
we
used
was
sentiment
at
the
text
analytics
for
sentiment,
analysis
and
then
computer
vision.
So
what
we
did
with
that
was
for
Instagram
posts,
whether
it's
pictures
or
video.