►
From YouTube: Running Multi-site, SAP Applications on Kubernetes and CoreOS by Nishi Davidson & Victoria Rozhina
Description
Running Multi-site, SAP Applications on Kubernetes and CoreOS - Nishi Davidson, SAP & Victoria Rozhina
"SAP Labs uses Kubernetes to deploy and scale containerized applications in select private clouds.
Kubernetes promises an environment consistent enough to optimally deploy and fail over workloads.
We’ll demo and talk about how we move from the current state of Enterprise applications, virtualization in private datacenters, to a world where we run (and fail over) workloads between clouds using containers and Kubernetes."
About Nishi Davidson
Nishi has been in the enterprise cloud infrastructure and application space for 13 years working across engineering, customer engagement/architecture and product management in South East Asia and the US markets. Currently she is responsible for SAP’s private cloud, container and cluster manager architecture and engineering initiative. Along with her team, she primarily works with SAP business units to prototype and steadily transition their workloads/use cases to containers running on Kubernetes and Mesosphere. In the past, Nishi led customer engagement teams and introduced multiple cloud products/solutions to the market while working at HP, Juniper Networks, NetApp and DSSD (EMC). Nishi holds an MBA from Massachusetts Institute of Technology, Sloan and a bachelor’s in Electrical and Electronics Engineering from College of Engineering, Guindy, Anna University.
About Victoria Rozhina
Recent graduate with a masters in CS and work experience at Cisco, Kaspersky Lab, currently working at SAP.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Alright
how's
everyone
doing
this
afternoon.
Awesome
I
am
ready
to
be
done
with
a
successful
demo,
so
I
can
go
drink
some
scotch,
but
thank
you
so
much
for
giving
us
the
opportunity
to
talk
here
today.
This
is
basically
a
12-month
journey
for
us
working
on
kubernetes
and
trying
to
actually
take
a
legacy
environment
and
make
it
micro
services
friendly.
As
you
can
guess,
it's
a
very
difficult
job,
so
my
name
is
Misha.
Davidson
I
actually
represent
sa
P
labs,
which
is
a
research
group
inside
sa
P.
A
We
work
with
a
lot
of
lines
of
businesses
inside
sa
P
and
we
have
our
enterprise
software
product
group
that
builds
software
like
ERP
financial
systems,
HANA
databases
so
and
so
forth.
But
over
the
years
because
of
competition,
we've
acquired
a
bunch
of
Lyon
SAS
companies
like
Reebok
and
Kerr
SuccessFactors,
and
all
of
these
guys
have
different
data
center
designs
and
different
types
of
doing
things.
90%
of
our
environment
is
virtualized,
but
most
of
the
SMP
applications.
A
Work
are
built
for
Linux
and
they're,
run
on
bare
metal
containers,
bare
metal,
infra
environments,
believe
it
or
not.
It's
also
less
twelve
and
it's
another
challenge
that
we
have
to
work
with
inside
sfe.
So
today
we
are
going
to
talk
about
how
we
are
running
a
pilot
and
a
design
concept
of
first
using
SVP
containers
and
running
them
on
a
kubernetes
infrastructure
that
we
have
built
on
top
of
core
OS.
A
B
A
So
with
that,
we'll
start
with
the
demo
finish
it,
so
we
can
all
breathe
and
have
fun.
So
we
will
actually
talk
about
four
setups
today.
This
is
the
best
we
could
do
after
talking
to
our
recipe
data
center
guys.
So
the
first
setup
is
SFE
data
center,
one
that
has
15
servers
inside
one
be
LAN
and
it
is
running
NetApp
nas.
That
setup
is
being
used
by
our
pilot
line
of
business,
so
we
are
not
going
to
run
any
automation
software.
A
On
top
of
that,
we
have
a
second
data
center
site
where
we
have
seven
servers
and
we're
also
going
to
show
you
two
setups
in
the
public
cloud.
One
is
in
Amazon,
that's
running
five
ec2
instances
in
a
B
PC,
along
with
s3
at
the
backend
and
the
other
is
GCP
with
six
instances
all
exposing
persistent
disks.
The
first
demo
that
we
want
to
show
is.
We
have
written
an
answer
based
simple
automation,
software
that
basically
brings
up
our
overall
kubernetes
design
on
top
of
any
infrastructure.
A
B
A
First,
we
have
to
the
minimum
change
that
we
have
to
make
is
actually
figure
out
how
our
environment
will
look
like.
As
you
can
see,
we
have
one
at
C,
D
node
in
this
cluster.
You
could
add
one
three,
five
or
seven
as
many
as
you
need
to
we've
put
one
master
node.
You
can
obviously
put
two
so
that
they're
in
an
H
a
set
up
and
the
remaining
are
our
worker
nodes.
A
These
are
our
Amazon
ec2
instances,
each
with
a
private
IP
and
a
public
IP
which
we
enter
here
and
we
tag
these
nodes
as
master
and
workers
same
setup
for
GC
P.
So,
as
you
can
see,
I
have
one
at
Etsy
d,
node
for
worker
nodes,
one
master
and
we
are
tagging
it
so
with
that
I
will
just
run
the
automation.
A
On
both
environments,
so,
as
you
can
see
what
this
is
doing
is
I'm
outside
the
VPN
and
via
HTTP,
we
are
able
to
connect
to
these
public
IPs
and
essentially
run
our
ansible
automation
from
the
Mac
in
order
to
bring
up
the
entire
kubernetes
cluster.
So
while
this
automation
is
ongoing,
I'd
also
like
Victoria
to
bring
up
the
ASAP
data
center,
the
same
software
is
just
a
different
set
up
altogether.
So
you
guys
can
believe
that
we,
our
software,
works
in
every
environment,
so
I'll
go
to
Google
hangout.
A
The
reason
we
built
this
software,
you
guys,
might
have
heard
of
cargo
and
cube
spray
out
there.
That
actually
does
similar
stuff
and
essentially
the
idea
is
to
bring
up
everything.
So
this
discovery
load,
balancing
firewall,
Sdn
all
of
those
aspects
on
top
of
core
OS
with
kubernetes
using
cube
spray
or
cargo.
We
actually
tried
using
it
in
our
own
environment,
but
the
nature
of
the
enterprise
infrastructure
is
such
that
our
private
VLANs
public
VLANs
all
are
created
in
such
a
convoluted
way
and
there's
also
different
name
servers:
different
DNS
--is.
A
That
became
really
hard
for
us
to
track
proxy
everywhere
inside
cube
spray,
and
we
realize
that
it
would
be
more
of
an
effort
to
like
try
to
customize
it
strether
than
write
our
own
automation
just
across
the
simple
design.
So
that's
why
we
went
ahead
and
created
our
own
little
tool.
In
fact,
it's
ready
to
be
open
sourced
and
we
will
most
probably
you
know,
make
it
available
for
other
people,
other
enterprises
to
use,
and
very
very
soon.
So
that's
my
excuse
as
to
why
I
did
not
use
tube
spray
or
cargo.
A
So
while
this
comes
up,
this
actually
takes
six
minutes.
To
be
honest,
so
while
this
comes
through,
let
us
talk
about
really
what
it
is
doing
so
essentially,
this
is
the
slide
that
talks
about
how
the
design-wise
is,
what
we
are
bringing
up.
So
if
you
actually
notice
during
these
last
eight
months
or
10
months,
that
we've
been
working
on
kubernetes
mesosphere,
we've
actually
evaluated
five
stacks
of
cluster
management
software.
A
If
you
will
so
in
Sapa,
there
are
certain
groups
that
also
used
using
Hoshi
carp,
so
a
combination
of
nomad
console
and
terraform
in
order
to
bring
their
micro-services
up,
we
have
tested
mesosphere
as
well.
It's
a
very
good
product,
which
is
you
know,
production
ready
and
it's
a
complete
product
that
has
monitoring
logging.
Everything
enabled,
but
the
interest
in
mesosphere
is
primarily
from
the
distributed
application
side
and
a
lot
of
our
I
Oh
D
guys
are
using
open
source
of
Maysles
because
they
want
to
include
the
information
and
push
it
into
a
spark
cluster.
A
We
chose
to
go
with
kubernetes,
primarily
because
this
whole
environment
is
extremely
new
to
all
of
us
and
it
was
painful
for
me
to
sit
and
manually
set
up
every
single
layer
of
the
architecture.
So
you
deploy
core
OS,
you
then
setup
plan,
LD
and
calico.
You
can
add
docker
to
it,
you
deploy
SVD,
and
then
you
have
to
do
the
firewall
load,
balancing
the
registry,
the
service
load
balancer.
All
of
that
it
is
painfully
slow
and
it
is
extremely
long.
A
However,
what
we
realized
is
by
doing
it
step-by-step,
like
this,
our
concepts
become
really
clear
and
as
we
containerize
the
application,
we
know
what
is
going
wrong
in
the
environment
when
we
want
to
solve
a
problem.
So
therefore
it's
going
to
be
easy
for
us
to
actually
help
the
engineering
team
understand,
what's
happening
and
also
train
the
operations
guys
in
debugging
and
getting
ready
for
production
deployment.
A
That
being
said,
we
have
also
found
that
there
are
many
challenges
with
the
environment
in
that
sometimes
many
times
my
kubernetes
nodes,
just
you
know,
suddenly
shut
off
for
some
reason.
Proxy
is
shut
off
and
then
the
master
cannot
go
into
individual
nodes,
so
I
have
to
go
in
log
in
and
manually
change
and
update
stuff.
A
So
we're
also
trying
to
build
a
tool
that
will
allow
us
to
go
into
a
cluster
of
twenty
six
or
hundred
nodes
and
modify
all
the
cubelets
at
one
go,
because
otherwise
it
becomes
really
difficult
for
a
for
an
infrastructure
person
to
go
login
to
every
single
machine
and
change
the
individual
settings
that
you
might
have
in.
So,
as
you
actually
see
there,
the
automation,
software
that
we
are
running
right
now
and
you'll
see
that
it
comes
up
and
the
whole
cluster
is
up.
It
not
just
includes
service
discovery,
the
basic
OS
kubernetes
features.
A
It
also
includes
Kanno,
which
is
a
combination
of
flannel,
tee
and
calico
that
we
are
using
for
load,
balancing,
sorry,
not
load,
balancing
firewall
and
network
user
Sdn.
We
also
have
integrated
Prometheus
with
DRA
fauna,
so
you
just
have
to
run
the
proxy
and
the
graph
on
our
dashboards
come
in
in
our
environment,
in
SA
P,
with
the
lines
of
businesses
that
we
working
with.
They
also
have
specific
requirements
for
the
application
metrics
to
be
integrated.
A
So
we
are
actually
trying
to
customize
zabbix
docker
containers
so
that
we
can
pull
the
container
application
metrics
and
push
it
to
the
backend
from
ETS
database
on
the
logging
side,
there's
a
huge
privilege
of
Splunk
in
SA
P,
which
is
going
to
be
true
for
many
enterprise
companies.
What
we're
finding
is,
even
though
Splunk
is
great,
the
cost
of
actually
having
Splunk
and
managing
it,
and
the
data
center
is
increasing
at
a
pace
which
is
unbelievable.
A
A
There
are
certain
aspects
of
log
context
and
audit
features
that
are
cool
about
Splunk,
but
I
think
that
going
forward
there's
going
to
be
more
development
in
the
Cabana
elasticsearch
space
and
we'll
be
able
to
customize
the
logs
and
remove
the
cost
problem
from
s
ap
data
centers.
Apart
from
that,
if
you
look
at
the
topmost
layer,
we
these
are
the
applications
that
are
in
their
early
phases
of
containerization,
so
Hana
physics
LMS.
There
are
some
s,
ap
Jam
community
pages
that
can
easily
be
containerized
and
going
forward.
A
A
Amazon
environment
came
up
and,
while
the
GCE
environment
excuse
me
I'm
sorry,
I
have
a
really
bad
cough.
No
one
will
fall
sick,
but
I
am
anyhow
going
to
go.
Keep
coughing,
so
I
will
open
a
new
window
here
and,
as
you
guys
remember,
I
had
showed
you
the
hosts
for
Amazon,
and
here
the
master
is
35.1
62.8
e
7.14.
For
so
you
just
have
to
choose
the
right
certificates
and
the
dashboard
comes
up.
So
this
is
the
default
kubernetes
dashboard
and
if
you
notice,
we
talked
about
flu
and
elasticsearch
gras
fauna,
hipster
Prometheus.
A
All
these
components,
along
with
the
registry
and
the
service
load
balancer,
also
Calico.
All
of
these
components
come
up
ready-made.
So
this
is
the
first
step
where
we
are
saying
that
you
can
use
any
infrastructure.
If
you
want
to
it,
can
be
in
Amazon.
It
can
be
in
s
ap
data
center.
It
can
be,
you
know
anywhere.
Our
automation.
Software
will
work
as
long
as
certain
conditions
RNA
both
and
it
will
give
you
the
base
infrastructure
to
start
deploying
your
Katinas,
which
is
great.
So
that's
the
first
step.
A
A
A
A
We
would
like
to
deploy
some
of
our
ICP
containers,
which
is
a
fun
part,
so
we're
going
to
do
this,
basically
inside
our
first
data
center,
where
we
couldn't
run
the
automation,
because
it's
being
used
by
our
pilot
customers,
we
are
going
to
deploy
six
ASAP
applications
as
pet
sets
and
then
we're
going
to
fail
two
of
those
containers
into
our
second
s,
AP
site
and
two
of
those
containers
into
Amazon.
That's
the
flow
of
the
demo,
and
now
you
all
should
wish
me
luck.
So
we
thinks
don't
crash.
Did
you
want
to
go?
B
A
Now
you
must
be
thinking
why
not
write
this
in
a
script,
six
iammoz
one
at
a
time.
There
is
a
reason
we
found
the
VPN
connectivity
to
or
whatever
connection
here
it's
just
difficult
to
reach
our
s,
AP
data
center
and
then
ASAP
applications
are
monolithic,
applications
we're
trying
to
containerize
them
the
processes
don't
easily
closed
or
shut
down.
So
if
I,
if
we
try
to
rush
through
things,
it
usually
results
in
corruption.
A
A
So
this
is
the
yamo
that
Victoria
is
trying
to
show
of
each
of
the
Hana
containers
that
we
deployed.
As
you
can
see,
the
server
there
is
the
NFS
server
for
the
first
site,
which
is
10.
Excuse
me,
ten
dot,
125
dot,
one
dot,
two
for
one,
it's
in
our
first
primary
data
center
and
now
bringing
up
the
SAV
containers
is
going
to
take
a
while.
So
while
that
happens,
we
can
talk
about
some
of
the
issues
that
we've
been
facing
so
like
I,
highlighted
highly
monolithic
applications.
A
Some
of
the
binaries
are
as
big
as
three
and
a
half
gig.
So
imagine
if
you
want
to
include
and
installed
container
it's
sometimes
it
grows
to
be
as
big
as
five
gig,
but
this
is
a
database.
If
you
look
at
some
other
s,
ap
applications,
they're
smaller
they're,
also
very
strongly
built
for
the
base
kernel.
So
we
have
found
that
there
is
incompatibility
between
C
groups.
A
So
typically,
what
happens
is
many
of
the
applications
initialized
from
the
kernel
resource
level,
so
they
do
not
think
of
the
cgroups
at
all
and
we
checked,
and
there
are
many
applications
that
show
this
kind
of
problem
in
that,
even
if
you
start
the
application
inside
the
docker
container,
it
still
initializes
from
the
kernel
path
and
therefore
it's
it.
It
knows
that
it
has
100
CPUs,
whereas
docker
limits
it
to
20
CPUs.
We
don't
know
what
the
challenges
of
that
kind
of
initialization
would
be.
A
We
have
seen
that
there
is
no
violation
of
the
C
groups
per
se,
as
in
the
application,
never
crosses
out
of
20
and
goes
and
uses
hundred,
so
docker
is
good,
but
then
it
still
might
result
in
some
additional
threads
being
spun
up
or
some
type
of
issue
that
we
don't
know
yet,
and
we
have
to
do
more
testing
around
that.
So
that's
one
open
area.
The
other
problem
we
have
found
is
many
of
the
applications
require
privileged
mode,
which
in
itself
is
a
security
problem.
You
don't
know
what
is
running
from
the
container.
A
If
everything
needs
privileged,
then
it's
a
problem
so
we're
trying
to
fix
that
and
the
other
issue
is
a
lot
of
the
config
files
are
spread
all
over
the
kernel
and
so
trying
to
understand
how
to
pick
these
config
files
and
customize
them
inside
the
container
is
a
huge
challenge.
It
is
a
challenge
that
can
be
solved,
but
nevertheless
is
an
issue.
The
other
issue
that
we
found
with
core
OS
was
Hana,
for
example,
typically
during
its
installation,
it
uses
tempest
core
OS
does
not
allow
it.
A
So
sometimes
the
operating
system
has
its
restrictions
and
the
installation
might
fail
so
we're
coming
across
these
huge
set
of
issues
that
doesn't
make
the
migration
into
kubernetes
and
the
docker
world
that
easy
per
se
also
in
most
cases
as
far
as
enterprise
applications
are
concerned,
they're
very
closely
tied
to
hardware,
as
well
as
firmware
there's
enough
dependency
in
terms
of
how
much
CPU
and
memory
is
also
required.
So
that
makes
it
hard
because
the
enterprise
guys
over
long
period
of
time
have
usually
tested
every
type
of
server.
A
Every
type
of
virtual
machine
and
they've
published
a
road
map
before
a
customer
can
choose
to
deploy
it
now
in
this
new
world,
it
becomes
really
difficult,
because
a
customer
has
the
expectation
to
go
and
deploy
a
container
in
any
environment.
So
that's
also
extremely
difficult
to
manifest,
and
therefore
production
releases
of
any
of
our
containers
is
going
to
take
more
time,
but
we're
making
the
right
move
in
the
right
direction.
A
Another
big
problem,
everything
is
stateful.
Nothing
is
stateless,
except
for
the
web
layer.
Even
there
you
want
some
stickiness
with
regard
to
the
application
IP,
and
that
causes
a
lot
of
challenges.
So
my
colleague
from
concur
is
here.
So
these
are
some
of
the
issues,
and
maybe
it's
ringing
home
with
you.
I
don't
know,
but
concur
is
obviously
a
little
bit
lighter
in
terms
of
applications,
whereas
the
basic
legacy
enterprise
applications
are
extremely
difficult
to
deploy.
Are
we
good,
yeah?
Ok,
looks
like
the
damn
demo.
Gods
are
with
us.
B
A
The
other
thing
we
have
done
is
we
have
created
a
very
simple
UI
that
actually
queries
the
database
and
loads
it
at
the
top
and
as
soon
as
we
deployed
the
six
containers,
the
UI
goes
and
talks
to
the
backend
and
populates
the
data.
So
the
six
containers
that
Victoria
had
deployed
in
the
SMB
data
center
site,
one
just
came
up
and
you
guys
have
to
believe
us
we're
querying
the
database
and
uploading
stuff
up
here
now.
A
A
Okay,
so
we
deleted
Hana
won
and
Hana
to
just
now,
and
there
are
four
other
databases.
Actually
we
deleted
Hana
one
two:
five
and
six
we're
going
to
bring
up
Hana
one
and
two
in
the
secondary
site
that
we
talked
about
and
five
and
six
in
Amazon.
Do
you
want
to
do
it
sure,
okay,
so
I'm
going
to
do
this?
Is
the
Amazon
environment
and
let's
check
what
the
current
context
is:
big.
A
A
You
can
see
it's
five,
one,
five,
six,
one,
six,
seven,
five
one!
So,
basically,
hopefully
you
followed
the
flow.
The
flow
is
we
just
killed
five
and
six
and
brought
it
back
in
Amazon,
and
the
query
against
the
load
balancer,
which
is
an
index
page,
is
showing
that
the
same
data
has
come
up.
Does
that
make
sense
or
did
I
confuse
the
hell
out
of
you?
A
Okay,
good,
so
going
back
to
Victoria's
system
she's
trying
to
deploy
now
the
first
two
containers,
one
and
two
in
the
secondary
data
center
with
s
AP,
all
these
yeah
Mose
are
will
show.
We
can
show
you,
the
yeah,
Mose
and
they're
attached
to
a
different
IP
address
a
different
net,
app
server,
not
the
same
net,
app
server
as
you
can
see,
it's
10.1
25.2,
dot,
77
the
first
one.
If
you
recall
I,
showed
you
as
10.1
25,
.,
1.2
41.
B
B
A
So
we
have
Hana
one
and
Hana
two,
and
if
you
go
back
to
the
original,
the
original
UI
page,
where
all
the
six
containers
were
running
you
can
see,
the
data
is
consistent,
correct
the
data
is
seven
one
five
and
if
you,
if
she
goes
back
to
her
other
load
balancer
at
seven
one,
five
so
point
being
you
can
failover.
As
long
as
the
data
is
consistent,
it
works
fine.
The
problem
is,
you
have
to
manage
your
own
storage
and
you
have
to
make
sure
how
it
gets
replicated.
A
A
A
I
had
promised
entertainment
dentists,
oh
okay,
awesome,
so
we
did
a
couple
of
things.
This
is
a
traditional
data
center.
Many
of
you
might
be
actually
aware
of
how
this
happens.
We're
using
two
NetApp
filers
across
the
two
sites
that
we
just
showed
and
snapmirror
is
the
net
apt,
replication
technology
that
replicates
asynchronously
or
synchronously.
We
obviously
showed
you
asynchronous,
because
there
is
a
DR
scenario
that
we
are
trying
to
simulate.
A
If
you
wanted
near
to
close
h.a,
then
you'd
have
to
do
some
more
techniques
of
having
a
fiber
channel
connect
between
the
two
sides
and
doing
stuff
like
that.
But
for
our
simple
scenario,
we
just
copied
the
data
across
the
two
NetApp
filers
and
then
all
we
did
is
we
killed
up
one
and
two
in
the
primary
data
center
and
brought
it
up
in
the
secondary
data
center.
So
that
was
clear
through
our
first
demo.
A
The
second
thing
that
we
tried
to
do
was
the
Amazon
demo
right,
so
in
Amazon,
NetApp
has
its
own
on
tap
software
available
as
a
VM.
So
you
get
the
same
aspects
of
the
replication
software
called
snapmirror,
backup
software
called
snap
all
stuff.
Like
that
and
what
you
could
do
is
you
could
just
spin
up
the
VM
on
command
is
their
management
layer
and
then
on
top
is
their
software.
So
basically
you
create
a
file
system
on
top
of
f3
or
EBS.
A
A
So
this
is
our
ec2
environment,
where
we
did
the
demo
setup
and
we
go
to
ec2
here
we
see
there
are
11
instances
running
as
you
can
see.
There
is
one
thing
that
says
on
tap:
there's
another
thing
that
has
a
tag
on
command
cloud
manager.
So
if
you
actually
go
here
to
on
command
cloud
manager,
it
asks
you
for
setup
information,
you
login,
and
these
are
where
are
snapshots.
These
are
the
snapshots
where
we
can
do
a
cloud
sync,
and
this
is
one
volume
that
is
been
created,
120
gig
in
size.
A
You
can
go
to
this
particular
instance
and
see
which
are
the
ec2
instances
connecting
to
that
on
top
software.
So
it's
very
simple
you're,
basically
doing
an
asynchronous
cashing
in
the
cloud
of
your
on-premise
filer
and
you
could
have
a
VPN
connection
between
the
two
and
it
will
copy
the
data
and
make
sure
that
you
have
some
infrastructure
available
in
the
cloud.
So,
frankly,
you
know
having
double
the
amount
of
infrastructure
and
not
being
able
to
use
the
cloud
as
not
really
sensible
anymore.
A
Of
course,
a
lot
of
our
guys
are
scared
of
doing
anything
with
the
database,
which
makes
sense
and
for
the
database
you
might
want
to
have
a
physical
infrastructure
replicating
data
all
the
time,
but
as
far
as
the
application
and
the
web
and
web
containers
are
concerned,
there's
no
need
to
actually
have
everything
running
all
the
time,
especially
because
the
DR
recovery
policies
today
are
as
bad
as
4
to
7
hours.
So
unless
it's
like
seconds,
why
would
you
want
to
have
double
the
amount
of
infrastructure?
Is
the
point?
Oh
three,
more
minutes.
A
C
A
We
haven't
coordinated
it
yet,
so
we
will
be
using
at
CD
log
going
forward
and
will
not
allow
the
auto
upgrade.
This
has
really
really
brought
down
my
cluster.
It
is
20
times
and
we
have
learnt
it
the
hard
way,
because
sometimes
you
know
they
have
moved
from
docker
to
rocket
in
their
flannel
code
base
and
adding
auto
upgrade
it's
just
not
the
right
way,
so
will
most
probably
go
to
ed
city
lock
and
try
to
do
it.
That
way.
C
C
C
A
Well,
some
of
the
config
files
were
not
being
set
properly,
EDC
password,
EDC,
just
hostname,
and
the
way
they
have
some
encryption
written
in
between
some
of
the
config
files
we're
just
not
getting
loaded
properly.
We
could
talk
more,
and
maybe
you
can
give
me
some
suggestions.
We
are
not
that
sure
what
docker
has
improved
in
this
particular
area,
so
it
could
be
a
feedback
for
me
and
I.
Just
don't
know
enough,
and
maybe
what
you're
talking
about
to
just
solve
all
our
problems.
I
just
don't
know
it's
like.