►
Description
NERSC Data Seminars Series: https://github.com/NERSC/data-seminars
Title:
Composable Platforms for Scientific Computing: Experiences and Outcomes
Speakers:
Erik Gough, Brian Werts, Sam Weekly - Rosen Center for Advanced Computing, Purdue University
Abstract:
The Geddes Composable Platform is an on-premise Kubernetes-based private cloud hosted at Purdue University that’s designed to meet the increased demand for scientific data analysis and to promote "SciOps" — the application of DevOps principles in scientific computing. The platform has supported research groups and data science initiatives at Purdue, enabling as many as sixty users from a variety of scientific domains. In this seminar, we will give a technical overview of the platform and its components, summarize the usage patterns, and describe the scientific use cases the platform enables. Some examples of services deployed through Geddes include JupyterHubs, science gateways, databases, ML-based image classifiers, and web-based BLAST database searches. The same technology behind Geddes is found in Purdue’s new XSEDE resource named Anvil, which provides composable computing capabilities to the broader national research community.
Bios:
Erik Gough is a lead computational scientist in the Research Computing department at Purdue University. He has been building, maintaining and using large scale cyberinfrastructure for scientific computing at Purdue since 2007. Gough is a technical leader on multiple NSF funded projects, including an NSF CC* award to build the Geddes Composable Platform.
Brian Werts is the lead engineer for the design and implementation of Purdue’s Geddes Composable Platform and a HIPAA aligned Hadoop cluster for researchers that leverages Kubernetes to help facilitate reproducibility and scalability of data science workflows.
Host of Seminar:
Jonathan Skone, Advanced Technologies Group
National Energy Research Scientific Computing Center (NERSC)
Lawrence Berkeley National Laboratory
A
So
I
it's
my
pleasure
to
introduce
several
folks
from
purdue's
ro
rosen
center
for
advanced
computing,
so
we
have
eric
goff
and
sam
weekly,
as
well
as
brian
wertz,
who
are
going
to
share
with
us
their
experiences
working
with
the
composable
platform
gettys,
which
is
a
rancher
kubernetes
cluster
and
the
use
cases
that
they've
supported
for
this
resource.
So
go
ahead.
Guys.
B
You
know
not
getting
too
technical,
but
we
do
want
to
talk
about
the
technologies
and
the
components,
and
we
also
want
to
highlight
some
of
the
use
cases
that
we've
seen
in
the
past.
I
think
this
has
been
up
for
maybe
almost
nine
or
ten
months
now
so
we'll
cover
some
of
the
some
of
the
use
cases
we've
seen
in
the
past.
B
So
first
just
the
thing
about
just
to
mention
purdue.
So
you
know
we're
a
r1
university
located
in
beautiful
west
lafayette,
indiana,
there's
a
student
population
of
about
50
000
and
about
12
000
ish
of
those
are
our
graduate
students,
our
department,
which
is
I.t
research,
computing
or
also
rcac,
which
is
a
a
previous
name.
B
We've
had
the
rosen
center
for
advanced
computing,
we're
a
unit
of
of
itap,
which
is
the
information
technology
organization
at
purdue,
and
what
we
do
is
provide
advanced
computing
resources
and
services
to
all
of
all
the
purdue
faculty
and
staff
that
want
to
do
research
and
our
goal
is
to
sort
of
be
like
the
one
stop
shop
or
one
stop
provider
of
choice
for
research,
computing
and
data
services.
So,
if
you
want
to
you
want
an
hpc
system,
you
want
access
to
hpc.
B
B
Research
groups
can
buy
into
those
for
just
the
cost
of
a
node
and
there's
a
lot
of
benefits
to
that
model.
So
people
from
itap
sysadmins,
you
know
they
support
the
thing.
So
your
graduate
student,
isn't,
you
know
an
expert
in
biology
and
also
expert
in
hpc.
They
can
just
focus
on
their
research
there's
low
overhead,
in
that
we
provide
all
the
infrastructure
for
these
systems
and
also
the
composable
systems
too
networking
and
things
like
that
power.
B
It's
cost
effective
because
we
can
leverage
group
purchasing
power.
You
know
dell's
going
to
give
you
a
better
deal
if
you're
buying
600
nodes
as
opposed
to
to
six
and
then
compute
nodes
are
shared
among
partners.
So,
if
I'm
not
using
the
nodes
I
purchased,
then
other
people
can
jump
on
those
for
a
for
a
short
period
of
time,
four
hours
and
and
use
those.
So
we
can
get
more
utilization
out
of
the
system
in
a
in
a
community
model
like
this.
B
B
B
But
what
we
see
is
that
researchers
come
to
us
with
these
ideas
or
things
they
want
to
do
that
need
more
diverse
computing
resources,
so
they
want
cloud
style
flexibility
they
might
want
to
deploy
a
persistent
service
like
a
database.
Do
some
scalable
data
analytics
with
something
like
das
or
spark
or
deploy
an
entire
web-based
science
gateway
and
have
it
interact
with
our
hpc?
B
So
a
lot
of
these
are
either
complementary
services
to
hpc
or
some
sort
of
elastic
software
stack
that
that
can
run
in
through
some
sort
of
scheduler,
and
these
are
these
are
sort
of
cloud
applications
right
they
would.
They
would
find
a
good
home
in
the
public
cloud,
and
so
why
don't
we
just
you
know,
make
you
know,
tell
research
researchers
to
get
their
credit
cards
and
you
know
go
to
go
to
google
or
wherever.
B
So
the
problem
with
that
is
that
it
doesn't
exist
or
it
doesn't
leverage
the
existing
infrastructure
that
we
have
on
campus
already.
You
know:
we've
got
high-speed
networks
and
the
large-scale
storage
systems
that
people
have
their
data
on.
The
data
is
close,
so
they
can
use
a
local
system
to
access
their
data
more
easily.
B
You
know,
if
you
go
to
public
cloud,
you're
magically
become
your
own.
Security
team
and
cloud
can
obviously
be
costly
for
for
people
who
aren't
familiar
with
it.
So
we
want
to
get
a
path
to
go
from
campus
to
public
cloud
or
something
like
jet
stream,
which
is
an
exceed
cloud
platform,
but
I
don't
think
people
need
to
go
there.
First,
they
can
have
an
intermediary
step
at
the
campus
level,
so
the
goals
that
we
have
are
to
create
a
campus
ecosystem
for
cloud
technologies.
B
We
want
to
share
this
technology,
but
also
the
knowledge
sharing
and
support.
We
want
to
empower
faculty
to
integrate
these
cloud
computing
technologies
in
research
and
in
their
courses
and
instruction.
So
we'll
see
some
of
that
in
the
use
cases
later
we
want.
We
really
want
to
give
people
the
ability
to
deploy
their
own
services,
and
they
can
do
these
things
that
are
very
important
for
reproducibility,
like
version
control,
they
can
do
automation
and
then
also
ci
and
then
evaluate
a
sustainable
model
for
campus
private
clouds.
B
B
An
idea
of
what
this
is
since
you're
you're
managing
your
own
system.
That's
like
this,
but
we're
providing
access
to
a
pool
of
resources.
You
can
spin
up
portions
of
that
resource.
However,
you
want-
and
you
can
do
this
using
devops
principles
and
things
like
infrastructure
as
code
version
control
automation.
B
A
B
Your
researchers
and
the
engineers
will
deploy
something
like
a
apache
or
tomcat,
or
something
like
that
and
they'll
try
to
make
that
accessible
to
the
developers
to
deploy
their
code
or
deploy
a
scientific
application
on
it.
And
since
the
engineering
team
has
root,
access,
they've
got
the
keys
to
the
castle
and
they
don't
want
to
give
the
developers
that
access
you
get
into
this
situation.
Where
there's
a
lot
of
back
and
forth
between
developers
and
operations
where.
A
C
C
A
B
Yeah,
okay,
I
it's
funny.
Last
time
we
gave
a
a
workshop
here
I
got
my
the
power
went
out
of
my
house
too.
So
every
time
I
talk
about
this
stuff,
something
happens
so
yeah.
So
it's
it's
a
platform
that
people
can
use
seamlessly
and
they
don't
have
to
interact
with
engineering
every
time
they
want
to
do
so,
make
some
little
change
to
their
system.
So
this
is
a.
This
is
sort
of
a
vision
for
the
the
platform.
B
You
can,
you
know,
build
a
an
image
that
contains
your
software
and
the
environment
to
run
that
you
can
store
that
either
in
a
public
or
private
registry.
You
know
you
can
use
docker
hub
github
or
we
have
a
a
private
registry
that
people
can
access.
You
can
incorporate
version
control
there
and
then
you
can
deploy
with
our
container
orchestration
systems,
which
we
use
rancher
to
manage
downstream
kubernetes
clusters,
so
use
kubernetes
to
to
deploy
the
application
use
kubernetes
to
open
up
api,
endpoints
or
ports.
B
B
Currently,
the
first
is
a
getis
composable
platform,
and
so
we
name
all
of
our
resource,
all
the
resources
we
have
after
purdue
faculty
or
alumni
that
have
you
know,
made
an
impact
here
at
the
university,
so
the
gettys
platform
is
named
after
lanell
gettus,
who
was
department
head
of
the
school
of
nursing
here
for
many
years.
So
this
is
our
central
community
cloud
system
for
general
purdue
research
use,
and
this
one
you
can
do
these
supporting
services.
We've
talked
about.
B
So
the
remainder
of
the
talk
focuses
on
this
implementation,
but
we
also
have
the
anvil
composable
subsystem,
which
is
it's
really
an
equivalent
to
the
spin
resource.
In
my
mind,
in
that
it's
intended
only
to
complement
batch
workflows
that
run
on
the
anvil.
Hpc
system,
currently
don't
do
large
scale
computing
there,
but
in
maybe
year
two
of
the
anvil
project.
We
might
try
to
incorporate
anvil
compute
nodes
into
the
kubernetes,
so
we
can
scale
out
data
analysis
workflows
there
and
we
also
have
the
archie
healthcare
analytics
platform.
C
C
So
I'll
start
off
with
giving
a
high
level
overview
of
the
getus
platform
so
for
the
physical
compute,
we
are
leveraging
dell
poweredge
servers,
amd's
7662
cpus,
and
we
have
a
small
subset
of
gpu
nodes
that
are
utilizing.
The
nvidia
a100s,
which
we
specifically
picked
due
to
them.
Having
something
called
a
multi-instance
gpu
feature
that
I'll
go
into
more
detail
further
down
in
about
two
two
more
slides
here
for
storage.
C
C
I
believe
eric
just
alluded
to
some
type
of
integration
with
community
cluster
integration
as
planned.
We
have
a
roadmap
for
this.
This
is
something
that
we're
currently
working
on
at
this
moment,
but
essentially
we'd
like
to
be
able
to
dynamically,
add
and
remove
nodes
to
the
kubernetes
clusters
that
we
build
from
our
community
clusters
idle
community
cluster
nodes.
C
C
Scd
has
their
own
physical
servers
and
the
control
planes
have
their
own
physical
servers
for
overall
storage,
we
offer
we
have
384
terabytes
of
spinning
disk
672
terabytes
of
sata
ssd
and
192
terabytes
of
nvme,
which,
as
of
right
now,
I
think
we
have
in
the
works
here
pretty
soon
to
add
an
additional
one.
Petabyte
of
spinning
disk
storage
that
my
colleague
sam
here
will
probably
go
over
in
more
detail
later
on.
C
We
have
eight
overall
nvidia
a100s
again
with
the
with
the
big
multi-instance
gpu
feature,
enabled
physical
networking.
We
have
25
gigabit
ethernet
straight
to
the
workers,
100
gigabit,
ethernet
straight
to
the
storage
servers
and
200
gigabit,
aggregated
uplinks
to
the
core
networks.
We
also
plumb
in
private
and
public
vlans
that
we
utilize
for
the
load
balance
services
that
we
deploy
via
mail
to
lb
that'll,
go
in
a
little
bit
more
detail
or
showcase
in
the
next
slide
or
two
and
our
entire
kubernetes
infrastructure.
C
C
C
So
we
do
utilize
rancher
we
were
talking
about
this
earlier.
Rancher
runs
on
his
own
three
node
tributes
cluster
as
an
upstream
cluster
and
from
there
that
cluster
is
used
to
provision
all
of
the
user
downstream
clusters.
So
get
us
in
this
situation
and
or
possibly
others
we
have.
We
have
tested
deploying
a
secondary
cluster
that
we
used
with
decommissioned
hardware
just
to
test
how
to
backfill
workloads.
C
If
that
comes
to
a
need
that
we
need
to
to
attest
for
so
we
use
a
centralized
authentication
with
shibboleth
and
that's
tied
into
purdue's
two-factor
authentication
system
and
the
overall
authorization
is
done
and
integrated
via
something
called
halcyon,
which
is
our
web-based
cluster
management
tool.
So
the
idea
behind
this
is
users
will
be
able
to
go
to
our
web-based
cluster
management
tool
and
add
remove
users
as
they
see
fit
to
their
projects.
C
C
So
this
gives
us
the
means
of
being
able
to
give
them
a
private
registry
means
for
their
own,
their
own
groups
and
bypasses
the
need
of
you
know.
Most
of
the
solutions
online
for
private
registries
cost
some
type
of
money.
So
this
is
the
resource
they
get
for
free
for
buying
in
and
opting
into
the
get
us
composable
infrastructure,
and
this
is
also
where
we
house
docker
docker
hub
cache.
C
So
we
also
learned
pretty
early
on
actually
the
hard
way
that
docker
hub
has
rate
limits
to
how
many
times
you
can
pull
images
per
six
hours.
I
believe
it's
a
hundred
images
per
hour
or
per
six
hours
and
after
that
they
they
actually
block
the
address
that
has
pulled
across
that
limit.
So
we
actually
ended
up
getting
our
entire
entire
net
address
pool
banned
from
docker
hub
at
one
point.
So
this
is
something
that
we
had
that
we
knew
we
had
to
implement
and
we
actually
did
end
up.
C
Implementing
and
harbor
is
a
really
easy
tool
to
use.
It
was
really
easy
to
do
and
yeah
so
docker
hub
cache
on
that
end.
For
the
kubernetes
configuration
portion
for
our
container
network
interface,
we
use
canal,
which
is
a
mixture
of
flannel
and
calico
for
the
network
policies,
and
this
is
the
layer
that
really
helps
us
implement
that
that
rancher
project
level,
isolation
between
workloads,
we
utilize
middle
lb
for
load,
balancing
services.
C
We
utilize
k8's
gateway,
which
is
a
core
dns
plugin
for
ingress
and
our
load.
Balancer
dns
records,
so
user
users
are
able
to
supply
any
host
name
or
any
name.
They
want
for
their
applications
ingress
as
long
as
it
ends
in
the
dot
get
as
domain,
and
that
dns
record
will
automatically
be
provisioned
and
resolvable
after
that.
C
C
This
means
that
we
don't
have
to
do
any
of
that,
physically
by
building
images
with
predetermined
versions
and
controlling
all
the
version
control
through
the
image
builds.
This
is
done
through
cremates
itself,
so
I
keep
saying
mig
big
multi-instance
gpu
if
to
clarify
what
that
means,
it's
a
feature
via
the
a100s
that
allows
you
to
take
a
physical,
gpu
and
segment
it
into
seven
separate
instances
of
a
gpu,
and
this
really
helps
you
bypass
the
typical
scenarios.
C
C
C
So
overall
we
have
a
total
of
32
gpu
profiles
and
compared
to
having
eight
physical
gpus.
You
know
where
we
have
eight
physical
gpus
that
we're
able
to
overall
split
into
individual
profiles
and
allow
up
to
32
overall
users
to
have
gpu
access
now,
instead
of
eight,
and
these
numbers
are
easily
changeable.
If
we
find
that
the
whim
of
gpu
usage
changes-
and
we
can-
we
can
associate
and
build
new
profiles
for
that.
C
We
do
have
a
profile-
that's
the
entire
gpu
itself,
so
we
just
because
we
can
segment
the
gpus.
We
definitely
wanted
to
be
in
a
situation
where
we
are
still
able
to
supply
full
gpu
capabilities
to
to
to
researchers
if
they
need
so
and
down
below
is
just
a
quick
snippet
of
of
kind
of
how
it
how
easy
it
is
to
actually
opt
in
and
pulling
these
particular
gpu
profiles
under
resource
limits.
C
You
just
specify
the
tag
for
that
particular
profile
and
how
many
of
those
gpus
you
want
and
then
under
node
selector.
You
would
utilize
a
label
that
is
assigned
via
the
gpu
operator
itself
to
determine
that
that
physical
node
has
has
a
actual
gpu
in
it,
and
we
do.
This
is
a
bit
nuanced
and
it
may
be
out
of
scope,
but
we
do
isolate
our
gpu
nodes
via
taints,
so
that
normal
cpu
workloads,
don't
if
they're
not
utilizing
the
gpus,
don't
spawn
the
gpu
nodes
themselves.
So
I
believe
that's
it.
For
this
slide.
C
So
security-
and
I
I
touched
a
little
bit
on
most
of
these
throughout
what
I
just
talked
about,
but
we
do
have
integration
with
the
campus
single
sign-on
role-based
access
control
again
through
halcyon,
and
this
is
kind
of
left
up
to
the
users
to
decide
who
they
want.
You
know
access
to
their
name,
spaces,
their
projects,
the
resources
inside
their
projects
and
that's
all
managed
via
ranchers
api
system
via
halcyon
pod
security
policies
are
in
place
to
prevent
privileged
escalation
in
general
project
level.
C
Network
isolation
that
I
touched
on
a
little
bit
through
our
cni
network
container
security
scanning
via
harbor,
so
an
additional
feature
to
harbor
is
it
allows
us
to
scan
for
vulnerabilities
we've
used
that
once
or
twice
already
and
monitoring
by
our
central
computer
ids
system.
So
a
few
years
ago
we
had
built
a
system
called
pulsar
that
is
an
ids
system
that
monitors
all
of
the
research
network
in
and
out
of
the
research
network
connections
which
this
particular
resource
lives.
C
D
D
D
It's
also
made
upgrades
and
things
significantly
easier
out
of
the
storage
services
that
we've
offered
a
lot
of
users
have
opted
to
try
out
the
s3
storage.
That
seems
really
popular.
We
expect
that
when
we
expand
our
disaster
recovery
and
backup
cluster,
that
we
will
be
adding
more
s3
storage
to
that
as
more
of
a
general
campus
resource
to
complement
the
users
using
these
platforms
as
well
as
provide
that
more
broadly,
can
you
advance
this
light
again,
all
right
so
for
the
application
deployment?
We
we've
done.
D
We
had
two
students
from
the
university
of
washington
and
they
helped
us
develop
a
lot
of
these
we're
calling
them
like
one
click
or
push
button,
applications
that
are
available
in
the
gui
to
users
who
don't
need
or
want
to
to
write
their
own
code
for
things.
Another
benefit
of
this
is
it.
It
allows
researchers
to
write
applications
that
are
accustomed
to
the
cluster
that
their
group
is
running
and
provide
an
easy
way
to
share
that
amongst
their
colleagues.
D
D
This
is
handy
we've
seen
instances,
for
example,
where
maybe
a
researcher
has
a
small
website
that
has
been
living
on
a
random
place
for
some
amount
of
time
and
hasn't
seen
a
lot
of
traffic
and
then
say
they
go
to
a
conference
and
give
a
presentation.
Then
suddenly
2000
people
go
and
check
out
this
website
that
that
has
had
relatively
low
traffic
in
the
past.
D
Things
like
that
are
really
useful,
because
this
auto
scaler
can
see
that
load
coming
in
and
dynamically
allocate
more
instances
to
help
keep
the
service
happy
and
when
you
deploy
with
this
they're,
they're,
typically
dynamic
assets.
So
so
the
application
is
aware
of
this
being
present
advanced,
please
thanks,
and
so,
as
eric
mentioned
earlier,
we've
had
this
in
production
for
some
time
now
we
have
we've
so
far,
as
I
should
say,
as
a
beta
test
system
as
we
move
into
production.
B
Yeah,
so
let
me
go
over
a
few
of
the
use
cases
we've
seen
over
the
past
over
the
past
year,
so
the
the
data
mine
is
one
of
these
residential
learning
initiatives
that
that
is
happening
at
purdue,
where
students
are
are
introduced
to
data
science
concepts
and
they
they
live
in
this
research-based
community.
Actually,
everybody
lives
in
the
same
dorm
room
and
they
have
advisors
and
faculty
visits
to
this
particular
residence
hall.
B
B
They
had
an
application
called
swat
chair
that
our
engineering
team
was
hosting
in
a
in
a
virtual
machine
and
if
you
think
back
to
that
developer
operations
thing-
I
talked
about
previous.
This
is
the
this.
Was
that
so
red
hat
six
vm
managed
by
our
engineering
team
developers,
don't
have
a
lot
of
access
to
it
and
there
was
a
lot
of
a
lot
of
back
and
forth
and
they
went
through
and
redesigned
this
with
containers
on
gettys
and
migrating
from
this
traditional
deployment
to
a
devops
model.
B
The
system
is
also
integrated
into
our
cms
tier
2
analysis
facility,
so
this
is
for
data
analysis
on
the
cms
experiment.
At
the
lhc
we
have
swann,
which
is
a
notebook
provisioner
service
and
cernbox,
which
is
data,
storage
and
data
sharing
integrated,
so
cms
researchers
can
use
jupyter
notebooks
to
either
scale
out
analysis
on
the
kubernetes
or
they
can
interact
with
our
community
clusters
to
scale
out
analysis
there
and
also
interact
with
some
some
public
cloud
services.
B
We
have
a
group
called
iron
hacks
that
have
developed
a
kaggle-like
platform
for
these
virtual
data,
science,
hackathons
and
challenges,
and
they
had
a
cloud-based
platform
that
was
running
in
digitalocean.
That
was
providing
this
service
as
the
hacks
got
bigger,
the
data
sets
got
bigger
and
they
needed
access
to.
You
know
different
different
computing
computing
things
like
gpus.
B
They
found
that
their
current
cloud
provider
couldn't
give
them
those
services.
So
they
talked
to
us
about
incorporating
the
gpus
in
gettys
to
to
do
a
hack
that
used
more
ai
or
ml
techniques
on
larger
data
sets,
so
they
migrated
their
jupiter
hub
from
the
public
cloud
in
status.
Brian
worked
with
them
to
integrate
the
gpus
into
the
containers
and
they
will
use
this
for
a
gpu
based
ai
hackathon
in
in
april.
It's
coming
up,
so
we're
excited
to
see
how
how
that
goes.
B
So
reo
is
the
automated
reconnaissance
image
organizer
and
you
can
think
of
a
reconnaissance
team
going
out
to
the
site
of
a
natural
disaster
taking
pictures
and
sending
these
pictures
back
to
a
service
that
would
automatically
classify
them.
You
can
kind
of
see
some
of
the
different
classifications
that
happen
here
so
image,
location
component
type,
damage
level
and
things
like
that
so
get
us
can
be
used
to
expose
api
endpoints
the
models
that
are
doing
this
classification-
and
this
was
implemented.
B
I
think,
sometime
early
on
in
the
in
when
gettys
was
deployed
and
they've
done
cpu-based
prediction
and
are
going
to
switch
to
incorporating
the
gpus
and
the
the
triton
server
sometime
in
the
near
future,
and
then
this
reo
application
that
currently
runs
in
somebody's
on
a
system
in
somebody's
research
lab
will
be
transitioned
over
to
us
as
well.
So
that's
like
the
web
web
front
end
that
can
do
management
of
the
data
sets
that
are
that
are
created.
B
One
of
the
things
that's
become,
maybe
the
most
popular
thing
is:
is
this
concept
of
a
personal
science
gateway?
These
are
a
lot
of
times
deployed
either
using
voila
or
are
shiny,
and
what
we
see
is
that
people
want
to
provide
access
to
data
sets,
but
they
also
want
to
provide
access
to
some
to
to
give
the
a
person
the
ability
to
do
some
sort
of
data
analytics
on
that
data
set
as
well.
B
B
They
would
provide
a
data
set
to
the
customer,
and
then
there
would
be
a
lot
of
back
and
forth
between
the
bioinformatician
and
the
customer.
Can
you
tweak
this?
Can
you
tweak
this?
Let
me
see
what
it
looks
like
this
way
and
eventually
they
decided
hey.
I
can
provide
these
things
via
our
shiny
app
and
I
can
give
the
customer
a
data
set
and
allow
them
to
do
all
this.
B
B
It
can
be
very
easy
to
deploy
things
in
kubernetes,
but
the
you
know:
building
your
troubleshooting
skills
in
this
type
of
environment,
a
lot
of
the
tools
break
down
things,
you've
been
relying
on
forever,
like
ping,
don't
no
longer
make
sense,
and
so
getting
your
staff.
You
know
familiar
with
the
the
platform
and
familiar
with
how
to
troubleshoot
the
platform
can
can
be
a
bit
of
a
can
be
a
bit
of
a
challenge,
and
some
work
needs
to
be
put
into
that
this.
This
environment
is
rapidly
involving
rapidly
evolving.
B
So
keeping
those
ftd
snapshots
somewhere
else
and
backed
up
is
definitely
a
good
idea
and
finding
the
right
balance
between
flexibility
and
security
in
an
environment
like
this
can
be
difficult.
A
lot
of
applications
are
written,
so
people,
so
they
expect
they
expect
people
to
have
a
their
own
kubernetes
cluster,
where
they
have
some
sort
of
administrative
privileges,
and
in
our
case
we
have,
to
you,
know,
clamp
down
some
of
that
access,
because
we're
running
a
multi-tenant
environment,
so
finding
that
balance
is
is,
can
sometimes
be
a
challenge
on
the
user
support
side.
B
Training
is
important.
Providing
tutorials
and
workshops
is
very
important,
and
you
know
users
will
leave
random
things
up
and
they
will
forget
about
them.
So
it's
good
to
sort
of
scan
what's
running
in
your
system
and
sort
of
take
inventory
of
what's
what's
being
made
available
to
to
the
the
internet,
especially
so
we're
trying
to
evaluate
the
gettys
cost
model
versus
public
cloud.
So
people
would
buy
into
this
buy
into
this
platform
at
a
subscription
fee,
which
is
300
for
a
four
core
and
16
gigabyte
chunk
per
year.