►
From YouTube: Towards Trading on Kubernetes: Operating Multi-Tenant and Se... - Andrew Kochut & Javier Diaz-Montes
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Towards Trading on Kubernetes: Operating Multi-Tenant and Secure Clusters - Andrew Kochut & Javier Diaz-Montes, Two Sigma
Two Sigma, a financial company, performs large-scale data processing for modeling and trading while facing risks, such as data exfiltration. We present how we addressed this by building multi-tenant Kubernetes clusters to run over 500 services on 30K cores and 200TB of RAM. These include parts of our trading system and our document translation system, build and test farms, and artifact caches. Kubernetes doesn't provide full tenant isolation so users often create per-tenant clusters. Two Sigma has many teams with unique data and service access needs, so such a model would have large overheads. We built multi-tenant clusters by coupling namespace, RBAC and PSPs with Two Sigma’s entitlement system. We also integrated Kerberos via annotations to inject tickets, keytabs, and SSL certs into Pods. We discuss lessons operating this both on-prem and public cloud, including pros and cons of GKE.
To learn more: https://sched.co/GrSA
A
Hello,
thank
you
very
much.
First
of
all
for
choosing
our
presentation,
it's
our
honor
and
well.
We
have
great
conference
this
year
right
so
Xavier
and
hi.
My
name
is
Andrew
will
will
not
tell
you
much
about
new
features
or
new
capabilities.
Instead,
we
will
kind
of
report
on
our
journey
with
kubernetes
that
started
couple
of
years
ago.
A
A
Just
that's
our
standard
disclaimer.
This
should
be
not
treated
as
anything
more
than
education.
I
won't
make
you
read
all
of
that.
Okay,
so
who
we
are
actually
so
two
sigma.
It's
a
systemic
investment
manager
which
most
people
know
as
hedge
funds,
based
in
New
York.
The
film
was
founded
by
essentially
a
pair
of
engineer
and
a
mathematician
that
was
back
in
roughly
2000,
so
we
are
17
years
old
and
since
then,
since
the
founding,
the
firm
grew
quite
a
lot.
A
We
have
more
than
1,200
people
now
and
that's
kind
of
between
engineering
research
and
some
business
functions
in
supporting
call
of
debt.
So
we
see
ourselves
actually
not
as
much
as
a
financial
company
that
you
could
think
of,
but
more
as
a
platform
company
that
builds
solutions
and
approaches
to
essentially
fulfill
our
mission,
and
the
mission
is
kind
of
simple
discover
value
in
the
world's
data.
So
that's
what
we
ventured
out
to
be,
and
of
course
the
first
main
application
is
is
is
finance,
but
it's
not
the
only
one,
so
2
Sigma
also
has.
A
Insurance
branch
that
pretty
much
attempts
to
apply
analytics
to
insurance,
business
and
more
will
come
I
am
sure,
as
the
film
matures
and
the
fact
that
we
are
really
a
technology
companies.
These
numbers,
more
than
70%
of
our
employees,
actually
do
not
come
from
any
financial
background
whatsoever.
So
this
is
a
combination
of
engineers
and
and
researchers.
Participation,
and
you
know
all
kinds
of
folks
who,
from
various
walks
of
life
but
rarely
from
finance
and
essentially
building
platforms,
is
what
we
believe
will
allow
us
to
scale
how
to
apply
analytics
across
the
board.
A
So
we
are
a
financial
sector
company.
So
what's
unique
about
that,
of
course,
we
are
under
significant
pressure
to
meet
security,
resiliency
compliance
requirements
and
essentially
meet
our
responsibilities
for
investors,
who
trust
us
with
our
money
and
and
that's
the
primary
thing
that
we
have
to
care
about.
A
Last
but
not
least,
we
cannot
be
you
know
proverbial
asleep
at
the
wheel.
At
the
same
time,
we
have
to
remain
agile
and
while
having
all
of
these
requirements
about
resiliency
and
security,
and
all
of
that,
we
still
have
to
keep
our
eyes
open
to
remain
competitive,
essentially
to
be
able
to
deliver
new
versions
of
analytics
process
data
faster
all
of
these
things,
so
this
is
kind
of
a
trade-off
that
we
end
at
in
line.
We
always
have
to
work
how
to
marry
all
of
these
aspects
of
our
mission.
So
why
kubernetes?
A
Well
so
before
that
we
had
pretty
successful,
run
with
private
cloud
based
on
OpenStack,
and
so
it
obviously
brought
a
lot
of
advantage
to
our
film
to
our
engineers
and
let
us
scale
much
better,
but
of
course
it
fell
short
in
many
respects.
That
kind
of
the
key
ones
are
listed
here.
So
we
did
not
grow
a
significant
ecosystem
of
cloud
native
apps
because
really
OpenStack
didn't
encourage
that
truly,
it
kind
of
more
brought
in
our
traditional
way
of
building
systems
with
it.
A
We
ended
up
with
significant
sprawl
of
VM
images,
VMs,
which
really
were
not
are
not
as
easy
to
manage
as
we
would
like,
and,
as
you
can
imagine
in
many
types
of
our
workloads,
we
actually
do
care
about
performance,
especially
both
data
rates.
Latencies
and
virtualization
is
not
the
best
still
even
today.
For
these
reasons-
and
we
see,
kubernetes
is
essentially
a
orchestration
platform
and
also
software
deployment
mechanism
that
can
actually
address
all
of
these
things.
A
So
our
first
target
was
essentially,
let's
see
whether
we
can
develop
and
build
our
code
test,
our
code
on
kubernetes
at
scale,
whether
that's
gonna
work
right,
so
that
was
kind
of
initial
objective
throughout
early
to
meet
2017,
and
only
after
we
gained
some
confidence
around
it.
We
went
into
scaling
it
out
a
little
bit
more,
so
we
actually
g8
within
two
sigma.
A
Of
course,
our
offering
opening
up
the
door
to
more
than
just
very
selected
use
cases
build
out
a
little
bit
more
capacity,
even
though
we
are
not
very
aggressive
about
that,
because
we
still
see
public
cloud
and
kubernetes
has
kind
of
a
unifying
platform
between
public
cloud
and
on-premise
as
a
way
to
go.
So
we
are
not
kind
of
thinking
of
investing
too
much
into
into
on-premise
capacity
and
then,
since
March,
we
kind
of
took
on
way
more
types
of
use.
A
Cases,
including
some
that
are
more
sensitive
ones
as
well
as
started
a
little
bit
of
alpha
type
of
applications
for
our
our
trading
systems,
and
this
is
very
early.
Obviously,
we
are
essentially
trying
to
see
to
what
extent
kubernetes
is
really
the
right
answer
for
fleet-wide
process
and
software
deployment
across
those
via
a
variety
of
things.
So
this
is
kind
of
where
we
are
and
I'll
hand
it
over
to
Javier.
To
tell
you
a
little
bit
more
about
how
we
how
we
came
here
right
so.
B
Basically,
as
Andrew
mentioned,
like
security
is
pretty
important
for
us,
so
this
translate
into
having
stringent
security
constrained
within
the
frame
right
like
there
is
no
general
root
access
for
our
engineers,
not
even
in
container.
We
are
limiting
that
the
users
can
only
be
specific
role
accounts,
so
a
role
account
is
something
it
looks
something
like
this
or
you
basically
have
liked
useful,
which
is
these
Linux?
B
U
UID
and
GID,
and
for
example,
in
this
case
it
can
be
used
by
Andrew
myself,
and
these
fou
is
member
of
the
group
this
bar
and
this
bar
right.
So
these
are
kind
of
like
a
part
of
our
central
management
system
that
we
will
have
to
make
sure
that
we
don't
bioW
late
when
movie
people
out
into
corner
these.
So
like
we
also
sundry
mention
like
we
have
large
number
of
Virginia's
applications
of
limited
size,
some
of
them
are
large,
but
in
general
we
have
very
true
genius
applications.
B
At
the
same
time,
like
we
had
to
integrate
like
all
these
platform,
with
our
entitlements
for
access
control,
as
well
as
with
cameras
for
authentication.
This
is
kind
of
normal
in
finance,
so
like
something
special
about
us
is
like
we
have
for
one
reason
or
another.
We
came
up
to
a
point,
but
we
have
like
large
size
of
bill
artifacts
whenever
we
want
to
deploy
some
software,
like
the
solar
itself,
with
all
these
dependencies
plus
runtimes
and
that's
being
like
in
the
order
of
gigabytes.
B
B
Because
people
don't
have
root
in
the
company
like
they
couldn't
build
doctor
containers,
so
what
we
they're
doing
is
creating
like
a
building
a
building
pipeline
that
will
allow
people
to
basically
specify
what
they
want
to
have
on
their
containers,
and
then
we
will
go
and
build
it
for
them
and
medic
it
available
to
them
right,
so
they
can
specify
like
here.
We
have
a
simple
example
of
how
this
manifests.
Look
like
where
you
can
say:
hey,
I
want
to
be
based
on
Debian
stretch
and
I
want
to
have
these
deployments.
B
That
is
associated
with
the
code
bases,
and
this
is
the
name
of
my
specification
right.
We
also
took
the
concept
of
namespace
from
cornetist
in
a
way
that
we
will
map
it
namespaces
between
our
building
pipeline
and
coordinate,
ease.
The
way
that
we
manage
this
large
artifacts
today
is
like
like
us,
as
the
diagram
shows
like
we
have.
Builders
that
are
in
each
data
center
could
be
on-prem
could
be
on
public
cloud,
and
then
we
will
drop
these
artifacts
into
distributed,
store
file
system
on
Prem.
We
are
using
like
a
safe
public
cloud.
B
We
are
using
like
Google
persistent
disks
right,
so
the
idea
is
that
we
pay
the
cost
of
unloading
everything
once
we
make
it
there,
then
we
snapshot
it
to
make
sure
this
read-only.
So
we
can
still
keep
the
immutability
of
containers
and
then
at
runtime.
We
we
attach
it
to
the
containers.
In
that
way,
you
are
like
the
container
start.
Time
is
pretty
low,
because
the
base
image
is
kept
small,
a
runtime.
We
also
had
to
make
some
modifications.
B
This
is
a
simplified
version,
so,
basically,
because
of
Kerberos,
we
had
to
translate
a
Kerberos
identity
into
something
and
kubernetes
understands.
So
we
built
assist
series
that
uses
the
kubernetes
webhook
authentication,
and
this
service
is
kerberized,
so
we
translate
a
curved
arrows
identity
into
a
very
token
that
then
is
sent
to
the
API
server
and
then
API
server
is
the
line
is
missing
here,
but
the
API
server
will
go
back
to
this
authentication
service
and
ask
okay.
B
Is
this
token
value
I
mean
if
you
saw
it
like
through
so
another
thing
that
we
have
on
top
of?
These
is
a
manifest
service.
We
call
so
basically
Cornett
is
Jamel
files.
We
modify
a
little
bit
the
image
field
to
kind
of
make
reference
to
the
previous
system
where
we
built
our
containers.
So
we
can
say:
hey
I,
want
to
use
this
image
from
their
namespace
contest
and
the
name
of
the
image
is
demo
flash
up,
and
then
this
manifest
service.
What
it
does.
It
call
goes
to
the
system
find
out.
B
What
is
the
docker
image
find
out?
What
are
the
octave
fact
that
needs
to
be
attached
to
the
system
and
they
enhance
the
the
Jamel
file
that
the
user
has
provided
with
all
these
volume
mounts
additional
additional
things
to
make
sure
that
things
like
Kerberos
work
properly
inside
things
like
SSD
for
Linux
identities
also
work
right,
so
once
it
goes
there,
the
API
server
will
create
pulses.
B
Normal
I
mean
if
the
user
has
provided
a
notation
saying
that
they
want
any
Kerberos
credential
like
hey
I
want
to
have
available,
make
ever
stikat,
so
I
can
talk
to
kerberized
services
or
I
want
to
have
available
key
taps,
because
I
am
a
service
that
is
going
to
accept,
request
against
myself
and
is
using
cameras,
so
the
couplets
will
basically
go
and
retrieve
dot.
Those
artifacts
keep
material.
B
So
here
like
how
do
we
enforce
like
users,
identities
right?
So
we
are
trying
to
build
a
multi-tenant
system
and
we
want
to
make
sure
that
again
we
don't
violate
what
our
central
like
corporate
identity
system
is
saying
right.
So
what
we
did
is
we
use
our
back
rules,
so
we
are
first
mapping
every
namespace
to
a
specific
role
account
or
identity
and
within
the
namespace
we
only
let
them
be
the
user
that
they
should
be,
and
then
like.
We
do
this
like,
for
example,
with
the
name.
B
A
space
is
full
they're
going
to
have
a
role
binding
connecting
to
the
cluster
role,
edit,
so
the
user
TS
who
can
operate
in
this
full
name
space,
and
then
we
have
a
cluster
or
binding
time,
TS
full
to
a
PSP
that
we
have
created
like
a
possibility
policy
and
then
the
possibility
policy.
Basically
we
say:
hey:
this
user
can
only
be
this
Linux
UID.
B
This
user
has
to
be,
can
only
have
these
supplemental
groups
and
that's
how
we
are
ensuring
and
then
the
additional
piece
to
make
sure
that
they
don't
overt
overcome
this
going
around
us
is
to
have
an
admission
controller
to
ensure
that
the
security
context
run
as
user
is
set
in
the
manifest.
In
that
way,
we
are
making
sure
that
if
somebody
is
trying
to
use
a
user
ID
that
is
not
allowed
for
him
to
use
within
an
image
space,
then
they
get
rejected
right.
So
basically,
Paul
won't
start.
B
We
also
build
like
system
to
synchronize
kubernetes
with
our
cooperative
DC
system.
So
basically,
we
have
some
soup
subscription
mechanism
that
whenever
we
identify
that
there
are
changes
like,
for
example,
this
fool
has
been
added
to
a
new
member
is
as
being
added
to
a
new
group,
so
we
will
detect
these
and
update
kubernetes
or
if
it
has
to
be
removed
from
a
group.
We
will
do
that
or
like
in
this
case,
if
we
add
or
remove
people
that
can
use
this
name
a
space,
we
will
also
synchronize
right
in
this
case.
B
Only
unreal
and
myself
will
be
able
to
run
containers
on
this
namespace
because
we
are
the
only
ones
that
can
log
in
as
that
enemy
space
next
trying
to
describe
like
a
little
bit
without
entering
too
much
detail.
What
is
was
our
first
attempt
to
integrate
his
camera
to
integrate
Kerberos
within
qualities,
so
basically
the
user
interface.
B
We
try
to
keep
it
very
simple,
so
we
use
like
a
kubernetes
annotation
to
say:
hey
I
want
to
have
these
key
tabs,
I
want
to
have
tickets
or
certificates,
and
then,
under
the
hood,
we
made
a
few
modifications
to
kubernetes
to
make
sure
that
we
can
translate
these
into
depositing
decry
key
material
in
the
right
place
right.
So
here
in
this
figure,
we
have.
B
The
orange
parts
is
where
we
made
modifications
like,
for
example,
the
scheduler,
what
it
basically
does
whenever
users
say:
hey
I
want
to
have
my
ticket
Prestage
on
my
container,
so
this
is
scheduler
is
going
to
get
encrypted,
put
it
into
the
manifest
of
a
container,
send
it
over
and
then
the
couplet
once
it's
received
this
manifest
with
decrypted
and
make
sure
that
is
in
the
right
place
in
the
container.
That
has
to
be
so.
B
Similarly,
if
the
user
says
I
want
certificates
or
kidnaps,
the
couplet
will
register
itself
or
create
the
Kerberos
logical
host
and
as
for
the
key
materials
negotiated
with
curved
arrows
and
all
that,
make
sure
that
everything
is
in
the
container
and
we
have
additionally
a
controller
that
is
going
to
make
sure
that
everything
is
up
to
date
and
unfresh
right
like
a
new
certificate
or
the
key
material
like
for
for
curved
arrows.
So
these
mostly
works.
B
B
B
Yeah
so
and
then,
finally,
like
we
have
exposed
our
number
of
issues
related
to
the
scalability
of
our
cameras
infrastructure.
So
we
been
having
a
hard
time
to
keeping
up
I
mean
security
has
been
having
a
hard
time,
keeping
up
with
your
native,
so
we
had
to
basically
implement
some
rate
limiting
before
they
got
up
to
speed,
to
make
sure
that
we
didn't
take
down
their
infrastructure.
B
So
that
was
the
first
right
so
now
like
we
are
working
on
our
second
attempt.
Basically,
what
we
are
trying
to
do
is
extract
everything
out
of
kubernetes
source
code,
so
we
are
not
tied
to
versions
and
it's
so
hard
to
upgrade.
So
we
are
rebuilding
everything
out
of
three,
so
the
mechanisms
are
similar.
It's
just.
B
The
only
thing
to
notice
is
that
we
are
leveraging,
in
this
case
flex
volumes
to
make
sure
like
we
can
ask
for
the
key
materia
from
the
worker
nodes,
and
the
other
change
is
that
now
we
are
centrally
going
to
manage
Kerberos
cluster
memberships
from
the
from
the
control
plane
front
from
the
controller.
A
few
machines
that
will
be
the
control
now
like,
for
example,
could
be
coordinating
masters
in
that
way.
B
The
worker,
the
only
thing
that
needs
to
do
like
a
run
time
when
it's
going
to
attach
the
directory
with
a
flex
model
in
the
Flex
volume,
will
run
a
set
of
commands
to
get
the
key
tabs
or
tickets
or
whatever
it
needs
to
be
in
case
that
users
decide
that
they
wanted
that
yeah.
So
and
then
we
probably
consider
two
open
sources
at
some
point,
so
anybody
can
take
advantage
of
that.
B
B
We
are
working
to
expand
to
public
cloud,
but
it's
still
in
the
early
stages.
So
we
have
on
board
like
us
a
few
like
critical
infrastructure.
Like
our
billing
test
form,
we
have
integration
testing.
We
have
some
distributed
memory,
cache
for
serving
artifacts.
We
have
some
components
of
the
training
system
as
mostly
like
a
post
rating,
and
then
we
have
other
workloads
like
as
part
drivers
or
some
translation
systems
like
HR
has
a
bunch
of
his
stuff.
B
They
have
their
own
dedicated
class
that
we
mentioned
before
so
so
in
terms
like
a
sim
farm
like
this
is
kind
of.
Besides
we
are
talking
today
like
we
are
building
around
250,000
bills
per
day
and
testing
an
average
of
3
million
unit
tests,
and
then
we
generate
and
serve
I
mean
we
serve
around
hundred
terabytes
of
artifacts
per
day.
And
again
this
comes
back
to
the
bloated
artifacts
that
we
have
today,
because
our
dependencies
so
the
benefit.
B
The
main
benefit
for
this
team
has
been
that
before
maintenance
was
a
pain
they
had
to
maintain
multiple
versions
of
the
operating
based
operating
system
in
different
parameter
machines.
They
had
to
themselves
like
a
hundred
at
infrastructure.
They
have
issues
with
noisy
neighbors,
they
didn't
have
proper
isolations
and
they
also
cooling
a
scale
because
they
were
a
nimble
right
like
they
were
very
slow
moving,
because
bare
metal,
so
cornetist
kind
and
in
general
container,
is
like
what
it
gives
the
option
to
solve
many
of
these
problems.
B
So
they
are
pretty
happy.
So
the
other
use
case
is
continuous
integration.
So
we
have
Jenkins
based
continuous
integration.
We
had
two
main
use
cases,
one
where
different
teams
get
their
own
Jenkins
to
basically
define
whatever
pipelines
they
want
to
create
for
integrations,
and
then
we
have
a
kind
of
big
jenkins
for
the
whole
firm
to
run
like
automated
post
push
tests,
integration
tests
right
so
so
here
like
we
have
in
the
like,
centralized
Jenkins
for
everybody.
B
We
have
around
5,000
pipelines
that
it
tests
around
300
around
30,000
tests
a
day
right
so
and
these
like
basically
give
them
the
flexibility
to
very
easily
like
hand
out,
like
jenkies
instances
to
people
and
easily
to
scale
them
up
and
down,
because
basically,
workers
are
being
started
as
containers.
So
it
gives
you
a
lot
of
power.
B
So
we
have
a
few
challenges
here,
like
drinking
doesn't
handle
where
it
doesn't
handle
very
well
being
terminated
because
it
loses
the
state
and
kill
everything
and
it
doesn't
recover
work
well,
so
Jenkins
managed
the
state
using
like
tiny
files
in
discs.
So
it's
not
great
for
persistence.
Like
we
try,
I
mean
this
team.
Try
and
fs
didn't
work
very
well.
B
So
today
they
are
using
persistent
volume
claims
backed
by
self,
and
it
seems
to
have
like
much
better
performance
than
NFS,
but
we
find
out
that
the
RVD
driver
is
not
super
reliable,
like
they
had
some
race
conditions
you
make.
Kubernetes
is
slow
to
mount
and
sometimes
times
out
so
we
been
working
and
we
have
deployed
some
flex
volumes
that
basically
implement
these
making
sure
that
it
works
a
little
better.
B
So
this
is
another
use
case,
the
news
team.
He
had
like
a
large
number
of
news
articles
in
all
kind
of
languages
and
they
were
trying
to
use
some
commercial
solution
to
translate
them.
So
we
help
them
to
move
it
in
to
govern
at
ease,
I
mean
the
vendor,
provided
them
a
container,
and
then
we
onboard
them
into
kubernetes
and
then
the
main
issue
that
they
had
was
capacity.
They
wanted
a
lot
of
capacity
and
we
kind
of
didn't
have
that
much
capacity
for
them.
B
So
what
we
ended
up
doing
is
creating
like
auto
scaling
solution
for
them
again,
because
we
are
in
all
kubernetes.
We
couldn't
take
advantage
of
all
the
new
features:
that's
starting
191
10
without
the
scaling
and
preemption.
So
we
believe
our
own
that
basically
what
it
does
is
scales
down
the
test
farm
and
they
scale
them
up
right.
So
here
in
this
graph,
we
have
at
the
top
like
the
queue
of
tests
for
the
test
farm.
So
on
here
in
the
middle,
we
have
the
number
of
test
workers.
B
B
So
we
have
like
on
board
like
few
training
is
cases
one
of
them
is
this
post
processing
training
pipeline?
Basically,
we
have
some
custom
workflow
engine
that
it
will
take
information
about
the
trades
from
the
colors
and
it's
going
to
process
some
kind
of
workflows
to
generate
reports
that
we
had
to
hand
over
to
the
authorities
and
like
compliance.
B
So
this
has
to
be
like
a
pretty
low,
latency
high
performance,
so
they
were
testing
through
communities.
They
were
able
to
satisfy
their
needs
and
has
been
working
fine.
So
far,
so
we
are
trying
to
onboard
more
people
as
they
get
interesting
into
communities,
so
the
domain
of
bondage
that
they
had
is
that
until
now
they
were
kind
of
starting
things
manually
or
with
crunchers,
so
cornutus
kind
of
gives
you
a
better
way
of
managing
these
processes.
B
Although
yeah
they
don't
use
a
lot
of
the
things
that
cornetist
gives
you
like,
for
example,
they
are,
there
is
resisting
to
kind
of.
Take
advantage
of
coordinate,
is
keeping
your
containers
up,
so
they
prefer.
That
is,
the
pod
goes
down.
They
want
to
look
at
it,
no
understand
why
he
went
down
and
they
started
again
because,
for
example,
if
you
run
out
of
memory,
so
their
system
is
deterministic,
so
you
will
run
out
of
my
money
again,
so
they
want
to
make
sure
that
they
increase
it.
B
So,
in
general,
with
kubernetes,
we
try
to
avoid
a
mistake
that
we
make
with
OpenStack
and
we
are
trying
to
push
for
a
cultural
change.
Like
to
move
toward
more
cloud
native
applications,
so
here
we
introduce
the
cows
engineering
by
default
on
our
general-purpose
clusters
for
now,
and
we
have
created
like
a
process
to
triage
those
applications
that
are
not
cloud
native.
Basically,
we
have
a
panel
of
people
that
are
going
to
talk
with
the
customers
that
think
that
they
are
not
clone
native
and
they
deserve
an
exception.
B
And
then
we
try
to
understand
if
they
really
need
the
exception,
because
the
application
doesn't
support
it
or
is
that
they
don't
want
to
do
the
work
to
really
be
cloud
native
right.
So,
like
so
far,
we've
been
pretty
successful.
We
have
a
few
exceptions
like
Sesay
Junkins.
It
doesn't
handle
great
being
terminated,
saying
with
Postgres
like
we
don't
have
today
a
good
HS
34
pro
Postgres,
so
we
gave
them
a
session
for
now
so
lesson
learned
from
operating
our
clusters.
So
basically
it's
pretty
hard
to
like
build
well-behaved
applications
in
cornetist.
B
Like
I
in
general,
it
requires
lot
of
communication
with
users
trying
to
educate
them,
trying
to
make
sure
that
they
use
cornetist
primitives
like
watching,
instead
of
being
in
busy
wait
all
the
time
make
sure
that
they
can
scale
horizontally
and
like
they
don't
create
problems
in
general.
Kubernetes
hasn't
proven,
proven
to
be
very
resilient
and
is
scalable
without
a
lot
of
effort.
So
we
having
spent
a
lot
of
time,
trying
to
tune
different
parameters
to
make
sure
that
it
works.
So
it's
pretty
much
works
or
box,
the
runtime
yeah.
B
B
Docker
has
been
giving
problems
here
and
there
operating
system
issues
like,
for
example,
we
have
some
issues
out.
We
don't
know
why
the
file
system
gets
full,
we
had
to
reboot
it
to
empty
and
we
have
our
engineers
trained
to
look
into
it.
It's
maybe
because
we
are
in
all
version
of
Debian,
but
we
haven't
been
able
to
figure
out
what's
going
on
so
another
problem
we
observe
is
a
community.
Scheduler
is
pretty
deterministic.
B
So
whenever
you
have
a
bun
node
like
the
bun
node
is
going
to
swallow
a
lot
of
pots
and
it's
going
to
basically
crush
them
or
a
victim
so
like
we
had
to.
We
have
invested
and
we
are
keep
working
on,
creating
more
monitoring
for
early
warnings
actually
proving
to
detect
these
bad
nodes
to
make
sure
that
we
take
them
off
production
and
even
better.
We
cannot
fix
them
and
put
them
back
into
production.
So
so
far
we
don't.
We've
done
I
think
like
to
two
or
three
upgrades
in
place,
so
it
mostly
worked.
B
So
it
requires
a
lot
of
work
because
we
had
to
basically
train
ourselves
a
lot
with
the
upgrade
make
sure
that
we
understand
what
has
been
changed
and
trying
to
see.
What's
going
on
to
be
prepared
for
when
we
do
the
actual
upgrade
in
production.
So
we
minimize
they
use
the
issues
that
we
caused
the
people
right.
B
So
we
are
in
process
of
kind
of
creating
some
workflows
where
we
will
allow
people
to
test
their
application
like,
for
example,
some
staging
cluster,
where
people
can
start
testing
the
new
version
of
communities
and
then
we
scale
up
and
down.
This
is
staging
cluster,
as
as
we
try
to
change
versions,
but
at
the
end
we
do
they
are
trained
in
place,
at
least
for
now.
B
So
finally
like
what
we're
working
on
so
basically,
we
are
working
to
trying
to
create
a
self-healing
infrastructure.
So
we
want
to
build
more
automation.
We
want
to
be
more
tools
to
identify
issues
to
fix
issues
with
a
cluster
because,
as
we
grow
like
this
will
become
very
important.
So
we
are
today
spending
significant
amount
of
time
trying
to
figure
out
issues
and
bugs
with
infrastructure.
B
So
we
are
also
like
working
to
expand
our
self
to
kubernetes
in
public
cloud.
So
again,
because
security
constraints,
we
cannot
take
advantage
of
the
full
public
cloud,
so
we
had
to
be
big
in
inside
secure
zones
like,
for
example,
Google
cousin,
like
these
VPC
secure
zones.
So
we
face
several
challenges
and
whether
we
were
when
we
were
deciding
whether
to
go
with
just
virtual
machines
and
deploy
our
cluster
or
GK.
So
we
evaluated
took
into
consideration
a
few
things
like
our
custom
operating
system.
B
Our
integration
with
Kerberos
like,
for
example,
at
the
time
GK
didn't
have
internal
endpoints
for
the
API
server
or
hard,
like
son,
worked
hacked
to
do
it.
So
eventually
we
are
there
with
our
own
cluster.
We
are
just
running
VMs
and
we
are
trying
to
expand
over
there
so
we're
also
working
on
schedulers.
Over
the
summer
we
have
an
intern,
the
she
was
working
on
trying
to
create
our
scheduler
to
improve
quota
management.
So
today
we
have
set
fixed
quotas
to
people
for
people,
and
this
kind
of
they
are
over
provisioned.
B
So
if
everybody
tried
to
use
the
full
quota,
they
won't
be
able
to,
because
there
is
not
enough
capacity.
So
here
we
are
trying
to
kind
of
make
use
of
the
new
features
at
kubernetes,
provide
with
preemption
to
have
like
a
dynamic
quota
management
and
to
have
different
type
of
workloads
with
different
priorities.
To
make
sure
that
we
can
adopt
cases
like
the
news
that
I
mentioned
before,
to
scale
them
up
and
down
as
is
required
and
yeah.
B
So,
like
yeah
so
like,
we
don't
have
a
real
solution
because
we're
oh
so
he's
mentioning
like
we
are
in
Google
in
BBC,
so
in
within
the
VP
CEO
Google.
They
don't
offer
like
a
proper
load
balancer.
There
is
like
the
internal
load.
Balancer
buddy
has
limitations
on
premiums
and
also
on
program.
Give
me
some
plan:
okay,
so
yeah,
so
I'm
Prem,
like
our
load
balancer,
like
we
think
clusters.
What
we
have
is
a
bit
of
a
special
set
up.
We
use
like
BGP
to
do
routing.
B
So
basically,
every
every
node
in
cornet
is
is
reka,
bgp
router,
and
then
we
take
advantage
of
the
service
service
IP
type.
So
basically,
every
machine
within
the
cluster
announced
the
same
service
service,
IP
range,
and
then
they
additionally
announced
their
own
cider
for
pots,
I
piece
right,
so
they
are
announcing
to
so
with
this
and
koala
I
mean
quality
the
framework
by
bgp,
but
basically
let
us
use
this
like
internal
service
to
route
traffic.
B
B
B
C
B
Yes,
so
basically
he's
asking
about:
how
do
we
implement
chaos
engineering?
So
basically
we
took
this
open
source
project
about
called
cube
monkey.
We
made
a
few
modifications
to
it
and
basically,
what
this
project
does
is
it
generates
a
list
of
candidates
to
be
killed
at
one
point
in
a
day
and
then
it
kills
them
through
the
day.
So
that's
the
basic
idea.