►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Kubernetes on Supporting $8 Trillion Card Payments in China - Xin Zhang & Deyuan Deng, Caicloud (Any Skill Level)
Kubernetes use cases are far less heard in the financial industry in China, because the core financial system consists of complex brownfield; applications that are not Kubernetes-friendly, and require better quality of service, fine-grained resource management, and stringent security guarantees. Nevertheless, the digitalization trend urges Chinese banks to transform their business to the Internet-style where Kubernetes can shine. We share a real journey of augmenting Kubernetes to support core services for the largest financial institute in China with $8 Trillion card payment value yearly. We share our architectural design and techniques to overcome the aforementioned challenges, including a unified networking control plane, a hardened security and access control-plane, a full-fledged multi-tenant management layer, supporting FSI applications with custom controllers, etc.
About Xin
Xin is currently CEO and co-founder of Caicloud (https://caicloud.io), a startup that fosters Kubernetes community in China and provides Kubernetes-based products and services for Fortune 500 Chinese enterprises. Before founding Caicloud, Xin was a Googler for almost 4 years working on Cluster Management and Google Cloud Platform, with close relationship to Borg and the early Kubernetes. Xin obtained his Ph.D in Computer Science from Carnegie Mellon University, during which he has published 30 research articles in top-notch venues and delivered about 10 academic speeches at international conferences around the globe to over thousands of audience.
About Deyuan
Deyuan is currently CTO and co-founder of Caicloud (https://caicloud.io), a startup that promotes Kubernetes in Chinese community and enterprises. He leads the in-house Kubernetes research and development and has successfully launched Kubernetes-based toolings and platform in tens of Chinese enterprises in production. Deyuan has given talks in Kubernetes meetups across multiple cities in China. He also presented in infoQ and CSDN in China to promote enterprise Kubernetes use cases to contribute to Kubernetes ecosystem. Before co-founding Caicloud, Deyuan was a Googler for about two years working on cluster management system. He was also one of the early members in Kubernetes. Deyuan obtained his Master’s Degree in Computer Engineering in Carnegie Mellon University. He is a technology enthusiast - he has participated in international robotics contest (Robocon), in which he claimed the Grand Prix and published papers based on his innovation.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Good
morning,
guys
and
I
hope
you
are
coming
to
the
right
talk,
I'm
Shin
from
tae
cloud
together
with
Derek,
so
we're
going
to
share
a
project
or
a
story
that
aims
to
have
most
card
payments
in
China
to
go
through
a
cornetist
cluster.
So
it
sounds
pretty
exciting.
So
this
talk
is
basically
two
parts.
So
the
first
one
is
me
doing
some
storytelling,
followed
by
some
tech
technical
meets
from
Derek.
So
to
better
understand
the
things
we're
going
to
share.
It
helps
to
first
know
us
a
little
bit
more.
A
We
are
active
contributors
to
kubernetes
tensorflow
and
a
rising
new
project
called
cube
flow.
So
it's
name
pretty
much
speaks
what
it
does.
So
it's
a
kubernetes
based
machine
learning
stack.
We
we've
also
done
a
lot
of
community
work
for
these
open
source
projects,
especially
in
China,
including
translating
the
official
kubernetes
websites
and
documents
into
Chinese,
as
well
as
hosting
and
running
the
official
tensorflow
community
in
Chinese
partnering
with
Google.
And
finally,
we
are
a
growing
business
providing
kubernetes
and
a
tensorflow
based
enterprise
solutions
for
a
lot
of
fortune,
500
customers
and
very
interestingly
I.
A
Think
two
months
ago,
CN
CF
did
a
survey
for
the
container
usage
patterns
in
China
and
all
these
customers
are
from
us.
So
all
the
story
begins
with
a
Chinese
banking
giant
seeking
for
help.
So
let
me
first
give
you
a
little
bit
idea
of
what
the
Banking
Authority
is.
So
it
is
pretty
much
the
equivalent
of
the
Visa
or
MasterCard
in
the
Western
world.
It
provides
the
only
inter
banking
connections
in
China
and
enables
transactions
across
different
banks
across
different
regions
and
now
even
across
over
a
hundred
and
sixty
countries
all
over
the
world.
A
A
See
it
in
Copenhagen
Airport,
so
we
just
took
the
the
pictures
from
yesterday,
so
it's
pretty
like
prevailing.
So
it's
indeed
a
giant
right.
So
everything
looks
pretty
good
so
why
it
is
looking
for
help
and
on
what.
So
the
answer
is
digital
transformation,
alright,
so
in
China,
so
I
can't
even
remember
the
last
time
when
I
used
a
cache
or
a
cart
to
pay
for
anything.
So
everything
every
payment
can
go
through
the
phone
right
using
apps
like
this,
like
Ali
pay
or
Tencent
pay.
So
every
time
I
go
out.
A
I,
basically
don't
carry
a
wallet
except
now
I'm
traveling
overseas.
So
these
new
form
of
payment
poses
great
challenge
to
these
traditional
banks
right
because
it
provides
much
better
user
experience
that
not
many
people
are
using
credit
cards
or
like
banking
cards
anymore.
So
all
banks
today,
especially
in
China,
face
a
choice
so
whether
they
need
to
catch
the
wave
of
teachers
of
transformation
by
delivering
something
similar
or
even
better
for
the
customers
or
their
business
may
be
hurt.
So
it's
a
do
or
die
decision.
A
But,
however,
so
in
order
to
bring
their
business
online
or
to
convert
some
or
to
invent
some
new
like
online
services,
these
banks
need
to
have
a
agile,
scalable
and
elastic
infrastructure
that
can
serve
the
surging
user
requests
with
very
high
availability
and
automation.
But
again
so
if
we
look
at
the
brutal
reality
of
UnionPay,
which
has
already
been
a
already
adopter
for
new
technologies
in
China,
so
their
current
infrastructure
is
like
a
brownfield.
So
we
have
to
cross
a
chasm
your
order
to
move
to
that
desired
state.
A
So
now
come
back
to
the
point,
so
we
all
know
what's
good
at
all
these
nice
things,
UnionPay
or
other
banks
won't
want
to
have
right
like
to
be
agile.
Scalable
available
and
having
a
lot
of
automation,
so
I
hope
we
can
shout
out
the
answer
right.
So
it's
a
kubernetes
right,
so
hopefully
no
one's
that
messes
or
swarm
all
right.
So,
okay,
so
at
this
point,
kubernetes
is
pretty
mature
right.
It's
free,
it's
open
source
does
kubernetes
automatically
solve
all
the
problems
of
the
world.
Right
is
at
least
four
UnionPay
right.
A
So,
let's
see,
if
we
look
at
the
the
up,
the
shelf
Corbin
at
ease,
it
provides
a
lot
of
building
functionalities,
like
native
name,
spacing
different
storage
options,
flat
network
and
a
bunch
of
others.
But
if
we
also
look
at
what
a
enterprise
level
production
grade,
banking
cloud
would
need.
The
list
of
requirements
actually
goes
on.
A
For
example,
they
need
to
have
multi
class
term
multi-zone,
multi-tenancy
and
multiple
isolated
networking
planes
you
order
to
achieve
better
security
or
to
serve
their
regular
regulation
purposes
and
to
shape
their
our
technical
architecture
to
be
aligned
with
their
organizational
architecture.
So
they
also
need
to
have
static
IP,
which
is
very
different
from
what
kubernetes
natively
does
right
and
also
sophisticated
Sdn.
A
You
order
to
implement
their
complicated
network
policies
again
for
better
security
and
regulations
and
if
they
wanted
to
have
commercial
storage
solutions
for
better
performance,
and
we
can
name
a
bunch
of
other
requirements
right.
So
even
though
kubernetes
really
made
great
progress
right
toward
this
this
end,
but
there
is
still
a
gulp,
or
at
least
a
crack
that
requires
the
last
mile
delivery
from
a
helper
like
us.
A
So
they
adopted
micro
services.
You
are
to
deliver
their
online
software
in
a
faster
pace,
but
very
soon
so
the
next
year.
So
they
realized
micro
services,
even
though
it
speeds
up
their
development
process,
but
it
poses
more
burden
on
the
admin
or
on
the
operation
side,
because
once
you
split
the
whole
like
the
binary
into
a
lot
of
different
components
with
thousands
or
even
more
containers
right.
So
they
don't
have
a
management
platform
to
manage
all
these
moving
parts.
A
So
that's
when
they
came
to
us
and
they
wanted
to
have
a
fully
automated
platform
to
manage
all
these
like
moving
containers.
So
now,
let's
go
go
to
2017
starting
from
July.
We
officially
kick
off
the
project
and
then
we
spend
a
three
month.
So
it's
a
big
chunk
of
time
to
figure
out
what
they're,
actually
requirements
and
knees
are
and
also
performed,
the
project
planning
and
the
technical
design
for
some
critical
architectures.
A
We
also
spend
two
following
month
to
heavily
developed
a
alpha
version
of
the
banking
cloud
on
top
of
coronaries,
so
that's
basically
our
velocity
by
five
engineers
and
then
in
the
next
two
months.
So
we
did
fast
iteration
made
necessary
adjustments
based
on
their
requirement
changes
and
also
performs
some
optimization
in
order
to
meet
their
performance
requirements.
So
and
so
the
exciting
thing
happened
in
February
this
year,
so
after
rigorous
testing
and
the
staging
so
as
of
today,
so
many
of
their
online
payments
actually
go
through
our
platform,
which
in
turn
go
through
kubernetes.
A
So
today,
so
every
payment
we
made
through
Union
pay
is
basically
advocating
or
supporting
for
criminales.
So
that's
pretty
exciting,
so
before
I
hand
it
over
to
my
colleague,
I
want
to
first
summarize
the
things
we
have
delivered
and
some
high-level
lessons
we
have
learned.
So
the
first
thing
is
on
the
business
side
filled
by
the
power
of
containers
and
kubernetes.
So
currently
there
are
new
online
businesses
like
the
wallet,
the
quick
pay,
which
is
pretty
much
like
the
diali
pay
or
Tencent
pay.
I
just
showed
in
a
previous
slide.
A
So
you
can
see
so
now
you
don't
have
to
use
a
card,
but
it
can
just
swipe
your
phone
in
order
to
finish
the
transaction,
and
this
accounts
for
the
eight
trillion
card
payments
I
put
on
in
the
title
so
besides
or
behind
behind
these
technical
in
a
business
innovation.
So
we
have
made
some
technical
contributions
along
the
way
which
some
of
them
are
not
in
the
upstream
kubernetes
repo,
for
example,
we
have
implement
single
sign-on
with
Keystone
and
also
multi-tenancy,
not
only
the
our
back.
A
The
name
spacing
thing,
but
also
support
the
access
privileges
for
like
looking
at
metrics
and
logs
and
perform
other
jobs
and
also
regarding
high
availability,
is
not
only
having
a
che
kubernetes
masters,
but
also
having
multiple
in
the
container
registries
that
can
sync
between
each
other
automatically,
as
well
as
having
a
configurable
static
IP,
so
that
the
bank
admin
can
better
manage
their
like
network
address
allocations.
So
the
list
goes
on
and
my
colleague
will
show
more
details
later,
probably
not
as
obvious
as
the
business
value
and
the
technical
innovation.
A
A
We
basically
help
the
customer
to
define
a
IGEL
process
using
containers
that
includes
how
to
do
code
review,
how
to
do
release
how
to
do
get
branching,
versioning
naming
in
a
sensible
way,
and
also
we
have
helped
them
to
split
they're,
still,
monolithic
applications
into
a
more
micro
service
style
and
also
we
have
set
up
the
continuous
integration
which,
again
so
in
in
one
of
the
keynote.
This
morning
was
mentioned
as
a
critical
piece
in
a
cloud
native
world.
A
So
and
finally,
we
have
accumulated
various
documentation,
ranging
from
the
user
guide,
the
admin
guide
to
the
incident,
playbook
and
various
like
performance
reports,
etc.
Alright,
so,
and
after
all
these,
so
we
did
learn
some
lessons
from
the
project,
so
the
first
one
is,
enterprises
are
really
complex
right.
So
it's
not
in
a
like
in
a
greenhouse.
You
have
to
deal
with
a
lot
of
real-world
noises.
For
example,
there
is
very
restricted
internet
accessibility
in
their
production
site
because
of
the
sensitive
sensitivity
of
the
data.
A
So
you
have
to
find
a
way
to
deploy,
upgrade
and
keep
running
your
system
without
like
free
internet
access,
and
also
we
realize
sometimes,
decisions
are
not
merely
driven
by
technical
merits,
but
you
also
have
to
cope
with
some
organizational
or
like
policy-wise
constraints,
and
so
we
also
learned
that
we
have
to
remain
open-minded
during
the
course
of
the
projects,
because
market
is
changing
rapidly.
So
as
the
the
requirements.
A
So
we
have
to
use
a
very
extensible
and
agile
way
to
cope
with
any
unpredicted
changes
and
finally,
so
be
prepared
for
like
multiple
all-nighters,
when
your
system
comes
online
or
you're
doing
a
like
online
operate
and
be
sure
to
comfort
your
project
members
so
that
they
don't
get
burnout.
So
with
that,
oh
so
there
is
one
last
slide,
so
I
just
want
to
give
a
like
look
forward
of
how
the
project
is
going
so
in
in
this
year.
A
So
after
using
kubernetes
and
containers
to
feel
they're
like
computer
computing
power
and
it
needs
so
we
further
use
it
as
a
vehicle
to
deliver
intelligence
to
further
help
they're
like
a
business
innovation.
So
during
this
conference,
you're
gonna
hear
a
lot
of
talks
talking
about
a
project
called
cube
flow
which
I
also
like
alluded
to
in
the
beginning.
A
So
we're
also
using
like
tensor
flow
on
top
of
our
communities
platform,
to
enable
their
developers
to
quickly
develop
AI
modules
to
help
them
do
things
like
customer
profiling,
emotion,
detection,
like
intelligent
recommendations
and
some
other
innovations
like
this
all
right.
So
that's
the
overview
of
the
project
and
now
is
Derek.
B
Okay,
thank
you
for
the
introduction.
So
now
in
your
field,
infolding
slides,
I'm
gonna
talk
about
some
of
the
technical
details
and
before
we
go
into
that,
I
want
to
say
thanks
to
our
teams,
and
actually
one
of
them
is
in
the
front
line
and
Steven,
and
there
are
other
folks
and
a
lot
of
folks
in
China
is
working
on
this
project.
B
It's
I
think
most
of
our
folks
is
working
on
this
project
and
part
of
he's
working
with
our
customers
day
by
day,
and
some
of
them
is
just
sitting
in
the
back
line
and
support
those
guys.
So
a
little
bit
overview
of
this
system
a
pair
for
won't,
be
a
platform
if
there
is
no
application
running
on
top
of
it
right.
So,
just
as
Sheen
just
introduced,
there
are
a
couple
of
applications
running
and
kubernetes,
and
there
is
some
front-end
applications.
These
are
the
applications
serving
some
static
contents.
It's
not
just
a
simple
web
application.
B
There
are
some
customizations.
We
have
to
do
to
run
them
and
kubernetes,
and
there
are
some
core
applications
like
the
UnionPay
wallet
and
the
user
verification
system.
Those
are
running
on
top
of
kubernetes
to
price
their
core
business,
and
the
goal
of
this
container
platform
is
to
translate
their
applications
and
microwaves
are
monolithic
applications
to
containers.
The
goal
is
to
adopt
the
trending
technologies
like
devops
micro
services.
There
are
actually
some
technical
details
we
have
to
deal
with
before
this
can
go
into
pub
production.
B
They
are
Network,
side
story,
side,
authentication,
side
and
we
will
go
through
that.
A
little
bit
later
here
is
a
the
overview
of
this
system
on
the
right
side.
Yeah,
it's
right
beside
this.
The
red
boxes
are
the
existing
ones.
As
you
can
see,
the
application
is
running
on
top
of
s
platforms.
Those
are
application,
those
are
OpenStack
instances
and
the
OpenStack
is
integrated
with
safeness
and
some
Sdn
solutions
either
it's
based
on
hardware
and
or
software
and
a
blue
side.
The
pool
epoxies
are
the
new
components
we
have
to
integrate
there.
B
B
Here
is
a
final
deployment
that
we
adopted
in
the
production
environment,
the
red
the
red
box
is
a
firewall
on
the
left
side.
This
is
a
behind
the
firewall,
an
upside.
This
is
a
cargo
with
cargo.
It's
actually
some
enhanced
version
of
talk,
rich
histories.
There
are
multiple
cargos
and
we
were
going
go
to
some
details
later
of
why
we
run
such
that
multiple
instance
of
token
registries
and
the
green
box
is
a
compass.
We
call
upon
compass,
it's
just
a
some
organic
grouping
of
kubernetes
clusters.
You
can
think
of
it
like
a
federated
kubernetes.
B
There
is
a
control
cursor.
We
run
all
the
control
components
in
this
in
this
cluster,
like
some
application,
administrator,
helm
or
permissive
running
on
top
of
this
control
cluster
and
for
user
clusters.
These
are
dedicated
to
run
their
business
applications
with
separate
these
two
kind
of
applications
to
improve
availabilities,
and
there
is
another
cluster
running
side
by
side
running
across
the
firewall.
A
control
cluster
is
running
behind
the
firewall
and
the
user.
A
public
user
clusters
is
running
on
the
Zizi,
so
it's
a
DMA
zone
here
are
some
details.
B
We
will
just
skip
it
first,
I'll
go
to
so.
The
first
thing
that
we
go
through
is
the
SSO
they
say
so
is
that
we
want
to
login
the
requirement
from
our
customers
question.
Who
are
you
logging
once
and
then
you
looking
everywhere
now
everywhere
means
Kuban.
It
is
an
open
stack.
So
if
I
think
folks,
here
already
familiar
with
kubernetes
and
companies
provides
a
lot
of
options
for
you
to
do
authentication
authorization
and
other
automation
control,
but
our
customers-
they
don't
know
anything
about.
B
This-
is
just
water
output
out
of
box
solution,
so
they
have
some
existing
environments.
There
is
a
OS
system.
This
OS
system
contains
all
the
organization
and
April
employee
information
and
there
is
an
identification
service.
It's
a
SSO
server.
It
contains
employee,
ID
and
password.
So
when
we
say
log
into
Union
pay
internal
system,
it's
locked
into
this
identification
service
and
these
two
services
are
the
core
applications
inside
of
their
systems.
B
The
Keystone
is
a
kind
of
external
service
that
integrates
with
this
ID
service
to
do
a
single
to
the
single
sign-on
and
that
it
pulls
information
from
this
organization
or
office
automation
daily
at
midnight
to
sync
with
the
internal
services-
and
this
is
the
actually
they
are.
They
are
portable
to
log
in
so
I,
say
so,
is
actually
the
one
of
the
simplest
simplest
thing
with
the
we've
done
in
Union
pay.
So
I'll
just
go
through
this
in
one
slide.
B
So
first,
when
a
user
log
into
this
system
and
it
access
a
council
web,
is
a
council
and
then
obviously
the
user
is
not
locked
in
then
it
it
accesses
a
council
web
and
then
log
in
with
the
tight
X
it's
a
fork
of
Coria
stacks,
which
which
does
a
light
East
a
connector.
We
implement
a
connector
instead
of
a
text.
Now
it's
not
working
and
texts
will
redirect
our
user
to
this
SSO
service
and
then
the
user
input
the
username
and
password.
Now
there
is
a
bunch
of
interactions
between
the
SSO
server
and
text.
B
Now,
with
this
interaction,
the
text
can't
has
enough
information
to
pull
information
from
Keystone
and
auth
our
own
implementation
overall
system
and
has
a
user
ID
it
get
OpenStack
service
account.
Then
it
includes
all
this
information
into
ID
token.
This
ID
token
contains
OpenStack
Keystone
token.
Then
it
uses
all
this
this
token
to
access
our
own
services
as
well
as
OpenStack
service.
This
is
a
whole
workflow
and
there's
another
service,
as
well
as
synchronously,
to
synchronize
keystone
information
to
our
own
implementation
of
us.
B
It's
all
on
running
daemon,
running
errand,
just
as
wasting
kiss
no
information
from
the
OS
system.
Okay,
the
other
thing
is
that
tenant.
So,
as
we
all
know,
there
is
no
concept
of
a
tenant
in
kubernetes
and
it
is
also
some
ongoing
discussion.
I
think
the
improvers
Austin
coop
can
about
how
to
design
a
tenant
in
kubernetes,
and
we
cannot
wait
until
there's
some
solutions
in
upstream,
so
we
have
to
do
it
ourselves.
So,
in
this
tournament
system
we
have
a
system
admin.
The
system
admin
will
is
a
kind
of
a
boss
of
this
platform.
B
B
I
can
allocate
some
clusters
and
then
give
Jing
the
attendant
admin
to
these
clusters
and
for
the
tenant
admin
it
manages
a
single
tenant,
a
user
can
be
a
tenant
of
multiple,
a
user
can
be
a
tenant,
eternal
enemy
of
multiple
tenants,
distant
enemy
will
add
and
delete
groups
of
users,
and
it
can
allocate
a
resource
for
machines.
Per
group.
B
Here
is
a
definition
of
the
team
and
users
so
team,
a
team
and
a
group
is
used
interchangeably
in
our
environment.
It
holds
the
permissions
of
resources.
Just
like
a
pack
right.
We
give
some.
We
came
back
to
a
user,
another
user
can
have
create
these
create
and
delete
permissions
for
them
spaces.
So
we
give
it
term
the
team,
because
that's
more
nature
to
users.
If
this
all
sounds
confusing,
we
can
go
some
snapshots
of
the
system,
so
this
is
a
view
from
tenant
former
system
admin.
B
This
is
a
like
operation
process
of
the
UnionPay
infrastructure,
and
it
only
cares
about
his
page
at
left
side.
There.
Some
note
Craster's.
As
you
see
it,
only
cares
about
these
resources
and
as
the
front
upper
side
is
attendance,
it's
a
Q&A
control
tenant
in
this
tenant.
We
give
it
some
permissions
and
resources.
If
you
see,
there's
a
multiple
tenant
and
across
circulate
know
each
cluster.
B
So
in
this
tenant
we
give
two
clusters
to
this
tenant
and
in
the
first
cancer
we
give
it
four
core
and
eight
gigabytes
of
memory,
and
then
we
can
also
give
you
some
number
of
volumes
to
use
and
the
total
capacity
of
this
volume.
This
load-balancing
there's
a
lot
of
things
we
can
give
to
our
users
and
all
these
are
implemented
inside
of
kubernetes
and
from
the
point
of
view
of
a
tenant
admin.
It
has
different
panels.
It
cares
about
where
to
run
our
applications,
and
it
cares
about
DevOps.
B
It
cares
about
logging
and
monitoring,
so
it
can
create
groups.
Each
line
is
a
group
with
a
set
of
users
and
informations,
for
example,
in
the
SI
team.
The
SRE
team
has
permissions
to
view
logs
to
build
some
more
insurance
standards
and
also
cares
about
pipelines
and
inside
of
this
tenant
inside
of
these
groups.
I
can
allocated
permissions,
for
example,
in
this
page
I
give
you
a
test
against
permission
to
demo
space,
so
these
are
all
the
concepts
and
screenshots
of
tenant
systems.
B
So,
in
a
quick
summary
of
this
page,
the
yellow
box
represents
some
resources.
For
example,
there
is
some
CDs
registries
and
they
have
four
clusters
voice
for
developed
cluster,
wise
for
testing
wise
for
staging
and
wise
for
production
and
they're
the
horizontal
lines
the
dotted
box
represents
a
tenant,
so
a
tenant
can
have
resources
across
different
clusters,
so,
for
example,
I
have
a
tenant.
This
is
a
union
pay.
Business,
9
and
I
can
allocate
development
Custer.
Some
resources
of
a
developed
cancer
to
it
and
I
can
locate
some
production
resources
to
this
tenant.
B
So
we
can
maintain
different
Sol
Israel
for
different
clusters
instead
of
managing
multiple
clusters
for
each
product
line-
and
this
is
a
kind
of
a
tree
like
descriptions
of
this
concept,
the
resources
can
be
divided
into
different
tenant
and
if
different
tenant
had
has
its
own
different
code
has
and
the
attendant
can
have
different
teams
and
a
team
has
multiple
users
and
user
is
called
a
member,
and
a
user
can
be
part
of
can
be
multiple
rows.
It
can
have
multiple
rows
and
each
row
has
a
h
team
has
a
set
of
permissions
to
resources?
B
A
user
itself
does
not
have
any
does
not
hold
any
permissions.
It
inherits
permissions
from
the
groups.
They
are
part
of.
So
as
just
as
Shing
just
mentioned,
we
have
to
map
this
whole
concept
into
OpenStack.
There
is
already
there
is
already
some
established
conventions
increment
in
OpenStack,
so
we
have
to
compromise
a
lot
of
our
concepts
to
OpenStack.
This
mapping
is,
is
described
from
network
point
of
view.
B
The
first
one
you
will
map
the
tenant
to
opens
their
project
because
there
is
no
concept
of
tenant
incriminated,
so
we
used
week
just
name
it
our
own
system
and
we
map
kubernetes,
name
space,
pod
and
service,
where
he
two
different
opens
their
concepts
and
each
car.
Each
namespace
has
different
OpenStack
resources,
and
this
is
a
picture
of
this
mapping.
B
B
Then
this
is
a
choosing
a
tenant
in
OpenStack
and
then,
when
you
create
a
new
namespace
in
OpenStack
in
kubernetes,
you
have
to
give
the
some
of
the
OpenStack
options
to
kubernetes
to
our
system,
so
it
so
that
we
can
provision
the
namespace
a
partition.
Is
our
concept?
Is
our
web
or
on
namespace,
because
user
tend
to
create
applications
without
resource
limit,
so
we
do
not
expose
them
space
directly
to
our
users.
We
expose
their
face
plus
resource
courthouse,
so
it's
a
wrapper
of
namespace.
B
Now
that
we
have
a,
we
don't
have
much
time
left
so
I'll
just
just
keep
this
one
so
for
network
click
civil
network,
the
requirements
are,
we
want
real
IP.
We
don't
want
users
to.
We
don't
want
the
traffic
to
go
through
different.
We
want
all
the
traffic
inside
of
our
cluster
to
have
a
real
IP,
so
we
don't
want
le
black
hole
in
the
system
and
we
want
configurable
traffic
shaping
and
we
need
to
have
multiple
airplanes.
B
So
the
whole
system
is
based
on
open
source
project
grid
kubernetes,
but
with
our
own
modifications
because
incorrect
binaries
there
is
no
concept
of
come
out
a
tenant,
so
we
have
to
customize
it
to
have
the
concept
of
multi
tenant
and
we
also
want
to
provide
chef,
shaping
and
isolation.
So
we
support
tbp
group
based
policy
in
this
system,
and
this
is
a
kind
of
detailed
picture.
B
Ok
for
network
multiple
planes,
so
this
is
a
quite
well
established
conventions
in
OpenStack,
but
we
have
to
plug
our
system.
Inside
of
this,
there
is
a
management
plane
which
runs
which
runs
the
portal
and
there
is
a
control
plane
which
we
run
all
the
container
from
system
components.
Instead
of
this
network
plane
and
there's
a
storage
places,
a
peace
plan
to
run
all
user
applications-
and
here
is
a
detailed
picture-
we
can
even
know
all
the
parts
of
the
picture,
but
we
can
focus
on
the
cream
part.
B
And
for
that
so
I
think
I'll
just
quickly
go
through
the
service
for
the
service.
We
have
a
customized
version
of
the
service
called
tenant
service.
When
we
run
we
create
a
service
just
a
class
or
IP
or
not
poor
service
in
kubernetes.
This
will
not
be
recognized
by
OpenStack.
We
need
to
integrate
the
system
into
OpenStack
because
all
the
union
player
systems
are
running
on
top
of
OpenStack,
so
we
have
to
create
a
concept
call.
B
It
analysis
service,
this
tunnel
service,
integrated
with
load
balancing
as
a
service,
and
it
has
its
own
virtual
IP
from
the
oil-based
virtual
IP.
This
is
a
mapping,
I'll
skip
it
and
that
there's
another
thing
called
T&S
in
kubernetes.
This
is
a
quite
challenging
so
because
there's
a
the
tinniest
part
needs
to
access
kubernetes
api
server
right.
It
pulls
all
the
service
and
points
information
from
kubernetes,
but
it
runs
in
the
control
plane
and
our
application.
Our
pin
is
traffic.
Our
business
applications
needs
to
access
this
DNS
server
to
query
DNS
records.
B
So
this
is
a
problem
we
have
to
solve,
because
this
tooth
planes
cannot
access
each
other
and
both
applications
need
to
access
each
other,
and
the
solution
to
this
is
to
run
our
team
as
part
directly
as
the
host
network
in
the
host
Network,
and
we
run
a
version
of
cootie
on
coop
DNS,
the
upstream
version
of
cube
DNS
in
the
management
caster
and
the
user
caster.
We
do
not
run
cube
DNS.
Instead,
we
run,
we
do
not
run
this
mask.
You
is
wrong.
The
cube
DNS
and
the
ten
years
of
masks.
B
You
will
clearly
other
records
from
different
cultures
and
our
users
are
using
applications
were
directly
query.
The
DNS
records
from
this
thing
as
mask
you.
So
this
is
the
solution
we
provided
to
solve
this
problem
and
it
served
quite
well
in
production
traffic.
There
is
some
pay
attention.
There's
a
VIP
for
this
studying
is
server
and
for
the
so
inside
there
is
no
other
box
solution.
Kubernetes
and
but
enterprises
customers
care
about
data.
We
don't
reinvent
the
wheel,
because
story
is
not
something
we
need
to
provide.
B
B
We
don't
care
about
what
you
use
for
each
registry,
but
we
provided
ability
to
run
multiple
registries
and
it
turns
out
they
run
two
registries,
one
in
the
behind
the
fire
or
behind
the
firewall
and
the
wrong
one
registries,
for
they
are
testing
and
staging
and
run
one
registry
for
the
production
traffic,
and
there
is
a
page
and
they
can
also
automate
the
process
to
think
the
images
between
the
two.
The
histories.
I
think
this
is
the
last
item
for
it
registry
tray.