►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Towards Serverless As Commodity: A Case of Knative - Nima Kaviani, IBM & Dmitriy Kalinin, PivotalSDCC
Serverless computing promises to evolve cloud computing architecture, reducing complexities of managing and scaling underlying infrastructure. However, one of the primary drawbacks customers face when deciding to use serverless is the potential for getting locked-in with a particular provider. This used to be a concern with Platform-as-a-Service (PaaS) offerings too. However with Kubernetes emerging as the industry standard, PaaS is closer to becoming commodity with the Kubernetes API as its common interface. The question is if a similar unification for the API interface and runtime contracts can be achieved for serverless. As contributor to Knative with in-depth understanding of its internal design, we use Knative as the baseline and compare its API interface and runtime contracts against other prominent serverless platforms. We identify commonalities and differences, discuss challenges, and review evolution we expect to see as serverless platforms reach commodity status.
A
Perfect,
alright,
hello,
everyone
so
in
this
talk,
we're
gonna
discuss
and
what
it
takes
for
serverless
to
become
commodity.
So
hopefully
at
one
point
and
we
start
moving
on
to
other
interesting
stuff
so
to
get
us
started,
my
name
is
Naimah
kavion
ii
and
I'm
an
engineer
with
IBM
and
I'm
here
with
my
friend
dmitri
from
payroll,
who
clearly
you
can
tell,
has
come
from
the
east
coast.
A
Alright,
so
I
think
at
this
point
in
talk
all
of
us
know
the
pros
about
several
lists.
Obviously,
a
lot
of
people
want
to
use
serverless
and
enjoy
using
server
list,
because
it's
easy
to
manage
and
it's
cost
effective.
Obviously,
it's
easy
to
manage,
because
there
is
less
that
you
have
to
care
about,
and
you
know
back
to
evans
talk.
It's
cost
effective
in
many
ways,
particularly
saving
on
the
operations
cost.
A
On
the
negative
side,
however,
one
of
the
biggest
problems
that
people
deal
with
when
talking
about
serverless
is
that
the
they're
afraid
of
getting
locked
in
with
a
particular
vendor.
Essentially,
the
reason
is
because
you're
only
in
charge
of
a
tiny
layer
for
your
software,
you
know
for
every
other
service,
you
have
to
rely
on
the
service
provider
or
the
cloud
platform
provider
to
build
your
application.
A
So
this
is
to
the
point
that
you
know
some
people,
including
alex
Pavia
of
Korres,
consider
surrealists
as
one
of
the
worst
and
proprietary
logins
in
human
history
right
also
so
other
people,
including
Kelsey,
acknowledged
this,
and
you
know
one
way
for
a
lot
of
companies
to
try
and
solve
this
is
by
providing
an
open
source
layer
that
can
go
on
top
and
then
would
give
you
the
flexibility
to
migrate
from
one
cloud
provider
to
another.
If
you
choose
to
so
for
us
to
kind
of
understand
what
it
takes
for
several
is
to
become
commodity.
A
We
started
looking
at
kubernetes
as
a
good
example,
because
I
think
at
this
point
pretty
much.
Everyone
agrees
that
kubernetes
is
the
orchestration
war
and
continued
as
a
service
is
essentially
kubernetes
right.
So
the
question
that
we
started
asking
ourselves
was:
okay:
what
does
it
take
for
service
to
become
commodity
and
who's?
Gonna
win
the
serverless
war,
so
we
started
looking
at
why
kubernetes
was
so
successful
and
identified
three
different
stakeholders
and
the
reason
they
actually
liked
kubernetes
a
lot
so
from
a
providers.
A
Perspective
kubernetes
was
very
appealing
because
it
was
open
source
and
it
was
eyes
agnostic,
so
they
could
contribute
to
you.
They
could
see
what's
happening
inside,
they
could
take
it
and
deploy
it
on
their
infrastructure
right
from
an
operator's
perspective.
It
was
interesting
because
it
offered
the
declarative
model
of
deployment,
which
was
consistent
so,
regardless
of
the
type
of
application
that
you
want
to
deploy,
you
need
to
know
the
building
blocks
and
then,
once
you
know
the
building
blocks,
you
put
them
together.
A
You
get
your
application
up
and
running,
so
it
was
very
easy
to
get
onboard
with
kubernetes
and
finally,
from
developers
perspective,
it
was
appealing
because
it
offered
a
consistent
API
across
all
the
platforms
right,
so
it
actually
allowed
them
to
migrate
from
one
providers
to
another
if
they
weren't
happy
with
their
existing
provider.
So
we
decided
to
actually
take
these
and
look
at
what
it
takes
for
a
service
platform
to
repeat
kubernetes
success.
A
What
did
I
need
to
do
right,
and
so
you
know
the
points
that
I
discussed
being
open
source,
eyes,
agnostic,
consistent
deployment
deployment
model
and
consistent
API.
The
first
three
appear
to
be:
you
know,
satisfied
by
essentially
every
service
less
platform,
that's
built
on
top
of
kubernetes
these
days
right
all
of
them,
and
here
today
use
the
utilized.
The
API
that
kubernetes
offers,
and
essentially
there
are
a
lot
of
open
source,
is
agnostic
platforms
that
can
be
built
on
top
of
kubernetes.
A
A
So,
yes,
we
started
looking
and
a
consistent
API
model,
and
in
order
to
do
that
and
next
slide
in
order
to
do
that,
we
actually
started
looking
at
different
and
severest
platforms,
and
we
started
looking
at
different
criteria
for
what
the
API
is
supposed
to
be
across
these.
In
order
to
compare
and
contrast
and
come
up
with
a
final
ideal
that
we
think
is
essential
for
a
service
platform.
So
I'm
going
to
end
it
to
Dimitri,
hey.
B
B
Take
a
look
at
the
seven
criterias
over
here
and
find
out
what
are
the
different
implementations
like
aw
aslam,
the
open
way,
scope
and
fast
cube
list.
Key
native.
Do
do
note
that
we're
not
experts
in
any
of
them
really,
but
we
did
look
into
different
implementations
as
deeply
as
we
could.
We
looked
at
documentation
we
played
around
with
some.
We
do
occasionally
contribute
to
key
native.
We
do
use
AWS
lambda
for
a
few
of
our
own
projects
and,
of
course,
neem
is
from
IBM.
B
So
is
more
familiar
with
open
risk,
but
do
take
the
information
with
a
grain
of
salt
and
do
let
us
know
if
something
is
incorrect.
So
let's
take
a
look
at
number
one
packaging
contract.
So
when
we
say
packaging
contract,
what
we
mean
is
what
does
the
developer
need
to
provide
to
the
platform
to
run
their
function
right?
B
There
you
go
and
then
Kay
native,
because
it's
based
on
kubernetes
it
only
supports
Osei
image,
though
I'm
sure
people
who
you
know
use
cloud
native
bill,
paxton
whatnot
can
ultimately
avoid
osei
image
directly,
but
you
know
really
convert
into
it
before
giving
it
to
K
native.
So
let's
take
a
look
at
next
one.
Next
one
is
runtime
invocation
contract,
so
we
kind
of
split
the
I
guess
the
the
system
into
two
parts,
one
part
between
the
platform
and
the
runtime
and
runtime.
We
have
this
kind
of
a
may,
be
a
wrapper
around
your
application.
B
That's
what
we
call
runtime
and
so
we're
gonna
talk
about
specifically
between
the
platform
and
the
runtime,
so
let's
dig
in
so
lambda
as
we
found
out
actually
as
runtime
pooling
lambda
API
right.
So
this
is
somewhat
different
from
some
of
the
other
implementations
where,
for
example,
open
whisk
allows
you
send
a
message
over
HTTP
to
you
know
the
end
application
right,
which
is,
you
know,
obviously
a
quite
a
bit
different
implementation
right,
open
fast,
quite
similar.
It
pushes
over
HTTP
cumulus
again
and
we
scan
it
if
there
is
actually
no
runtime
wrapper
right.
B
B
You
know,
depending
how
much
functionality
you
want
to
reuse
from
the
platform
you
can
bypass
some
of
the
layers
I'm
sure
some
of
these
other
implementations
love
you
also
to
remove
some
of
the
I
guess
built-in
default
processing,
but
that
that's
that
our
current
summary
of
this
so
between
the
runtime
and
application
right.
So
if
we
imagine
that
this
runtime
thing
is
a
wrapper
around
your
application,
what
does
it
take
for
the
runtime
to
actually
pass
the
information
down
to
your
function?
So
in
lambda?
B
If
you
look
into
it
into
the
underlying
libraries
and
SDKs
that
they
provide,
there
is
some
custom
JSON
enveloping
going
on
with
open
risk.
There
is
two
options:
there
is
a
JSON
envelope
which
I
believe
is
custom
for
open
risk,
and
then
you
can
also
just
directly
accept
http
stream,
as
if
it's
coming
from
you
know,
your
regular
low
bouncer
open
fast,
we
saw
that
there
is
a
standard
in
standard
out
began,
corrects
for
wrong.
No
tell
us
what
it
is.
B
Okay,
yeah
so
sounds
like
we
should
have
talked
to
open,
fast
maintained,
there's
a
little
bit
deeper.
We
we
didn't
take
a
look
at
open
fast
to
too
much
besides.
The
docs,
like
I,
said
not
an
experts
in
a
lot
of
these
things,
but
it
sounds
like
HTTP
acts
as
a
basis
of
it
and
I'm.
Assuming
there
is
a
watchdog
or
some
other
mechanism
that
translates
it
into
standard
instant
without
for
some
of
the
stuff,
no.
B
Interesting
so
I
guess,
based
on
my
understanding,
some
of
the
examples
did
show
that
you
can
just
write
some
kind
of
a
bash
function
or
whatnot.
That
just
accepts
some.
You
know
input-output
kind
of
a
thing,
but
maybe
that
wasn't
it
right
right
so
that
that's
what
we're
kind
of
talking
about
this
wrapper
rate
that
right.
But
this
is
exactly
what
we're
talking
about
so
runtime
is
that
potentially
optional
wrapper
rate
that
ultimately
results
in
what
the
end
user
looks
like
back.
B
B
B
One
interesting
thing
that
you
know
Kennedy
when
lambda
we
saw
was
that
with
lambda.
You
do
get
to
use
a
lot
of
the
SDK
and
I
guess:
proprietary
lambda
technology
to
wrap
your
application
around
with
skinny
rivers.
It's
slightly
more
familiar
check
to
an
average
web
developer
right
that
you
don't
have
to
really
write
any
kind
of
you
know
wrappers
or
you
don't
have
to
use
any
kind
of
additional
libraries.
You
can
just
use
directly,
let's
say
in
goaling
net
HTTP
or
whatever
else
it
may
be
right.
B
B
We
saw
that
there
is
a
well
personally:
we've
used
synchronous
functionality
in
most
of
the
cases,
but
we
did
observe
that
you
can
configure
some
of
this
functionality
to
actually
run
asynchronously
for
open
risk.
We
have
synchronous
asynchronous
as
well.
Open
fast
also
supports
that.
So
that's
cublas,
however,
in
key
native
we've
only
seen
synchronous
functionality
being
supported
by
default.
Now,
obviously,
you
know
anything,
that's
synchronous.
B
You
can
kick
off
asynchronous
tasks
within
your
synchronous
requests
and
maybe
fully
complete
it,
but
with
that
comes
interesting
questions
of
when
does
your
function
terminate
and
how
you
can
manage
it
over
time
or
visibility
into
that
stuff.
I
believe
you,
my
actually
made
some
issues
to
discuss
that
further
with
the
key
Native
community.
I,
don't
quite
know
where
that
is
at
this
point.
B
Let's
see
what
else
we
got
retry
model
with
retry
model.
It
was
quite
actually
interesting.
Most
of
the
most
of
all,
things
did
not
support
any
kind
of
a
built-in
platform
provided
retry,
with
one
exception
of
lambda
and
open
fast
with
lambda,
though
it
wasn't
three
tries
based
on
if
the
platform
is
failing,
was
mostly
retries.
Based
on
your
application
logic,
we
don't
have
any
kind
of
retries,
one
I
guess
asynchronous
workloads.
Well,
no
a
synchronous,
workloads,
no
trice,
so
that
was
a
that
was
kind
of
an
interesting
observation.
There.
Yes
question.
B
So
the
question
was,
since
we
mentioned
cloud
events:
does
that
mean
that
we're
also
putting
key
native
eventing
and
key
native
serving
under
the
under
this
title
of
key
native?
Yes,
and
no
so
again,
none
of
this
is
you
know
specific
with
particular
examples,
but
we
did
see
that
key
native
is
aware
that
cloud
events
is
an
emerging
standard,
so
it
is
being
used
in
one
part
of
a
system,
but
not
as
alien
Adam,
but
mostly
focused
on
key
native,
serving
in
this
case.
A
So,
even
though,
obviously
you
can
you
can
implement
a
sync
and
do
retries
with
with
cloud
events
and
can
I
live
event
in.
There
is
no
standard
for
how
you
do
it
like
there
is
no
general
accepted
community
way
that
okay.
This
is
how
you
do
a
sync
with
Kay
native,
in
other
words
with
the
other
ones.
There
is
like
a
very
clear
way,
for
example
with
lambda.
You
can
just
pass
the
invocation
time
and
that's
it
right.
B
All
right,
concurrency
model
and
auto
scaling,
so
lambda
is
mostly
request
based
and
you
know,
given
that
their
auto
scaling
based
on
a
queue
length
again,
not
a
lambda
engineers,
so
they
might
be
doing
some
more
magic
stuff
behind
the
scenes.
That's
what
we
observed
based
on
api's
and
usage,
open
whisk,
was
mostly
request.
Based
open,
fast
request,
based
and
resource
based,
and
when
we
mention
resource
based
will
mean
here
is
based
on
CPU
and
all
kinds
of
other,
potentially
custom,
metrics
sure
NH.
So
so.
A
B
So,
finally,
I
think
this
is
the
last
one
traffic
splitting
so
for
traffic
splitting.
It
seems
that
lambda
decided
to
go
with
offering
a
first
class
in
their
API
first
class
support
for
a
provisions.
Open
wisk
did
not
have
such
functionality,
and
I
think
it
requires
external
load
balancing
to
separate
between
different
a
provisions.
So
if
actually
you
end
up
deployment,
multiple
applications,
not
iterating
on
the
same
application
for
open,
fast
external
service
mesh
was
the
choice,
seemed
to
be
the
choice.
B
Any
corrections
they're
sitting
on
okay,
perfect
so
using
service
mesh
interface
that
came
out
of
I.
Think
this
and
then
cublas
same
kind
of
a
situation,
external
service,
mesh
and
finally,
key
native.
We
have
built-in
functionality
done
through
revisions.
So
revision
is
a
you
know.
Copy
of
your
application
over
time
and
cainy
also
support
some
configuration.
Some
first-class
routing
specific
objects,
CR
DS
that
could
be
used
to
integrate
with
whatever
you
want
to
integrate
with.
B
A
Obviously,
like
Demetri
said
like,
if
we
give
it
in
give
this
presentation
in
any
other
conference,
we
we
have
some
opinions
or
some
ideas
about.
You
know
what
is
the
design
for
a
service
platform,
but
here
because
there's
a
lot
of
experts
in
the
room,
we
thought
we
would
turn
it
over
and
ask
you
guys
what
we
think
would
be
the
proper
API
designed
for
for
several
s.
So
the
first
thing
that
we
started
packaging
contracts.
A
So,
as
we
mentioned,
there
are
a
number
of
choices
for
packaging
contract
like
OCR
images,
custom,
packaging
and
custom
packaging
into
OCR
images
right
so
just
by
a
show
of
hand
who
thinks
you
know
the
the
proper
packaging
contract
or
packaging,
your
packaging
contract
would
be
OCI
images,
okay,
custom,
packaging
of
zip
files
and
stuff.
Okay,
one
person
custom
packaging
into
OCR.
You
inches,
okay,
great.
What
is
it
one?
Entry,
yeah
yeah,
so
the
perfect
option?
Combination
about
see
our
images
and
custom
packaging
into
OCR
images.
A
Sure
so
we
thought,
like
our
opinion,
was
OCR
images
and
the
the
main
reason
was
that.
Well,
obviously,
you
can
do
custom
packaging
into
OCR
images,
but
then
you'll
have
to
make
your
users
to
also
customize
their
OCR
images
and
provide
them
potentially,
if
there's
a
huge
dependency
for
your
application,
huge
set
of
dependencies
for
your
application,
because
otherwise
you
know
you
can
rely
on
kpac
or
some
other
technologies
or
build
packs
to
kind
of
bring
it
into
the
into
the
OCR
image.
A
B
B
All
right,
so,
what
what's
being
said
for
people
who
didn't
hear
is
that
it's
really
the
question
of
what
kind
of
user
if
you're,
targeting
with
your
platform
right
some
users
are
very
comfortable
with
doing
let's
say
traditional
non
or
CI
artifact
delivery
to
their
platform,
so
they're
happy
in
that
world
right
versus
maybe
a
newer
kubernetes
based
systems
right.
They
really
prefer
the
OCI
image
base.
B
Yeah
I
think
we
generally
agree
is
if
it
becomes
more
of
a
question:
where
do
you
draw
the
boundary
of
the
service
platform
right
and
at
least
both
of
us?
Don't
necessarily
think
that
you
know
yeah
it's
fair
to
put
that
functionality
into
the
service
platform,
but
we
do
believe
that
it's
fair
to
put
it
into
an
additional
product,
maybe
on
top
of
that
right,
so
yeah
all
right.
B
So
let's
take
a
look
at
the
next
one
run
time
invocation
contract,
so
keep
in
mind
that
again,
we've
defined
the
run
time,
as
that
lightweight
wrapper
around
the
application
right.
It's
mostly
for
convenience
for
the
end
user
to
be.
You
know
more
potentially
familiar
with
the
with
the
contract
right.
So
this
this
piece
is
really
maybe
on
the
implementation
detail
to
the
platform
right,
so
we
think
there's
two
options
here:
there's
an
option
where
runtime
calls
the
platform,
so
this
is
mostly
for
style
off.
B
Let's
say
your
platform
has
some
kind
of
an
internal
cue
right
where
individual
functions
get
spun
up.
Let's
say
and
pull
from
the
pool
from
the
platform
and
an
alternative
is
platform,
calls
the
runtime
right.
So
that's
more
similar,
that's
a
2k
native
where
there
is
a
platform
component.
It's
smart
enough
to
scale
up
some,
come
your
function
and
then
send
the
request
to
the
function
so
for
number.
One
any-any
ends
that.
B
Talk
we
picked
this
one
I
think
so.
This
is
by
the
way.
I
was
actually
surprised
when
they
were
looking
at
lambda.
We
actually
deployed
a
few
applications
online
and
run
it
for
a
while,
but
we
didn't
know
how
exactly
it
worked.
So
we
looked
into
it
a
little
bit
and
we
actually
found
out.
It
was
actually
two
api's
exposed
where
she
is
a
little
bit
like
3
or
4
API
endpoints
exposed
where
individual
containers
when
they
get
spun
up.
B
They
actually
say:
what's
the
next
request
and
then
they
pull
it
and
they
work
with
it.
We
think
it's
a
little
bit
more
flexible
approach.
It
removes
a
few
problems
that
you
typically
end
up
dealing
with,
if
you
prefer
number
two
for
auto
scaling
and
whatnot,
but
there's
a
big
downside
right
is
that
if
there
is
some
existing
applications
that
you
want
to
put
in
this
service
world
and
they
deal
with
HTTP
workloads
right
and
they
just
want
to
accept
the
HTTP
streams
or
whatever
it
may
be
right.
B
There
is
some
difficulty
there
now
lambda,
for
example,
they
they
had
some
libraries
and
whatnot
to
make
it
easier
right.
So
once
you
connect
once
you
connect,
let's
say
alb
like
we
do
in
our
case
with
the
lambda
and
use
some
additional
libraries.
You
can
still
do
you
know
basic
HTTP
stuff,
but
there
is
some
shims
involved,
any
other
thoughts
from
the
audience.
B
B
A
So
the
point
is
there
is
like
different
types
of
workloads
and
for
one
particular
type
of
workload.
That's
streaming
always
the
amount
of
one
it's
not
going
to
work,
but
if
you
put
that
one
aside,
then
the
benefits
that
you
get
and
I
think
in
the
other
ones.
In
the
other
points
or
criteria
we're
going
to
get
to
it.
The
benefits
that
you
get
from
model
one
makes
it
appealing,
potentially
and
I,
think
we
can
discuss
it
in
the
next
slide.
I
Ono
application.
Okay.
A
So
before
we
get
to
that
application,
invocation,
contract
right
and
the
application
invocation
contract
I
think
is
the
layer
that,
in
your
runtime
calls
or
basically
the
the
layer
that
your
platform
uses
in
order
to
make
data
or
transfer
data
to
your
application
right.
So
the
three
or
the
four
different
models
that
we've
seen
is
that
there
is
like
a
custom
message
envelope
that
can
be
transferred
and
like.
This
is
the
example
with
lambda.
For
example,
you
get
a
JSON
envelope
from
the
platform
and
your
application
deals
with
it.
A
There
is
the
CDN
the
standard
in
and
standard
outs,
and
it
happens
in
some
of
the
platforms
and
that
you
know
the
parameters,
get
passed
the
application
and
execute
the
binary
for
the
application,
or
it
can
be
HTTP
similar
to
what
Kay
native
does
and
basically
making
a
direct
call
to
the
application,
or
it
can
be
a
combination
of
HTTP
and
cloud
events.
Basically
passing
the
message:
wrapping
it
in
an
envelope
and
passing
it
in
the
form
of
cloud
event
into
the
application
right.
A
A
Custom
message,
envelopes
I
saw
one
to
standard
in
and
standard
out.
There
is
one
HTTP
yeah,
there's
a
few
and
then
HTTP
and
cloud
events.
There
you
go
okay,
so
for
us
we
also
think
that
the
last
one
is
the
better
option,
mostly
because
you
know
piggyback
on
the
on
the
previous
point
that
we
made
about
the
runtime
contract.
A
If
your
runtime
contract
actually
allows
for
pulling,
then
the
way
we
thought
of
the
platform
is
that
you
actually
store
that
information
that
comes
in
or
the
request
that
comes
in
in
the
form
of
the
package
or
a
client
event
package
envelope,
and
then
you
keep
it
and
pass
it
through
to
your
application.
As
you
process,
these
requests
right
and
for
it
to
be
a
standard
like
cloud
event,
is
potentially
more
useful
because
then
you
know
there
is
again
like
given
level
of
flexibility
between.
A
B
Little
quicker,
we
have
about
five
minutes
sure
so
execution
model.
You
know
some
of
the
platforms
offer
only
synchronous
some
offer
both
some
of
her
well
I.
Don't
think
any
of
them
offer
just
a
synchronous.
Maybe
some
do
I'm
sure
someone
does
so
what
is
it
any
hands
for
just
synchronous,
just
asynchronous
both
alright?
Well,
everybody
got
both.
We
also
think
both
finally
yeah
retry
model.
So
this
one
you
know
you
can
really
slice
it
and
dice
it
all
kinds
of
ways.
B
You
know
there
is
something
that
could
be
platform
provided
and
obviously
platform
provided
also
goes
in
all
kinds
of
various
shades
of
that
right.
There
is
a
you
know:
there
could
be
internal
application,
internal
platform,
problems
that
they
asked
to
deal
with,
or
he
has
to
be
maybe
only
application,
specific
retry
logic
or
alternative.
You
just
go
ahead
and
go
leave
it
to
the
client
right.
B
So
what
do
you
think
number?
One
enhance
a
couple
ends
mostly
alright.
We
pick
the
other
one.
The
reason
we
pick
the
other
one
is
it's
mostly
because
we
think
similar
to
how
service
meshes
work
where
they
shift
some
of
that
responsibility
from
the
end-user
from
the
application
writer
right
into
this.
You
know
additional
component
that
you
don't
really
have
to
think
or
worry
about.
We
think
that
the
platform
can
do
a
similar
kind
of
help
or
give
similar
help
to
the
application.
Writer
I'd
only
think
we
had
any
other
specific
reasons.
A
B
B
B
Potentially,
but
then,
if
you
have
deployed
an
application
that
you
know,
let's
say
you've
said
hey,
this
application
is
a
single-threaded.
It
only
deals
with
one
requested
time
right.
You
gave
it
some
initial
sizing
right,
maybe
that's
enough,
not
sure
all
right
traffic
splitting
I
think
this
might
be
the
last
one
we
have.
B
We,
we
kind
of
decided
to
so
I
guess
offer
two
options
over
here
that
there
are
not
necessarily
related
to
traffic
splitting
but
really
influence
traffic
splitting.
So
we
thought
native
a
provisions
that
facilitate
traffic
splitting
would
be
nice
or,
alternatively,
independent
apps,
where,
if
actually
you
have
to
push
your
app
multiple
times
to
get
different
versions
and
then
externally
deal
with
how
you
route
to
those
things
might
be
another
good
option,
thoughts
number
one
who
thinks
number
one
is
good
all
right,
there's
a
few
of
them
number
two.