►
From YouTube: Kubernetes on Supporting 1 Million Bike-Taxi Drivers in Indonesia - Giri Kuncoro & Iqbal Farabi
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes on Supporting 1 Million Bike-Taxi Drivers in Indonesia - Giri Kuncoro & Iqbal Farabi, GO-JEK
At GO-JEK, we build products to help 100 millions of Indonesians commute, shop, eat and pay, every month. With the pace of GO-JEK business that has been growing 6666x in 36 months, we need an infrastructure platform that could scale quickly and deliver products to customers faster. Three years ago, all of GO-JEK services ran in our baremetal machines in Jakarta. Today, they are running across multi-cloud: GCP, AWS, and on-premise datacenter. GO-JEK is in the process of migrating applications to containers, centered around Kubernetes. We chose Kubernetes because of its portability across cloud and dynamic scaling. We will talk about the work that went into building Kubernetes custom resources for our workload deployment, custom controller, integration with Consul to discover external services, and how GO-JEK got to where it is today: processing 350 million internal API calls per second.
To learn more click here: https://sched.co/FuJX
A
A
Okay,
I'll
just
hold
it
in
my
hand:
okay,
yeah,
hello.
Everyone
thanks
for
having
us.
This
is
my
first
ever
keep
corn.
So
my
first
ever
trip
to
China
my
first
ever
trip
to
Shanghai,
of
course,
so
this
is
my
first
ever
time
to
have
having
to
use
VPN
to
access
Twitter
my
work,
email
and
etc.
So
this
new
experience
to
me,
okay.
A
So
today
we
are
going
to
talk
about
Cuban.
It
is
on
supporting
1
million
bike
taxi
drivers
in
Indonesia.
So
first
let
me
introduce
ourselves.
My
name
is
Sigma
Fabi
and
my
friend
here
is
Greek
winter.
Oh
yeah!
We
are
both
our
engineers
from
Gajic
Indonesia,
so
most
of
you
probably
have
not
really
been
familiar
with
go-jek,
so
let
me
introduce
go-jek
first
as
well,
so
we
are
from
Jakarta
Indonesia.
Anyone
has
ever
been
to
Jakarta
before
okay,
two
people.
A
So
if
you
have
been
to
Jakarta,
you
might
be
familiar
with
this
image
right
here,
yeah,
so
Jakarta.
We
have
a
serious
problem
with
our
traffic
there.
So
this
is
what
it
looks
like
in
Jakarta
main
streets
in
rush
hours,
so
our
traffic
jam
is
basically
dysfunctional
and
so
lately
this
has
been
like
this
for
not
in
rush
hours
as
well.
I
mean
almost
every
time,
but
we
Jakarta.
We
are
a
bunch
of
resilient
people,
so
the
lack
of
yeah,
so
the
lack
of
public
infrastructure
enabled
us
to
instead
develop
our
own
transportation
mode
back.
A
Then
we
call
it
object.
So
what
is
an
object?
You
might
wonder
no
Jake
is
this
is
I,
don't
think
you
have
this
in
China.
This
is
it.
It
looks
like
this,
so
it
is
actually
like
a
bike
taxi.
So
like
people
will
drive
a
bike
and
then
you
can
ran
the
if
they
can
pick
you
up
and
then
write
you
to
some
places,
but
it
is
only
it
only
works
for
short
trips.
I
assume
I,
wonder
it.
It
comes
with
a
lot
of
like
problems
because
it
doesn't
really
serve
the
public
effectively.
A
So
in
2010
Noddy
macadam,
our
CEO
and
co-founder
of
go-jek,
he
created
something
that
he
caught
go
Jack.
So
what
is
go
Jack
so
at
the
time
go
Jack
only
serves
as
a
call
center.
Basically,
you
can
call
go
Jack
and
you
ask
them
to
find
new
bike
taxi
driver
and
then
what
it
does.
It
only
lets
say:
matchmaking
you
into
bike
taxi
drivers
and
of
course,
obviously
it
did
not
take
off,
because
such
a
surface
is
very
limited
in
its
scope
and
it
in
what
it
can
serve.
But
then
something
happen
in
2015.
A
The
adoption
of
mobile
phone,
smart
phones,
especially
Android
phones,
have
become
very
o
big
theaters
in
Indonesia.
So
this
is
the
period
in
which
most
Indonesians
can
afford
to
buy
hundred
phones,
and
then
gojek
launched
its
first
mobile
app
in
2015.
So
it
started
with
three
surfaces:
the
first
three
icon
in
this
slide
in
the
screen,
as
you
can
see
for
csgo
right,
of
course,
this
bike
taxi
that
the
second
one
is
go
mark.
A
It
is
say,
if
you
want
to
have
your
grocery
sent
to
your
home
or
some
address
you
can
use,
go
mat
and
then
go
send
is
to
basically
send
small
items
from
one
point
to
another.
It's
it
starts
very
with
this
three
humble
surface,
but
in
the
span
of
three
years
from
2005
Steen
to
2017,
Gajic
has
become
a
big
surface
in
your
Nisha.
We
have
at
least
18
surfaces
if
I
count
correctly
I.
A
Don't
really
remember
the
correct
number,
the
exact
number
so
we
have,
and
most
some
of
our
surfaces
are
the
market
leader
in
Indonesia.
So,
for
instance,
we
have
go
food.
Raw
food
is
the
biggest
delivery,
surface,
I
think
I,
don't
know
in
the
world
outside
of
China,
of
course,
because
Chinese
you
China,
have
more
much
more
people
than
assets.
That's
not
fair!.
A
But
we
are
bigger
than
all
the
leafy
resurfaced
in
India
combined,
so
the
tree
delivery
surface
in
India,
Zomato,
swiggy
and
uber
food
yeah.
We
are
bigger
than
that
and
then
we
have
go
pay.
The
blue
alert
that
you
can
see
right
there.
It's
the
biggest
evil
act
surface
in
Indonesia.
Of
course,
our
go
ride.
Our
riding
surfaces
as
well
combined.
We
serve
more
than
1
million
surfaces
per
day.
Oh
no,
no,
no,
sorry,
100
million
surfaces
per
day,
and
then
we
went
from
only
a
handful
of
drivers.
A
Back
then
in
2010
it
was
just
probably
around
20
to
40
drivers
into
1
million
drivers.
Today,
in
2018
we
go
check,
acquire
three
financial
technology
companies
in
Indonesia
and
of
the
first
one
is
me
trans.
It
is
the
one
of
the
largest
online
pavement
surface
in
Indonesia.
The
second
one
is
car2go.
It
is
one
of
the
largest
offline
pavement
surface
in
Indonesia
and
then
the
third
one
is
Maupin.
It
is
one
of
the
largest
financial
surface
for
the
unbanked
in
Indonesia,
so
we
also
just
recently,
two
months
ago
we
launched
our
surface
in
Vietnam.
A
It's
got
no
fear.
We
are
now
in
beta
eating
my
correct
intel
on
beta
fashion,
and
then
we
are
going
to
launch
soon
in
Singapore,
so
yeah,
that's
the
story
of
go-jek.
Of
course,
with
this
great
growth
and
great
number
of
apps
comes
great
headache,
so
I
will
show
you
a
typical
workflow
of
an
app
in
go
check.
So
this
is
an
anonymized
architecture
of
micro
surfaces
involved
in
processing.
A
single
order
in
go
pay
our
award
surface.
So
all
of
this
should
happen
in
order
of
less
than
three
to
five
seconds.
A
Otherwise
the
customer
will
get
upset.
So
we
have
some
surfaces,
we
have
some
workers,
we
have
some
reading
and
writing
to
and
from
Kafka
and
then
reading
and
writing
to
and
from
database.
Of
course,
managing
this
many
services
could
be
a
nightmare.
Even
the
stories
taught
that
in
the
our
dark
days
early
days
it
takes
for
engineers
to
have
their
infrastructure
ready.
It
takes
like
a
week,
but
there
was
dark
ages.
We
are
way
past
day.
So
what
we
did
was
we
try
to
automate
everything.
Obviously
we
went
with
infrastructure
as
code.
A
B
So,
by
keeping
this
in
mind,
we
began
our
gojek
journey
to
the
land
of
containers.
This
is
a
typical
picture
that
you
would
see,
there's
a
bunch
of
good
news,
they're
being
shipped
by
what
do
you
call
that
ship?
Ok?
So
anyway,
we
decided
to
migrate
from
VMs
to
containers
because
of
the
pickle
3
advantages
that
containers
can
bring.
Right.
B
B
B
So
at
that
time,
as
an
entire
organization.
Obviously
we
were
very
new
at
that
time
to
containers
and
containerization
system,
so
we
envisioned
Project
X
to
enable
the
product
engineers
or
developers
to
create
their
applications
easily
in
our
container
based
infrastructure.
We
envisioned
Project
X
so
that
the
developers
could.
Similarly,
the
expression
would
be
similar
to
creating
applications
on
Heroku,
for
instance,
and
then
by
abstracting
away
the
complexity
of
containers
and
consume
system
from
developer
perspective.
B
We
hope
that
developers
don't
have
to
gain
the
knowledge
of
container
and
container
acquisition
system
here
when
we
design
first
the
project
X.
We
were
also
new
as
part
of
the
infrastructure
team.
We
were
actually
very
small
and
we
were
new,
so
we
were
not
only
looking
on
kubernetes
at
that
time.
We're
looking
on
mesos
as
well
and
nomads
do
help
us
in
deploying
container
application.
So
we
hope
to
learn
as
we
go.
We
decided
that
Jax
could
deploy
in
both
kubernetes
and
Nomad
infrastructure.
B
Here's
a
typical
workflow
that
a
developer
will
face
wherever
they
wanted
to
deploy
their
application,
so
they
will
go
to
project
X,
web
UI
or
CLI
and
then
configure
their
metadata,
whatever
information
required
on
their
application
and
then
Project
X
will
generate
something
called
Gerald
manifests,
which
is
a
generic
manifest,
which
represents
our
infrastructure
based
infrastructure.
In
this
case
and
then
Project
X
will
package
the
applications
into
containers
and
push
to
to
docker
registry.
B
In
this
case,
Project
X
will
generate
in
this
case,
depending
on
the
choice,
either
commodities
or
nomad,
manifests
and
then
deploy
to,
depending
again,
depending
on
the
choice,
kubernetes
or
Nomad
infrastructure.
So
the
idea
here
developers
did
not
have
to
care
where
their
applications
were
running.
That
was
the
idea
and
then
we
as
an
infrastructure
team.
We
had
a
privilege
to
mandate
everyone
to
migrate
to
this
new
pooling,
and
then
we
started
rolling
out
what
happened
at
that
time.
It
went
very
badly.
B
We
faced
a
lot
of
issues
in
both
technical
as
well
as
well
as
non-technical
issues.
At
least
we
faced
three
technical
issues
here.
First
Project
X
did
not
have
a
well-designed
abstraction,
because
obviously
we
were
trying
to
make
a
super
generic
attraction
that
are
good
for
both
kubernetes
and
nomad
in
this
case,
so
the
implication
of
this
is
Project.
X
was
not
easily
extendable,
as
you
guys
know,
the
kubernetes
community
is
is
actually
very
dynamic
and
very
fast
pace.
There's
always
really
something
new.
B
Every
three
months
and
probably
every
week,
there's
always
new
patches
or
new
bug
fixes
something
like
that.
So
this
makes
it
hard
to
adopt
bit
changes
that
community
bring
to
our
rulings,
and
last
thing
we
tried
to
attract
too
much
so
that
this
deployment,
who
became
a
black
box,
became
a
black
box
for
the
developers.
B
So
most
of
the
time
whenever
their
production
issues
arise
or
deployment
issues
arise,
they
always
blame
us.
They
always
blame
us.
The
infrastructure
team
and
blame
the
project.
X
where's,
the
the
real
issue.
Most
of
the
time
are
in
the
company's
configuration
itself,
for
example,
and
then
we
the
mistake
at
that
time,
we
did
not
give
capability
to
developers
to
see
their
application
logs
because
of
this
as
well.
That
was
one
of
the
biggest
complaints
coming
from
the
developers.
B
Apart
from
technical
issues,
we
were
also
facing
at
least
two
non
technical
issues.
At
that
time,
the
the
knowledge
was
new
right.
The
entire
organization
was
new.
We
were
pretty
small
at
that
time.
The
migration
effort
was
to
centralize
centralizing
within
the
small
infrastructure
team
and
the
knowledge.
The
knowledge
that
we
were
trying
to
gain
from
the
community
was
siloed
within
that
single
team.
B
Only
so
we
could
not
keep
the
learning
pace
with
changes
in
the
community
and
and
the
knowledge,
and
then
the
the
biggest
mistake
was
start
started,
starting
to
roll
out
with
a
bang,
so
we
mandated
everyone
all
teams
in
the
company
to
migrate
at
one
time,
instead
of
onboarding
small
number
of
teams
at
at
a
time.
So
the
result
of
this
is
multiple
issues
were
raised
by
multiple
teams.
B
At
the
same
time,
and
since
we
were
very
small,
we
did
not
have
any
bandwidth
to
squash
the
ball,
the
other
bugs
iterate
and
move
on,
and
things
like
that.
However,
not
all
is
vain
in
winning
we
earn
in
losing
we
learn.
We
do
learn
from
all
of
our
mistakes
so
from
the
mistakes
we
decided
to
decentralize.
Instead
of
centralizing,
the
the
migration
import
we'd
be
centralized
the
VM
to
container
migration
effort
to
all
the
teams.
B
The
implication
of
this
is
some
teams
have
different
approaches.
A
lot
of
approaches
erase,
however,
in
a
run
well
organization,
best
ideas
will
eventually
converge.
So,
as
I
told
you,
there
are
multiple
approaches
here.
Some
teams
decided
to
run
container
workload
in
AWS
running
on
kubernetes
cluster
deployed
by
cops,
for
example,
for
their
specific
teams.
They
were
using,
for
instance,
a
Debian
I
plug
in
because
they
had
a
hybrid
workload.
B
Some
teams
decided
to
use
help
to
package
the
application
into
helm,
charts
and
then
deploy
using
help
within
their
see
ICD.
They
didn't
use
Project
X.
Some
teams
actually
built
their
own
custom
controller
using
good
builder
pod
whitewabbit
this.
This
was
their
because
they
have
a
specific
use
case
to
have
multiple
deployments
at
the
same
time
for
serving
one
single
application.
B
This
is
a
good
use
case
in
our
data
science
infrastructure
team,
because
they
wanted
to
test
multiple
different
predictive
models,
for
instance,
for
one
single
application
to
test
which
one
works
best
for
particular
use
case
or
or
customers,
and
then
from
the
mistakes.
We
could
improve
our
project
ex
base
deployment
tool.
B
We
work
with
small
number
of
teams
at
one
at
a
time
and
then
fix
issues
there,
iterate
and
then
move
on
and
then
on
board.
Another
team,
as
a
result
of
this
project
ex,
has
become
primary
deployment
tool
for
our
go-jek
international
expansion
projects.
As
Iqbal
mentioned
earlier,
we
just
launched
in
we
just
launched
project
services
in
Vietnam,
and
then
we
will
be
launching
in
in
Singapore
in
a
few
weeks
and
Thailand
soon
very
soon,
because
of
adopting
the
containers
and
and
kubernetes,
we
were
able
to
set
up
vietnam
infrastructure
much
much
faster.
B
It
took
us
only
four
days
to
set
up
from
from
scratch,
really
scratch
there
in
vietnam
and
then
because
of
this.
This
is
by
the
way
called
cookie
cutter
model
because
of
the
portability,
a
nature
that
kubernetes
offered.
We
were
able
to
repeat,
or
basically
replicate
the
infrastructure,
to
run
the
whole
business
very
very
quickly.
For
instance,
here
we
launched
in
in
Vietnam
it
took
us
very
fast
and
then
we
were
able
to
biggest
companies
it's
very
portable.
We
were.
B
We
were
also
running
state
pool
services
on
kubernetes
in
in
Vietnam
here.
As
a
result,
the
DBS
had
fewer
much
periodic
and
improve
their
reliability
and
then
with
containers
we
improve
our
civilization
and
memory
utilization
became
very
efficient.
Then
we
extended
Project
X
with
some
central
conflict
management
with
namespaces
to
enable
easy
multi
deployment
and
multi
data
center
and
then
finally,
the
most
important
one.
We
could
save
up
to
60%
infrastructure
costs
per
year
per
country
in
Vietnam
case.
B
A
As
Jerry
mentioned
earlier,
one
of
the
problem
that
we
faced
when
we
were
launching
with
project
access
that
project
X
itself
became
a
black
box
for
our
product
engineers.
It
is
a
problematic
because
one
when
things
went
wrong,
our
product
engineers
will
not
be
able
to
figure
out.
Work
went
wrong,
so
we
created
a
centralized
logging
system
called
burrito
lock.
So
the
picture
you
see
here
is
the
burrito
refer
in
Indonesia,
so
people
what
people
do
in
burrito,
reforest
de
transport
would
lock
from
the
forest
to
the
city,
and
then
they
have
like
floating
market.
A
A
So
what
is
burrito
lock
so
actually
the
it
has
several
elements
and
of
course
it
utilized
several
CNCs
projects,
so
in
damar
cut
what
you
can,
what
a
product
engineers
can
do
is
they
can
register
their
surfaces
there
and
then
once
they
register
their
surfaces,
their
burrito
rock
will
install
an
agent
in
their
surfaces
and
then,
with
with
the
router
burrito
log,
we'll
be
able
to
route
lock
their
logging
data
from
from
your
surfaces
into
several
lock
clusters.
So,
for
instance,
in
here
there
is
a
lock
Laster
a
and
then
once
the
lock
is
received.
A
A
So
that's
so
far
now
burrito
lock
has
been
in
production
since
what
I
think
several
months
ago,
and
then
right
now
we
are
even
pass
the
1,000
transaction
per
second
and
if
I
recall
correctly
at
one
time,
we
had
trouble
with
with
this,
and
we
like,
consult
several
people
and
find
out
that
I
think
Google
Apps
driver
is
only
processing
less
than
that.
But
I
don't
know.
A
The
other
problem
that
DeeDee
mentioned
earlier
is
that
the
knowledge
about
containers
and
cloud
native
applications
and
kubernetes
have
became
siloed
in
one
particular
team.
At
the
time
when
we
launched
project
X
and
it
it
is
not
good.
We
are
not
saying
that
centralized
effort
will
always
be
bad.
In
fact,
for
burrito
lock,
we
are
doing
centralized
effort
there,
but
what
we
are
saying
that,
when
knowledge
of
a
particular
technology
that
our
organization
is
adopting
siloed
into
one
only
handful
of
people,
it
is
not
good.
A
So
what
we
another
remedy
that
we
took
is
that
we
create
a
session
called
cognitive
saturdays.
This
is
basically
knowledge
sharing
sessions
that
we
conduct
every
Saturday
within
our
organization.
The
goal
of
this
session
is
to
be
able
to
be
able
to
spread
the
knowledge
of
cognitive
applications,
of
course,
cognate
cognitive
technologies,
of
course,
including
kubernetes,
and
all
the
CNCs
projects
to
the
whole
organization.
B
We
have
been
running
100%
container
workload
in
Vietnam
on
kubernetes,
and
then
we
have
been
also
running
100%
containerized
workload
for
data
science
infrastructure,
but
we
at
current
state
we
are
still
in
the
middle
of
migrating,
the
entire
existing
production
traffic.
So
we
might
be
able
to
share
that
learnings
next
year,
hopefully,
and
then
the
next
thing
is,
we
are
looking
into
migrating
stateful
applications
as
well.
We
have
proven
that
running
DB
services
in
Vietnam
improve
reliability
on
kubernetes.
B
So
next
we
are
looking
into
menu
to
transform
communities
TVs
into
something
like
object:
storage,
s3
to
to
store
our
massive
amount
of
kyc
images,
for
example,
and
our
driver
profile
pictures,
for
example,
because
of
compliance
reason
and
we
are
actually
adopting
stolen.
We
have
our
own
Fork
of
stolen.
If
you
guys
don't
know,
stolen
stolen
will
help
us
in
configuring
high
availability,
high
available
Postgres
clusters
on
kubernetes
in
vietnam.
B
C
C
Yeah
yeah,
you
are
migrating.
The
applications
from
the
we
entered
to
the
kubernetes,
so
they're,
possibly
Debu.
Some
situations
that
the
hyper
hyper
network
that
you
need
to.
For
example,
there
is
some
application
in
the
Vienna
need
to
communicate
with
the
applications
inside
in
the
in
the
kubernetes.
Sometimes
the
the
application
in
the
kubernetes
need
to
communicate
with,
via
the
applications
outside
the
kubernetes
lab.
B
B
In
our
head
of
this
cloud
infrastructure,
for
instance,
we
were
using
a
double
AC
and
I
plug
in
so
that
the
pods
get
the
same.
Ip
ranges
in
the
v
pc
the
pods
get
routing
through
the
AWC
fabric,
so
that
vm
can
reach
to
the
pod
direct
and
then
pod
can
be
TM
directly,
as
as
we
don't
have
any
picture
here,
but
they
they
don't
differentiate
that.
So
this
is
the
exact
use,
because
we're
migrating
step-by-step
and
we
are
starting
to
hit
this
issue.
B
There
are
many
Center
plugins
that
can
do
that,
especially
for
instance,
calico
Coleco,
you
can,
if
you
can
figure
it
properly,
you
can
have
a
routable
IP
s.
If
you
do
bgp
peering
and
all
that
you
can,
you
can
configure
pots
to
be
routable
directly
to
reach
the
VMS
in
the
same
VDC
and
and
many
other
options
actually,
but.