►
From YouTube: Improving Dev Experience: How We Built a Cloud Native Dev St... Srinidhi S & Venkatesan Vaidyanathan
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Improving Dev Experience: How We Built a Cloud Native Dev Stack At Scale - Srinidhi S & Venkatesan Vaidyanathan, Razorpay
The session will shed light work of improving Razorpay's dev experience using a bunch of open source tools that scales to 100's of engineers,in a secure and compliant fashion We talk about extending cloud native development to local desktop,how it integrates with our overall kubernetes driven CI/CD workflows.In a nutshell,the session describes building a dev centric packaged environment for reducing their cognitive load while developing sofware This talk brings clarity to the application cluster development , and shows the work being done on aggregating various open source solutions like helmfile for describing and setting up a micro service fleet , traefik routing,header propagation for ephemeral service access ,helm hooks for auxilary app requirements like queues,databases,vendor cloud components, hot reloading and devspace for integrated dev local development/debugging and autoscaler,janitor,botkube etc for cluster segregation and management In the end ,this talk hopefully aligns the developers,practitioners and operators to the benefit of local development with faster iterations , customizable dev tools in remote kubernetes cluster with an extremely simplified , cost effective ,git ops native and agile solution impacting the entire org's dev productivity
A
Today
we
would
love
to
talk
about
improving
the
developer
experience
and
how
we
build
a
cloud
native
dev
stack
for
hundreds
of
engineers,
a
quick
overview
into
what
razer
pay
does
razer
pay
enables
frictionless
payment,
banking
and
lending
experiences
for
different
classes
of
merchants
of
various
scales
and
sizes.
Today
we
process
billions
of
transactions
for
millions
of
merchants
across
the
country.
A
Razer
bay
has
been
at
the
forefront
of
innovation
over
the
last
several
years
in
terms
of
transforming
the
financial
ecosystem
in
the
country,
a
big
motivation
into
why
we
actually
embarked
upon
this
journey
over
the
last
four
years.
Our
engineering
head
crown
head
count,
has
actually
grown
up
by
10x
in
that
process
we
have
scaled
up
our
teams
into
full-fledged
pods
and
views
four
views
and
30-plus
bots.
We
had
embarked
upon
microservices
journey
a
couple
of
years
back
and
today
we
have
about
100
plus
micro
services
on
it.
A
Just
in
the
last
two
years,
the
number
of
micro
services
has
increased
to
about
50.
we've
had
three
company
acquisition:
three
acquisitions
across
the
course
of
time
that
has
led
to
a
polyglot
stack
of
various
languages
like
php,
gold
and
python
java,
etc.
Overall,
we
do
roughly
about
thousand
finder
deployments
per
month.
A
Now
a
quick
look
at
what
our
ci
cd
practice
looks
like.
This
is
not
really
very
different
from
the
way
many
companies
operate,
but
just
to
give
a
perspective
and
sort
of
build
up.
The
motivation
for
the
kind
of
problems
that
you're
dealing
with
at
a
very,
very
high
level
developers
commit
code
into
github.
A
We
heavily
use
github
ci
integration,
github
actions
for
our
ci
integration,
so
the
git
of
actions
basically
builds
images
which
are
docker.
Images
runs
a
variety
of
unit
tests
and
pushes
it
into
our
private
doc
registry
as
soon
as
these
images
are
available,
but
the
developers
typically
start
deploying
that
code
with
spinnaker
spinnaker
is
a
open
source
developer
cd
platform
created
by
netflix.
So
the
deployment
process
is
very
similar
in
a
pre-broad
environment.
A
They
they
run
a
variety
of
integration
tests,
load
tests
if
needed,
and
a
bunch
of
other
tests.
Once
the
developer
is
satisfied
with
the
test
reports,
the
developer
goes
on
to
deploy
the
code
in
a
canary
environment.
Again
via
spin
occurs.
Kayanta
we
run
a
variety
of
canary
threshold
metrics,
and
what
that
means
is
that
we
sort
of
leverage
our
monitoring
and
tracing
infrastructure
for
actually
making
sure
that
the
candidate
has
passed
once
the
candidate
thresholds
are
met
or
processed.
A
The
deployments
deployment
is
moved
forward
into
production
in
case
that
running
calorie
thresholds
do
not
pass
the
deployment
is
rolled
back
in
general,
we
use
aws
as
our
infrastructure,
where
most
of
our
all
our
code
runs.
We
are
a
heavy
problem
of
kubernetes,
very
specifically
somewhere
in
2016
late
2016
is
when
we
start
a
quick
community
journey.
We
can
proudly
say
that
we
are
we're-
probably
the
first
company
in
the
country
to
have
actually,
you
know,
managed
a
full
production,
great
kubernetes
ecosystem.
A
Well,
today,
we
run
managed
kubernetes
via
eks,
like
I
mentioned,
because
of
the
fact
that
we
are
on
kubernetes.
All
our
all
of
our
packaging
is
done
by
apple.
Spinnaker
is
something
that
we
mentioned.
Our
entire
infrastructure
is
on
code
via
terraform
and
specifically
for
sensor
infrastructure
provisioning.
We
have
enabled
atlantis
where
developers
can
actually
come
and
provision
any
cloud
resources
that
they
probably
want
and
again
github
actions
is
something
that
we
use
for
all
the
continuous
integration
and
the
docker
image
building.
A
So
with
this,
you
know
principle
in
mind.
We
can
sort
of
notice
that
there
is
a
tremendous
amount
of
deployment
time
that
the
developers
spend
in
terms
of
you
know
rolling
out
their
features.
You
know
faster
and
actively
into
production,
with
specifically
in
the
development
dependencies
themselves
because
of
the
number
of
engineers
and
the
number
of
micro
sources
that
you're
sort
of
dealing
with
it.
A
This
leads
to
a
sequential
development
process,
and
what
that
means
is
that
imagine,
there's
a
single
service
and
there's
a
bunch
of
developers
who
are
working
on
those
features.
Each
developer
will
have
to
take
some
time
to
actually
deploy
their
code
into
a
pre-broad
environment,
and
this
development
process
is
largely
sequential
and
they
have
to
coordinate
with
other
developers
in
terms
of
deploying
so
that
their
changes
are
not
overwritten
by
other
developers.
The
other
part
of
the
problem
is
because
there's
a
heavy
dependency
on
aws
resources.
A
Most
of
the
aws
resources
are
mock
on
the
local
development
environment,
which
leads
to
a
lot
of
you
know,
confidence
issues
in
terms
of
pushing
the
code
to
production
testing.
Again,
even
a
single
line
of
code
change
requires
the
entire
development
process.
Entire
deployment
process
before
the
developers
can
start
testing
their
code.
The
on
the
right
hand,
side
is
another
set
of
problems
with
the
shared
environment
and
what
I
mean
by
shadow
environment
is
this.
A
So
I,
as
a
developer,
I'm
working
on
service
a
while
service
a
has
a
dependency
on
service
b.
Obviously,
because
my
change
is
only
affecting
service
a,
I
don't
need
to
necessarily
build
service
b,
while
I'm
testing
my
service,
imagine
there's
a
developer
who's
working
on
service
b
and
they
end
up
changing
the
deployment
of
service
being
on
a
pre-broad
environment.
A
My
test
starts
breaking
and
how
do
you
sort
of
cost
effectively
scale,
not
just
my
own
individual
services,
but
also
the
shared
environments,
and
today,
that's
quite
difficult
unless
and
until
there's
a
lot
of
costs
thrown
at
it?
The
other
part
of
the
problem
is
that
this
leads
to
also
a
fact
of
the
shadow
environment
imposing
a
lot
of
constraints
in
terms
of
being
able
to
demo
the
product
or
the
features
to
the
business
of
the
product
counterparts.
A
Again
there
could
be
demo
restrictions
or
there
could
be
you
know,
as
I'm
doing
things
can
change
up.
So
this
leads
to
a
lot
of
you
know
time
loss
in
one
sense,
the
third
part
of
the
problem
in
the
nature
of
the
environment
that
we
live
in,
we
indulge
in
a
variety
of
integrations
with
third
party
gateways,
payment
gateways,
partners,
etc.
A
Many
of
these
integrations
sometimes
can
run
from
weeks,
maybe
even
sometimes
a
month,
and
what
that
means
is
that
every
single
such
integration
requires
a
separate
service
fleet
of
sorts
which,
which
is
useful
for
that
particular
integration
themselves
itself.
So
how
do
you
sort
of
really
build
these?
These
kind
of
you
know
shared
environment
in
a
cost-effective
fashion.
A
While
all
of
this
is
there,
there
are
also
challenges
on
the
infra
provisioning
and
specifically,
like
I
mentioned
all,
our
infrastructure
is
on
code
and
terraform.
This
requires
developers
to
go
through
a
cognitive
overload
of
learning
data
from
dsl
for
provisioning,
even
simple
things
like
an
sql
endpoint
or
an
sms
endpoint.
More
specifically,
like
I
mentioned,
because
of
the
entire
development
process,
debugging
even
trivial
one
line
change
sometimes
requires
the
developers
to
context
switch
between
the
actual
application
and
probably
like
a
logging
platform
or
a
distributed
tracing
platform.
A
More
specifically,
there's
a
lot
of
time
that
is
spent
in
just
building
and
deploying
the
images
themselves
depending
on
you
know
the
complexity
and
the
nature
of
the
tests
or
the
unit
tests
themselves.
There's
a
significant
amount
of
time-
and
this
means
that
feature
development
is
largely
sequentially
iterative
and
it's
not
independently.
It
cannot
be
done
today.
A
Need
to
simplify
the
developer
workflow,
we
need
to
reduce
the
rollout
time
to
rollout
or
the
cycle
time,
as
I
say,
in
an
independent
fashion
and
in
a
in
a
simple
fashion.
With
that,
I
would
like
to
hand
over
to
srinidhi
my
colleague
who
will
walk
you
through
the
journey
on
how
you're
solving
these
problems.
B
B
The
third
important
one
is
to
have
a
hassle
free
onboarding,
which
means
that
we
wanted
to
onboard
application
with
minimal
changes
and
also
not
drastically
change
the
deploy,
development
or
deployment
life
cycle
and
last
but
not
the
least.
We
wanted
to
have
a
cost,
effective
solution
that
will
make
sure
that
we
don't
burn
cash
in
order
to
get
ephemeral
infrastructure.
B
So
the
code
name
of
this
particular
solution
is
dev
stack
and
this
is
a
razor-based
journey
towards
a
better
development
life
cycle.
So
let's
try
to
understand
the
solution
by
asking
a
series
of
questions
and
answers
and
coming
to
a
logical
conclusion
before
the
demo.
The
first
one
is
on:
how
do
we
bypass
the
ci
cd
loop
for
the
iterative
development
and
the
need
is
to
have
or
directly
expose
the
local
code
on
to
the
cloud
environment?
B
So
as
we
one
version
of
the
solution,
we
went
ahead
with
daily
presence
which
followed
a
proxy
based
approach.
So
telepresence
was
a
client
that
was
sitting
in
both
the
local
system
and
the
remote
cluster
and
made
sure
the
connection
happens
through
a
tunnel.
It
took
care
of
the
dns
resolution,
volume
mounting
and
also
networking,
but
there
was
a
major
drawback
due
to
the
whole
tunneling
approach,
which
was
that
the
responses
were
slow.
B
Also,
has
port
forwarding
and
lock
streaming
that
provided
scope
for
further
features,
as
as
it
all
looked
good,
there
was
a
hiccup
sort
of
a
limitation
per
se
due
to
the
container
restarts,
so
the
container
restarts,
which
are
bound
by
kubernetes
probe,
were
flaky
at
times
and
also
are
not
completely
reliable.
So
we
wanted
something
that
was
better
and
faster.
So
the
next
question
is
on:
how
do
we
avoid
this
container
restarts?
B
So
in
order
to
avoid
the
container
restarts
we
had
to
put
in
a
library
inside
the
container
that
will
take
care
of
the
hot
reloading?
So
there
are
a
lot
of
libraries
per
language
like
compiled,
email
for
golang,
nodemon
for
node.js
and
so
on.
Who
rely
on
watching
for
the
files
that
are
changed?
These
are
the
files
that
are
synced
by
dev
space
and
then
it
rebuilds
the
binary
and
then
runs
the
new
one
which
is
made
available
to
the
server.
So
this
effectively
avoids
the
container
restart,
thus
not
breaking
the
flow.
B
So
we
could
have
a
generic
docker
container
per
language
built
and
use
that
container
with
the
dev
space
command
in
order
to
make
sure
this
on
boarding
is
seamless.
So
all
of
these
three
questions
helped
us
to
solve
the
problem
of
local
sync,
so
we
now
short
circuited
the
feedback
loop
by
just
running
a
command.
B
So
next
question
is
on:
how
do
we
provide
or
orchestrate
multiple
services?
How
can
the
developers
declaratively
define
and
apply
service
dependencies
and
the
solution
for
that
is
helm
file,
which
is
a
wrapper
on
top
of
health?
So
hello
file
helps
us
to
compose
several
charts
together
to
create
a
comprehensive
deployment
artifact
for
anything
from
a
single
application
to
the
entire
infrastructure
stack.
B
So
we
define
a
term
called
service
fleet
here,
which
is
a
collection
of
micro
services
that
are
required
by
the
developer
for
his
or
her
workflows.
So
this
helm
file
works
seamlessly
with
the
existing
helm
packages,
as
all
of
our
applications
already
had
helm
packages,
and
we
didn't
want
to
or
need
to
write
extra
thing.
B
We
just
had
to
wrap
all
of
them
in
a
single
aml
file
and
provide
it
as
it
is
given
in
the
in
this
right
side
of
the
of
the
screen
so
helen
file
when
took
care
of
all
the
cuban.
It
is
resources
orchestration
and
that
solved
major
of
a
problem
with
respect
to
providing
kubernetes
resources
like
your
deployment
service,
ingress,
job,
cron,
job
and
whatnot,
but
the
application
is
not
wholesome
without
the
other
requirements
which
are
like,
for
example,
sqs
or
databases
or
aws
resources
which
are
not
completely
on
kubernetes.
B
So
the
next
question
is
on:
how
do
we
provision
ephemeral
infrastructure
resources,
so
we
used
or
relied
upon
help
hooks
for
this
provisioning,
so
helmholtz
provided
a
plug-and-play
model
to
maintain
the
dynamics
of
applications,
auxiliary
requirements.
What
we
mean
by
that
is,
we
have
written
two
to
three
or
per
requirement
base
helmholtz,
which
can
be
plugged
into
application
based
on
a
requirement.
So
if,
for
example,
an
application
is
using
sqsqs,
they
can
plug
in
a
sqs
configurator
as
a
hook,
whereas
an
application
using
kafka
queues
could
plug
in
a
kafka
configurator.
B
Similarly,
for
all
the
other
resources,
the
plug-and-play
model
will
fit
in
and
in
order
to
also
avoid
the
aws
overridden
or
maintaining
aws
resources,
which
was
done
by
a
terraform.
We
used
local
stack,
which
provided
a
framework
for
mocking
aws
components.
So
this
is
how
a
helm
file
workflow
will
look
like
on
running
the
command.
The
chart
is
being
verified
and
it
runs
a
bunch
of
pre-installed
books
which
make
sure
that
the
auxiliary
requirements
for
the
application
is
up,
and
then
it
loads
the
charts,
the
kubernetes
resources
and
deploys
them
into
the
kubernetes
cluster.
B
And
then
some
post
install
books
as
well,
which
will
which
can
include
like
the
english
route,
configuration
or
validation
of
the
manifest
generated
and
make
sure
that
the
ephemeral
service
or
the
fml
infrastructure
per
developer
is
available
by
just
running
one
command.
So
this
is
how
we
solve
the
problem
of
having
a
streamlined
workflow
for
the
developers
in
order
to
bring
up
their
ephemeral
service
fleet.
B
So
now
with
fml
infrastructure
using
helm
file
and
local
sync
using
devspace.
There
is
one
piece
in
the
whole
puzzle
that
needs
to
be
sorted
which
is
on.
How
do
we
make
sure
the
traffic
is
routed
to
the
right
user
service
so
for
that
we
use
header-based
routing
our
ecosystem
or
our
kubernetes
cluster
have
have
been
using
traffic
for
ingresses
and
we
upgraded
that
to
traffic
2.0
with
supported
header
based
routing
out
of
the
box.
B
So
the
traffic
ingress
route
configuration
will
have
multiple
rules
in
order
to
guide
it
to
the
dynamic
routing.
So,
for
example,
in
in
the
right
hand,
side
of
the
presentation
we
have
two
services
which
are
api
web
and
api
and
based
on
the
header
traffic,
will
make
sure
that
the
request
that
comes
with
the
header
is
propagated
to
the
api
web
studio
service,
which
in
turn
puts
the
request
into
the
api
web
streaming
the
deployment
or
a
part.
B
So
this
is
how
we
have
enabled
header
based
routing
to
solve
the
routing
problem,
and
the
next
set
of
question
is
on:
how
do
we
make
sure
the
upstream
services
are
routed
properly
as
well,
and
we
use
header
propagation
there
and
we
have
piggybacked
on
open
tracing
where
open
tracing
by
default,
propagates
the
header-
and
we
rely
on
that
where,
at
every
service,
the
traffic
will
read
the
header
and
route
it
to
the
appropriate
service.
Let's
walk
through
a
use
case
on
on
how
does
that
routing
work?
The
routing
overview,
for
example?
B
So
we
have
three
developers
who
are
working
on
both
of
these
micro
services
where
developer
one
is
explicitly
working
on
app
one
and
developer.
Two
is
working
on
app
two,
whereas
developer
three
is
working
on
a
feature
that
spans
between
app
one
and
app.
Two
now
by
running
the
helm
file
command,
the
developers
would
have
configured
the
infrastructure.
B
Now,
let's
see
how
the
request
routing
happens
so
developer.
One
on
passing
the
request
header
devon.
In
the
request,
the
ingress
route
that
is
present
in
front
of
app
one
will
make
sure
that
the
request
is
routed
to
the
dev
one
instance
of
the
app
one
and
then
the
request
will
propagate
into
the
app2
and
given
that
the
dev1
label
is
not
present
in
the
configuration
it
will
route
it
to
the
default,
shared
infrastructure.
B
That
is
running
because
the
configuration
would
be
there
and
also
in
the
last
case,
when
the
header
feature
one
is
passed,
the
request
will
be
routed
to
the
feature
one
instance
of
app
one
and
then
the
feature
one
instance
of
the
app
two
as
both
of
these
are
present.
So
this
is
how
the
request
routing
will
now
happen
across
the
microservices,
and
this
enables
us,
in
order
to
run
only
a
subset
of
microservices
required
for
the
functioning
of
the
application.
B
So
let's
move
on
to
some
of
the
practical
implementation
of
this
solution
and
see
how
it
works
in
real
world,
and
this
is
the
demo,
let's
open
our
terminal
and
run
a
command
now,
which
will
enable
us
to
create
a
the
fml
infrastructure
lm
file.
Sync
is
the
command
that
is
required
and
the
command
now
has
started
to
create
ephemeral
resources
for
a
couple
of
services
and
let's
look
into
what
they
are.
While
the
build
is
happening
or
the
deployment
is
happening,
so
we
have
a
file
called
helm.
B
The
image
here
is
the
commit
id
of
the
dep
of
the
branch
that
I
am
working
on
and
also
the
same
is
the
docker
image
tag,
so
the
image
tag
we
use
in
razer
pay
is
the
commit
hash
of
the
of
the
branch
that
we
are
working
on
and
just
take
a
notice
of
one
thing,
which
is
called
the
dev
stack
label.
So
this
particular
label
holds
the
key
for
the
fml
infrastructure.
All
of
our
infrastructure,
ephemerally
are
created
with
the
suffix
of
this,
this
particular
label.
B
The
same
thing
works
for
the
other
micro
services
as
well.
Let's
quickly
check
what
are
the
kubernetes
resources
that
it
has
created.
So
when
I
do
cubes
it
will
get
parts
hyphen
api.
We
can
see
that
there
is
a
screen
in
the
demo
that
is
running
and
also
if
we
can
just
check
the
dashboard
resources.
There
is
also
be
a
srini.
B
The
demo
that
is
running
also
take
a
notice
of
all
the
other
labels
that
are
working
this,
and
this
is
the
live
infrastructure
that
we
are
running
on
and
all
the
other
developers
are
parallely
working
while
we
are
demoing.
So
this
is
the
standard
dashboard
of
razer
pay.
Let's
see
how
we
can
access
our
specific
resource,
so
we'll
use
a
plugin
called
mod
header
which
injects
the
header
into
the
request
that
go
through
your
browser.
You
can
also
use
postman
by
adding
this
header
or
use
curl
request
to
to
add
the
header.
B
So
let
me
now
just
refresh
the
page
to
see
whether
the
changes
of
my
branch
are
being
reflected
and,
as
you
can
see,
the
color
has
changed
to
green,
which
is
the
feature
that
we
are
probably
demoing.
So,
let's
go
and
see
the
code
of
the
same
branch
that
is
there
and
the
color
is
green.
So
now,
at
this
point
of
time
we
have
created
ephemeral
infrastructure,
separate
from
the
all
the
developers
working
and
we
can
access
it
by
giving
just
a
header.
B
B
So
in
this
case
the
code
in
user
streaming,
the
code
app
api
is
being
synced
into
slash,
app,
slash,
app
and
also
the
service
php
file,
which
I
just
changed
is
being
shown
as
sync.
So
we
can
quickly
validate
that
by
refreshing,
the
page
and
seeing
the
seeing
whether
the
color
has
got
changed,
and
here
you
go.
The
color
has
now
changed
to
blue.
So,
let's
for
some
reason,
I
don't
like
this
color.
B
So
dev
space
in
the
back
end
would
have
synced
the
file
automatically
and
the
change
is
reflected
in
my
browser.
So
this
is
how
simple
the
workflow
will
be
with
the
ad
option
to
the
dev
stack,
and
this
particular
example
is
for
an
interpreted
language
like
php
and
we'll
just
jump
into
another
example
where
we
see
how
a
compiled
language
like
go
works
now
for
the
demo
purposes.
B
We
have
already
created
another
service
called
capital
cards
which
deals
with
the
cards
of
razer
pay
and
we'll
just
walk
through
the
helm
templates
before
going
into
the
demo.
So
there
are
multiple
deployments
that
are
present,
which
are
your
kubernetes
resources.
These
are
the
hooks
that
we
mentioned
about,
which
will
be
run
as
either
post
install
or
pre-installed.
So
this
is
an
user
configurator
which
will
update
the
ingress
route.
There
is
a
preview
url
hook
as
well.
B
There
is
a
secret
updater
hook
that
runs
which
updates
the
secret
and
there
is
also
a
sqs
configurator.
So
this
particular
application
is
an
asynchronous
one
with
the
web
worker
and
sqs
queues,
and
we
are
configuring,
sqs
queues,
dynamically
and
also
having
a
service
which
is
a
humanities
resource.
So
now
that
we
have
already
created
the
deployments,
let's
just
validate
that
so
on
doing
the
cube,
ctl
get
parts
there
are
the
web
and
the
worker
parts
that
are
created.
B
We
can
also
go
ahead
and
see
the
sqs
resources
that
are
created,
and
this
is
the
local
stack
ui.
So
this
is
the
particular
sqsq
that
is
created
for
capital
cards,
and
this
is
the
url
in
which
it
can
be
accessed.
So
likewise,
we
have
created
the
kubernetes
resources
and
also
the
ephemeral
infrastructure
resources.
B
So
these
are
the
files
that
are
probably
not
required
for
the
for
the
binary
to
work
and
also
it
has
locks
part
where
we
can
see
the
locks
of
the
container
that
we
have
synced
with.
Let's
see
the
status
of
the
dev
space
command
and
yes,
it
has
started
to
run
the
command.
It
has
started
to
sync,
which
is
basically
at
this
point
where,
where
it's
synced
the
code
and
then
it
is
running
the
build
command,
which
is
nothing
but
the
go,
build
hyphen
o
and
then
it
has
run.
B
B
B
Add
the
loggers
in
order
to
print
the
request
login.
So
there
are
two
loggers
that
I
have
commented
out:
I'll,
just
uncomment
it
and
save
the
file
so
right
now
the
workflow
is
that
dev
space
watching
these
file
changes
has
synced
the
file
of
helpers,
which
is
there
here
and
then
the
same
process
of
building
and
running
the
new
binaries
has
happened,
and,
let's
just
validate
that
by
getting
a
request-
and
here
you
go,
the
changes
that
have
done
are
being
reflected
dynamically
in
an
interpreted
language
as
well.
B
The
next
part
of
it
is
the
ephemeral
databases.
We
currently
support
three
different
ways
of
configuring:
the
database,
which
are
one
a
local
database
where
the
developers
run
their
db
instance
locally,
and
we
reverse
proxy
the
request
into
the
local
system.
The
second
one
is
the
ephemeral
database,
where
developers
can
span
ephemeral
databases
based
on
the
preceded
data
and
also
make
sure
that
the
schema
is
synced
with
the
pre-broad
environment
and
also
have
the
dev
migrations
run.
B
The
last
one
is
the
persistent
database
where
the
developers
can
opt
to
use
a
pre-existing
database
of
stage
or
beta
or
a
pre-plot,
which
is
regularly
controlled
by
the
data
ops
flow.
So
the
workflow
of
an
ephemeral
databases
is
that
we
copy
the
stable
stage
environment
into
s3,
which
is
acting
as
a
base,
and
then
the
data
container
based
on
this
will
be
generated
and
migrations
run
on
top
of
it,
which
gives
an
ephemeral
database
which
is
isolated
from
others.
So
are
we
really
looking
into
the
cost?
B
The
primary
goal
of
us
to
is
to
have
a
cost
optimal
solution
right,
so
cube.
Generator
is
the
tool
that
takes
care
of
all
the
cleanups.
So
our
resources,
all
of
the
fml
resources,
have
been
tagged
with
a
ttl
of
six
hours
by
default
and
the
developers
can
override
it
based
on
their
requirements,
so
generator
will
make
sure
that
it
cleans
up
the
resources
when
the
ttl
is
expired,
and
also
in
order
to
solve
the
requirements
of
ups
and
downs,
the
upscaling
and
downscaling
we
use
cluster
autoscaler
and
spot
nodes.
B
So
this
is
how
the
solution
looks
likes
in
from
the
angle
of
tools
where
we
have
different
components
of
dev
orchestrator
cluster
manager
in
from
monitoring
and
routing,
and
this
particular
diagram
specifically
explains
us
how
the
pre
and
post
time
stack
will
be
where
a
sequential
development
workflow
is
replaced
by
multiple
parallel
ecosystem
of
things
working
with
local
sync.
What
is
the
impact
on
their
productivity?
We
have
seen
a
10
to
15
x,
reduction
in
time
to
take
feature
live
because
per
feature.