►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Kubernetes to serve a fraud detection model - Gabriela Souza de Melo, Legiti
In this talk, Gabriela will present how Legiti has used a Kubernetes cluster for serving a real-time fraud detection machine learning application. This application is the most crucial API in Legiti, being the main point of contact with their customers, and being reached in a critical moment by their customers: in the middle of the transaction processing flow, and while the end users are waiting to know if their transaction has been accepted and their order placed. By leveraging Kubernetes' scaling capabilities, Legiti has ensured a reliable server, capable of handling their customers' transaction peaks without degradation of model performance, response quality or latency. Kubernetes also makes it easy to have auto deployments that have no impact on production reliability, with easy ways to set up automated rollbacks based on CI/CD end to end tests results. Using Kubernetes' Python client and pod management and configurations available by Kubernetes, Legiti was able to incorporate business logic that requires for the applications to get restarted with simple Python scripts.
Pearson Henri, Co-founder & CTO of Legiti, will join for the live Q+A session following this talk!
A
A
First,
I'd
like
to
give
a
bit
of
a
background
about
how
credit
card
fraud
works
in
brazil,
where
legit
is
based
credit
card
fraud
is
considered
in
some
circles
to
be
the
national
pastime.
Brazil
alone
accounts
for
45
of
global
credit
card
fraud
cases.
So,
let's
just
say,
there's
plenty
of
demand
for
a
good
and
I
fraud
solution
here.
A
Any
company
that
sells
any
sort
of
high
value
item
online
is
a
prime
target
for
fraudsters,
half-priced,
plane,
tickets,
discounts
and
pet
products.
Sure
why
not
when
frauds
fraudster
has
a
stolen
credit
card
in
their
hands,
anything
that
they
buy
is
essentially
free,
they're,
not
paying
the
bill
for
the
credit
card,
so
that
means
they
can
resell
whatever
they've
purchased
at
a
discount
price,
because
any
money
they
receive
receive
is
pure
profit.
A
A
A
A
So
the
moment
that
a
credit
card
company
refunds,
those
charges
to
your
card,
they
just
turn
right
around
to
the
original
merchant
who
authorize
that
purchase
and
they
want
the
money,
the
money
back
from
them.
So
all
that
money
that
the
credit
card
company
refunded
for
you
that's
actually
coming
straight
from
the
merchant
who
let
that
purchase,
go
through.
A
So
this
process,
in
which
the
credit
card
company
essentially
reverses
a
previously
approved
transaction
and
forces
the
original
merchants
to
reimburse
them,
is
called
a
chargeback.
They
want.
They
happen
all
the
time
and
can
be
disastrous
for
merchants
if
you're
a
platform
that
takes
only
a
small
person,
percentage
of
the
value
of
goods
sold
on
your
platform,
so
say
10
and
you
originally
took
in
10
out
of
a
100
transaction
that
later
comes
back,
comes
back
as
a
chargeback
you're,
now
going
to
need
to
pay
100
for
that
charge
back.
A
So,
if
you
made
10
on
that
sale,
you're
gonna
now
need
to
make
that
same
sale,
10
times
to
get
the
value
back
for
that
100
chargeback
that
you're
having
to
pay.
So
you
need
a
lot
of
sales
to
make
up
for
the
losses
from
one
single
fraudulent
transaction,
so
this
can
be
devastating
for
a
business
and
controlling
losses
from
chargebacks
is
obviously
a
very
major
concern.
A
But
one
thing
that's
important
to
notice
and
that
we're
trying
to
convey
with
this
image
here
is
that
stopping
losses
from
chargebacks
is
only
one
part
of
the
equation.
The
other
side
of
this
coin
is
conversion
so
lgd.
We
always
say
that
stopping
fraud
is
easy.
If
you
just
reject
all
of
your
customers,
purchases
you'll
never
have
fraud.
That's
your
solution
right
there!
A
Well,
that's,
obviously
not
applicable,
so
the
hard
part
is
balancing
your
fraud
prevention
with
conversion
or
more
concretely,
rejecting
as
few
valid
customer
purchases
as
possible,
while
blocking
as
many
fraudulent
purchases
as
possible.
So
you
want
to
reduce
your
your
error
on
both
sides
of
this,
not
just
by
not
letting
fraudulent
transactions
go
through,
but
also
by
reducing
the
amount
of
legit
transactions
that
you
block.
A
So
that's
where
legit
comes
in.
What
we
do
is
we
train
machine
learning
models
on
vast
core
pieces
of
credit
card
transaction
data
to
deliver
customer,
specific
anti-fraud
models
to
our
e-commerce
clients
and
we're
pretty
good
at
it.
So
we've
managed
to
get
credit
card
fraud
in
half
within
six
months
of
entering
at
one
of
our
customers
and
increase
conversion
of
valid
purchases
by
nearly
four
percent.
A
But
it
takes
a
lot
of
work
to
make
it
happen.
So
that's
the
part
I'm
going
to
dig
into
with
all
of
you
and
not
terribly
concerned
that
anybody
here
is
going
to
be
copying
our
work
because,
as
I'm
sure
many
people
here
know,
it's
pretty
hard
to
set
up
a
massively
scalable
data
collection
and
ml
model
deployment
system
from
scratch.
A
A
So,
let's
dive
in,
as
I
mentioned,
our
system
is
fundamentally
based
on
machine
learning.
So
every
time
a
user
is
trying
to
make
a
purchase
with
any
of
our
customers,
which
are
the
merges
that
integrate
their
platform
to
our
system,
our
customers
reach
our
api
and
we
consult
machine
learning
models
to
give
them
a
decision
on
whether
to
reject
or
approve
that
transaction.
A
The
data
gets
into
our
database,
so
this
is
one
example
here
of
a
pipeline
that
we
have
so
as
our
customers
reach
us
through
our
api
gateway,
and
we
start
passing
that
data
through
a
few
different
lambda
functions.
We
use
kinesis
sqs
to
get
the
data
in,
and
this
is
all
happening
within
seconds
and
then
the
date
is
available
for
us
in
our
database,
which
we're
using
amazon
rds
here.
A
However,
collecting
the
data
is
only
the
first
step
towards
being
able
to
feed
a
machine
learning
model.
After
that,
we
still
need
to
transform
the
raw
data
into
something
that
is
valuable
to
our
models.
That's
what
is
called
machine
learning
features
so
for
that
we
use
both
a
feature
store
and
a
feature
generation
python
package
to
generate
those
machine
learning
features.
A
A
These
jobs
fetch
some
data
from
our
database
process.
This
data
get
features
out
of
them
and
stores
it
in
two
different
locations,
so
we
have
an
online
store
and
an
offline
store.
The
online
store
is
a
very
low
latency
storage
solution,
so
we're
using
a
redis
cluster
running
on
aws's
elastic
cache
solution,
and
we
also
have
the
offline
store,
which
is
a
storage
solution
that
has
less
rigid,
latency
constraints,
but
is
able
to
to
store
larger
amounts
of
data,
and
that's
one
part
of
the
features
that
we
calculate.
A
So
the
other
parts
are
some
features
that
cannot
get
calculated
beforehand.
So
we
calculate
those
at
real
time
and
for
that
we're
sharing
code
between
model
training
and
inference
by
making
usage
of
an
internal
python
package
and
by
sharing
that
code,
we're
also
actually
reducing
the
risk
of
getting
training
serving
skew
in
the
future
values
that
we
use.
A
A
So,
in
that
case,
all
of
this
is
happening
while
the
customer
is
waiting
to
receive
feedback
on
their
purchase
attempt.
Any
delays
on
this
flow
cause
a
direct
impact
on
their
customer
experience.
This
means
that
this
api
is
under
critical,
latency
constraints
and
we
need
a
system
that
can
scale
without
causing
degradation
in
the
api's
latency.
A
So
we
have
a
kubernetes
cluster
deployed
on
eks
and
that
takes
away
some
of
the
work
that
would
be
involved
in
getting
the
cluster
up
and
maintaining
it
abstracting
away
the
lower
level
cluster
management
tasks
like
the
management
of
kubernetes
control,
plane
and
the
cluster
nodes
and
we're
using
ec2s
for
our
nodes
and
that's
very
easy
to
define
the
size
of
the
machines
we
want
to
use
for
our
nodes,
make
any
modifications
to
it.
Should
we
want
to
change
it.
A
A
A
A
So
we
parse
and
validate
the
content
of
the
request
we
receive
from
our
customers.
We
extract
some
of
the
features
from
the
feature
store,
which
pre-calculates
features
some
other
features
get
calculated
in
real
time
by
consulting
a
db
and
computing
features.
On
top
of
the
query
data,
we
consult
our
machine
learning
models.
We
might
even
consult
a
few
rules.
Our
customers
can
decide
themselves
in
the
form
of
allow
and
block
lists,
and
we
log
all
of
our
decisions
with
information
about
which
decision
was
used
and
that's
what
allows
us
to
monitor
our
metrics.
A
So
all
of
this
means
that
we
have
heavy
objects,
loading
to
memory
which
which
are
the
models
and
there's
some
intensive
cpu
usage
to
calculate
features
on
the
fly,
as
I
mentioned
before.
All
of
this
is
happening
under
very
strict
latency
requirements.
So
kubernetes
has
been
serving
us
very
well
here
by
providing
a
robust
way
to
serve
this
api.
A
Now,
we've
got
our
server
up
and
running.
Requests
are
coming
in
getting
features,
evaluated
and
models
consulted
for
decisions
we're
properly
sending
back
responses.
Everything
looks
great,
so
our
job
is
done
right.
Well,
what?
If
out
of
the
blue,
we
start
receiving
an
abnormal
amount
of
requests,
and
then
we
learn
that
one
of
our
customers
is
having
a
promotion
which
is
what's
accounting
for
the
peak
and
requests.
A
We
don't
have
any
time
to
make
any
changes
to
our
setup,
but
we
can't
always
foresee
this
type
of
increased
activity
beforehand
and
we
cannot
degrade
our
performance
due
to
that.
So
that's
where
kubernetes,
auto
scaling
functionality
comes
in
really
handy
for
us
we're
using
kubernetes,
horizontal
pod,
auto
scaler,
which
allows
for
an
extremely
easy,
auto
scaling
setup.
A
We've
configured
it
to
get
trigger
on
cpu
usage,
which
is
the
main
indicator
of
degrading
performance
that
we
have
as
an
overloaded
cpu
would
result
in
slower
feature
computation
and
hence
in
an
increasing
latency,
with
that
we
can
now
seamlessly
and
automatically
scale
without
degrading
performance,
and
it
really
just
takes
these
20
lines
of
codes
to
achieve
that.
So
it's
a
very
easy
solution
that
gets
a
lot
of
results.
A
However,
our
job
was
not
done
once
we
had
the
system
first
deployed,
either
we're
constantly
making
code
code
changes
for
bringing
in
improvements,
adding
new
features
or
making
adaptations
to
new
customers.
We're
also
constantly
changing
our
models,
because
we're
constantly
retraining
them
be
due
to
new
data
or
to
the
modification
or
addition
of
new
features.
A
So,
if
deployments
are
going
to
happen
frequently,
we
need
it
automated
to
allow
that
to
happen,
we're
using
ci
cd
so
that
changes
are
automatically
deployed
every
time.
There
is
a
merge
to
our
main
main
branch,
as
I
mentioned
before,
changes
trigger
new
containers
to
get
pushed
to
ecr,
and
then
we
just
use
helm
for
upgrading
our
application
release,
which
is
what's
happening
in
this
picture
here.
So
this
is
an
actual
part
of
our
ci
pipeline.
This
is
the
helm
deployment
step.
A
This
is
what
it
looks
like,
while
it's
updating
it
so
helm
allows
us
to
redeploy
our
services
live.
We
don't
get
any
downtime
when
we
do
that
and
we
make
usage
of
the
redness
probe
con
configuration
to
inform
the
deployment
of
when
and
whether
new
pods
are
actually
ready
to
start
receiving.
Your
requests,
if
that
doesn't
get
reached
in
a
given
timeout,
the
deployment,
is
aborted
without
any
effect
to
the
currently
running
application.
A
A
So
the
bad
thing
about
having
it
automated
is
that
it's
easier
for
errors
to
slip
in
as
no
one's
manually
triggering
and
watching
the
deployments
right.
Well,
not
in
our
case,
so
we've
built
automated
test
suites
that
run
not
only
at
the
unit
and
integration
level,
but
also
at
end
to
end
level.
That
means
that
each
new
deployment
starts
getting
tested
as
soon
as
it's
put
out
in
our
actual
production
environment
and
should
anything
go
wrong.
A
Rollbacks
get
automatically
triggered
to
restore
the
application
to
its
previous
version,
so
here
we're
also
using
helm
for
for
this,
we're
using
helm's
rollback
functionality
paired
with
s3
as
a
registry
where
we
stored
the
id
of
the
latest
successful
deployment.
So
if
our
entrance
has
failed,
which
is
the
case
in
this
picture
that
I
got
here,
helm
just
automatically
triggers
a
rollback
by
using
the
timestamp
or
the
id
for
the
last
successful
version,
and
if
end-to-end
tests
are
successful,
then
that
less
successful
release
identification
gets
updated.
A
So
that
is
the
one
that's
just
been
deployed
so
that
that
was
it
for
today.
We
hope
that,
with
this
talk,
we
could
show
you
how
we
use
kubernetes
to
build
a
real-time
fraud
detection
system
that
is
bringing
great
value
to
our
customers
and
that
even
a
simple,
low-maintenance
kubernetes
setup
can
get
you
a
long
way.
So,
thank
you
all
for
the
attention.