►
From YouTube: Building a Cloud Native Feature Store with Feast on Kubeflow - Willem Pienaar & Oleksii Moskalenko
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Building a Cloud Native Feature Store with Feast on Kubeflow - Willem Pienaar & Oleksii Moskalenko, Gojek
Features are at the heart of what makes machine learning systems effective. However, many challenges still exist in the feature engineering life-cycle. Developing features from big data is often an engineering heavy task, with challenges in both the scaling of data processes and the serving of features in production systems. Feast is an open-source feature store co-developed by Gojek and Google Cloud, which allows for the storage, management, access, validation, and reuse of ML features throughout an organization. Feast solves the key operational challenges with the productionization of features for both small teams and large organizations. The project has been built from the ground up to be cloud-native and has recently been included as a component within Kubeflow. In this talk we will explore the key challenges that ML teams face and how technologies like Feast help to solve them.
https://sched.co/ekCE
A
Hi,
my
name
is
velen
and
I'm
an
engineering
lead
on
the
data
science
platform
team
at
gojek,
I'm
joined
today
by
alexi
muskalenko,
who
is
my
colleague
a
senior
engineer
on
the
team.
Today
we're
going
to
be
talking
a
bit
about
building
a
cloud
native
feature
store
with
feast
today
on
agenda,
we're
going
to
talk
a
bit
about
the
data
challenges
that
teams
face
when
operationalizing
machine
learning
we're
going
to
talk
a
bit
about
how
feast
can
help
you
with
those
challenges.
A
A
So
how
does
data
science
work
at
most
teams
today?
These
projects
are
typically
started
because
you
want
to
target
some
kind
of
business
outcome.
Some
kind
of
metric
that
you
want
to
push
up
and
a
data
scientist
is
often
tasked
with
building
the
first
model
or
the
first
proof
of
concept
for
this.
So
they're,
given
data
and
they're,
asked
to
build
this
model
and
see
if
it's
actually
viable
and
so
often
the
basis
of
the
project.
This
you
know
the
inception
is
the
notebook.
A
This
is
a
linear,
end-to-end
flow
that
a
data
scientist
comes
up
with
and
you
evolve
that
notebook
into
an
end-to-end
machine
learning
system,
and
so
you
have
something
that
resembles
what
we
have
on
the
screen.
Now
on
the
left,
you
have
your
data
sources
and
then
you
have
transformations,
let's
say
batch
transformations,
you
have
model
training,
you
have
model
deployment,
you
have
model
serving
one
end-to-end
flow
and
this
often
represents
and
resembles
the
notebook
that
it's
being
evolved
from
and
the
final
product.
A
There
is
a
model,
that's
integrated
with
the
production
system
and
this
works,
and
most
teams
can
get
to
this
point
by
just
hacking
together,
open
source
or
proprietary
solutions
products.
At
some
point
somebody
will
say
we
need
real-time
features
and
real-time
data
and
hook
up
a
stream.
At
some
point
somebody
will
say
we
need
more
distributed
compute,
so
they
hook
up
spark,
but
ultimately,
you're
left
with
an
end-to-end
system,
one
monolith,
but
these
models
come
with
some
costs,
sometimes
very
big
costs.
One
of
the
first
problems
is
that
they're
very
slow
to
iterate
on.
A
They
all
want
to
iterate
at
different
frequencies
and
paces,
and
you
know
they're
different
teams
sitting
in
different
places,
but
they
have
to
work
in
unison,
because
it's
one
monolithic
system
and
this
slows
them
down.
The
second
problem
is
that
features
often
need
to
be
redeveloped
when
going
from
training
to
serving.
A
This
is
often
because
features
are
written
in
python,
and
you
know
they're
not
performant
enough,
and
they
need
to
be
written
in
java
or
go
and
there's
all
these
reasons,
but
inconsistencies
arrive
when
engineers
and
data
scientists
need
to
work
together
to
produce
one
final
output,
one
product.
The
third
problem
is
that,
because
of
these
rewrites
and
changes
from
one
environment
to
the
next
often
you
have
training
and
serving
inconsistencies
in
the
data,
and
this
can
lead
to
a
performance
drop
in
your
models.
The
fourth
problem
is
that
data
quality
problems
arise.
A
When
you
don't
know
that
you
don't
have
the
proper
monitoring
in
place,
you
don't
have
the
proper
metrics
in
place.
You
don't
have
validation
of
your
data.
Basically,
the
tools
that
are
available
today
are
not
meant
to
measure
data,
but
at
the
same
time,
this
data
is
affecting
the
business
outcomes.
A
So
we
believe
that
a
need
exists
for
a
battle
tested,
open
source
feature
store
to
address
these
problems
that
we've
highlighted,
and
we
think
feast
is
that
feature
storm
sofia
is
a
system
that
attempts
to
solve
the
key
data
challenges
to
production
machine
learning.
So
if
we
go
back
to
that
diagram,
I
showed
you
earlier.
Typically
what
you'd
have
on
the
left
is,
you
know,
you'd,
have
your
data
you'd?
Have
your
transformations
and
then
you'd
have
your
model.
A
A
Feast
decouples
this
completely?
So
if
we
talk
about
the
problems
that
we
were
highlighting
earlier,
the
first
one
we
were
highlighting
was
the
lack
of
iteration
so
now
you've,
decoupled
engineering
of
features
and
creation
of
features,
because
that
process
ends
with
feast.
That's
one
team
that
can
be
the
engineers
data
scientists.
Then
you
have
another
team,
that's
which
is
purely
iterating
on
model
training,
they're,
selecting
features
from
feast,
training,
a
model
and
shipping
it
to
a
registry.
A
Then
you
have
a
third
team.
This
can
be
a
production
team
engineers,
perhaps
they're
shipping
models
into
production,
they're
hooking,
those
up
into
you,
know
connecting
them
to
real-time
production
systems,
and
you
know:
they're
looking
up
online
features
at
low
latency,
and
they
can
do
this
confidently
at
scale
and
with
monitoring
and
instrumentation
that
it
requires
and
all
three
teams
can
iterate
independently,
and
so
this
both
solves
iteration
speed
and
also
solves
reuse,
because
teams
can
now
independently
use
each
other's
artifacts
with
their
models
or
features
in
this
architecture.
A
A
This
also
does
ingestion,
so
it
has
jobs
that
it
provisions
that
loads
data
in
from
upstream
sources,
whether
streaming
or
batch
into
the
stores
in
order
for
the
third
point,
which
is
serving
fees,
provides
a
point
in
time
correct,
a
temporarily
correct,
serving
layer
that
allows
you
to
look
up
data
at
scale
for
training
a
model
and
it
will
handle
the
joints
and
everything
you
need,
and
it
also
allows
you
to
do
online.
Look
up
so
low
latency
lookups
for
for
predictions
in
the
online
case.
A
A
You
push
your
data
into
feast,
and
this
is
done
through
the
triggering
of
ingestion
job
and
that
job
is
a
spark
job
and
this
job
will
write
your
data
into
the
stores,
so
in
this
case
consistent
copy
of
the
data
that
you're
ingesting
in
online
and
offline
stores
and
the
definitions
of
the
features
and
the
data
that
you're
ingesting
are
handled
by
core.
So
fist
core
is
that
central
registry
and
then
once
you've
ingested
your
data
into
the
storage
layer,
it's
then
available
for
all
teams
to
use.
A
So
what
is
feast?
Not
feast
is
not
a
workflow
scheduler,
it's
not
like
luigi
or
airflow.
It
doesn't
do
scheduling
it's
not
just
a
data
lake
or
a
data
warehouse
like
bigquery,
because
it's
got
the
online
functionality
as
well,
although
it
uses
bigquery
and
some
of
these
tools,
underneath
feast
doesn't
do
transformation.
So
it's
unlike
spark
and
pandas,
although
it
will
utilize
those
tools,
but
it's
made.
Those
are
upstream
tasks.
A
B
B
B
B
A
B
B
It
pulls
data
from
data
warehouse
or
data
lake
to
prepare
a
data
set
for
training,
both
those
jobs
aimed
to
provide
consistency
between
data
for
training
and
data
for
serving
these
sdk
kind
of
glues.
All
these
together,
it
used
for
creating
features
and
feature
tables
for
requesting
features
from
feed
serving
to
retrieve
online
features.
B
B
B
B
B
The
interesting
thing
about
those
features
that
some
of
them
are
updated
daily,
like
average
daily
trips
or
conversion
rate,
and
some
of
them
like
trips
today,
are
dated
in
real
time
and
despite
the
fact
that
the
all
of
the
features
are
connected
to
the
same
key
or
entity
driver
id.
I
created
here
two
feature
tables.
B
So
ingestion
job
here
pulls
data
from
the
watch
source
and
stores
it
into
the
feature.
Storage,
another
ingestion
job
consume
the
data
from
the
stream
source
and
also
stores
it
to
the
feature,
storage
and
serving
basically
pull
all
the
features
related
to
one
entity,
and
it
does
it
with
low
latency,
because
we
store
all
features
related
to
the
same
entity
together
in
redis.
B
B
B
B
B
B
B
Actually
your
machine
learning
project,
the
first
step
in
preparing
a
model
is
training
and
for
that
we
need
to
generate
a
training.
Data
set
important
part
about
generating
data
set
for
training
is
point
in
time
correctness,
as
you
can
see
on
this
graph,
when
we
are
making
a
prediction,
we
use
feature
values
with
various
timestamps,
so,
as
you
can.
B
B
We
are
making
this
point
in
time:
correctness:
correction
in
the
for
the
training
data
set
as
well,
so
data
scientists
must
specify
this
line
here.
Basically,
this
time
point
and
we
will
we'll
be
doing
backward
search
and
finding
the
most
recent
feature
value
in
relation
to
this
time
point,
let
me
show
an
example.
B
So
here
from
the
several
data
frames
that
I
generated,
we
are
using
entities
data
frame
to
create
a
request
for
training
dataset,
so
we
taking
a
sample
of
those
entities
and
adding
to
them
a
random
event
timestamp
right.
So
this
is
a
data
frame
which
we
will
provide
as
entity
source
for
the
our
get
historical
features
request.
B
B
B
In
order
to
do
that,
you
need
to
have
those
features
in
online
storage.
The
simplest
way
to
populate
the
storage
is
to
take
everything
you
have
in
your
box
storage
and
put
the
latest
values
of
each
each
feature
for
each
entity
into
online
storage.
So
now
we
will
do
exactly
this.
We
will
run
the
offline
to
online
ingestion,
which
also
starts
a
spark
job
which
reads
from
the
google
storage
and
store
to
raise.
B
B
As
the
next
step,
we
can
also
add
another
features
to
our
production
prediction,
namely
the
real-time
features.
So,
in
order
to
do
that,
we
need
to
add
a
streaming
source
as
one
of
the
sources
for
our
to
our
to
our
feature
table
with
this
streaming
source.
We
need
to
specify
also
bootstrap
server
and
topic
since
we're
using
kafka
source
and
also,
we
need
to
specify
a
message
format
which,
in
this
example,
is
avro
format
and,
of
course,
we
need
our
schema.
So
the
ingestion
job
would
know
how
to
decode
records
coming
from
kafka
topic.
B
B
A
A
Where
is
feast
now?
Well,
we
released
0.1
late
2018,
we
rolled
it
out
in
gojic
and
we've
gone
from
strength
to
strength.
From
0.2
onwards,
we've
been
working
with
our
community.
We
developed
decentralized
serving
we
developed
point
in
time
correctness.
We
added
project,
isolation
and
isolation
and
name
spacing.
We
simplified
concept
and
added
integration
points
we
integrated
with
t
of
x
and
t
of
db.
A
Well,
we've
been
working
closely
with
a
community
of
large
technology
companies
like
agoda,
that's
delivered
cassandra
support,
postmates
has
delivered
bigtable
support,
we've
added,
some
prison
has
added
oauth
and
we've
been
working
with
microsoft
and
farfetch,
and
they've
delivered,
delivered
data,
bricks
and
delta
support,
and
recently
we've
been
working
with
techton
who've,
contributed
spark
and
amazon
support,
so
there
are
over
43
contributors
and
six
plus
enterprise
customers
running
feast
in
production
today
at
scale.
Let's
talk
about
the
feast
vision
for
a
second,
so
there
are
three
aspects
that
we
want
to
look
at.
A
The
first
is
open
governance
and
standards,
so
there
are
teams,
large
and
small
that
currently
rely
on
feast
they're,
not
just
running
it
in
production,
but
they're
also
contributing
code
back
to
the
project.
They're
collaborating
with
us.
These
teams
need
to
know
that
they
can
rely
on
the
project
at
scale,
but
that
they
can
also
shape
the
direction
of
the
project.
So
we
believe
open
governance
and
the
transparent
process
is
important
in
driving
our
project
and
encouraging
further
adoption.
A
Secondly,
we
want
to
both
feast
into
a
multi-cloud
or
cloud
agnostic,
modular
and
production
grade
feature
store,
meaning
if
you're
a
team
using
fees.
You
should
be
able
to
cherry
pick
the
components
you
need
deployed
wherever
you
are
running,
your
existing
machine
learning
stack
and
then
run
that
at
scale
reliably.
A
Lastly,
we
want
to
make
feast
easy
to
integrate
into
existing
machining
tools
and
the
mml
tools
of
the
future,
meaning
if
these
are
machine,
learning
platforms
upstream
data
transformation
systems
downstream
model
serving
we
want
to
have
clear
api
contracts
and
clear
integration
points.
So
these
are
all
key
parts
of
our
vision
and
we
are
already
starting
to
work
towards
addressing
them
so
for
open
governance
and
standards.
A
We
are
super
excited
to
acknowledge
that
feast
is
now
officially
a
part
of
the
linux
foundation,
meaning
that
not
a
single
company
has
any
kind
of
special
privilege
on
the
project.
The
trademark
is
now
part
of
the
linux
foundation,
and
we
will
be.
We
are
already
operating
on
a
governance
structure
defined
by
the
linux
foundation,
which
means
it's
completely
neutrally
governed
and
the
process
and
structure
is
completely
public
and
transparent.
A
We
believe
that
by
joining
forces
with
techton
we'll
be
able
to
build
a
world-class
feature
store
that
teams,
large
and
small,
can
rely
on
and
together,
we
hope
to
set
the
standard
for
what
a
feature
store
is,
and
so
we're
super
excited
for
this
collaboration.
Opportunity,
and
all
of
this
is
happening
underneath
the
linux
foundation
umbrella.
A
Then,
finally,
we'd
like
to
announce
that
we
are
now
a
top
level
component
of
keyflow,
and
this
is
the
start
of
our
integration
journey.
Keyflow
is
one
of
the
best
machine
learning
platforms
out
there
completely
open
source,
so
it's
both
best
of
breed
and
an
end-to-end
system,
and
we
believe
that
by
integrating
into
key
flow
and
improving
that
integration
will
not
just
improve
the
integrations
of
feast
with
into
ml
platforms,
but
also
to
sister
projects
like
model
serving
or
pipelining
or
other
managed
services.
A
So
our
roadmap
going
forward
for
fees
0.9,
we
believe
which
is
currently
under
development.
We
believe
that
we'll
be
able
to
achieve
cloud
agnostic
deployments,
so
we've
already
got
as
we're.
Sorry,
we've
already
got
amazon,
google
cloud
support
and
we'll
be
launching
azure
support,
as
well
as
on-prem
deployment,
support,
we'll
be
adding
offline
storage,
support
for
delta
and
we'll
be
adding
feature
engineering
support
which
will
likely
be
through
spark
sql.
A
We
will
also
be
sending
out
requests
for
comments
on
on-demand
feature,
transformations
and
a
feature.
Discovery,
user
interface,
then
finally,
come
and
say
hello.
Our
home
page
is
over
at
fistdev.
Our
source
code
is
all
online.
It's
completely
open
and
apache
too
licensed.
We
have
a
feast
channel
on
the
keeper
slack
and
you
can
find
a
link
to
this
deck
on
tinyurl
on
the
screen
right
now.