►
From YouTube: How Event Driven Autoscaling in Kubernetes Can Combat Climate Change - Annie Talvasto & Adi Polak
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Event Driven Autoscaling in Kubernetes Can Combat Climate Change - Annie Talvasto, CAST AI & Adi Polak, Microsoft
Climate change is one of the biggest challenges of this century, and at the same time adoption of cloud services increases the world’s CO2 production. We will show a demo showcasing how we can combat climate change with Kubernetes, e.g. by using event driven autoscaling (KEDA), spot instances, aggregated data from multiple sources as well as CNCF projects. We will take you through the creation & ideation of the open source project and prove that by optimising your cloud & Kubernetes, you can also decrease the environmental impact of your compute.
A
B
Very
much
welcome
on
our
part
to
our
session
how
evan
drew
and
auto
scaling
and
kubernetes
can
combat
climate
change.
We
are
super
excited
to
be
here.
The
topic
is
very
close
to
our
hearts
and
we
hope
that
it's
close
to
your
hearts
as
well,
and
one
of
the
reasons
why
we
are
super
excited
to
be
here
as
well,
is
because
the
topic
is
very
closely
connected
to
the
this
year's
kubecon
themes
are
resilience
realized
and
I
think
it's
well
climate
change.
B
What
could
be
more
of
a
topic
where
resist
resilience
is
actually
needed
and
where
it's
a
very
important
topic,
so
today
we
will
take
you
on
a
journey
through
how
evident
even
auto
scaling
can
actually
make
a
difference
in
climate
change,
and
this
really
connects
to
the
fact
that
if
we
use
our
compute,
if
we
use,
if
we
build
our
software
in
a
way,
that's
not
wasteful
of
natural
resources,
we
can
actually
already
buy
that
be
in
a
quite
good
space
for
that.
But
I'm
super
excited
for
today's
topic.
A
B
A
Am
I
hi,
I'm
andy
folak
I
work
for
microsoft,
I'm
a
senior
manager
at
microsoft.
I
have
a
master
degree
in
computer
science
with
a
thesis
in
anomaly
detection
in
graphs
with
complex
node,
I'm
a
writer
of
upcoming
o'reilly
book
about
machine
learning
pipeline
with
apache
spark
ecosystem.
I'm
a
databricks
weekend.
I
worked
a
lot
with
a
databricks
and
overall
apache
spark
open
source
community
and
I
have
about
a
decade
of
experience,
hands-on
experience,
building
machine
learning
modules
and
also
scalable
infra
data
infrastructure,
annie.
B
B
What's
the
situation
at
the
moment-
and
I
think
particularly
in
this
sense,
when
you
think
about
industries
when
you
think
about
things
that
are
causing
a
lot
of
emissions
that
are
causing
a
lot
of
pollution
in
the
world,
we
don't
typically
think
about
the
technology
industry.
We
think
about,
for
example,
the
airline
industry,
and
I
think
it's
a
really
interesting
fact
that
actually,
the
tech
industry
is
either
causing
the
same
amount
of
emissions
every
year
or
even
more
than
the
airline
industry,
so
tech
or
airline
industry.
B
A
Yeah
tapping
into
it
the
rapid,
modern
technology
that
we're
seeing
together
with
the
digitalization
process
that
took
place
during
kovit
19,
where
a
lot
of
businesses
went
online,
open
their
either
e-commerce
or
websites,
or
anything
like
that.
We
definitely
see
a
huge
demand
for
compute
power,
which
impact
also
the
carbon
emission
that
those
machines
are
putting
into
into
a
world.
Basically.
B
Yeah
and
truly
because
of
these
two
factors,
essentially,
if
we
don't
act
or
we
don't
change
one
of
these
two
factors-
the
world
is
going
to
experience
an
irreversible
change
in
the
climate
that
we
can
then
no
longer
control.
So
something
has
to
change.
So
one
of
these
two
previous
points
will
have
to
kind
of
bend
and
obviously
we
as
technology
enthusiasts,
don't
want
it
to
bend
from
the
second
point,
probably
but
more
from
the
first
one
yeah
yeah.
B
A
On
top
of
that,
you
get
improved
performance
because,
for
the
same
amount,
the
same
unit
of
computes
that
we're
paying
for
we're,
basically
getting
more
workloads
done,
because
we're
proving
it
to
the
level
of
maximum
maximum
that
we
can
take
out
of
that
specific
compute,
which
means
we
also.
It
also
leads
to
cost
saving
because
hey
we
can
serve
more
customers,
we
can
generate
more
revenue.
We
can
make
more
business
with
the
same
compute
units.
So
once
we
are
improving
the
performance,
we
also
saving
on
costs.
B
B
So
this
will
help
companies
push
towards
that,
which
obviously
usually
leads
to
better
organizational
improvements
and
and
goals
as
well,
because
that's
really
nice
as
well
as
it
helps
us
to
meet
the
global
sustainability
goals
that
have
been
set
to
really
meet
those,
so
that
those
changes
would
not
happen
in
the
world.
Yeah.
A
And
actually,
in
certain
countries,
for
example,
in
europe
and
finland
right
there
are
specific
rules
about
carbon
emission
people
are
really
pushing
the
the
society
and
the
economy
are
really
pushing
towards
carbon
neutrality.
So
this
is
a
really
important
goal
for
some
of
the
countries
and
we
definitely
see
how
it's
going
to
spread
over
the
rest
of
the
world.
B
A
B
B
So
really
there
are
a
few
basic
things,
there's
carbon
and
electricity,
and
this
obviously
connects
to
how
much
carbon
does
your
compute
emit
or
then
how
much
electricity
is
needed
to
run
your
software
and
these
kind
of
things,
but
actually
the
thing
that
that's
not
so
often
may
be
discussed
is
the
fact
that
it's
not
actually
the
net
amount.
That's
produced
that
matters
as
much
rather
than
the.
How
much
value
do
you
get
for
that
that
amount
that
you
put
out
so
then
it
really
connects
to
carbon
incentive.
A
Intensity,
yeah,
carbon
intensity,
it's
basically
a
function
that
takes
into
consideration
your
carbon
units
and
your
electricity
units.
This
is
one
of
the
functions
that
you
might
want
to
optimize
for
so
it
will
give
you
another
set
of
tool,
another
set
of
metrics
in
order
to
understand
where
you're
standing
in
terms
of
care
consuming
up
sorry
producing
carbon
or
emitting
carbon
versus
consuming
the
electricity
for
the
computer
for
your
cluster.
In
another
point,
this
is
a
network
efficiency,
because
we
also
often
when
we
run
compute.
We
have
some
data
that
we're
transferring.
A
We
have
some
messaging
difference.
Transferring
we
want
to
be
aware
of
how
much
data
we're
transferring
where
are
we
transferring
it
to,
because,
let's
also
use
compute
use,
run,
use
a
lot
of
resources
and
that's
actually
going
to
shape
the
demand
that
net
neutrality
is
going
to
shape
the
demand
in
how
we're
using
software
and
how
we're
consuming
software
and
building
those
sustainable
software.
So
all
of
those
points
going
to
help
us
assess
how
are
we
impacting
the
the
world?
A
Once
we
get
eyes
into
what's
happening
in
a
cluster
using
observability,
we
can
start
making
decisions
right.
We
can
start
thinking
how
we
want
to
optimize
measure
the
optimization
impact
understand
if
it
really
had
an
impact
or
we
thought
it
had
an
impact,
so
really
try
it
and
start
to
play
with
that,
and
on
top
of
that
we
can
overall
add
a
layer
of
governance
of
how
much
polluting
our
software
is
if
it
is
even
polluting
or
if
we
are
at
kind
of
a
carbon
neutrality
space.
A
B
And
that
really
helps
with
the
governance
for
sure.
So
then
we're
gonna
move
on
to
giving
some
open
source,
tooling
tips
and
support
that
support
the
calls
as
well.
So
since
cncf
is
close
to
everyone's
heart
here,
cncf
projects
can
really
get.
You
started
on
your
journey
here
as
well.
Those
here
is
a
collection
of
cncf
projects
that
help
you
get
more
observability
or
or
more
optimization
into
your
clusters,
so
you
can
get
started
with
them,
so
you
can
use
cada
for
event
driven
auto
scaling.
B
We
will
do
the
deep
dive
decatur
next,
but
that's
already
a
tip
here
and
then
you
have
also
application
management
with
helm.
You
can
use
prometheus
for
monitoring,
you
can
use
flex
for
container
workflow
and
then
we
always,
of
course,
recommend
using
a
service
mess,
so
there's
linker
d,
kuma
and
whatnot,
and
of
course,
if
you
end
up
in
a
situation
where
you
have
multiple
service
messes
and
you
need
to
manage
all
of
them.
A
Absolutely
so
the
cncf
tools
are
great
in
helping
us
optimizing,
the
cluster
right
optimizing,
our
compute.
However,
we
still
need
to
add
this
layer
of
observability
to
understand
how
the
different
compute
translate
into
carbon
emission
and
encounter
footprint
overall.
So,
for
example,
here
there
is
a
project
named
impact,
it's
not
part
of
the
cncf,
but
it's
a
project
that
helps
us
track
and
kind
of
calculate
how
much
carbon
footprint
our
machine
learning
workloads
are
generating.
A
A
So
this
is
one
of
the
steps
that
we
need
to
take
in
order
to
create
that
observability
layer,
and
there
are
more
tools,
of
course,
part
of
the
opensustain.tech
that
help
us
track,
that
on
different
devices,
if
we're
using
iot
devices
or
if
I'm
using
a
different
kind
of
hardware,
all
of
those
tools
are
going
to
help
me
create
that
observability
layer.
In
order
for
us
to
estimate
how
the
great
cncf
projects
that
we
have
helping
us
optimizing
and
reaching
those
and
reducing
the
carbon
footprint
that
we're
creating.
B
B
So
they
really
aim
to
gather
a
lot
of
the
resources
that
we're
talking
about
here,
a
lot
of
the
people
that
are
excited
about
these
topics
under
one
umbrella,
so
that
we
can
all
get
together
and
work
on
these
issues
and
goals
and
all
of
these
things
together,
because
there's
more
power
in
community
and
teams
working
together,
obviously
as
well.
So
that's
really
wonderful,
absolutely
yeah
and
then
now
jumping
into
the
auto
scaling.
Event-Driven
cater
part
of
our
session
yeah.
B
Yes,
for
sure,
so
really
this
connects
to
what
I
was
talking
in
the
beginning.
If
you
auto
scale
your
clusters,
your
environment,
then
you're
not
wasting
any
resources,
because
you
are
only
acting
with
your
environment
when
there
is
actually
a
need
for
it
not
constantly.
So
that
really
leads
your
processes
as
well.
A
Well
well,
this
is
an
answer
that
a
question
was
answered
many
many
times,
but
we'll
just
repeat
it
we'll
make
it
very
short:
it's
basically
an
architecture
that
is
centered
around
event.
It
means
we're
producing
detection,
detecting,
consuming
and
reaction
to
events
that
we're
collecting
it
can
be
payment.
It
can
be
a
customer,
it
can
be
tweets
that
we're
sending
out
to
the
world
and
the
thing
is
with
event
driven
architecture.
We
need
to
make
sure
we're
able
to
scale
because
often
we're
going
to
hit
some
picks.
A
B
Important
things
for
sure,
so
what
is
keda,
as
we
have
mentioned
kayla
so
many
times
in
this
presentation
already,
we
gotta
go
deep
into
what
it
actually
is,
and
what
does
it
do
so?
Kata
really
answers
the
issue
of
that
default.
Kubernetes
scaling
is
not
really
well
suited
for
event,
driven
applications,
it's
more
from
memory
and
cpu.
B
So
this
is
exactly
the
issue
that
cada
is
is
solving
within
the
kubernetes
space,
so
it's
even
driven
scale
controlling
so
that
they
can
run
inside
any
kubernetes
cluster
so
that
it
can
monitor
the
rate
of
events
to
preemptively
act
before
cpu
is
actually
affected.
So
this
is
where
the
power
relies.
B
A
Yeah,
let's
assume
we
have
kind
of
a
generic
e-commerce
system,
so
we
know
we
have
customers
that
are
generating
events.
We
know
we
have
orders
right
they're
putting
in
orders.
We
know
we
have
shipments
that
also
generate
events,
and
then
we
have
events
from
the
warehouse
so
without
cada
there
we
need
to
handle
bernina's
help
us
handle
the
growth
in
in
the
need
to
process
it
using
something
called
hpi.
A
So
the
horizontal
hpa,
the
horizontal
auto
scaler
and
the
resultable
auto
scaler
is
based
on
mattresses,
that
related
to
compute
things
like
cpu
cycle
ram,
etc,
and
I
can
define
the
threshold
so,
for
example,
if
I'm
reaching
a
specific
amount
of
consumption
from
the
cpu
or
a
specific
amount
of
usage
of
ram,
then
it
will
then
companies
will
start
rolling
out
and
adding
more
and
more
pods.
Those
all
metrics
can
be
collected
also
and
sent
to
different
services
on
the
cloud.
A
So,
if
we're
running
and
using,
for
example,
aws,
we
have
the
metric
adapter
there.
It's
part
of
the
aws
cloud
watch.
If
we're
running
with
prometheus,
we
can
leverage
that
we
can
also
connect
it
to
some
information
that
we
can
collect
from
kafka
and,
if
we're
running
it,
for
example,
on
the
azure
cloud,
we
can
connect
it
to
azure
monitoring
and
then
make
decision.
We
can
bring
in
all
the
information
from
the
different
messaging
services
on
azure.
So
while
this
is
nice,
it
still
not
doesn't
know
how
to
support
the
amount
of
messages.
A
It's
true.
We
can
kind
of
leverage
prometheus
in
order
to
collect.
You
know
information
from
the
kafka
brokers
into
how
much
message
is
there,
but
it
doesn't
give
us
this
kind
of
a
generic
platform
in
order
to
connect
a
lot
of
different
services
into
it.
We
have
to
handle
each
one
of
them
individually,
and
this
is
why
we
want
to
have
cada
in
there
so
cada.
A
Basically,
it's
a
refined
service
that
we
can
deploy
on
top
of
kubernetes,
that
going
to
enrich
the
kubernetes
api
and
what
it
does
it's
listening
into
the
amount
of
messages
in
different
messaging
systems.
So
it
can
be
kafka,
it
can
be
a
service
bus.
It
could
be
any
other
messaging
system
that
we
want
a
lot
of
the
messaging
system.
Those
scholars
that
we
talked
about
more
than
30
scholars
are
already
supported
by
the
community.
So
thank
you
so
much
for
everyone
who
contributed
to
it
and
added
those
scalars
and
later
on.
A
It's
what
it
does
it's
taking
the
amount
of
messages
and
translate
it
into
matrices.
That
kubernetes
knows
how
to
make
a
decision.
If
you
want
to
scale
it
up
and
scale
it
down.
So
that
brings
us
to
the
next
slide,
so
it
does
what
it
does
inside
we're.
Defining
our
yama
like
how
many
for
how
many
messages
we
want
to
start
scaling
up
our
kubernetes
cluster
and
after
we
define
those
those
amount
of
messages.
A
A
It
helps
us
scale
up
before
we're
actually
reaching
the
limits
of
compute
of
ram
or
cpu.
Once
those
mentioned
messages
are
in
the
system,
it
already
sent
the
command
to
the
kubernetes
to
scale
up
my
cluster,
and
that's
really
great.
When
I
need
to
have
a
good
throughput
of
messaging,
when
there
are
peaks,
so
it
knows
how
to
scale
up.
It
also
knows
how
to
scale
down.
So
this
is
great
for
optimizing
our
compute.
A
B
Yeah,
so
now
getting
into
our
demo
for
today.
So
this
is
abstract
representation
of
the
demo
architecture.
So
you
have
the
order
portal.
There
order,
processor,
azure
service,
bus
order,
generator
all
of
these
parts
there's
the
front
and
the
back
end
and
whatnot,
and
this
is
a
very
abstract
representation.
But
what
you
can
think
about
this
demo
while
it's
running
is
that
we
are,
we
would
be
running
a
web
shop,
for
example,
for
halloween
masks
or
anything
that
has
a
very
kind
of
high
peak
of
demand
at
some
point.
B
B
I
will
so
then,
obviously
gonna
go
here
to
pick
up
all
of
these
things.
So
now
you
should
see
all
of
these.
Let's
switch
this
a
bit
closer
here,
so
we
see
a
bit
better.
So
in
here
we
have
you
see
a
few
of
my
terminals,
many
of
them,
because
there's
going
to
be
a
lot
of
things
happening.
So,
with
this
command
of
qctl
and
git
services,
we
will
get
to
see.
Okay,
we
get
the
external
ip
where
we
get
this
cada
samples
order
portal
that
will
be
seeing
a
lot
of
action
in
the
future.
B
I
have
already
opened
it
up
because
I'm
prepared,
but
let's
just
refresh
it
so
that
we
know
for
sure
that
it
is
working
very
well,
that's
beautiful,
and
then
we
get
to
add
a
few
more
of
these
terminals
working.
So
we
get
a
bit
more
visibility
into
seeing
what
will
happen
within
the
demo.
So
in
here
we
have
where
we
see
how
many
order,
processors
we
have
ready
and
up
to
date
and
so
forth.
B
And
then,
if
we
go
to
the
last
terminal
that
you
see
that's
very
small
here,
we
will
put
the
cube
ctl
kit
parts
here
so
that
we
see
what's
the
status
as
far
as
this
goes
so
is
it
running?
Is
it
creating
or
is
it
terminating
the
order,
processors
here?
So
then
we
get
to
the
active
and
very
honestly
fast-paced
part
of
this
demo,
which
is
going
to
be
very
fast.
So
don't
blink
or
you
will
miss
it.
B
So
we
put
this
command
in,
which
is
dot
net
run,
and
then
we
will
get
a
question.
Let's
use
some
orders.
How
many
do
you
want-
and
this
is
the
part
we
say
that
100
orders
are
coming
in
at
the
same
time
and
it's
going
to
be
crazy.
So
here
we
go,
the
orders
are
coming
in
it's
coming
very
fast.
We
see
the
order
queue
spiking
up
which
is
getting
to
the
100
mark.
We
see
here
everything
is
responding
correctly.
B
We
see
order
processors
pending
container,
creating
and
then
starting
to
go
to
the
running
phase
as
well.
So
this
is
okay,
then
meeting
the
demand,
that's
happening
with
the
100
orders.
Suddenly
just
jumping
in-
and
you
can
then
see
that
now
the
the
queue
is
being
handled,
so
the
order
queue
is
going
down.
So
we're
going
down
from
the
100
mark
very
nicely.
We
can
see
the
order,
processors
there's
six
out
of
eight
well,
I
was
too
slow,
so
there's
eight
or
eight
using
now
already,
and
then
we
have
still.
B
We
have
everything
running
now,
no
more
container
creating
or
pending,
for
example.
So
that's
where
we
are
at
the
moment,
and
now
we
see
the
order
queue
getting
even
slower
and
smaller
and
smaller.
So
then
we
are
simply
just
running
the
order,
processors
as
well
as
just
keeping
them
at
10
out
of
10..
So
everything
is
working
perfectly,
no
demo
issues
here
so
far,
and
I
know
that
was
quite
fast
and
it's
just
everything
moved
a
lot.
A
Yeah,
so
let's
break
it
down
a
little
bit.
We
actually
have
a
kubernetes
cluster
on
top
of
the
azure
cloud,
so
we're
leveraging
something
called
aks
azure
kubernetes
services
as
a
managed
service.
We
also
have
message
message,
bus,
so
message.
Bus
is
a
messaging
service,
also
part
of
the
azure
family.
A
What
we
did
here,
we
kind
of
mimicked
the
logic
of
web
of
e-commerce.
We
kicked
off
deployment,
we
deployed
and
kubernetes
the
order
processor,
so
it
crosses
all
the
events
that
comes
into
the
system.
We
also
kicked
off
an
application
that
generate
those
events,
so
we're
kind
of
mimicking
those
events
as
well,
and
he
decided
to
have
to
get
us
a
100
event.
A
We
can
also
support
a
thousand
event
if
you
want
we've
seen
on
the
order
q
length,
how
the
orders
are
getting
in
and
how
fast
it
goes
down
and
the
reason
it
went
down.
It's
because
kata
kicked
in
data
was
kicked
in
and
started,
creating
more
and
more
pods
for
the
same
deployment.
So
our
order
processor,
instead
of
having
one
replica,
now
got
multiple
replicas.
So
this
is
what
happened
and
now
after
we
finish,
processing
cada
is
checking
to
make
sure
everything
is
actually
finished
and
then
what
it's
going
to
do?
A
B
B
There
in
here
you
would
normally
see
the
replica
little
click
accounts
that
that
we
would
be.
You
know
saying
that
one
two
three,
you
know
the
set
amount
of
replicas,
but
in
here
we
don't
actually
see
them,
which
is
then
the
fun
part,
because
they're
actually
in
in
this
other
place
other
area.
So
in
here
we
actually
see
that
it's
not
a
set
replica
amount,
but
it's
a
minimum
replica
amount
and
then
the
maximum
replica
account.
A
So
if
we're
diving
a
little
bit
deeper,
we
see
this
file
is
actually
defining
our
deployment
right.
So
this
is
the
application.
That's
going
to
process
the
inputs,
and
you
can
see
here
it
named
order,
processor
and,
like
any
mentioned,
there's
there's
no
replica
there
right.
So
what
is
happening
behind
the
scene
with
the
auto
scaler,
so
here
we're
connecting
it
directly
into
cada.
This
is
kind
of
a
skilled
object,
so
kata
knows
what
to
do
and
we're
connecting
it
through
the
scale
target
reference
right,
so
we're
using
the
same
name.
A
B
Yeah
and
truly
the
environmental
impact
is
here
so
that
you
know
when
there
is
no
order,
processors
working,
there's,
no,
no
nothing
happening.
Then
we
are
simply
not
you
know,
destroying
the
trees.
We
are
not
consuming
any
wasted
resources
or
anything,
but
we
only
have
the
resources
that
we
need
at
that
point
when
we
actually
need
them.
So
we
are
then
using
the
resources,
and
then
we
are
scaling
down
on
exactly
when
we
don't
anymore
need
them.
B
So
this
means
that
we
are
being
very
conscious
of
how
much
of
cloud
waste
how
much
environmental
resources.
So
all
of
the
things
that
I
mentioned
before
electricity
carbon
all
of
these
things
are
being
conserved
at
a
really
good
rate,
so
that
we
know
what
we're
doing,
and
we
know
that
we
are
auto
scaling
to
the
need.
That's
happening.
A
B
B
Klita
is
terminating
them
one
by
one,
and
then
we
see
here
in
the
blue
terminal.
We
see
that
it's
down
to
zero
completely
already
so
that
was
that
fast
we
went,
we
had
100
orders
we
scaled
up
and
then
we
had
no
longer
100
orders.
We
got
the
queue
handled
and
now
it's
down
to
zero
again,
so
no
resources
wasted
all
done
automatically
by
cada.
We
did
not
have
to
you
know,
fiddle
with
anything
or
whatnot
during
the
order
amount
that
was
happening.
Yeah.
B
A
Cool,
so
we
covered
compute.
We
know
how
we
can
get
a
better
compute
for
and
get
very
throughput
based
on
the
messages
that
we
have
in
our
queue.
But
what
about
messaging?
Our
data
in
a
distributed
environment?
Right
because
once
I
have
a
kubernetes
cluster,
I
need
to
take
into
consideration
what
the
different
workloads
that
I'm
running
on
top
of
my
kubernetes.
What
is
the
compute
that
I'm
actually
doing?
Is
that
a
stateful
compute
or
is
that
a
stateless
compute?
And
if
I'm
going
with
state
flow?
A
It's
a
totally
different
game,
because
we
want
to
optimize
data
movement.
Often
when
we're
using
things
like
apache
spark
and
things
like
a
distributed
computing
environment.
We
want
to
make
sure
we're
optimizing.
The
data
movement
over
the
network,
if
you're
familiar
with
the
concept
of
shuffling,
for
example,
when
we
need
to
shuffle
the
data
into
multiple
machines,
and
we
also
need
to
adhere
to
the
needs
of
our
data
customers
in
organization.
We
need
to
make
sure
our
data
has
a
good
quality
in
it.
Everything
is
tracked,
everything
is
there.
A
So
let's
take
a
look
at
another
architecture
example
where
we
actually
need
exactly
once
item
put
in
c
computation
for
each
message,
message
and
item
potency
means
I'm
going
to
calculate
some
function
or
I'm
going
to
do
some
mathematical.
You
know
around
that
message,
but
I'm
going
to
make
sure
I'm
doing
it
exactly
once
and
not
more
than
that,
and
even
if
I'm
going
to
either
run
another
set
of
computation.
On
top
of
the
exact
message,
it's
not
going
to
change
the
outcome.
A
A
So
this
is
why
I'm
using
kafka
here,
but
the
challenge
the
challenge
here,
that,
after
everything
after
I
kind
of
consumed
that
message
and
I
did
that
commit-
I
need
to
make
sure
I
still
have
another
layer
that
helps
me
measure
the
data
quality
on
the
distributed
world
right.
Because
then
I
want
to
plug
it
in
into
some
other
reports,
maybe
other
compute
analytics
system
like
data
bricks
or
synapse,
or
our
data
hub,
or
anything
like
that-
that
I
want
to
work
with.
A
But
the
challenges
that
I
have
with
my
cloud
object.
Storage
is
sometimes
it
doesn't
give
me
that
capability
to
track
what
really
happens
with
my
data
throughout
between
each
distributed
compute,
and
this
is
why
we're
adding
here
another
layer
of
a
new
open
source,
a
new
kid
on
the
block
named
lakefest,
so
like
fest,
is
a
new
open
source.
That's
going
to
help
us
solve
a
little
bit
of
what
of
challenges
or
kind
of
the
problems
that
are
still
in
part
of
the
object
storage.
A
So
what
lake
fest
gives
us
lake
fest
gives
us
source
control
for
object
storage.
So
we
know
it
sounds
a
little
bit
weird,
but
we'll
touch
it
more,
but
zooming
into
that
specific
open
source.
It
helps
us
verify
the
output
so
for
every
inter
iteration
that
I'm
doing
I
can
track
and
verify
the
output.
I
can
verify
the
schema.
I
can
make
sure
I
don't
have
any
fault
data
in
it.
I
have
a
good
scheme
of
verification.
I
have
a
good
format
into
you.
It
helps
me
understand
the
delivery
guarantees.
A
I
need
to
make
sure
the
delivery
is
exactly
as
I
need
it
to
be,
and,
of
course
it
helps
me
measure,
quality
and
compliance.
So,
let's
take
a
step
back
and
think
wait:
source
control
for
object,
storage.
We
often
do
source
control
for
code
right
we
have
git
get,
is
a
source
control
for
code.
So,
like
we
control
the
same
way
that
we
control
code,
we
can
start
controlling
data,
so
thinking
about
github
for
data.
A
So
what
is
a
source
control
for
data?
Where
can
we
run
it?
We
can
run
it
in
a
production
environment
for
a
kubernetes
cluster,
on
staging
sandbox
that
everywhere
we
have
a
distributed
or
we're
dealing
with
big
data
on
top
of
on
top
of
kubernetes
or
other
distributed
environments.
Why
we
want
to
do
it?
We
want
to
have
complete
verification
of
the
data
before
it's
being
used
by
their
customers.
A
A
So
this
is
super
cool,
because
that
really
helps
me
make
sure
I'm
staying
in
the
space
of
carbon
neutral.
I
don't
want
to
copy
my
data
for
every
version.
I
want
to
have
zero
copy
branching
for
that
I
can
revert
because
I
have
branching.
So
if
something
bad
going
on
with
my
large-scale
data
in
production,
on
top
of
kubernetes,
some
pod
decided
to
die
to
me
in
the
middle
of
you
know
a
spark
job,
then
I
can
revert
without
any
issues
and
the
nice
thing
about
it.
A
A
So,
for
example,
we
can
leverage
yaml
to
create
those
web
hooks
for
validating
the
data
flows
in
our
production
environment.
So
we
are
doing
separation
of
controls
once
our
developers
are
developing
the
logic,
we
can
also
have
data
ops,
people
developing
those
web
hooks
to
make
sure
we're
not
exposing
any
secrets
or
any
information
to
kind
of
making
sure
that
data
has
the
best
quality.
A
If
anyone
here
ever
worked
with
the
hadoop
ecosystem
and
tried
to
kind
of
take
what
everything
we
had
on
hadoop
and
work
on
kubernetes,
it's
very
hard
because
we
don't
have-
we
don't
have
that
observability
into
what's
happening
with
our
data.
It's
mostly
around
matrices
that
related
to
compute
yeah,
and
with
that
we
want
to
connect
it
back
to
our
vision.
B
A
So
imagine
that
for
every
observability,
the
tools
that
you're
using
you're
actually
having
another
box
there
that
says
carbon
emission,
how
we
can
optimize
it
so
we're
actually
connecting
it
to
the
values
of
people.
What
we
want
to
achieve
and
kind
of
creating
those
green
green
world,
while
also
making
sure
we're
optimizing
for
compute
and
cost.
B
Yeah,
that's
the
plan,
that's
the
vision,
but
yeah.
That's
a
few
resources
here,
obviously
check
out
keda
and
thank
you
for
cater
for
helping
us
with
the
demo
and
everything
to
do
with
these
and
then
obviously,
here
are
a
few
resources,
a
great
short
course
on
sustainability
and
software,
and
the
engineering
for
microsoft
as
well
as
I
wrote
an
article
about
organizational
view
into
these
topics,
so
check
that
out
as
well
and
then
the
slides
and
materials.