►
From YouTube: Observability In ArgoCD/Rollouts Using Streaming ML For Reducing MTTR - Vigith Maurice & A Kalamkar
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Observability In ArgoCD/Rollouts Using Streaming ML For Reducing MTTR - Vigith Maurice & Amit Kalamkar, Intuit
Speakers: Amit Kalamkar, Vigith Maurice
At Intuit one third of P1/P2 outages are caused by a change. As Intuit runs ~2500 services on K8s we need to quickly detect and resolve problems using AIOps. Our talk focuses on how we built a K8s native DAG-based streaming processing platform (Numaflow) and streaming ML platform (Numalogic) which is open-sourced under Numaproj to address this problem. We will show how we collect, process, and analyze in-cluster data in real-time and how our Numalogic computes anomaly scores for each deployment. This DAG-based ML platform has now been adopted by Intuit and helps our ML engineers focus on writing just the inference and pre/post-processing logic while the platform takes care of building the dynamic execution model, retries, buffering between the vertices, back-pressure, conditional-forwarding, and auto-scaling. We will also show how we integrated Observability into Argo CD so users can understand and remediate the behavior induced by change and how this is helping Intuit reduce MTTD/MTTR.
A
Today,
we
wanted
to
talk
how
we
integrated
observability
into
Argo,
CD
and
Argo
rollouts,
which
is
backed
by
AI
Ops,
and
how
that's
helping
into
it
reduce
mptd
and
mttr.
My
name
is
Amit
kalamkar
I,
lead,
observability
and
analytics
at
Intuit.
I
have
with
me
visit
Morris.
He
is
a
principal
engineer
for
that
that
tech
tech
track
here's
the
agenda.
For
today
we
will
talk
about
a
specific
problem
that
is
change,
induced
incident
and
how
we
resolved
it
using
aiops.
A
A
B
A
Widget
belongs
to
developer
experiences
and
platforms.
Our
group
is
responsible
for
all
build
time
and
runtime
add
into
it.
Just
to
give
you
a
idea
about
scale.
We
run
around
2000
plus
services
on
kubernetes
and
the
investment
we
have
done
on.
Modernization
of
the
internal
platforms
have
resulted
in
6X
Improvement
in
development
velocity
at
into
it
into.
It
is
very
much
bought
into
open
source.
We
not
only
use
it,
but
we
have
active
contributors
and
maintenance
for
a
lot
of
CNC
projects,
including
arvo,
istio
and
others.
A
A
Let
me
first
start
with
giving
you
guys
an
idea
about
tech
ecosystem
at
Intuit.
We
started
the
modernization
of
this
platform
in
2018..
We
pretty
much
modernized
everything,
front-end
platforms,
back-end
platforms.
We
moved
all
our
container
payloads
as
part
of
this
two
kubernetes.
We
also
created
paved
roads
both
for
serverless
and
services
that
gave
our
developers
end-to-end
determination
from
commit
to
deploy.
A
A
This
data
we
use
to
derive
actionable
insight
for
different
areas
like
operational
excellence,
cost
security
and
so
forth.
To
derive
this
actionable
Insight
at
scale.
We
needed
a
platform,
a
IRS
platform
that
can
scale
and
that's
how
new
map
Raj
was
born,
is
a
kubernetes
native
data
processing
and
analytics
tools.
It
consists
of
two
areas:
one
is
pneuma
flow,
which
deals
with
data
processing.
It
makes
data
processing,
easy
scalable
and
reliable
and
numerologic,
which
we
have
open
sourcing,
a
lot
of
models
which
we
use
internally
and
which
runs
on
Remo
flow.
A
One
of
the
core
principles
at
Intuit
is
innovation.
We
want
to
make
sure
we
innovate
and
We
Know
It
Fast.
We
do
over
Thousand
Villages
per
day
in
production
and
that's
only
possible
because
the
investment
we
made
in
Argo
most
of
the
people
might
know
Argo
into
it,
created
Argo
and
open
sourced
it.
Over
100
companies
now
use
it
across
the
world
being
one
of
the
largest
SAS
company.
One
of
operational
excellence
is
always
at
the
Forefront.
For
into
it.
We
want
to
make
sure
that
our
products
are
always
available
and
If.
A
There
are
issues
we
resolve
them
fast,
so
there
is
less
empty
TD
and
empty
TR
for
for
our
incidents.
So
we
are
always
looking
at
ways
to
improve
this.
One
of
the
areas
we
saw
where
it
needed
Improvement
is
change
related
incident.
We
found
out
that
one
third
of
our
incidence
were
caused
by
change
and
their
mttd
was
higher,
and
then
we
dig
into
a
little
bit
more
into
the
data
as
why
it's
happening.
There
were
few
reasons.
A
Our
deployment,
as
well
as
operational
experiences,
were
disjoint,
so
people
were
deploying,
but
they
needed
to
go
different.
Dashboard
to
figure
out
what's
happening
to,
people
had
to
go
to
hundreds
of
Matrix
and
needs
to
be
really
know
the
application
very
well
to
understand
the
quality
of
the
change
and
they
depend
on
on
the
developer,
how
much,
how
much
they
know
the
application
to
resolve
three.
We
didn't
have
an
automatic
aioffs
based
rollback
through
our
go
rollouts
and
that
resulted
in
a
higher
MTT.
A
A
Instead
of
going
through
hundreds
of
Matrix
and
making
a
manual
judgment,
we
run
a
multivariant
model
which
is
powered
by
olamapraj,
where
you
get
one
signal
which
tells
the
quality
of
your
change,
whether
it's
good
or
bad.
Three,
we
removed
the
human
from
the
equation.
We
integrated
with
Argo
rollouts
so
that
if
the
change
is
bad,
it
automatically
rolls
backs,
and
that
has
helped
us
reduce
mttd
and
mttr.
So
let
me
hand
it
over
to
Vijay
to
show
a
demo.
B
Thank
you.
So
let
me
start
off
with
the
demo
where
like
listen,
this
is
the
Persona.
Is
a
service
developer,
so
let's
put
an
I.
Am
a
service
developer
and
I'm
gonna
make
it
change
the
change
I'm
about
to
make,
has
a
small
bug
inside
it
and
we
will
use
the
three
features.
I
will
talk
about.
That's
the
metrics.
We
have
integrated
the
time
series
anomaly
scores.
We
have
and
also
the
automated
rollback
that
auto
rolls
back
and
mitigate
the
problem.
B
For
that
first,
let
me
introduce
a
demo
app
okay,
so
what
it?
What
you
see
here
is
that
the
when
the
browser
renders
it
talks,
the
back
end
and
backend
gives
you
two
things.
One
is
the
type
of
the
fish
here
you
see
octopus.
This
is
the
version
one
we
are
running
and
the
color
of
the
fish
lotta
person
is
a
happy
state
that
meaning
we
are
getting
successful
requests
from
the
back
end
once
in
a
while,
you
will
see
a
red
unhappy
octopus
going
around.
This
shows
that
the
back
end
has
a
unhealthy
response.
B
B
Do
your
refresh
and
synchronize
the
change,
so
this
is
an
algor
roll
of
stage,
meaning
this
is
a
progressive
delivery
where
a
new
Canary
gets
deployed
as
part
of
the
change.
I
just
deployed
right,
and
you
will
see
that
the
new,
the
old
ports
are
getting
terminated
and
the
new
new
ports
are
coming
up
in
the
same
time.
What
happens
is
that?
B
So?
If
you
see
right
visually,
you
can
see
on
the
right
hand,
side
right.
There
is
a
new
set
of
slow
moving
right,
one
could
say
even
fish
is
moving
around
right.
This
is
the
new
version
of
the
application.
It
is
slow
and
you
see
that
it's
erroring
out
right.
The
point
is
that
in
real
life,
unlike
the
demo,
you
don't
go
to
let's
say:
I,
don't
deploy
and
go
to
Turbo
Tax
online
to
see
whether
there's
a
redfish
right,
the
meaning.
These
are
very
subtle
errors
right
and
the
way
you
detect.
B
This
kind
of
errors
are
looking
at
the
metrics.
Your
application
colors
this,
but
this
demo
is
just
to
show
the
effect
of
such
a
bad
change,
manifested
in
the
UI
right.
So
what
you
need
is
you
want
to
look
at
the
metrics
and
quantify
whether
it
release
is
good
or
bad
along
the
lines.
You
need
to
know
that
we
do
see
in
our
CD
right.
You
see
that
it's
healthy.
The
resources
are
shown
as
healthy,
though
the
application
behind
it
is
unhealthy.
B
Is
here
the
help
means
your
resources
is
healthy,
but
it
does
not
represent
the
state
of
your
application.
Now,
traditionally,
one
would
go
to
your
Matrix
provider
to
look
at
logs.
So
this
is
the
problem
statement
number
one
where
we
said
hey.
We
have
metrics
integrated
with
Argo
CD
right
now,
when
you
open.
What
you
see
here
is
the
default
template.
It's
written
golden
signals
as
the
tab
name,
it's
just
the
default
configuration.
We
have
deployed
it
into
it.
B
It's
just
a
collection
of
metrics
which
we
think
are
vital
for
our
application
is
from
the
family
of
Matrix
called
request,
errors,
duration
and
also
utilization
saturation
and
errors.
This
is
out
of
the
box
for
any
application
that
you
do
it.
You
can
reconfigure
it
to
whatever
you
think
is
the
right
thing.
It's
totally
detox
driven
the
the
tab
itself
and
the
way
you
read
this
Matrix
is
the
good
old
way
the
x
axis
is
per
minute.
That's
how
you
configure
it
and
the
way
to
interpret
the
metric
is.
B
The
latency
is
second
actually
on
the
left
side.
So
the
current
stable
hash
that
is
5
B7.
The
blue
line,
is
having
around
0.04
millisecond
latency,
while
the
new
Canary
hash
is
around
having
0.8.
Second
latency,
that's
what
you
clearly
see
with
the
new
Canada
reverse
of
the
old
octopus
release,
right,
I,
told
average.
It
doesn't
mean
you
have
to
see
average.
It's
also
the
even
better
case.
Config
driven,
you
can
point
it
to
percentiles
or
whatever
you
see
is
right.
I'll
talk
about
how
we
get
the
metrics
down
the
lane.
B
We
also
have
other
metrics
like
status
code,
that
is
total
number
of
requests
with
200
with
Canary,
and
so,
if
you
see
here
right,
we
do
see
some
successful,
yellow,
happy
fishes
too.
It's
just
that
the
number
of
requests
successful
are
very
low.
Along
the
same
line.
We
have
error
right.
You
can
clearly
see
there's
a
spike
in
error.
We
have
HTTP
traffic
and
just
to
show
that
this
charting
library
is
very
powerful.
We
added
a
pie
chart
out
there
yeah.
B
One
by
looking
at
here
could
say
that
hey
my
deployment
is
unhealthy
right.
My
errors
are
higher,
I
do
see
higher
latencies
and
can
come
to
that
conclusion,
but
what
we
found
out
is
in
real
life.
You
have
way
more
metrics
than
that,
and
you
should
be
a
very
experienced
application
developer
in
that
platform
to
really
understand
whether
you
can
qualify
or
quantify
whether
a
deployment
is
good
or
bad.
So
you
need
to
analyze
a
lot
of
metrics
holistically.
B
What
we
wanted
was
we
wanted
one
single
score
that
can
Define
it
for
you
right
that
is
the
AOS
part
of
it.
So
this
is
a
multivariant,
a
Ops
platform
that
looks
into
multiple
Matrix,
which
is
again
configurable.
You
can
configure
as
many
metrics
to
look
into
and
gives
you
one
single
score
to
make
the
decision
right.
The
way
you
have
to
interpret
the
score
is
if
the
score
is
between
0
to
3.
It
is
operating
within
the
normal
operating
pattern
between
three
to
seven.
It
is
deviating
and
Beyond
seven.
B
It
is
totally
deviant
right
and
if
you
see
on
the
right
hand,
side
the
role,
the
rollback
has
happened,
automatically
I'll
talk
to
I'll,
get
to
that
how
it
happened,
but
the
key
thing
is,
while
I'm
explaining
things
have
happened.
We
have
mitigated
the
problem,
this
anomaly,
unlike
the
demo
where
we
show
the
yellow
fishes
and
the
red
fishes.
This
is
the
real
thingy
we
have
been
using
it
into
it
for
last
four
years.
B
We
use
this
to
create
incidents
when
there
is
a
spike
in
user,
behaviors,
right,
anomalous,
behaviors,
they
are
saying
in
front
and
when
we
see
anomalous
behaviors
when
Services
talk
to
each
other.
So
this
is
the
real
thingy
and
that's
the
reason
we
open
source
it,
because
we
felt
that
the
community
could
use
it
and
we
that
is
the
new
my
logic
part
of
it,
where
we
opens
off
this
model,
so
you
could
use
it.
B
B
This
is
an
anomaly
which
just
looks
into
your
Canary
deployment,
unlike
the
entire
deployment,
just
just
focus
on
the
canary
Delta
small
deployment
you
have
made
and
the
success
criteria
for
this
deployment
is
that
the
anomaly
should
be
less
than
three
so
the
moment
it
sees
that
it
is
more
than
three.
It
goes
and
auto
rollbacks
right.
So
this
is
an
AV
testing
between
your
stable
hash
and
the
canary
hash
That's
The
Power
of
this.
So
you
can
do
anomaly
at
different
levels.
We
have
in
the
aops
level.
B
You
are
looking
at
the
application
as
a
whole.
Here
we
are
looking
at
the
Canon
audacities
and
you
can
see
that
it
failed
because
the
score
is
coming
very,
very
close
to
10
right,
of
course,
greater
than
three
is
a
failure,
but
you
can
clearly
see
the
number
of
errors
that
are
high
and
that,
hence
we
rolled
back
I
think
this
is
the
demo
we
have,
and
so
you
saw
three
things
one
is
the
metrics
I
just
wanted
to
finish
it
off
just
showing
that
right.
So
one
is
the
metric
you
saw.
That
is
the.
B
If
you
see
here
right.
So
if
you
see
even
the
metric
after
the
deployment
has
gotten
over,
your
values
are
smaller
right.
The
new
country
is
no
longer
seen
in
The
Matrix.
Then
you
saw
the
aops
that
looks
into
the
ndf
system
as
a
whole
and,
lastly,
you
saw
the
canary
deployment
the
canary
analysis
using
AO.
So
there
are
three
things
we
saw
now.
Let
me
get
into
the
architecture
on
how
we
are
doing
it
and
continue.
B
Okay,
so
how
are
we
seeing
it
right?
So,
when
user
lost
the
Argos
a
UI,
it
talks
Argo
CD
server.
Here
we
use
Argos
CD
extension
framework
to
see
whether
observability
has
been
turned
on.
We
developed
a
new
thingy
called
Argo
CD
metric
server,
which
is
for
this
project,
which
can
put
in
metrics
from
any
service
provider.
You
have
promises
attend
to
it,
we
use
from
it
since
Prometheus,
but
it
can
talk
to
any
service
provider.
This
is
a
decoupator.
Now
this
is
the
first
metrics.
B
You
saw
right
the
metrics
that
is
being
collected
by
from
this,
since
this
is
a
decoupled
architecture
to
see
the
anomaly
score.
All
you
need
to
do
is
the
pneumo
project
just
need
to
write
anomaly
back
to
the
Prometheus.
That's
the
Hub,
where
you
saw
the
AI
op
stamp.
This
is
how
it
is
fading
it
off
now
to
do
Argo
rollouts.
B
All
you
need
to
do
is
do
the
analysis
template
from
Prometheus,
so
Numa
approach
will
write
another
anomaly
score
just
for
the
canary
deployment
so
that
you
can
get
the
score
in
again
for
analysis
template
by
Argo
rollouts,
and
that
makes
the
decision
of
rolling
back
now
this
buttons.
In
the
second
question,
how
do
we
do
anomaly?
B
This
anomaly
is
computed,
even
though
you
are
not
looking
rendering
the
UI
or
not
even
when
you
are
just
deploying
it's
always
on.
This
is
a
streaming
system
which
always
inspects
and
says
whether
your
system
is
doing
well.
The
way
it
happens
is
Prometheus
has
a
feature
called
remote
writer,
so
it
for
every
metric
it
scrapes.
It
will
push
the
metric
real
time
streams
into
our
new
map
project
pneuma
flow
we
on
which
the
new
my
logic
runs,
and
it
does
feature
engineering
like
scaling
the
metric
and
so
forth.
B
Once
it's
does
that
it
does
the
inferencing
right.
It
assigns
an
anomaly
score
once
the
anomaly
score
has
been
computed.
We
do
post
processing.
This
is
to
normalize
it
to
a
human,
understandable
format
and
give
it
between
0
to
10..
Otherwise,
it
will
be
too
complex
for
them
to
interpret
it,
and
this
is
an
N
plus
one
anomaly
score,
so
we
compute
for
each
multivariant
aspect,
one
and
only
square,
and
one
plus
one
to
have
the
unified
anomaly
score.
B
Since
it
is
an
operational
system,
things
change
all
the
time.
We
cannot
always
configure
the
normal
system.
It
auto
will
discover
about
new
applications
that
come
online
new
configurations
that
come
online.
That
means
we
have
to
do
an
inline
training
all
the
time,
so
we
trigger
our
training
training
such
as
the
data
from
Prometheus
and
it
stores
it
in
model
storage,
model,
storage
out
of
the
box.
B
The
problem
number
one
is
a
lot
of
boilerplate
code
for
streaming.
What
it
means
is
what
we
found
out
was
our
ml
Engineers
are
spending
more
time,
writing
streaming
code
and
doing
the
infrastructure
for
streaming
like,
for
example,
if
you
use
Kafka,
creating
Kafka
topics,
how
to
reliably
read,
consume,
produce
data
and
so
forth,
rather
than
what
they
do,
the
best
that
is
ml
exploration
and
ml
experiments,
so
that
was
one
bigger
problem
for
us.
Second
was
auto
code
here,
adok
actually
means
non-standard.
B
For
example,
we
deploy
streaming
systems
as
a
deployment,
which
is
mostly
good
for
North
South
traffic,
while
streaming
systems
are
East-West.
So
there's
much
more
to
that,
meaning,
you
need
to
really
understand
how
the
pipe
is
basically
fluid
dynamics.
At
that
point,
streaming
systems
are
more
like
fluid
dynamics
right,
it's
not
a
north-south
one,
single
transaction.
It
has
much
more
to
that
context
is
very
important.
B
B
This
does
not
mean
that
there
are
no
other
data
streaming
systems
out
there.
There
are
excellent
systems
like
Apache
flinks
power,
streaming
Samsung
to
name
few,
but
those
are
very
complex
data
engineering
tools
meant
to
do
data
Engineering
in
a
data
centralized
way
in
this
demo.
What
is
in
earlier
demo?
What,
in
the
new
map
project
the
fall
system?
Pipelines
is
along
with
their
application
name
space.
It's
so
lightweight
that
it
sits
along
with
our
application,
namespace
and
completes
anomaly.
B
You
don't
need
to
do
that
like
so
we
move
all
the
problems
to
the
data
producer
side,
so
we
needed
a
very
lightweight
cost
efficient
and
a
lightweight
system
rather
than
complex
systems
like
data
engineering
tools
and
Frameworks.
We
have
so
for
these
reasons
we
did
experiments
and
then
we
ended
up
building
our
own
system.
This
is
the
new
flow.
B
B
Then
it
passes
to
the
first
user
defined
function.
All
it
cares
is
I.
You
are
given
a
key
and
a
value.
The
output
can
be
zero
or
one
or
many
k,
n
values.
The
moment
the
output
has
been
computed,
it
can
choose
which
path
it
has
to
take
either
it
can
take
all
the
path
or
a
subset
of
path.
This
is
for
a
b
testing,
weighted
experiments
and
whatnot.
B
That
is
called
conditional
forwarding
internally
right.
So
we
do
that
and
then
we
you
can
have
as
many
udfs
as
you
want.
In
the
end,
we
have
a
sink.
Sync
is
its
main
job
is
to
make
sure
that
we
persistently
write
the
data
to
a
stream
or
to
a
blob.
Stop.
You
can
easily
write
a
user
defined
string
because
they
are
like
you,
you
might
come
up
many
different
ways
to
write
it
and
we
Define
very
easy
way
to
write
UDF
sensings.
B
So
what
are
the
features
of
pneumo
flow?
The
most
important
I
would
say.
The
value
is
the
the
one
thing
which
we
hold
true
to
very
core
to
ourselves.
It
should
be
very,
very
easy
to
use.
You
should
be
able
to
get
it
up
and
running
in
five
minutes,
and
it
should
be
even
we
would
say
you
should
be
able
to
learn
it
in
less
than
five
minutes.
It's
language,
agnostic,
meaning
you
can
write
in
any
different
language.
You
like
each
udfs
and
the
things
it
is
lightweight
it's
native
to
kubernetes.
B
B
With
this
complex
systems,
what
we
need
is
separation
of
concerns.
Streaming
systems
are
very
complex.
It
can
get
very
very
with
a
bigger,
dag
right,
so
separation
of
conscience
is
very
important
and
we
separate
functions
as
units
and
each
functions.
We
guarantee
that
the
debt
a
transfer
between
them
is
exactly
one
semantic,
so
there
is
no
duplication
problems
also.
It
means
we
also
support
other
lower
demands,
but
semantics
like
at
least
once,
but
the
key
thing
is
we
support
exactly
upon
semantics
with
all
this.
We
have
these.
These.
We
are
standardized
streaming.
B
What
it
means
is,
if
you
were
to
change
in
the
demo
you
saw,
we
saw
writing
from
Prometheus
when
we
were
writing
back
the
Prometheus.
For
writing
the
anomaly
score.
You
can
easily
replace
both
the
source
with
kaf
canvassing
with
something
else,
but
the
core
logic
of
ml
system
still
works.
This
is
how
we
use
the
same
platform
for
customer-centric,
anomaly,
detection
and
whatnot,
meaning
the
core
anomaly
detection.
All
we
need
to
do
is
change
the
source
on
the
thing
right
things
will
work
and
we
also
do
back
pressure
handling.
B
B
Lastly,
anything
that
is
built
on
pneuma
Flow
should
be
operational,
excellent,
It's,
a
fire
and
forget
model,
because
your
input
never
ends
is
a
unbounded
stream.
So
you
turn
it
on
and
then
you
can
forget
about
it,
because
we
take
care
of
automatic
Ray
Price
spot
migration,
node
migration.
What
not
now,
let
me
quickly
introduce
new
my
logic.
Numerology
actually
is
the
ml
system
we
have,
and
this
is
powering
the
real-time
analytics.
B
This
is
a
library
actually,
but
it
works
very
well
when,
on
top
of
new
flow,
the
way
we
use
it
today
is
that
we
have
a
stream
coming
in.
We
do
pre-processing,
we
do
inference
post
processing
and
right
into
a
sync,
and
we
have
many
different
models.
For
example,
we
have
Auto
encoders,
clustering,
Logics
and
so
forth
for
different
use
cases,
but
on
a
very,
very
high
level.
This
is
how
the
deployment
looks
like
in
and
around
okay,
and
we
also
go
inline
training
using
the
same
system.
B
B
The
features
it
provides
is
a
numerologic
is
a
model
repository
for
aops
models,
which
is
very,
very
well
written
for
time.
Series
data.
There
is
a
lot
of
data
processing
toolkits
available
with
it,
which
helps
you
do
analyze,
understand
operational
data,
which
is
time
series
in
nature.
Online
training
is
out
of
the
box
a
bit.
Experimentation
is
one
core
feature
we
really
really
hold
hold
on
to
it,
because
we
experiment
a
lot
with
ML
models,
and
the
point
of
this
is
earlier.
B
We
had
a
lot
of
problem
with
ML
developers
having
tough
time
with
streaming
and
everything
this
improves
ml
development
velocity.
It's
a
cookie
cutter
development
right.
You
write
your
udfs,
you
deploy
it,
you
get
every
feature
of
numerology,
which
I
talked
about
and
much
more
than
what
minimal
logic
provides
on
top
of
it
with
this
actually
yeah.
The
earlier
slide
didn't
do
gestation,
human
flow
or
numerology,
because
it's
a
very
complex
system
come
to
our
board
talk
about.
B
A
Thank
you
Vijay
for
awesome,
demo
and
introduction
to
Nima
praj.
So
what's
next
for
us,
we
are
obviously
adopting
it
across
our
Argo
CD
and
our
go
roll
out
clusters.
We
have
also
very
rich
road
map
for
Nima
paraj
on
pneuma
Flow
side.
Some
of
the
things
which
we
are
open
sourcing
includes
windowing
and
aggregations,
which
we
use
internally
a
neurologic
side.
We
will
be
open
sourcing,
other
models
which
you
internally
like
forecasting.
A
These
are
our
gates.
So
if
you
go
to
the
any
of
these
Gates,
there
are
a
lot
of
examples
and
documentation
how
to
get
started.
You
can
take
a
look
at
it.
If
you
like
it,
you
can
start
it.
You
can
also,
like
we
just
said,
go
to
our
booth.
There
are
other
use
cases
for
pneuma
project.
You
can
look
at
including
streaming
workflows,
image,
processing
automatic
scale
up
and
scale
down,
so
it
means
you
can
visit
that
and
you
can
have
a
discussion.