►
From YouTube: AIOps SIG Full Mtg April 29 2019 OpenShift Commons
Description
co-chairs: Diane Mueller and Marcel Hild (Red Hat)
Guest Speaker(s)
Federator.ai OpenShift Cluster Optimization for Multi-Cloud – Brian Jeng (prophetstor)
Overcoming Tomorrow’s Operational Challenges with AIOps – Phil Tee (MoogSoft)
Log Anomaly Detection - Zak Hassan (Red Hat)
upcoming events:
AIOps SIG Meetup at Red Hat Summit - May 8, 2019 2:00 PM
location: OpenShift Commons Meet and Greet Booth in Community Central
2019 AIOps Exchange - Four Seasons Hotel, San Francisco, California | May 15, 2019
https://www.aiops-exchange.org/2019-aiops-exchange/
resources:
https://github.com/AICoE/log-anomaly-detector
https://groups.google.com/forum/#!forum/aiops
Next Meeting May 27th Monday 9:00 am Pacific
B
A
We're
also,
while
people
are
milling
about
where
somebody's
coming
up
and
Marcel
and
myself
will
both
be
there
and
we'll
be
holding
sort
of
a
an
open
ship,
meet-and-greet,
booth
and
community
central
and
I.
Think
Marcel
you
signed
up
for
an
hour
and
go
look,
look
up
what
our
that
was.
If
people
who
are
part
of
this
segment
to
come
and
meet
Marcel
and
myself
will
be
in
the
booth
at
that
appointed
hour
and
I'll
go
dig
up
what
that
is
in
the
next
few
minutes.
A
B
A
From
profits
store
from
Brian
Jiang
and
it
Brian,
if
you
would
like
to
try
taking
over
the
controls,
I
will
relinquish
the
screen.
Sir
and
I'm
done.
You
can
share
your
screen
and
it
going
and
while
you're
talking
there's
about
11
people
on
the
call
right
now
again
the
link
to
the
the
notes
where
you
should
add
your
and
your
affiliation,
if
you
would
is
in
the
chat-
and
this
is
the
AI
ops
egg-
and
this
is
our
second
meeting-
really
thankful-
that
you're
all
joining
us
to
have
this
conversation.
D
D
C
All
right,
I'm
Brian
I'm,
an
SI
over
at
prophets,
tour
and
I'm
gonna,
be
talking
about
federated
AI,
which
is
an
AI
op
solution
for
OpenShift
and
multi
cloud
environments.
Just
to
get
you
guys
a
little
bit
of
background
on
the
multi
cloud
market,
it's
growing
really
quickly,
right,
more
more
and
more
businesses
are
shifting
their
IT
infrastructure
to
the
cloud
and
it's
projected
to
go
from
20%
to
40%
by
the
end
of
2019
and
it's
already
over
a
trillion
US
dollars.
C
You'll
have
to
do
a
lot
of
research
on
your
own
and
what
we
ate
was
so
just
for
day,
one
you're
at
a
roadblock
right
and
then
for
day
two,
even
after
you
deploy
your
application
into
the
club,
you'll
have
to
regularly
monitor
and
manage
the
resource
usage
based
on
just
just
a
safeguard
against,
in
effect,
inefficiently.
Utilizing
your
cloud
instances.
If
you
over
provision,
you
waste
a
lot
of
money.
If
you
under
provision,
you
have
a
lot
of
application
issues
all
right.
C
So
that's
where
we
come
in
federated
a
I.
We
ran
a
off
solution
that
simplifies
the
cost
optimization
process
for
both
day,
one
and
day
two
operations
of
multi
cloud
environments.
So
for
day
one
just
give
us
the
application
and
optimization
policy
and
we'll
recommend
which
cloud
service
provider
instance
type
you
choose,
but
we
do
all
the
legwork
for
you.
You
just
have
to
tell
us
what
you
want
to
deploy
and
then
for
day.
Two,
that's
what
our
machine
learning
AI
actually
comes
in.
C
So
here's
some
more
details
about
our
day.
1
deployment,
the
user
will
notify
us
of
which
application
he
wants
to
deploy
approximately
how
many
requests
per
day
and
which
policy,
whether
it's
minimizing
costs
and
maximizing
performance,
maintaining
SLA
and
will
recommend
which
cloud
provider
instance
type
is
the
most
suitable.
And
if
the
user
wants
to
just
manually
deploy
into
a
cloud
of
his
choosing,
you
can
we
just
give
directly
the
amount
of
resources
you'll
need
just
just
straight
up
and
then
for
day
2.
This
is
the
where
our
machine
learning
AI
comes
in.
C
We
learn
the
future
usage
of
each
pod.
So
if
you
guys
look
at
this
graph
here,
there's
a
blue,
solid
line,
which
is
the
observed,
CPU
usage
and
then
there's
a
white
dotted
line,
which
is
the
predicted
CPU
usage,
and
you
can
see
we're
about
10
minutes
ahead
and
the
lines
are
really
intertwined
just
showing
how
accurate
our
prediction
engine
is.
C
Ok-
and
this
is
a
study
done
by
TSO
logic
in
2017-
is
this
kind
of
the
use
case
we're
going
after
immediately
about
84%
of
on-premise?
It
environments
were
over
provisioned
and
if
they
move
directly
to
the
cloud
of
the
direct
matching
of
their
resources,
they
would
actually
pay
more
than
they
currently
were
paying
on-premise.
But
if
they
fitted
their
clouds,
just
right
which
we're
aiming
to
do,
they
could
save
36%
in
terms
of
monetary
value
and
60%
in
terms
resources.
Right
and
let
me
just
go
over
to
it's-
not
really
a
demo.
C
This
is
this
is
just
the
side-by-side
comparison
of
the
native
kubernetes,
horizontal
pod
autoscaler
and
horizontal
autoscaler
using
our
federated
I.
We
found
benefits
in
these
three
main
categories.
We
can
serve
the
same
workload
using
19%,
less
replicas.
We
can
reduce
the
amount
of
CPU
over-limit
instances
by
61%
and
we
can
reduce
the
out
of
memory
instances
by
almost
90
percent
and
we
have
graphs
that
show
all
this
data.
C
But
then
the
big
one
here
is
actually
the
out
of
memory
instances
basically,
every
time,
an
application,
it's
an
out
of
memory,
it
stalls
out
and
crashes.
So
you
want
to
avoid
these
at
all
costs
and
we
definitely
you
can
see
where
the
green
line.
We
only
have
two
instances
here,
whereas
the
rest
is
just
a
native
kubernetes
h-tpa.
C
We
can
reduce
these
by
almost
90
percent
all
right
and
then
just
note
that
this
is
just
the
horizontal
comparison,
but
our
federated
I
can
be
applied
to
the
vertical
Patil
scalar,
the
cluster
autoscaler
and
the
native
kubernetes
scheduler.
So
all
these
different
facets
of
your
open
shifts.
Buster
can
be
optimized
using
machine
learning
with
our
with
our
solution
and,
what's
really
cool
is
once
we
have
those
usage
predictions
we
can
feed
it
back
into
our
day
one
tool,
and
now
we
have
a
new
recommendation
of
which
cloud
provider
and
instance
type
to
choose.
C
So
now
your
full
stack
from
your
resource
usage.
All
the
way
up
to
your
cloud
provider
is
fully
optimized
using
Federale,
and
I'm
just
gonna
go
over
to
operator
hub
io
over
here.
Just
to
show
you
how
simple
it
is
to
deploy
us
or
we're
already
listed
on
opera
github,
just
click
us
and
all
our
details
and
instructions
and
how
to
configure
the
custom
resources
is
over
here
how
to
install
it
and
yeah.
We
have
all
the
links
on
the
sidebar
and
that'd.
Be
that's
it.
Thank
you.
B
C
Well,
we're
gonna
do
it
based
on
a
policy,
a
policy
based
prediction.
So
if
you
want
like
like
a
cost
optimization
but
will
predict,
the
predictions
will
be
the
same,
but
the
scaling
will
be
based
on
the
policy,
though
the
modeling
item-
I'm,
not
I,
don't
think
will
be
changing
too
much
but
I'll
to
check
with
our
team.
If
we
wanted
to
change
that.
E
B
I
think,
from
a
from
a
source
perspective
and
from
an
ecosystem
perspective,
this
is
one
of
the
first
complete
end-to-end
solutions
that
run
on
native
kubernetes
and
OpenShift,
which
will
hook
some
sort
of
prediction
into
the
metrics
being
gathered
from
kubernetes.
So
I
think.
That's
that's!
A
really
cool
open
source
solution.
A
A
E
E
Super
well,
you
know
thanks
for
inviting
us
to
speak
along,
and
you
know
we're
we're
super
excited
to
see.
You
know.
Red
hats
are
really
dove
into
the
air
of
stuff,
and
you
know
it's
been
a
it's
been
an
area
and
a
market
that
have
moved
soft
we've
been.
You
know
somewhat
very
early
in
from
the
you
know,
from
the
history
of
the
company
going
all
the
way
back
to
2012.
E
E
I've
been
involved
in
service
assurance
and
operations
and
since
the
early
1990s
having
co-founded
Micronesia
wrote
a
protocol
neckhole,
and
you
know
that
a
bunch
of
startups
in
this
space,
culminating
in
the
move
soft
and
this
little
light
right
here
in
a
kind
of
gives
you
a
sort
of
a
brief
history
of
time.
In
terms
of
you
know
what
I've
seen
changing
morph
over
that
period-
and
you
know
if
you
kind
of
go
back
to
the
original
mainframe
era,
which
was
sort
of
dying
out
as
I
entered
the
market.
E
You
know,
service
assurance
was
actually
done
extremely
well
and
part
of
the
reason
for
that
was
you
know
a
single
vendor.
It
was
very,
very
simple:
it
was
pretty
low
scale
and
change
cycles,
were
you
know,
glacial
and
to
say
the
least.
But
of
course
we
live
only
through
dirty
computing
wave.
We're
in
it
is
started
to
erode
the
ability
to
you
know
to
deliver
service
in
a
very
straightforward
way,
just
because
scaling
and
clarity
started
to
enter
into
the
environment.
E
With
you
know,
multiple
vendors
standards
driven
best
in
breed
and
type
approach
and
really
what's
going
on,
is
the
kind
of
the
latest
phase
that
is,
you
know,
we're
all
living
in
the
cloud
residents
sort
of
digitally
transformed,
Enterprise,
these
spectacularly
complex
in
the
high
scale.
We
are
seeing
almost
a
sort
of
a
chaotic.
E
You
know
deployment
paradigm,
where
you
know
we
large
enterprises
have
moved
to
a
fully
agile,
devops
application
development
tool
chain,
so
change
cycles
are
almost
instantaneous
of
second,
very,
very,
very
large-scale,
and
you
know
in
requires
something
very
different
to
be
done,
and
you
know
this
is
kind
of
a
sort
of
almost
a
dimbo
slide
in
a
lot
of
ways,
just
to
sort
of
punch.
The
point
home
that
you
know
in
the
yesterday
datacenter,
you
know
if
you
were
writing
an
application.
E
You
know
you
wrote
an
app
to
run
on
a
specific
operating
system
to
run
on
natural
physical
server.
You
know
there
are
two
to
three
interactions
between
those
three
components.
You
know
you
have
to
worry
about
making
sure
the
right
version,
your
insistence
on
our
way.
You
know
the
app
is
built
in
the
right
way.
The
app
is
built
to
optimize
the
bare
metal
you
know,
but
still
you
know
it
was
pretty
contained.
Then
you
look
today
and
actually
this
is
a
slice
through
our
cloud
in
terms
of
how
we
and
deploy
our
application.
E
You
know
we're
also
navigating
our
way
through
the
whole
multiclad
and
instance
management
world
as
well.
Net
net
is,
is
the
complexity.
Is
a
complete
phase
transition
step
change
to
what
it
was
before
and
there's?
So?
What
about
that?
Is
you
know
that
this
is
there
for
a
reason,
and
the
reason
is
this
digital
transformation,
where
you
know
you've
gone
from
organizations
being
able
to
happily
live
in
the
world
of
a
much
more
regulated
economy.
E
Where
say
a
whilst
Fargo
completes
with
a
Bank
of
America
COMPETES
of
the
city
bank,
whereas
now
that
having
to
compete
with
digitally
transformed
startups-
and
this
is
across
every
segment,
every
part
of
the
economy
and
he
drives
a
step
change
in
how
you
manage
service-
and
you
know
it
really
comes
down
to
this
scale
in
complexity
that
is
driven
by
this
massive
upstage
in
you
know
the
number
of
moving
parts
in
modern
applications,
application
of
structures,
much
more
chatty.
Never
data
points
is
measured
in
billions.
E
From
a
while
back,
which
tends
to
120
by
the
way,
the
reason
I'm
an
ex
theoretical
physicist
reason
why
that
number
is
of
interest.
It
actually
is
more
information
that
is
stored
in
the
observable
universe.
To
give
you
an
idea
of
how
bad
it
is,
and
the
complexity
and
in
terms
of
what
people
are
having
to
deal
with
is
usually
significant.
E
So
what
to
do
and
well
you
know,
if
you
look
across
a
lot
of
our
customers,
you
know
what
they're
seeing
is
this
overwhelmed
by
data
and
lack
of
information,
he's
driving
all
kinds
of
consequential
issues
with
the
quality
of
service
assurance,
the
you
know
the
amount
of
data
that
how
deaths
and
s
our
reason
to
deal
with
the
amount
of
incidents
that
are
actually
being
captured
by
monitoring.
How
you
proceed
to
a
resolution
of
an
issue.
The
Australian
service
is
very
siloed
and
very
linear,
and
the
net-net
is
is:
is
it's
horrible
14:8?
E
So
this
is
really
the
appeal
for
the
insertion
of
AI
and
data
science
in
operations,
and
it
kind
of
stems
from
you
know.
Back
in
the
day,
when
things
were
a
lot
more
predictable
and
a
lot
more
low
scale,
you
have
this
kind
of
state
and
measurement
separation
of
the
world
where
you
could
analyze
the
system
that
you
were
monitoring
work
out.
E
What
the
potential
failure
modes
of
that
system
were
monitor
for
the
occurrence,
those
failure
modes
act
and
they
sort
of
continue
to
do
this
in
a
continuous
cycle-
and
you
know
the
the
time
around
if
you
like,
the
orbiter
of
that
left-hand
circle
was
a
lot
faster
than
the
change
cycle
of
the
system,
that
you're
monitoring
and,
in
any
case
the
scale
of
the
systems,
research
that
you
could
take
these
rules
based
approach.
Well,
you
know
in
the
modern
world,
in
the
modern
stack
you
know
you
can't
do
that.
E
It
is
so
complicated
that
there
isn't
the
ability
to
analyze
for
potential
failure
modes.
You
can't
work
them
out.
It
is
too
complicated.
So
you
can't
go
through
this
rules.
Model-Based
approach
to
monitor
for
the
known
failure
modes.
You
really
have
to
combine
state
and
measurement
and
use
the
data
that
you're
getting
from
the
monitored
systems.
E
How
do
we
do
this
MOOC
software?
Our
incarnation
of
this
of
this
inside?
Is
these
really
a
pipeline
of
algorithms?
The
starts
on
the
left
hand
side
with
we
take
anything
we
take
application
logs.
We
take
metrics
from
collect
these
stats.
D
we
take
traps.
We
take
indication
from
ones
like
management
systems.
We
really
don't
care.
E
We
use
a
bunch
of
information,
theoretic
algorithms,
to
suppress
noise,
because
by
the
way,
most
of
that
data
is
junk
and
useless,
and
then
what
we
really
doing
is
we're
looking
in
those
synthesized
flows
and
for
groups
of
South
correlated
events,
and
that
indicate
that
there
is
an
underlying
causal
thing
that
has
occurred,
which
has
caused
the
pattern
of
alerts
to
be
sent
to
us.
We
mine
that
data
for
these
groups
of
alerts.
E
You
know
for
future
occurrences
and
I'm
just
going
to
cause
the
animation
to
finish
this,
a
slightly
more
detailed
view
of
it,
and
so
what
it
really
boils
down
to
is
sort
of
four
pillars,
and
you
know
entropy
is
the
information
theoretic
algorithm
that
we
use
to
try
and
eliminate
the
noise.
This
correlation
is
grouping
these
combinations
of
unsupervised
and
machine
learning
and
supervised
machine
learning.
We
can
take
indications
from
time
to
Paula
gene
text
in
the
data
to
form
these
clusters
of
alerts.
E
We
then
have
a
series
of
information,
theoretic
and
and
supervised
machine
learning,
algorithms
to
look
for
root
cause
inside
of
those
groups
of
alerts.
Some
of
them
are
in
your
nature
of
them.
Some
of
them
are
used
in
terms
of
you
might
not
be
familiar
with
vertex
entropy,
which
is
you
know,
actually
fundamental
research
that
we
that
we
did
and
move
stop
into.
You
know
how
you
can
take
hints
from
interrelationships
of
entities
to
look
at
where
you're
more
likely,
where
the
high
likelihood
is
at
a
service
impacting
instant
or
a
riceball.
E
So
there
are
some
peer-reviewed
journal
articles
in
around
that
and
then
as
solid
AdWords.
We
have
this
and
this
collaborative
war
room,
we're
looking
for
things
like
situation,
similarity,
that's
cluster,
similarity
the
drive,
prompting
you
know
remediation
hints
and
to
accrete
a
a
sort
of
a
living
room.
Look
if
you
like,
and
if
there's
anybody
who
is
british
on
the
call
over
chuckle
and
I,
see
the
little
bottom
right-hand
icon,
which
is
supposed
to
be
a
brain.
E
So
you
know
we
like
to
think
of
ourselves.
The
nails
leader
and
I'll
give
you
a
little
bit,
as
you
know,
the
reason
why
we
made
that
claim-
and
you
know
we've
been
at
this
since
2011-
we
filed
50-plus
patents
being
granted
I.
Think
14
was
the
number
of
Friday,
and
you
know
it's
about
twenty
that
will
grant
this
year.
E
This
will
be
my
third
at
scale
company
in
this
space,
and
we've
got
leaders
from
you
know:
Splunk
Qualis
and
other
businesses
in
there
as
well,
and
and
if
you
look
at
the
the
breadth
of
the
at
the
IP
that
we've
built
I
mean
this
is
just
a
vista
across
places
where
we
have
either
both
father
patterns
all
produce
their
peer-reviewed
paper
and
there
is
generated
IP.
There
is
in
the
product.
So
everything
from
a
lot
of
significance.
E
Timestamp
patents
text
by
these
similarities,
use
of
topology
use
of
entropy
of
graphs
use
the
deep
learning
in
you
know
across
the
correlation
energy
and
feedback
and
root
cause
kind
of
form,
part
of
our
platform
and
and
and
that's
that's
it
for
me
again,
I'll
just
say
few.
You
know
anybody
is
welcome
to
come.
Join
us
on
the
15th
of
May
in
the
Four
Seasons
just
hit
our
website.
It's
pretty
prominent
any
questions
and
thank
you
for
listening.
A
A
D
Okay,
so
my
name
is
Zach
and
I
work
in
the
AI
and
our
excellence,
with
an
under
Marcel,
the
AI,
ops
group
and
I'm.
Currently,
the
lead
for
log
anomaly,
detection
project
and
I
wanted
to
over
and
present
some
some
of
the
things
that
we've
been
building
eternally
the
first.
The
use
case,
though
we're
building
this
log
anomaly
detector,
to
help
our
internal
customer
with
root
cause
analysis.
D
We
built
it
with
unsupervised
machine
learning
to
identify
the
anomalies.
We
found
that
you
know
with
with
machine
learning,
it's
it's
more
of
a.
They
give
you
a
probabilistic
answer
rather
than
a
deterministic
answer
where
probabilistic
is
generally,
it's
saying
that
you
know,
there's
a
percentage
of
being
correct
and
there's
a
possibility
of
having
all
false
positives
and
we'll
talk
a
little
bit
about.
How
are
we
built
in
designs
and
things
to
tackle
those
challenges
so.
D
Our
architecture
we
run
on
kubernetes
and
we
have
basically
the
core
component,
which
is
log
anomaly,
detector,
which
pulls
log
data
from
elasticsearch
trains
and
and
then
produces
two
models.
I'll
talk
more
in
depth
about
the
model
part,
but
for
now,
I
just
want
to
kind
of
give
it
a
high-level
overview
of
the
system
and
then
once
the
models
are
trained,
there's
inference
the
inference
step
also
also
basically
tags
anomalies
that
are
flagged
as
anomalies,
and
we
store
that
in
a
fact
store.
D
We
also
generate
a
prediction
ID,
but
we
can
reference
it
later
and
then
we'll
which
I'll
talk
about
more
in
the
next
slide
with
which
is
about
more
in
depth,
about
the
pack
store.
And
then
we
have
some
monitoring
in
dashboard
tools,
we're
using
Prometheus
in
Ratana
to
get
a
dashboard
to
see
the
feedback
they
were
getting
back
from
the
fax
store,
as
well
as
the
metrics
that
are
being
generated
from
the
model
training.
D
So
we
also,
we
also
in
the
inference
step.
We
also
send
out
messages
to
a
lossy
search
which,
which
triggers
a
alert
to
be
fired
off
and
send
an
email
sent
to
the
user
and
in
that
email
we
embed
a
link,
a
hard
link
where
we
use
that
prediction
ID
that
was
generated
in
the
email.
So
when
that
user
says
ok,
this
is
not
an
anomaly,
and
this
is
false
positive.
They
can
report
that
and
give
us
feedback
on
that
and.
D
D
So
the
the
system
itself
is
basically
a
Python
program
via
live
in
you.
Can
you
can
specify
if
you
just
want
to
train
or
if
you
want
to
train
and
do
the
inference
and
then
in
step
together
you
can
provide
a
config
map
or
a
llamo
to
inject
into
the
program
or
you
can
specify
environment
variables.
What
options
are
supported
and
you
can
either
pull
data
from
elasticsearch
or
you
can
pull
data
from
a
local
file
and
then
the
sink
that
you
would
write
to
would
be
either
elasticsearch
or
local
file.
D
Again
we
may
we're
actually
looking
at
maybe
trying
out
Kafka
next
to
try
to
see
if
we
can
it's
a
more
performance
out
of
it.
Now,
though,
the
facts
store,
so
the
facts
store
came
about
because
of
this
folks
were
we're
getting
basically
predictions
that
weren't
necessarily
right,
so
we
needed
human
feedback
to
verify.
Is
this
really
an
anomaly
or
not,
and
I'll
talk
a
little
bit
more
about
the
Assam
model
self?
D
So,
just
just
a
I
can't
take
the
credit
for
this
whole
system
itself,
as
I
had
a
great
data
scientist
on
the
team,
Michael
Clifford
and
I
had
an
awesome
architect,
varcek
Abilene,
who
also
were
involved
in
the
design
of
the
the
model,
training
of
the
somme
model
and
the
and
the
overall
system
I've
added
the
PAC
store,
which
was
an
additional
component.
This
system,
so
I'll
talk
a
little
bit
about
the
model
training
part.
D
So
there
are
two
models:
there
is
a
word
to
beck
model
and
the
word
feedback
model
does
the
machine
learning
and
basically
what
it
does.
It
lends
the
probability
of
words
occurring
with
each
other.
It's
we're
using
it
in
a
very
kind
of
unconventional
way,
but
basically
it's
kind
of
like
a
pre-processing
step,
basically
verify
basically
convert
the
raw
log
messages
into
vectorized
representations
and,
and
once
we're
done
with
that
in
the
training
step.
D
The
the
next
thing
that
happens
within
the
training
step
is
we
generate
another
model
called
Sol
model
self-organizing
map,
and
what
that
does.
It
compares
the
vectorized
representation
of
the
words
the
inference
step
and
basically
it
would
basically
friend
will
train
the
model
and
then
you
would
measure
distance
of
each
inference
message
to
the
different
nodes
on
a
map
and
then
determine
how,
like
or
unlike
they
are
to
the
nodes
on
the
map.
D
Basically,
it
generates
a
real
number.
If
it's
normalized,
it
would
be
between
0
&
1.
We
have
a
threshold
value,
it's
configurable,
but
basically,
let's
say
consider
if
it's
3
3
times
the
standard
deviation
plus
the
mean
that's
an
example,
our
calculating
the
threshold
value,
and
then
this
is
a
standard
way
to
like
determine
it's
outside
the
bounds
of
normal
behavior.
If
it's
outside
of
the
bound.
Nobody
who
like
this
is
an
anomaly
and
anything
that
like
will
make
sure
we
report
it.
D
D
All
the
next
thing
I'm
going
to
show
you
is
a
live
demo,
because
I
think
it's
flies
are
cool,
but
what's
what's
cooler
than
a
live,
so
I've
prepared
an
environment
here
for
us,
oh
I've
deployed,
so
the
database
that's
being
used,
we're
using
stick
welcoming
and
the
database
we're
using
is
my
sleepwalk.
Then
we
have
profound
and
the
facts
store,
which
is
a
classic
application
which
connects
using
mice,
connects
using
tickle
alchemy,
and
you
can
you
can?
D
D
The
data.
Sciences
can
also
look
at
a
graph
on
a
dashboard,
a
little
bit
easier
to
look
at
and
see
how
how
we're
performing
in
terms
of
the
the
system
itself
and
also
you
can
kind
of
view
some
of
the
log
messages
that
were
recorded,
the
prediction
ID
the
score
and
then
the
the
internal
customer
can
give
us
feedback
over
here.
So
I'm
gonna
go
and,
and
and
let's
say,
I
got
this
email.
D
Would
use
query
string,
but
then
it
would
auto
fill
these
these
values
here
basically
I
would
say.
Is
this
an
anomaly
false,
and
why
is
this
not
an
anomaly,
because
it
is
just
wrong
and
once
I
submit
that
that
feedback
goes
in
it's
now
in
the
dashboard?
We
can
see
this
anomaly
wrong
and
and
that's
it
and
pretty
much
we
iterate
through
this
and
did
you
develop
him
and
that's
pretty
much
it
for
demo?
D
So
we
have
this,
we
have
this
internal
machine
learning
platform
where
we
have
OpenShift
and
we
have
been
more
components
like
Alice
research
Cubana
as
well
as
logstash
in
Africa,
and
some
systems
are
deployed
up
there
and
their
logs
are
streaming
through
Kafka
and
then
into
elasticsearch.
Some
customers
would
need
would
meet.
The
would
want
would
have
a
requirement,
the
machine
learning.
Then
we
deployed
this
service
as
service
that
they
would
utilize
eternally.
B
Yeah,
so
the
internal
customer,
Diana
I,
think
that
was
also
your
question
is
a
team
within
red
head
that
is
maintaining
build
pipelines
for
containers
and
they
are
using
this
tooling
to
identify
anomalies
in
locks
of
their
deployed
system
and
they
are
running
on
an
internal
deployed
openshift
instance.
So,
basically,
you
could
deploy
the
same
setup
that
Zack
just
highlighted
to
any
other
team.
That
also
runs
on
top
of
openshift
and
I.
Think,
that's
that's.
Why
we're
trying
to
go
from
a
prototype?
B
A
Are
there
any
other
questions,
thoughts
from
folks
on
the
call
or
for
Zach
or
for
Brian
or
for
Phil?
Well,
we
have
them
here
and
if
there
are
other
topics
you
folks
would
like
to
hear
about,
we
will
be
meeting
again
in
another
month,
post
Red,
Hat
summit
and
COO
con,
so
there'll
be
lots
of
busyness
in
between.
So
if
you've
got
topics
that
you
learn
about
there,
that
you
want
to
talk
about
afterwards,
just
ping
Marcel
or
myself
and
we'll
get
them
added
to
the
hinge
of
the
next
agenda.
Just
one.
E
B
Sure-
and
it's
all
going
to
happen
on
that
very
github
repo,
where
we
try
to
post
the
data
sets
and
actually
that's
a
good
cue
for
maybe
one
of
the
next
topics.
So
one
very
dear
topic
to
my
heart
is
that
that
there's
a
yeah
there's
like
a
lack
of
public
datasets,
where
you
could
do
some
research
on
operational
data.
B
Data
for
metrics,
like
you,
would
see
it
in
the
usual
machine
learning
space
where
you
have
kaggle
and
all
sorts
of
competitions,
so
I'm
I'm
really
eager
to
build
out
the
collaboration
amongst
folks
here
on
the
call
to
sort
of
curate
datasets
and
strive
interest,
also
from
academia
and
to
apply
their
magic
tools
and
wisdom
to
those
datasets
that
that
we
can
bring
to
the
table
as
operational
folks.
So
yes,
Phil
I'm,
really
keen
to
collaborate
on
that.
E
Icon
dataset,
which
is
pretty
well
known,
the
the
artifically,
have
just
created
this
new
way
of
curating
these
things,
so
they
get
a
DIY
numbers.
Well,
so
it's
very
good
from
a
academic
research
perspective.
So,
let's
connect
up
afterwards.
There
are
issues
around
an
anima
zation
that
are
well
understood
and
well
known,
but
I'm
sure
between
you
know
us
customers,
your
customers.
We
can
maybe
work
something
out.
B
Yeah,
awesome
and
I
actually
have
a
question
to
your
presentation
Phil.
So
it
seems
that
you're
said
I
was
pretty
much
agnostic
to
what
kind
of
operating
systems
and
what
kind
of
environment
you're
running
in
or
do
you
have
some
sort
of
preference,
whether
it
be
a
virtual
machine
environment
or
a
containerized
environment?
I'm?
Not
talking
about
where
you
run
your
software
stack
on
the
the
Mook
soft
stuff,
but
the
environment
that
you
are
monitoring
and
applying
reactions
to.
E
What
I
will
say
is
obviously
you
know
what
tends
to
play.
Is
you
know
where
we
go
where
we
wanted,
and
you
know
today,
an
awful
lot
of
that
is
is
very
much
sort
of
cloud
focused
and
in
a
new
stack
focused
that
we
tend
to
work
a
lot
with
you
know:
managing
the
hypervisor
managing
docker,
mounting
kubernetes,
managing
the
modern
compute
stacks,
or
the
very
different
elevation
tend
to
find
in
it.
B
Great
yeah
and
I
actually
would
love
to
see
a
hands-on
demo
at
some
at
some
point.
Maybe
we
can
connect
to
that
and
maybe
even
to
a
kubernetes
environment,
because
I
think
that's
what
most
of
the
folks
here
on
the
call
are
interested
in
and
most
of
the
people
that
will
see
it
not
life
but
posted
on
the
on
the
websites.
I
think
we
have
more
people
viewing
these
recordings
afterwards,
then
actually
live
I,
guess
and
as
they're
posted
in
the
context
of
the
openshift
Commons
I.
Think
that's
where
the
interest
of
most
viewers.
A
E
D
E
A
A
So
we
do
have
a
bunch
of
upcoming
events
and
I
know
they'll,
be
at
least
a
lot
of
Red
Hatters
at
them,
and
hopefully
some
other
folks
from
different
AI
sig
meetings.
Odst
is
coming
up
this
week
and
so
there'll
be
a
lot
of
people
in
Boston
and
then
right
after
that
is
Red.
Hat
summit
and
I
did
dig
up
the
time
it's
2
p.m.
on
the
8th.
A
If
you
would
like
to
meet
everybody
in
person,
you
can
find
out
what
Marcel
looks
like
in
person
and
myself
and
come
hang
out
at
the
cig,
meetup
in
the
beat
and
greet
booth
in
community
central
on
the
expo
floor
and
as
Phil
I
think
it
was
that
mentioned
it.
The
AI
ops
exchange
is
happening
in
San
Francisco
and
there's
a
link
in
there
for
that.
A
Anyone
else
on
this
meeting
that
has
topics
that
they
project
they're,
working
on,
or
initiatives
they're
working
on
inside
of
Red,
Hat
or
side
of
Red
Hat.
Please
do
propose
them
and
if
any
of
you
want
to
work,
the
booth
that
Red
Hat
summit
at
the
expo
hall,
let
me
know
plenty
of
space
and
hours
still
open
the
operators
or
the
Commons
kubernetes
one
or
the
meet-and-greet
space.
So
let
me
know
I'm
always
looking
for
people
and
it's
a
great
way.
A
A
Well,
thanks
again
and
for
all
of
you
for
coming
we'll
post
the
video
of
this
and
the
notes
on
the
Google
Group
if
you
haven't
joined
that
yet
please
do.
The
link
is
just
AI
ops
on
Google
Groups
I'll,
throw
that
in
the
notes
to
hear
if
you
haven't
seen
that
if
you
haven't
joined
that
that's
where
we're
using
it's
a
very
public
forum.
So
just
be
aware
of
that
and
we're
just
happy
that
you
came
and
listened
and
really
glad
that
both
profit
store
and
massage
and
Zach
jarred.