►
Description
Presentation: November 5th, 2021
Moderator: Florence Hudson
Presenters: Jay Yang, Richard Biever, and Inna Kouper
Bios:
Jay Yang: S. Jay Yang is currently a Professor in the Department of Computer Engineering and Director of Global Outreach for Global Cybersecurity Institute at Rochester Institute of Technology. Supported by NSF, NSA, IARPA, DARPA, AFRL, ONR, and ARL, his research team has developed several pioneering machine learning, attack modeling, and simulation systems to enhance cyber situational awareness and enable anticipatory cyber defense. His earlier works included FuSIA, VTAC, ViSAw, F-VLMM, CASCADES, CAPTURE, and attack obfuscation modeling. More recently, his team has developed ASSERT, HeAT-PATRL, and CLEAR-ROAD. He was a NSF Trusted CI Open Science Fellow in 2019 and TTP Fellow in 2020, and received IEEE Region 1 Outstanding Teaching in an IEEE Area of Interest Award – for outstanding leadership and contributions to cybersecurity and computer engineering education
Richard Biever: Richard Biever is Duke University's chief information security officer and a senior director in the Office of Information Technology. He has served in previous roles with the Georgia Institute of Technology's Office of Information Technology and Hewlett Packard. Under Richard’s leadership, Duke University has developed a strong, collaborative office working with university departments, research faculty, other higher education partners to develop and enable effective cybersecurity defenses at Duke. STINGAR, a Duke-developed and NSF-funded threat intelligence and sharing initiative, is currently being deployed beyond Duke with a number of higher education partners. More information on STINGAR may be found at https://stingar.security.duke.edu.
Inna Kouper: Inna Kouper is Director of Researcher Engagement for the ResearchSOC and a Research Scientist at the Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington. She studies emerging technologies and data practices and works on projects that advance open science and data sharing. She is involved in several large-scale projects funded by the National Science Foundation and the Institute of Museum and Library Services to study communities of data producers and consumers, promote data stewardship and develop innovative methods of data collection, analysis, and preservation. Kouper has undergraduate and masters degrees in information systems, a PhD in sociology and a PhD in information science.
A
So
today
we
have
a
really
cool
panel
to
talk
about
cyber
security
and
big
data,
science
and
research
and
education,
and
so
actually
you've
heard
from
our
three
speakers
today.
Ina
will
be
speaking
as
well
as
jay
yang
and
richard
beaver,
and
so
perhaps
we
can
do.
Ladies
first
and
ina,
would
you
like
to
present
first?
Is
that
okay
tell
us
a
little
bit
about
where
you're
coming
from
you
know
the
work
you
do
with
the
research
sock
and
then
how
this
might
apply
to
the
community.
A
B
So
it's
it's
a
big
project
and
it
has
multiple
objectives
and
the
main
one
is
to
provide
operational
security
services
for
nsf
scientific
cyber
infrastructure,
but
also
part
of
that
and
richard
beaver
is
actually
a
collaborator
on
the
project
as
well,
so
we'll
work
together,
yeah.
So
the
two
other
objectives
is
to
build
the
threat,
intelligence
network
and
communities
of
practice
and
outreach
to
higher
education
professionals
in
information,
security
and
cyber
security.
But
the
part
that
I
am
involved
in-
and
I
am
actually
the
director
for
researcher
engagement.
B
This
is
focusing
on
enabling
cyber
security
research
and
particularly
doing
that
through
providing
access
to
operational
security
data.
So
this
is
what
we're
trying
to
do
and
how
we're
doing
that.
We're
trying
to
learn
about
researchers
needs
for
data
and
tools,
researchers
who
do
cyber
security
research,
and
through
doing
that,
we
are
building
a
secure
environment
to
access
the
data
and,
as
you
know,
operational
security
data
has
a
lot
of
sensitivities
and
pretty
much
no
one
likes
to
share
this
type
of
data.
B
So
this
is
part
of
my
goals
is
to
you
know,
participate
in
building
this
community
and
bringing
together
security,
analysts,
researchers
and
other
people
who
are
interested
in
sharing
data
and
working
with
data
with
this
data,
but
also
building
tools
and
kind
of
you
know
growing.
This
community,
so
the
secure
environment
that
we're
building
and
I
will
not
go
into
the
details
of
the
architecture,
but
this
is
basically
what
we're
doing
we're,
calling
it
an
enclave
and
we're
we've
developed
the
prototype.
B
Data
is
coming
in
and
can
be
organized
in
a
specific
way,
but
then,
when
users
are
accessing
it,
they
cannot
take
it
out
right,
the
all
the
ports
and
and
other
types
of
configuration.
It's
it
protects
data
from
download.
So
then,
to
enable
you
know
the
research
we
need
the
middle
part.
You
know
we
need
a
flexible
environment
and
we're
that's.
Why
we're
learning
about
researchers
needs?
You
know
what
kind
of
tools
they
will
be
working
with,
how
they
will
be
processing.
B
This
type
of
data,
et
cetera,
we're
also
trying
to
provide
access
to
various
types
of
data
and
masking
the
most
secure
parts.
So,
even
though
we
just
think
you
secure
enclave,
our
security
team
still
thinks
that
this
is
not
allowable
to
share
raw
data.
So
it's
still,
you
know
masking
in
and
sanitization
is
still
a
big
component
of
this
effort,
but
then
we
think
with
some
tools
that
we've
developed.
B
Some
data,
like
vulnerability
scans,
for
example,
could
be
shared
outside
of
the
enclave,
but
some
data
will
still
be
staying
within
the
enclave
strictly,
so
what
kind
of
data
is
staying
in
the
enclave?
So
far?
It
is
zeke
network
monitoring
data,
and
you
see
that
you
know.
We've
focused
on
sort
of
these
five
fields
that
that
were
you
know,
because
z,
clock
data.
B
It
has
tons
of
information,
and
it's
interesting
that
once
we
started
working
with
the
prototype,
we
haven't
really
considered
the
amount
of
resources
that
it
would
take
to
provide
access
to
this
data.
So
we
ran
out
of
resources
pretty
quickly
particularly
storage
and
compute.
So
then
we
had
to
limit
the
amount
of
data
that
we're
working
with.
For
prototyping
purposes,
and
so
currently
it's
the
destination,
ip
and
port
network
transport
protocol
and
the
source
ipn
port.
So
this
is
a
limited
amount
of
data.
B
I
I
hope
it
still
is
useful
for
research
purposes,
but
then
you
know
so
the
access
that
we
provide
to
this
data
is
currently
done
through
kibana.
That
allows
to
work
with
data
in
various
ways.
Also
do
some
interesting
visualizations.
So
this
is
the
dashboard
that
you
can
see,
and
you
know
it's
it's
interesting.
I
think
the
the
timing
of
this
panel
is
very
good
because
we
just
basically
completed
the
full
pipeline
and
the
prototype
is
working,
and
you
know
the
dashboard
that
you
see.
B
This
is
the
actual
data
flowing
in
which
is
I'm
very
proud
of
that
we
were
able
to
complete
that
and,
as
you
can
see,
you
know
the
flow
like
there
are
some
gaps,
for
example
for
particular
days,
so
this
is
some
troubleshooting
that
will
be
happening
and
we'll
see
you
know
why
what
is
happening
and
why?
But
even
with
this
limited
data,
like
I
said,
there's
some
interesting
opportunities.
B
You
know
the
ways
you
can
analyze
it
and
look
at
the
types
of
destination,
ips
and
sources,
and
then
the
types
of
connections
that
are
being
made
and
also
you
know
the
protocols
and
what
kind
of
traffic
is
happening.
So
we're
hoping
that
in
the
future
we
will
be
able
to
expand
what
we
call
a
data
catalog
the
amount
and
types
of
data
that
is
available.
B
Because
then
you
encounter
like,
I
said,
a
lot
of
storage
issues,
legal
issues,
policy
issues
so
we're
in
addition
to
the
technical
components,
we're
also
developing
agreements.
You
know
and
access
protocols
and
everything
like
the
whole
infrastructure
around
data
sharing
that
I
think
there
are
a
lot
of
kind
of
thorny
questions
to
talk
about.
B
A
B
A
Yeah
this
is
excellent.
Thank
you.
I
think
what
we'll
do
is,
let's
have
all
three
presentations,
so
we
have
a
feel
for
you
know
what
the
whole
the
landscape
looks
like
and
then
we'll
go
to
q
a
and
the
panel
discussion
that'll
be
great
and
so
richard,
why
don't
we
have
you
go
next?
Are
you
ready?
I
see
you.
C
D
A
C
So
those
are
kind
of
the
the
four
goals
we
do
use
honey
pots
on
the
collection
side.
I
do
want
to
know
on
this
particular
slide.
We
use
low
and
high
interaction,
honey
pots
for
those
that
aren't
aware.
Honeypots
are
vulnerable
systems
that
we
just
put
out
on
the
network.
They're
designed
to
be
compromised.
C
High
interaction
means
that
we
allow
an
attacker
to
actually
connect
to
the
honeypot.
They
can
run
some
commands.
They
can
do
some
different
things,
etc.
Before
we
block
them.
Low
interaction
is
a
type
where,
once
they
touch
the
honey
pot,
we
block
them
right.
We
just
assume
they're
malicious
and
we
don't
capture
any
additional
data.
C
So
the
biggest
thing
from
our
perspective
operationally,
you
know
why
do
people
want
to
participate?
We
actually
started
this
approach
back
in
2014,
and
that
was
really
starting
with
just
traditional
log
data
that
was
saying
nsm
and
which
was
helpful.
But
what
we
were
trying
to
figure
out
is:
how
can
we
generate
our
own
threat?
Intelligence
right?
You
know
with
it,
because
we,
when
we
talk
to
vendors,
the
costs
involved
in
buying
threat,
feeds,
etc,
was
so
high.
C
C
Is
we
really
kind
of
went
live
with
this?
We
saw
an
85
reduction
in
in
our
intrusion
prevention,
our
commercial
intrusion
prevention
alerts.
That
doesn't
mean
that
that
you
know
you
stop
buying
intrusion
prevention
devices.
What
that
means
is
it
took
all
of
kind
of
the
the
the
brute
force,
the
the
scanning
attacks.
It
took
all
that
off
the
network
and
we
were
able
to
see
you
know
much
more
interesting
kind
of
attack
showing
up
in
ips
and
we
were
sharing
about
17
000,
unique
ip
addresses
at
that
point.
C
C
C
That
ena
mentioned,
and
we've
worked
with
at
least
two
researchers
on
the
data
that
we
have
and
the
latest
researcher
we
worked
with
at
duke
they
were
taking
the
honeypot
data
and
combining
it
with
z,
data
to
get
kind
of
that
full
view
of
of
the
network
traffic
and
what
was
happening.
So
I'm
not
going
to
spend
a
lot
of
time
on
this.
I
I
did
want
to
point
out
a
couple
data
points
here,
really
quick.
C
So
we
can
move
on
to
jay
and
his
his
presentation,
but
the
stinger
threat
feeds
we
find
are
highly
relevant
over
40
percent
of
all
the
indicators
of
compromise
are
new
or
unique,
or
never
seen
before
that.
That
speaks
volumes
to
me
from
an
operational
standpoint
of
how
quickly
attackers
are
cycling
through
infrastructure
to
target
organizations.
C
C
So
you
start
seeing
a
little
bit
of
variation
there,
which
we
we
thought
was
fascinating
about
20
of
all
ip
addresses
are
seen
by
at
least
two
or
more
partners
with
an
average
time
between
attacks
of
you
know
within
24
hours.
That's
a
positive
thing
to
me.
That
means,
if
we're
sharing
the
data
around
as
soon
as
somebody
sees
something
bad,
other
people
are
protected
and
then
from
a
rich
data
perspective.
Much
like
nina
noted.
C
The
thing
that
we're
really
interested
in
is
getting
beyond
just
the
the
metadata
associated
with
network
connections
and
into
that
deeper
level.
So,
if
you're
running
kind
of
the
high
interaction
honeypots,
we
start
you
know
being
able
to
do
some.
As
you
can
tell
here.
It's
a
little
on
the
rudimentary
side,
but
we
start
getting
more
information
that
we
can
do
additional
analysis
on.
C
So
in
this
case
we
start
seeing
the
types
of
usernames
and
passwords,
so
the
attackers
are
just
automatically
trying
give
judication,
perhaps
of
of
what
they're
interested
in
and
then
we
also
capture
quite
a
bit
of
the
the
commands
that
they're
running
so
again,
it
kind
of
starts
speaking
to
building
a
profile.
If
you
will
types
of
groups
and
the
commands
that
people
are
running.
So
we
we're
pretty
excited
about
this.
C
We
we
think
it's
got
some
value
here,
potentially
and
we'd
love
to
continue
working
with
the
community
on
it
if
you're
interested.
Obviously
you,
you
know
more
than
happy
to
talk
to
you
and
you
can
also
visit
our
website
and
and
happy
to
to
help
out
where
we
can.
So
I
will
stop
there.
So
jay
can
go.
A
That's
wonderful,
thank
you
richard
and
jay
yang,
since
we
have
multiple
j's.
If
you'd
like
to
start
sharing
your
screen
and
richard
it's
been
so
cool
watching
you
grow
up,
you
know
not
you
I
mean
you
know
stinger,
you
know
from
when
you
first
got
the
award
and
to
how
much
impact
you're
having
and
it's
wonderful
thank
you
for
how
you
want
to
continue
your
broader
impact
in
working
with
more
people
to
add
more
value,
gather,
more
data
and
add
more
value.
That's
wonderful!
So
thank
you
for
that
too.
E
Thank
you,
florence,
so
I'm
not
going
to
present
the
whole
just
just
this.
I
think
it's
easier
for
me
to
to
see
all
the
people's
name,
and
here
there
very
nice
to
hear
ina
and
richard's
presentation
and
questions
for
them
already
all
right.
So
I'm
a
researcher
I
work
at
rit
and
I've
been
working
on
ai
ml
for
kind
of
understanding,
predicting
modeling
cyber
attacks
for
quite
a
long
time,
15
years.
E
E
Most
of
these
tools
really
give
you
a
lot
of
statistical
summary
of
like
how
many
ipa
addresses
come
from,
where
I
think
that
you
already
just
saw
richer,
showing
that
I
think
asan
that
that's
one
of
my
questions
I'll
address
a
little
bit
later,
so
basically
see
that
what
kind
of
threat
from
coming
from
where
and
to
what
kind
of
servers
really
like
numbers
like
counts.
That's
the
the
key
things
to
look
at
the
the
rich
data
analysis
really
relies
on
people
right,
the
human
expertise,
that's
the
solution
and
that's
the
problem.
E
So
today
that
very
experienced
gray,
analysts
will
take
a
lot
of
time
really
to
get
a
very
rich
analysis.
What
kind
of
threat
what
kind
of
cyber
attack
campaign
is
going
on
my
network,
I
really
like
what
richard
presenting
kind
of
benchmarking
between
one
university
against
all
the
data
fees.
I
think
that's
very
exciting,
because
I
also
was
trying
to
do
that
as
well.
E
The
bottom
line
is
that,
if
we
want
to
do
more
than
just
summarizing
percentage
of
ip
address
percentage
of
core
numbers
like
summarizing
more
than
basic
statistics,
we
need
some
some
help
right.
The
human
analyst,
the
human
cognition
is
not
it's
just
not
there
to
process
millions
of
alerts,
millions
of
unlocked
entries
every
day
every
hour.
E
E
They
like
acronyms
right,
so
I'm
going
to
briefly
talk
about
what
they
are
actually
just
add
one
slide,
because
I
think
that's
very
relevant
to
what
richard
said
about
rich
data
analysis,
so
I'm
just
going
to
quickly
see
that
some
of
the
example
results,
processing
real
alerts
from
either
real
socks,
or
you
know,
like
I
said,
exercise
kind
of
data,
so
the
first
one.
This
is
actually
processing
data
from
a
real
stock
operation.
E
E
That
actually
seems
like
a
real
situation,
probably
due
to
a
misconfiguration
something
that
continues
going
on
for
the
network.
Meanwhile,
there
you
see
that
orange
was
in
the
order
of
10
and
100.
The
y-axis
is
a
volume
in
the
logarithmic
scale,
so
before
august
18
2020
it
was
at
10
100
level
that
level
of
scale
and
then
certainly
jumped
to
ten
thousand
hundred
thousand
kind
of
level.
The
the
ms
sequel,
we
see
a
lot
of
coming
in
control
for
throughout
the
sequel.
At
that
time
frame.
E
I
do
not
have
reason
behind
I'm,
not
the
operator.
I
cannot
tell
you
why
that's
the
case,
my
hypothesis,
is
that
there's
a
time
that
school
starts,
the
fall
semester
starts,
so
there
may
be
something
going
on
with
a
different
configuration
or
users,
certainly
swam
in
to
create
a
lot
of
traffic
in
that
area.
We
do
not
expect
that
all
of
these
are
malicious,
so
this
could
be
misconfiguration,
so
this
may
be
due
to
users
again.
E
E
Learning
approach
that
we
developed
called
patrol
patrol
stand
for
pseudo-active
transfer
learning,
and
the
idea
here
is
that
so
richard
talked
about
varied
reach,
like
data
analytics
in
terms
of
the
signatures
in
the
alerts
or
some
of
the
logic
sentences,
you
have
cvds,
you
need
to
kind
of
really
analyze.
What's
what
was
actually
happening
for
given
log
entry,
so
we
have
developed
a
method
to
help
analysts
to
interpret
those
cryptic
analytics
to
understand
the
potential
intended
consequence
of
the
anniversary.
E
So
we've
done
that
I
will
demonstrate,
with
our
more
theoretical
result,
that
we
can
translate
cericada
alert
signatures
into
what
we
call
a
10
stage
by
reference
to
miter's
attack
framework
right
using
that
can
do
even
more
so
this
is
the
slide
that
I
was
showing
you
earlier.
I
need
to
go
into
presentation
mode,
so
you
can
see
the
animation
so
idea
here
that
we
give
it
a
lot
of
data.
This
actually
comes
from
collegiate
pencil
testing
competition,
so
we
collect
the
data
we're
looking
at
for
particular
iocs.
E
What
are
the
previous
steps
that
lead
to
the
ioc?
So
we
used
the
previous
that
patrol
thing
and
we
interpret
the
data
we're
able
to
see
how
the
prior
steps
from
recon
scanning
data
exploration
all
the
way
to
have
a
code
red
alert,
they're
happening
at
the
end,
so
about
expanding
over
a
few
hours
during
the
competition
we
can
reconstruct
the
attack
campaign
for
that
purpose,
all
right,
so
the
last
one
is
gordon,
was
doing.
E
The
previous
work
is
done
by
steve,
and
this
was
done
by
gordon,
so
gordon
has
developed
patent
mining
algorithm
again
to
help
analysts
interpreting
what's
happening,
so
he
processed
both
the
real
world
data
and
the
cpdc
yeah.
What
he
has
found
is
that
if
you
look
at
the
code
red
when
people
execute
correct
what
might
come
before
and
after
with
a
pattern
right,
be
the
odds,
be
the
statistical
ask
that
this
is
happening.
E
We
have
found
that
there
are
two
ways
to
see
the
correct
one
is
with
alerts
before
and
after
in
milliseconds,
you
can
see
the
top
portion
of
it
is
in
mini
seconds,
and
a
button
is
a
different
way.
Code
red
is
being
seen
and
the
the
attack
actions
before
and
after
are
more
in
the
order
of
30
seconds.
It's
older,
some
magnitude
differences.
So
we
can
conclude
that
hey.
E
There
are
two
very
common
ways:
people
use
code
red
and
then
provide
that
insight
that
through
intelligence
to
endless,
maybe
in
the
future,
when
we
see
this
kind
of
root
access
over
here
very
soon
we're
going
to
see
code
red.
Well,
if
you
see
this,
then
it's
going
to
be
another
30
seconds,
not
that
many
second
30
seconds.
This
is
very
different
for
human,
but
probably
very
different
for
machines
right.
A
E
Around
your
voice,
right
here,
okay,
so
yeah,
so
thank
you
for
listening.
I
think
that
the
main
point
I
want
to
convey
is
that
researcher
does
get
a
lot
of
benefit
to
understand
the
true
challenges
in
data
here:
we're
presenting
the
results
using
this
unsupervised,
non-label
data
trying
to
gain
insights
on
how
we
can
help
analysts
really
looking
at
the
rich
data
kind
of
analytics
right.
It's
not
just
summarizing
ib
address
it's
not
just
about
the
poor
numbers,
but
also
a
little
bit
more
about
what
kind
of
attacks
are
happening.
E
A
This
was
excellent,
you
you're
all
doing
such
valuable
work
and
very
impactful
and
very
complementary,
so
great
job.
Thank
you
all
for
presenting,
and
so
I
always
have
questions,
but
I
don't
want
to
go
first.
I
want
the
community
to
ask
questions.
So
is
there
anyone
who
has
a
question
or
a
comment
or
a
different
point
of
view,
we're
very
open
and
inclusive
here
when
it
comes
to
thinking
christine?
I
think
you
have
your
hand
up.
F
Yeah,
I've
got
maybe
one
and
a
half
questions
we'll
see
so
back
on
stinger
and
maybe-
and
I
think
this
probably
will
expose
more
of
my
outdated
understanding
about
the
work
of
the
security
teams.
But
when
you're
looking
at
you
know
identifying
different
ip
addresses,
how
do
you
sift
out
or
does
it
matter
people
are
doing
ip
spoofing,
like
you
know,
is
it
help
if
you're
blocking
a
whole
bunch
of
people
who
it's
that's,
not
really
where
they
are
anyway?
Tell
me
more.
C
Now
from
an
ip
spoofing
perspective,
you
know
sure,
but
we
we
also
work
pretty
closely
with
the
networking
team
to
enable
some
of
the
anti-spoofing
controls
that
are
on
the
external
routers
things
like
that,
so
hopefully,
by
the
time
it
gets
to
the
honeypot,
that's
not
an
issue.
I
will
tell
you
I
haven't
seen
a
we
haven't
had
a
false,
positive
okay,
which
is,
which
is
a
good
thing.
C
We
used
to
have
false
positives,
all
the
time
with
ssh
connections
coming
through
the
ips
and
b,
the
ip
spoofing,
isn't
what
I've,
what
we've
run
into?
We
have
run
into
the
situation
occasionally
where
google
or
microsoft
just
you
know,
slaps
a
new
subnet
in
for
a
little
while
and
we've
had
to
go
adjust
for
that.
F
Well,
and
specifically,
I'm
wondering
if
you're
able
to
work
across
organizational
barriers.
You
know
our
main
security
team
is
in
central
I.t,
the
san
diego
computer
center.
We
collaborate
with
them
from
time
to
time,
but
it
can
be
very
difficult
to,
even
in
the
same,
at
the
same
university
to
work
together.
E
Right,
I
think
that
richard
probably
have
a
lot
of
insights
on
that,
so
I
can
tell
you
that
I
share
echo
richard's
comment
in
his
presentation
about
they're
looking
at
the
threat
intelligence
from
cross
organization
and
comparing
that
against
that,
and
that
chart
is
extreme
for
me,
I'm
being
looking
at
how
we
can
get
that
kind
of
data
to
do
it,
how
to
overcome
that
christine.
I
don't
know
the
answer
to
that,
but
I
do
have
a
vision
right.
E
My
vision
is
that
if
we
can
somehow
find
a
way
to
summarize
the
data
in
a
way
individual
organization
or
individual
sub-group
organization
doesn't
feel
as
sensitive
right.
So
if
we're
summarizing
a
way,
so,
for
example,
richard
you
mentioned
about-
I
remember
correctly-
vietnam
and
russia
versus
china.
E
If
that
kind
of
information
is
not
a
sensitive
people
are
feeling.
Okay
with
that.
This
is
great,
and
likewise
I
just
share
about,
for
example,
comment
and
control
through
ms
sql
component
control
through
domain
controllers.
Is
there
something
that
organization
will
be
concerned?
What,
if
I
share
it,
you
know
where
this
aggregate
right
so
out
of
20
organizations.
Work
with
these
are
top
five
and
they're
aggregates,
it's
not
necessary
from
one
or
two
organizations.
E
C
Yeah,
I
you
know,
because
you
raised
two
really
good
points
here,
both
of
you.
The
the
first
is.
Let
me
take
j
the
point
you
just
made
first.
So
so
my
philosophy
is,
I
don't
care
who
knows
what
I
p
addresses?
What
domain
names
are
attacking
duke?
I
I
don't
care,
I
don't
care
if
they
know
what
ports
they're
attacking
over
right.
I
do
care.
C
C
We
are
not
doing
a
good
enough
job
sharing
the
information
like
jay
just
described.
We
need
to
do
more
of
that
and
we
need
to
provide
here,
the
iocs
and
the
ttps
that
we
have
seen
hit
us.
If
you
don't
want
attribution
to
your
organization
great,
I
totally
understand
that,
but
there
is
benefit
to
sharing
that
information.
So
that's
one
comment
and
as
far
as
a
data
analyst
I
I
will
tell
you
so
I
was
somewhat
lucky
in
this
regard
twice.
The
first
is
we.
C
We
bought
a
log
analysis
system
about
seven.
Eight
years
ago
we
bought
splunk
right
around
that
time.
I
hired
a
new
security
engineer
in
the
past
seven
years.
I
can't
get
him
out
of
splunk
right.
He
he
just
he
lives
in
splunk.
He
loves
it
and
he's
built
a
lot
of
our
operational
dashboards.
But
these
are
point
in
time
dashboards.
C
We
built
a
data
analytics
team
inside
of
central
I.t
with
data
scientists,
data
analyst
data
engineers
and
I
pay
for
part
of
a
person.
Actually,
she
works
on
the
rsoc
project
on
gog
and
deep
core
and
she
lives
in
both
worlds,
and
so
she
and
eric
are
our
splunk
guy.
They
can
talk
the
same
language
and
she's
doing
these
deeper
level
analyses
the
data
that
we
showed
here.
That's
a
result
of
her
work
and
that's
just
kind
of
scratching
the
surface.
C
F
B
So
that's
interesting.
Just
to
add
to
that
you
know
the
point
about
successful
models.
I
think
there's
three
sort
of
related
points.
One
is
in
sharing
data,
the
understanding
that
the
data
will
be
shared
needs
to
be
at
the
beginning
of
the
conversation
and
not
an
afterthought,
because
this
is
what
we
saw.
You
know
if
you,
if
you
start
with
that,
then
you
can
work
with
organizations
even
if
they're
very
uncomfortable,
sharing
their
data
right.
There
is
a
process
and
then
you
know
agreements
and
things
like
that.
B
If
it's
an
afterthought,
it's
it's
very
hard
to
to
do
that.
But
then
so
you
know
then
the
second
point
sort
of
related
to
that
is
to
think
through
the
agreements
and
and
being
transparent
right.
That's
that's
another
part
that
you
know
everything
that
goes
into
documentation,
both
parties
kind
of
think
about
that,
because
I
think
the
security
folks
and
the
you
know
data
scientists
on
the
research
side.
B
They
have
different
objectives
right,
so
aligning
aligning
those
objectives
is
another
component
which
doesn't
happen
if
you
don't
think
about
it
from
the
beginning,
but
another
aspect
of
it.
I
think
in
the
end
partially,
what
we're
doing
you
know
it's
bringing
students
in
there
and
not
sort
of
establish
data
scientists.
Again
it's
it's
harder
to
start
with
when
people
already
have
their
frameworks
and-
and
you
know,
set
minds,
but
the
students
who
are
hopefully
more
flexible
and
and
bringing
them
into
the
teams.
A
Very
interesting,
the
cultural
phenomenon,
the
human
condition,
as
we
say
you
know,
whenever
whenever
we
get
to
that
we'd
say
what
about
the
humans
and
sometimes
people
look
at
you
like
what,
especially
this
your
technical,
you
know
group
it's
very
important
to
think
of
the
human
condition.
So
I
think
the
next
hand
up
was
matthew,
wang
and
then
jim
wilkin
bush,
matt.
D
Hi
everyone
thanks
for
three
great
presentations,
ina,
I'm
sorry
I
walked
in
in
the
in
the
middle
of
your
presentation.
So
I
apologize
if
you
actually
covered
what
I'm
about
to
ask,
but
but
I'd
like
to
ask
this
actually
across
the
the
presenters
delighted
to
hear
I
don't
spend
my
my
time
in
security
full-time,
but
I
do
spend
my
time
building
ontologies,
and
so
I'm
gonna
push
my
agenda.
D
That
also
it
would,
you
know,
potentially
include
a
knowledge
graphs
of
models
that
represent
how
people
are
attacking
right.
So
we
could
be
sharing
not
just
the
data
but
the
models
as
well,
and
I'm
wondering
what
is
there
an
appetite
for
that
and
and
what
that
might
look
like
in
each
of
your
projects.
B
Well,
I
will
yeah,
I
can
say
about
our
projects,
since
our
main
goal
is
to
share
operational
data,
whatever
is
being
collected
from
clients
that
use
research,
stock,
operational
security
services,
so
we
are
focused
on
data,
but
looking
into
the
future,
I
agree
with
you.
I
think
it's
important
to
share
models
and
I
hope
that,
as
part
of
our
future
work,
we
would
encourage
researchers
who
work
with
our
data,
share
their
work
and
and
hopefully
contribute
to
model
sharing,
and
you
know,
tools
and
software
sharing
as
well.
E
I'll
come
a
little
bit
so
in
the
security
academic,
kind
of
research,
environment,
even
industry,
there's
an
increasing
trend
to
make
the
both
the
data
and
model
being
shared.
We
call
it
artifacts,
and
so
there
are
indeed
increasing
number
of
reaches
doing
that,
of
course,
not
everyone.
Yet
the
challenges
I
see
is
that
the
current
majority
of
the
machine
learning
ai
work
is
in
the
realm
of
supervised
learning,
which
I
personally
do
not
agree
is
my
opinion.
E
You
don't
need
to
agree
with
me,
but
I
personally
do
not
agree
that
that's
very
useful
for
cyber
security.
In
some
cases
they
are,
but
in
most
cases
I
don't
think
they
are
simply
because
label
data
doesn't
really
cannot
exist
in
the
real-time
operation
like
updated.
Unless
you
can
hire
someone
afford
to
hire
someone
who
label
data
for
you
every
single
hour
and
day,
which
I
don't
think
is
possible.
So
that's
that's
that's
my
opinion.
E
My
view
going
forward
in
the
future
is
that
publishing,
publishing,
meaning,
disseminating
model
data
as
well
as
that
findings
right,
so
I've
been
hoping
to
get
a
project
funded
to
really
have
like
creating
a
gateway,
a
gateway
where
we
can
share
a
hundred
institutions,
attack
models,
attack,
behaviors
online,
so
assume,
and
it
based
on
real-time
data
right.
So
it's
aggregates
if
I
see
something,
that's
really
start
rising
from
like
the
orange
line.
E
I
talk
about
if
the
origin
line
starts
occurring,
not
just
in
one
organization
but
multiple
organizations
around
the
same
time,
whether
it's
misconfiguration
because
microsoft
didn't
update
or
because
this
malicious
campaign
going
on
doesn't
matter,
then
everyone
sees
that
then
do
something
about
it.
Instead,
you
tried
to
decipher
hey
what
about
my
log
versus
what
happened
to
your
log
right,
so
that
that
would
be
my
hope
in
the
future,
not
just
sharing
the
data
and
model,
but
also
showing
the
insights
that's
obtained,
based
on
real
data.
C
I
I
think
jay
and
nina
covered
it
pretty.
Well,
I
guess,
from
a
you
know,
kind
of
coming
at
this
from
an
operational
perspective,
we're
interested
in
actionable
information,
and
so
part
of
the
challenge
is
right.
We
one
of
the
things
that
we
enjoy
doing,
is
working
with
the
research
community
on
the
model
piece
because
we
recognize
we
need
actual
information,
but
it's
also
you
look
at
how
fast
the
speeds
are
going
right
now,
with
network
traffic,
etc.
Then
you
know
we
we
have
to
make
decisions
faster
and
better.
C
F
So
I
just
I
just
spent
the
last
week
for
work
looking
into
fhir,
which
is
a
interoperability
standard
for
healthcare
data
and
I'm
like
wow
and
they're.
Talking
about
all
the
problems,
I'm
like,
we
have
exactly
the
same
problems,
there's
too
many
tools
and
too
much
data,
and
we
can't
speak
the
same
language
and
it's
just
so
much
work
to
to
get
all
this
data
together
into
some
sort
of
reasonable
into
some
sort
of
reasonable
format.
A
It's
a
very
good
point:
I'm
really
glad
you
brought
that
up.
I
actually
worked
on
trust,
identity,
privacy,
protection,
safety
and
security
standards
for
clinical
iot.
You
know
with
ieee.
I
lead
a
global
working
group
on
that,
and
so
fire
is
a
part
of
what
there
is,
but
there
are
still
holes
there
too,
that
we're
working
on
at
the
interoperability
level
for
all
these
new
devices.
So
we
should.
We
should
talk
about
that
a
little
bit
later,
but
I
want
to
let
jim
wilkin
bush
ask
his
question:
if
that's
okay.
G
You're
really
kind
of
talking
about
cyber
security
researchers
and
it
kind
of
causes
my
mind
to
bend
a
little
bit
because
I'm
thinking
of
sort
of
domain
researchers,
often
at
the
same
time-
and
I
wonder
to
what
extent
you
know
here-
I
am
you-
know-
running
a
large
center
with
a
huge
number
of
projects
and
and
dealing
with
compliance
across.
You
know
a
broad
spectrum
like
a
lot
of
us.
G
I
wonder
you
know,
to
what
extent
do
your
teams
purposely
interface
with
some
of
these
domain
areas
to
domain
science
areas
to
sort
of
infuse
the
application
of
this
work
into
some
of
the
infrastructures
that
you
support
on
campus?
I
know
you
alluded
to
some
of
that,
but
I
was
kind
of
hoping.
Maybe
you
could
just
highlight
some
examples
where
your
teams
are
doing,
you
know
really
sort
of
moving
some
of
this
innovative
work.
You
know
into
your
your
active
infrastructure,
your
your
current
infrastructure.
E
Maybe
I'll
address
a
little
bit
jim,
so
I
I
was
thanks
to
florence.
That
was
one
of
the
ttp
early
people
on
the
trust
ci,
so
I
actually
work
with
omnisoc
and
try
to
help
deploy
that.
I
will
tell
you
that,
that's
again
the
dream
of
me
right
with
a
gateway
I
talked
about
earlier,
but
also
all
the
prototype
we
developed
can
be
used,
but
it
takes
time
it
takes
time
and
effort
the
being
academic
as
a
researcher
as
a
professor
students
right.
E
At
the
same
time,
they
put
down
their
code
that
can
be
put
in
on
github
gitlab
and
things
like
that.
But
those
codes
are
still
very
challenging
to
be
download
hit
play
so
my
mac
right,
so
I
can
download
it
hit.
Install
everything
works.
Well,
basically,
it
is
very
challenging
to
do
that.
I
love
to
work
with
richard.
I
can
see
from
his
presentation.
There
are
a
few
things
that
his
team
could
experiment
with
me.
E
I'll
really
follow
up
with
him
on
that
you
know
the
type
of
data
I'll
follow
with
you
a
little
bit
more
about,
and
maybe
it
doesn't
one
of
the
questions
I'll
ask
you,
but
with
the
ip
and
port
the
certain
thing
can
be
done,
but
also
their
things
may
not
be
able
to
get
done.
So
I'm
curious,
whether
it's
something
that
I
can
help
you
to
make
your
operation
easier,
so
that
you
can
share
more
data
without
by
alleviating
concerns.
So,
yes,
jim.
E
I,
I
really
like
your
question,
because
I
think
that
should
be
every
single
research,
whether
a
cybersecurity
researcher
or
some
other
researchers
go
right.
The
the
research
should
have
some
impact,
whether
it's
immediate
impact
or
futuristic
impact.
I
think
that
that's
that's
important
for
researchers.
A
And
you
know
there
may
be
an
opportunity
for
us
to
think
about
what
some
things
we
could
do
in
the
future.
That's
one
of
the
things!
That's!
Why
we're
having
a
cyber
security
discussion?
Now
we
had
another
one
in
april
that
jay
helped
lead
for
us
across
the
community,
with
the
data
sharing
cyber
infrastructure,
working
group
and
caroline
and
matt
know,
there's
a
number
of
you
were
there
too.
A
You
know
maybe
there's
there's
something
we
should
put
together
regarding
the
domain
scientists,
the
cyber
security
experts
and
researchers
and
data
scientists.
Maybe
we
should
try
to
put
together
a
workshop
like
that
at
this
point
it
might
have
to
be
virtual.
Nsf
said
you
can
ask
for
permission
to
use
money
in
person,
but
people
have
to
be
allowed
to
go
so
it's
kind
of
complicated,
but
have
we
all
done
that
as
a
team?
Yet.
C
Thing
yeah,
I
mean
I,
I
think
that's
a
great
idea.
One
thing
I
would
like
to
point
out
on
that
because
I
I
do
think
this
is
a
great
question
and
a
challenge
jim.
So
I
worked
at
georgia
tech
before
coming
to
duke-
and
you
know,
georgia,
tech
at
the
time
had
this
massive
I
mean
obviously,
you've
got
gtrc.
There.
You've
got
the
college
of
computing
there
they're
doing
a
lot
of
really
really.
You
know
back
then,
and
even
now,
they're
doing
some
very
cool
things
in
the
cyber
security
space.
C
What
I
would
say
is
since
being
a
duke,
we
actually
have
found.
What
I
like
is.
Our
cio
has
really
pushed
central
I.t
involvement
in
the
research
space
and
the
what
we
call
co-curricular
space
things
like
that
working
with
students.
So
you
know
two
two
examples:
we
are
working
with
a
professor
at
duke
right
now
on
a
grant
funded
project,
that's
going
to
involve
it's
involving
building
anomaly
detection
models
and
we're
using
the
honeypot
data,
along
with
the
zeke
data,
to
help
prove
out
the
modeling
for
it.
C
I'm
interested
in
that
because,
like
I
said
if
it
improves
the
effectiveness
and
the
efficiency
of
it,
that's
less
work.
My
team
has
to
do,
but
it
has
to
be
actionable
and
it
has
to
be
applied.
I
guess
maybe
is
is
the
right
word
to
say
so,
very,
very
interested
in
that
we
also
worked
with
another
researcher
on
an
initial
dod
grant.
Didn't
it
ended
up
making
it
phase
two,
but
there
is
that
interest
there.
G
Yeah
you
hit
it
right
on
the
head,
richard,
that's
those
are
the
kind
of
examples
I'm
looking
for,
because
I
mean
I'm
I'm
the
first
one
to
respect.
You
know
really
sort
of
leading
edge
research
and
that
sometimes
it
needs
to
be
so
far
ahead
of
the
application
that
it's
not
practical
in
many
many
instances.
But
you
know
there
are
real
opportunities
in
this
space
to
to
bring
it
closer
where
there
can
be
some
win-wins
and
just
curious
about
sort
of
how
that
works,
because
I'm
I'm
looking
for
those
at
the.
G
U,
we
have
lots
of
opportunities
where
you
know
we
can
bring
those
to
my
own
university
and
and
usually
the
way
it
usually
works.
Is
you
know
we
could
say
hey,
let's
look
at
duke
as
a
model.
They've
done
some
things
that
we
could
actually
sort
of
inspire
some
people
to
see
tangible
ways
in
which
things
that
aren't
necessarily
in
our
typical
wheel
of
bucket
or
toolbox,
you
know
could
be,
could
be
implemented.
B
G
Yeah,
that's
great
yeah,
good,
good
question
so,
like
I
mean
I
know
that
you're
you're
working
in
a
lot
of
sort
of
health
sciences
or
with
many
health
sciences.
In
terms
of
probably
the
data
that
I
saw
just
in
the
brief
overview
that
you
gave.
But
that's
that's
one
area,
but
then
there's
others
where
you
know
obviously
cmmc
around
cui.
G
B
G
G
B
You're
talking
about
those
groups
sharing
their
data
and
not
them
using
operational
security
data.
Is
that
correct?
Because
I
I
I
think
I
didn't
really
understand
so.
For
example,
you
know
we're
we're
providing
access
to
operational
security
data,
and
I
mean
I'm
curious
to
learn
if
there
are
domain
scientists
who
would
be
interested
in
that
type
of
data,
but.
B
I'm
working,
I
have
another
project
that
actually
we're
working
with
libraries
and
creating
an
environment
for
accessing
digital
collections
in
libraries,
they're
protected
with
copyright,
and
you
can't
download
those
and
work
with
those,
and
I
think
there
are
various
enclave
solutions.
In
our
experience.
B
It's
still
it's
kind
of
a
very
customized
process
to
provide
access
to
restricted
data.
So
you
know
there's
no
like
plug
and
play.
You
know,
take
install
and
and
work
with
it
and
then
partially,
because
you
have
to
work
with
various
institutional
setups
and
the
way
institutions,
control
access
to
networks,
access
to
physical
versus
virtual
resources
and
all
that,
so
we
have
to
kind
of
work
through
all
these
hurdles,
to
figure
out
what
the
right
configuration
is.