►
Description
Date: 07/01/21
Presenter: Geoff Dobson
Institution: AIr Force Research Laboratory at Wright-Patterson, AFB
Bio: Geoffrey Dobson is a PhD candidate in Carnegie Mellon's School of Computer Science, focusing on agent-based modeling approaches to simulating cyber team performance. Geoff has been in the Air Force for 16 years, currently serving as a reservist assigned to the AIr Force Research Laboratory at Wright-Patterson, AFB in Dayton Ohio. In his full-time employment with the Software Engineering Institute at Carnegie Mellon, Geoff works on cyber war exercises with various Department of Defense teams. In this role he's helped deliver over one hundred high fidelity, live-fire cyber war exercises, which has helped shape the ideas and concepts paramount to his research.
A
The
few
announcements
we
have
is
next
week
we
have
our
sbp
brims
conference
going
to
happen
virtual.
The
workshop.
Sorry,
the
program
for
the
conference
is
online.
You
can
take
a
look
at
the
website
and
we
have
some
exciting
tracks
on
disinformation
challenge
and
also
some
domestic
extremism
track
on
parlor
data
challenge.
So
if
you're
interested
take
a
look
at
the
conference
website,
we
still
have
a
few
student
scholarships
or
early
career
scholarships.
A
So
if
you
have
students
or
colleagues
who
would
like
to
attend,
feel
free
to
share
the
information
with
them,
they
can
get
registered
to
the
conference
and
the
scholarship
will
cover
the
registration
cost.
So
in
case
you
are
looking
to
attend
the
conference
just
get
registered
and
when
registering
you
will
find
an
option
for
applying
for
scholarships
all
right.
So
those
were
some
of
the
announcements,
and
today
we
have
joff
dobson
with
us
who's
going.
Sorry
can.
B
I
do
an
announcement-
real
quick,
sorry,
oh
sure,
absolutely
yeah.
Just
as
a
reminder,
I
put
into
the
chat
a
link
to
a
microsite
registration
page
for
the
all
hands
meeting
for
the
self
big
data
hub
and
I'm
going
to
see
if
I
can
share
my
screen-
real,
quick
just
to
hopefully
you're
seeing
it
now,
and
this
will
be
a
virtual
meeting
on
the
28th
to
the
30th.
B
Much
like
you
might
see
in
a
trade
show,
but
it'll
be
virtual,
and
so
it's
an
opportunity
to
showcase
research
posters
activities
website
whatever
and
then
I'll
follow
up
with
you
on
talking
about
that.
But
it's
a
great
feature.
No
cost
it'll
run
the
three
days
with
some
plenary
sessions
each
morning
and
then
tracks
in
the
afternoon
around
each
of
the
priority
areas
or
the
topics
and
what
the
platform
allows.
You
can
come
in
and
look
at
the
booths
and
decide
which
track
you
want
to
go
to.
So
it
provides
optimal
flexibility.
B
A
Awesome
sharon.
I
will
certainly
follow
up
with
you
on
how
to
set
up
the
virtual
booth.
We
have
some
ideas.
We
kicked
around
those
ideas
over
the
email
for
the
social
cyber
security
working
group,
as
well
as
the
cosmos
center
that,
as
you
suggested,
that
will
be
an
awesome
opportunity
for
cosmos,
also
to
showcase
some
of
the
work
that
is
relevant
to
the
working
group.
A
I
think
I
see
all
familiar
names,
so
we
can
skip
that
and
directly
go
to
job's
presentation.
So,
first,
let
me
introduce
jeff.
It
is
my
pleasure
to
introduce
joff
dobson
who's
going
to
speak
today
at
the
working
group.
Joff
is
a
computer
engineer
at
carnegie,
mellon
university's
software
engineering
institute
where
he
helps
design
and
deliver
cyber
warfare
exercises
to
the
us
department
of
defense.
A
This
includes
working
with
regional
cyber
centers,
bringing
teams
in
for
competitions
and
coming
up
with
experiments.
I'm
sure
that
must
be
very
exciting
job.
We
look
forward
to
hearing
all
of
that.
Job
is
also
a
grad
student
associated
with
the
center
for
computational
analysis
of
societal
and
organizational
systems,
also
known
as
casos,
which
is
part
of
the
societal
computing
program
in
the
school
of
computer
science.
Joff
is
actively
researching
cyber
team,
behavior
and
performance.
A
C
C
So
I
know
about
this
working
group
and-
and
I
I
part
of
the
thing
that's
pretty
cool-
is
the
the
sharing
of
data
and
whatnot
and
that's
kind
of
like
what
you
know.
The
theme
of
of
this
presentation
will
be
is:
is
I'm
going
to
show
you
like
what
my
model
is
and
then
and
and
essentially
my
model?
Is
a
data
generation
machine
to
try
to
figure
out
how
cyber
teams
perform
so
I'll
share?
My
screen
now.
A
A
C
B
C
C
C
C
I've
been
doing
that
for
seven
years
now
and
over
the
course
of
those
years
and
standing
and
watching
a
lot
of
these
exercises.
I
got
the
idea
that
maybe
we
should
write
some
code
that
you
know
simulates
those
things
that's
kind
of
where
I
got
started
with
this
research,
and
then
I
read
you
know
some
documents
that
was
essentially
calling
out
this
problem,
so
there
this
is
the
dod
cyber
strategy.
C
When
confronted
with
multiple
contingencies
and
then
what
I
found
is
when
I
talk
to
troops,
it's
and
I
ask
them:
what's
your
mission
objective,
even
that's
hard
hard
to
explain,
then
there
was
another
document
that
kind
of
guided
some
of
my
thinking
here,
and
there
was
a
defense
science
board
report
that
came
out,
and
it
was
basically
saying
that
we
need
to
have
a
dashboard
that
essentially
looks
like
this.
That
can
track
all
these
different
things.
You
know
in
real
time
and
whatnot,
and
when
I
looked
through
those
I
thought
to
myself.
C
A
lot
of
these
are
actually
what
I
would
consider
team
performance
measures,
things
like
time
to
detect
and
availability
and
how
what
the
status
of
systems
are,
that
you're
charged
with
protecting
and
this
problem
is
actually
very
expensive,
as
you
can
imagine
so,
there's
always
headlines
like
this
and
a
different
military
newspapers.
I
follow
like
400
million
for
training
platform
and
150
million
for
exercises.
C
Some
of
these
things
are
things
that
I
get
involved
with
over
at
sei
and
then
the
last
one
that
was
that
was
pretty
interesting.
Was
this
this
executive
order
on
america's
cyber
security
workforce
from
two
years
ago
that
dhs
was
charged
with
having
an
annual
cyber
security
competition
and
the
goal
was
to
identify
the
best
teams.
So
really,
what
is
the
scoreboard
when
you
have
a
cyber
conflict?
C
And
I
and
you
know-
I
think
I
thought
a
lot
about
that,
and
I
think
that
asian-based
modeling
would
be
a
great
approach
to
trying
to
to
just
figure
that
out.
Essentially
just
you
know,
start
from
the
ground
and
figure
out.
What
are
these
these
scores
and
whatnot
so
and
as
most
of
you
on
this
group
probably
know
agent-based
modeling
at
the
center
is
agent-based
artificial
systems
that
have,
and
each
agent
has
it
has
environmental
variables
and
rule
sets
that
that
provides
emergent.
C
Behavior
there
on
the
left,
then
up
to
the
top
you're
going
to
propose
how
these
things
kind
of
interact
with
each
other
and
then
finally
to
the
right.
Then,
then,
you
can
kind
of
come
around
and
do
some
some
computational
experimental
design,
so
I've
been
kind
of
like
traversing
through
this
loop
over
and
over
again
for
several
years
agents.
C
According
to
some
of
the
the
leaders
in
this
in
this
area,
bonabu
mccollum,
north,
so
agents
are
individually
assessed
at
situation,
makes
decisions
on
the
basis
of
a
set
of
rules
and
an
agent
is
identifiable
situated
global
directed,
autonomous
and
flexible.
So
I
said
to
myself:
what
are
the
agents
here
when
we're
looking
at
a
cyber
conflict,
and
I
think
that
you
really
have
two
main
types
of
agents.
C
Now
you
have
to
take
the
terrain
and
maneuver
to
the
terrain,
and
things
like
this
so
like
it
says
there,
the
force
agents
are
the
military
personnel.
The
terrain
agents
are
the
military
computers
and
then
so
then,
once
you,
you
kind
of
wrap
your
head
around
that
you
can
start
to
ask
questions
like
the
questions
that
you'd
want
to
know
from
a
military
planning
perspective.
Like
is
the
operation
effective.
Is
the
cyber
terrain
vulnerable?
Have
we
disrupted
the
maneuver
of
the
enemy?
How
many
cyber
forces
are
necessary?
C
C
I
wanted
to
make
a
a
minimally
viable
model
to
run
proof
of
concept,
virtual
experiments
to
make
sure
the
the
the
most
basic
interactions
and
rules
were
working
so
in
in
version
one.
We
have
two
types
of
forces,
defensive
and
offensive,
so
obviously
the
defensive
ones
are
protecting
the
terrain
and
the
offensive
ones
are
trying
to
attack
the
terrain,
and
that
gives
them
goals
to
to
go
through
in
the
simulation.
C
C
What
the
what
types
of
attacks
might
work
against
those
things,
and
then
the
interac
interactions
are
directed
links
from
one
agent
to
another
at
any
given
time,
and
then
I
also
had
a
concept
of
environments
in
the
early
versions
where,
depending
on
what
environment
you're
deployed
to
like,
like
a
military
cyber
team,
you
would
have
different
types
of
systems
present
in
in
the
the
vulnerability
growth
rates
will
be
different.
Based
on
those
environments,
so
I'll
go
past
that
and
then
there's
just
an
example
of
the
codes.
C
The
first
three
versions
are
all
written
in
netlogo.
I
don't
know
if
anyone's
familiar
with
that,
but
it's
a
pretty
powerful,
but
yet
simple,
scripting
language
for
specifically
built
for
agent-based
modeling,
and
that
so
I
had
a
version.
One
interface,
where,
on
the
left,
you
could
set
different,
independent
variables
on
the
right.
It
would
collect
all
the
dependent
variables
as
the
as
the
simulations
were
running
and
what's
cool
is
once
you
do
that
you
can,
you
can
have
virtual
experiments
so,
for
example,
I
was
trying
to
think
of
virtual
experiments.
C
That
would
be
like
what
would
the
what
would
the
colonel
want
to
know
at
the
cyber
protection
brigade
and
they
they
asked
questions
like
this,
like
what
is
the
expected
effect
on
cyber
terrain
if
the
adversary
switches
from
a
15-day
routing
protocol
attack
to
a
denial
of
service
attack
in
a
base
environment
with
six
troops
deployed
to
that
conflict,
and
so,
if
you,
if
you
so,
I
would
run
the
simulation.
You
know
30
times
in
a
different,
different
different
cells
of
that
experiment,
and
then
I'd
say
well.
C
It
turns
out
that
in
in
this
situation,
we
would
see
that
the
servers
will
experience
a
lower
compromise
rate
than
than
the
networking
systems
based
on
that
setup.
So
the
google
version
version
one
was
complete.
I
moved
on
to
version
two
where
I
wanted
to
add
empirical
data
to
make
the
make
it
a
little
more
realistic.
C
So
I
kind
of
you
know
I
looked
around
in
the
literature
and
I
actually
came
across
this
research
group
at
brims
a
couple
years
ago
from
temple
and
they
had
gone
around
to
different
cyber
war
exercises
and
actually
counted
how
long
it
took
for
the
bad
guys
or
the
attackers
bad
or
good,
depending
on
you
know
what
side
you're
on,
but
the
the
they
would
watch
the
attackers
and
actually
time
them
through.
What's
called
the
intrusion
chain,
and
that
was
an
interesting.
C
You
know
more
realistic
addition
that
I
could
put
into
my
model
because
before
the
attackers
would
just
always
attack
a
system
but
in
real
life
you
have
to
go
through.
What's
this
concept
of
the
intrusion
chain,
lockheed
martin
probably
has
the
most
famous
one.
Their
picture
looks
like
this,
which
is
the
they
call
it
the
cyber
kill
chain.
Where
you
have
to
do,
recon
you
have
to
weaponize,
then
you
have
to
deliver
the
payload.
Have
the
exploitation
has
to
happen?
C
C
So
I
you
know
going
if
the
exploitation
success
rate
is
very
high.
Obviously,
they're
just
going
to
be
able
to
go
through
the
kill
chain
very
quickly,
so
where,
where
do
we
actually
slow
down
the
adversary
and
that's
just
a
little
code
snippet
of
where
that
exploit
success
rate
is
set,
and
then
and
then
this
is
the
result
of
this
virtual
experiment,
which
has
kind
of
two
cool
takeaways.
The
first
one
is
the
the
exploit.
C
Success
rate
has
a
larger
effect
on
the
delivery
phase,
which
is
the
phase
before
that,
and
that's
mainly
because
if
the
exploit
success
rate
is
lower,
then
when
the
attacker
gets
to
that
point
they
have
to
they
have
to
start
over
and
it's
actually
putting
them
in
that
delivery
phase
longer.
So
you
kind
of
have
these
takeaways,
like
defensive
forces,
should
ensure
that
your
cyber
security
tools
oops,
I
can
alert
on
on
that
phase,
but
then
the
other.
The
other
interesting
thing
is
a
lot.
A
great
thing
with
agent-based
modeling
is
trend.
C
Analysis,
and
you
can.
You
can
see
here
that
once
you
get
past,
10
or
so
exploit
success
rate,
there's
really
no
difference
between
10
and
40.
So
you
really
have
to
focus
on
where
you
you
make
the
biggest
difference
and
clearly
the
biggest
difference
is
when
you
go
for
from
something
like
five
to
two.
C
You
can
have
a
real
big
difference
on
how
how
much
you
can
slow
down
the
adversary,
so
that
was
version.
Two
I'm
actually
gonna
skip
version
three
this
presentation's
a
little
longer
than
I
want
to
be,
and
then
this
and
then
so.
I
got
to
the
point
where,
after
I
did
three
experiment,
three
versions
that
was
mainly
through
coursework
and
stuff,
and
then
I
I
basically
proposed
version
four.
C
As
my
thesis
I
mean
to
complete
my
thesis
at
cmu
last
year,
and
that
proposal
was
accepted
last
year
and
I'm
actively
working
through
version
four,
but
as
you
can
see,
I
tried
to
start
really
basic
and
then
and
then
spiral
up
from
the
foundation
to
now
version.
Four
is
all
about
defining
like
a
comprehensive
set
of
performance
measures
of
cyber
teams.
C
So
you
know
just
a
reminder.
What
I
started
with
was
trying
to
make
a
system
that
can
project
cyber
mission
force
to
achieve
its
mission
objectives
and
that
and
that
ends
up
being
this
list,
which
is
my
working
list
of
what
I
would
consider
like
a
really
close,
or
at
least
a
good
first
crack
at
a
comprehensive
set
of
performance
measures.
So
things
like
time
to
react
and
time
to
restore
and
operational
effectiveness
and
train
vulnerability,
change
and
rate
and
compromise.
I
have
some
network
measures
in
there
and
then
some
attacker
based
measures.
C
So
so,
essentially,
as
I
wrote
the
code
in
version
four,
I
was
just-
I
was
ensuring
that
I
could
computationally
with
the
software,
get
get
a
computational
measure
out
of
every
simulation
and
then
eventually,
I'm
going
to
try
to
build
a
dashboard.
That
looks
like
that,
one
that
the
report
was
calling
for.
C
So
I
have
description
in
question
here:
we're
actually
going
to
move
a
little
quicker
just
to
get
to
this
this
version,
and
then
this
is
just
a
screen.
This
is
a
screen
capture
of
how
I
have
cyber
mission
capability
rate,
so
I've
I've
I've
worked
out.
You
know
I'm
trying
to
work
out
the
math
for
each
one
of
these
formulas
from
an
agent
based
perspective,
so
essentially
I'm
gathering
sets
of
of
of
data
on
each
agent,
so
in
this
in
it.
C
For
this
example,
cyber
mission
capability
rate
comes
down
to
how
many
fulfillment
requests
come
out
from
a
friendly
force
and
then
how
well
the
request
is
fulfilled
by
the
cyber
terrain,
which
is
essentially
the
cyber
mission.
The
the
the
goal
of
the
cyber
mission
is
to
provide
information
to
the
friendly
forces
that
are
operating
there
so.
C
C
C
C
Repast
is
java
based
and
it's
maintained
by
argon
national
lab,
and
it
gives
you
a
lot
of
the
basics
of
agent-based
modeling
in
libraries,
and
then
you
can,
you
know,
inherit
from
their
turtle
agents
and
whatnot,
but
the
nice
thing
is
in
repast.
You
have
a
it's
pretty
easy
to
collect
the
data
on
each
of
the
classes
and
the
agents.
C
So
this
is
a
cyber
battle
being
played
out
between
a
adversary
force
of
three
over
to
the
right
and
a
defensive
force
of
nine
over
to
the
left
and
at
the
top
is
four
kinetic
missions
and
there's
a
bunch
of
friendly
forces,
the
little
little
white
troops
they
are
actually
making
requests.
I
just
don't
like
those
up
because
it
would
be
just
completely.
You
wouldn't
be
able
to
be
able
to
see
anything
because
there's
so
many
there's.
C
To
finally
get
to
the
point
where
the
fight
was
fair,
you
know
because
there's
a
lot
of
rules
in
here
where,
if
you
just
tinker
with
them
a
little
bit,
then
you
know
the
attackers
win
every
time
or
the
defenders
win
every
time.
So
I
I
spent
a
lot
of
time
tuning
it
to
get
it
to
the
point
where
this
is
actually
a
pretty
fair
fight
and
I'll
show
you
the
data
on
that
now
see.
If
I
can
switch
back.
C
Cool,
thank
you
so
so
yeah,
each
sim.
So
each
simulation
is
a
10-day
simulation,
so
this
and
I
did
that
just
to
get
somewhat
of
a
realistic
like
time
frame.
So
you
would
imagine
that's
like
a
10
days
of
a
cyber
conflict
where
a
team
is
deployed
and
and
going
against
an
adversary.
So
that
means
fourteen
thousand
four
hundred
ticks,
which
is
there's
fourteen
thousand
four
hundred
minutes
in
a
ten
day
period,
and
then
I
can
chunk
those
up,
and
you
know,
group
the
data.
C
However,
I
want
you
know
I'll,
just
and
I'll
show
you
some
of
those
results
and
each
simulation,
so
each
one
run
is
about
690
megabytes
of
all
of
those
measures.
I
showed
you
before
you
know:
I'm
I'm
collecting
information
requests
the
type
of
request,
the
type
of
operation,
whether
or
not
the
defender
was
successful
or
not
successful.
C
How
long
the
attacker
takes
to
do
things
whether
or
not
their
exploit
was
successful.
What
type
of
vulnerability
it
was
some
of
it,
I'm
not
even
using
yet
I'm
not
sure
what
to
do
with,
but
I
just
wanted
to
get
it
all.
And
then
you
know
look
through.
What's
what's
the
most
interesting
irrelevant
yep?
So
let
me
actually
switch
to
this
view,
and
so
here's
like
an
example
of
terrain
vulnerability
rate.
This
is
one
of
those
ones.
C
So
in
the
upper
left
there
we
see
the
terrain
vulnerability
so
also
keep
in
mind
when
I,
when
I
start
the
simulation
every
terrain,
every
computer
has
zero
vulnerabilities,
so
it's
kind
of
like
the
the
first.
You
know,
zero
to
a
hundred
is
just
like
that
natural
you,
you
open
computers
and
they're
going
to
become
vulnerable
over
time
if
they
have
zero
vulnerabilities.
C
So
after
100
hours
now
we're
kind
of
getting
to
that
point
where
it's
it's
leveled
off
and
they're
keeping
the
vulnerability
rate
around
point,
one
anywhere
from
point:
zero,
eight
up
to
like
point
one
one
and
that's
the
total
number
of
vulnerabilities
out
of
total
possible
vulnerabilities
in
the
in
the
terrain.
They're
responsible
for-
and
you
know
you
can
see
at
that
point-
there's
there's
it's
going
up
and
down,
and
it's
still
trending
up
a
little
bit.
C
So
I
did
you
know
I
fit
the
curve
there
below
a
logarithmic
curve
and
then
on
the
right.
That's
the
that's
the
graph
of
the
derivative
of
that
other
curve,
which
is
you
know,
dx
dy,
so
the
change.
So
you
can
see
that
it
it's
hovering
around
zero
for
for
most
of
the
end
of
the
of
the
simulation,
which
means
it's
a
it's
a
good
fight,
essentially
the
the
attackers
are
keeping
the
defenders
busy
and
the
defenders
aren't
able
to
make
the
vulnerable
the
terrain
that
much
vulnerable.
C
It's
like
an
even
fight
and
then
similarly
with
terrain
compromise
rates,
the
number
of
systems
that
are
compromised
at
any
given
time
over
divided
by
total
number
of
systems
and
a
little
more
variance
in
that
one.
But
you
can
see
the
the
compromise
rate.
Change
is
pretty
steady,
so
as
the
as
the
attackers
are
able
to
affect
a
machine,
the
defenders
eventually
notice
it.
They.
They
see
that
it's
compromised,
they
message
each
other
and
then
they
go
and
restore
that
system.
C
So
one
thing
that
I
found
is
that
if,
if
you
tune
that
just
a
little
a
little
more
in
the
attackers
make
it
make
it
a
little
easier
for
the
attackers
either
add
more
attackers
or
less
defenders
or
make
their
success
rate
a
little
higher.
It
quickly
becomes
like
you
know,
one.
The
attackers
will
just
take
over
every
machine
eventually
so
and
then
some
other
things
that
I
collect
like
interaction
data.
C
So
this
is
a
a
this
is
the
like:
a
dynamic
network
analysis
of
total
degree
centrality
of
the
messages
between
amongst
the
cyber
team.
So
there's
been
some
research
on
this
before
with
how
how
well
teams
interact
like
in
cyber
exercises,
so
I
actually
have
a
colleague
I've
worked
with
at
army
research
lab
that
has
done
some
things
like
this,
where
they
try
to
observe
interactions
between
teammates
and
so
I'm
doing
the
same
thing
in
the
simulation
every
time.
C
There's
a
message
sent
then
I
I
track
that
data
and
then
you
can
see
these
are
the
top.
So
this
this
is
from
this
is
a
output
from
from
our
network
science
to
aura
and
defender.
60
is
clearly
the
highest
level
of
total
degree
centrality
and
that's
because
defender
60
is
the
team
lead
and
the
others
are.
You
know
squad
members,
so
so
you
so
it
is
behaving
as
I'd
expect.
The
team
lead
is
going
to
have
more
total
degree,
centrality
most
interactions
with
others.
C
Part
of
the
code
is
when
you
find
compromises,
you
immediately
alert
to
team
lead.
You
know,
for
example,
but
then
on
close
to
centrality
defender,
68
isn't
in
the
thought.
It
is
actually
last
because
when
there's
problems,
the
team,
the
the
higher
skilled
team
members,
will
take
care
of
the
problem
and
also
the
message
message:
team
members
within
their
team.
C
So
just
like
in
real
life,
you
know,
if
you
start
an
operation
and
there's
a
completion,
it
needs
to
be
done
in,
let's
say
60
minutes.
If
you
can
do
it
in
in
15
minutes,
then
that's
a
much
higher
you're,
much
more
efficient
than
than
if
you,
the
person,
got
it
done
in
60
minutes.
So
every
operation
that
comes
out
of
the
simulation-
I
run
it
through
this
algorithm
and
then
then
we
can
see
by
operator
who
is
the
most
efficient
so
part
of
what
I'm
doing.
C
C
C
That
might
probably
seems
a
little
high
like
in
real
life.
I
I
would
assume,
but
maybe
not
like
this.
This
is
one
of
those
ones
where
it's
it's
really
hard
to
validate,
because
you
know
who
knows
how
many
who
knows?
What,
like
you
know,
this
cyber
criminal
group's
actual
attack
success
rate
is
but
it's
nice
to
try.
C
Well,
I'm
gonna
conduct
several
virtual
experiments
and
this
is
actually
the
work
that
I
just
began
is
now
funded
through
an
air
force.
Research
lab
grant
where
one
of
the
the
call
outs
was
to
try
to
understand
how
does
knowledge,
skills
and
experience
differentiate
outcomes.
So
this
I
haven't.
So
this
is
an
example
of
a
virtual
experiment
table
that
I
could.
C
I
could
now
run
with
this
version,
where
I
could
change
the
skill
set
low,
medium
and
high,
and
I
could
change
that
on
knowledge,
skill
and
experience
and
then
you
know
run
run
different
just
to
run
some
simulations
and
see.
Are
you
better
off
with
higher
knowledge?
Are
you
better
off
with
you
know,
more
experience
and
the
research
questions
there
are
around?
C
C
Is
it
better
to
shadow
more
experienced
team
members
like
that's,
you
know
on
the
job
training,
should
we
reward
troops
with
more
experience,
so
we
don't
have
them
get
this
really
awesome
experience
and
then,
and
then
quit
the
army
and
and
join
a
private
company
for
twice
the
money
like
a
lot
of
them.
Do
when
I
talk
to
them
at
the
at
fort
gordon.
So,
yes,
it's
kind
of
like
what
you
know.
C
What,
where
do
you
get
the
most
bang
for
your
buck
when
you're
thinking
about
something
on
an
agent
on
an
agent
level
or
troop
level
of
knowledge,
skills
and
experience?
So
that's
where
I
plan
to
do
the
night,
the
next
several
months
of
work
on
this
okay.
So
that
concludes
my
presentation
and
I'm
happy
to
take
questions
or
talk
about
what
your
thoughts
are.
A
Jeff
that
was
fascinating.
It's
always
interesting
to
see
how
simulation
environment
can
lead
to
decisions
in
the
real
world,
and
I
think
we
have
a
question
to
that
direction.
From
shannon
on
the
chat-
and
I
can
read
the
question
shannon
has
said:
how
do
you
validate
an
agent-based
model
in
this
context?.
C
Yeah,
so
I've
thought
a
lot
about
that
with
my
advisor
dr
carly,
and
negotiated
that
a
little
bit
and-
and
so
actually
my
next
nice
couple
slides
when
I
first
got
the
invite
for
this,
I
wasn't
sure
if
it
was
a
full
hour
or
what
the
time
was
so
I
actually
had.
I
actually
had
some
model
validation
plans
on
on
my
slide.
So
I'll
just
present
those
next
couple
slides
here,
because
this
is
exactly
what
I'm
thinking
about.
C
First,
according
to
mccall
in
north
this
book,
managing
business
complexity
is
really
good.
If
I
was,
if
I
ever,
you
know,
become
a
professor
in
teaching
class
they'll
be
required
reading
for
agent-based
modeling
because
they
actually
go
through
what
they
describe
as
the
seven
types
of
agent-based
model
validations
and
they
are
requirements,
data
phase
process,
model,
output
and
agent
in
theory.
So
you
know
so
I
kind
of
I
read
through
all
those
those
chapters
in
the
book
and
just
thought
about
specifically
for
my
simulation.
C
What
is
the
most
you
know,
what
could
I
actually
do
and
the
first
one
is
requirements,
validation
and
you
know
so.
I
actually
just
thought-
irb
approval.
A
couple
weeks
ago
to
to
do
a
survey
and
then
focus
group
with
military
planners
so
essentially
walk
through
well.
First,
the
survey
I'm
asking
questions
just
out
to
the
you
know
to
the
field
of
how
do
you
interact
with
your
team
when
you're
engaged
with
an
incident?
C
How
do
you
determine
your
performance
measures?
What
are
your?
What
is
your?
What
is
your
knowledge,
skill
and
experience
like
a
self-assessment
of
their
knowledge,
skill
and
experiment
experience?
So
it's
the
idea
is
from
that
survey.
I
can
actually
seed
some
of
the
virtual
experiments
that
then
I
can
talk
to
a
focus
group
about.
So
that's
like
requirement.
Validation,
face
validation
is
just
kind
of
like
well,
basically,
what
I
was
talking
about
before,
like
after
you
run,
the
simulation
does
does
0.12
attacker
success
rate
seem
reasonable.
C
You
know,
so
I
actually
could
go
through
that
question
with
experts
on
all
the
different
data
that
comes
out
of
this
and
then
the
most
interesting
one
for
sure
is
the
model
output
validation.
So
I'm
actively
looking
at
ways
to
get
some
data
from
competitions
and
exercises
this.
This
is
for
sure
the
hardest
one,
because
I've,
actually
there
are
some
data-
sets
out
there.
This
is
what
I
like
about.
C
This
group
is
talking
about
this
type
of
thing,
because
there
are
some
data
sets
out
there
from
some
cyber
exercises,
but
you
know
like
I
have.
I
have
like
bro
data
malware
data,
some
net
flow
data,
but
the
problem
is
it's
like.
I
don't
know
how
many
people
were
on
the
team.
I
don't
know
when
the
attacks
happened.
C
C
I
could
at
least
know
the
total
number
of
nodes
in
that
network
and
how
often
they're
communicating
and
that's
something
that
I
can
easily
simulate
so
that
that
that's
probably
my
best
guess
at
like
output
validation
as
of
now
and
theory.
Validation
is
a
little
bit
more.
You
know
it's
like
the
more
abstract
one,
where
you're
essentially
baking
some
theory
into
the
the
model.
So
theory
validation.
Is
you
get
it
re
peer,
reviewed
and
published
yeah
there
you
go.
C
B
Would
actually
not
follow,
but
you
know
the
next
step.
Skills,
knowledge
and
experience
right
is
that
it
skills
knowledge.
Experience
are
those
the
three
yeah
yeah,
so
that
seems
to
me
like
you,
would
have
applications
in
any
space.
Do
you
mean
is,
is,
do
you
think
there'll
be
some
unique
features
to
cyber
threat
analysis,
or
do
you
think
that
the
model
could
be
used
elsewhere.
C
C
C
So
this
is
the
agent-based
rule
set
within
the
code
where
so,
basically
the
each
agent
at
whenever
they
need
an
operation,
because
they're
they're
either
done
with
something
or
they
messed
up.
They
select
an
operation
and
immediately
they
decide.
Can
they
do
that
and
that's
based
on
knowledge
right
so
and
I
think
that's
pretty
generalizable,
like
you
know
you
think,
of
a
a
work
crew
and
it's
like
hey.
We
want
you
to
do
this.
Why?
I
can't
do
that.
I
don't
have
the
knowledge
to
do
that
thing.
C
That's
where
the
stochasticity
comes
into
play
and
based
on
their
skill,
they're
more
likely
to
get
that
correct
and
then
and
then
they
and
then
they
stay
in
that
operation
for
for
a
number
of
ticks
all
based
on
their
experience.
So
those
things
play
together,
but
yeah,
I
think
it.
I
think
this
is
pretty
a
pretty
good
general.
You
know
way
things
things
go
in
the
real
world
on
pretty
much
any
task.
A
Okay,
I
want
I
wanted
to
ask.
Maybe
this
can
help
elaborate
a
little
bit
more
on
the
discussion
side,
so
the
cyber
attack
or
the
cyber
warfare
in
general
has
seen
a
lot
of
evolution
in
the
recent
past
right.
So
how
do
you
look
at
that,
and
specifically
from
the
perspective
of
asian-based
models
and
how
to
track
that
evolution
and
be
able
to
adapt
the
agent-based
models
to
perhaps
simulate
the
evolution
as
closely
as
possible?.
C
C
C
Mitre
attack
has
a
huge
database
and
they
even
track
like
what
are
the
most
frequent
types
of
attacks,
and
you
can
look
at
that
year
every
year
and
it
does,
it
does
evolve,
based
on
changes
in
and
technology
and
operating
systems,
and
everything
like
that,
so
I
I
abstract
that
away
and
in
so
in
the
code
the
attackers
do
go
through
the
kill
chain
and
what
they
do
is
they
look
for
vulnerabilities
in
the
system.
The
vulnerability
is
a
number
between
zero
and
one
hundred.
C
Zero
represents
a
zero
day,
so
only
certain
type
only
the
highest
tier,
the
the
highest
sophistication
level
attacker
can
do
a
zero
day.
All
the
others,
based
on
their
tier
level,
can
do
certain
types
of
attacks,
so
a
weaker
attacker
can
only
can
only
affect
you
know,
attacks
one
to
twenty,
for
example.
C
But
but
the
reason
I
do
that
is,
I'm
I'm
trying
to
abstract
away
all
the
complexities
in
the
attacker
space
and
just
get
it
to
the
basics
of
they're.
Looking
for
vulnerabilities,
they
build
up
a
list
of
vulnerabilities
that
they
found
in
in
in
this
in
the
machines
and
then
there's
a
waiting
time
which
simulates
them
weaponizing
and
preparing
the
attacks
and
then
and
then
delivering
the
attacks.
C
So
that
that
way,
the
attackers
are
different,
each
other
based
on
their
sophistication
level,
but
I'm
I'm
I'm
just
abstracting
that
away
too
oh
attack
sophistication
level.
Five,
you
have
everything
available,
except
for
a
zero
day,
sophistication
level.
Four,
you
have,
you
can't
do
a
zero
day
and
you
can
do
attacks
that
are
rated
one
to
eighty,
I'm
also
basing
that
on
the
a
defense
science
board
report.
So
that's
that's.
That
was
their
findings
from
looking
at
like
sophistication
levels.
They
they
categorize
adversaries
into
six
levels
of
sophistication,
tears,
yeah.
A
For
instance,
when
you
look
at
the
type
of
recent
attacks
on
colonial
pipelines
or
the
the
beef
manufacturer,
I
think
this
first
one
of
the
largest
beef
manufacturer
in
the
us
so
does
that
consider
how
impactful
that
is
to
the
economy
of
the
the
region,
or
maybe
even
you
know,
disturbing
the
integrity
of
elections
and
so
on
and
so
forth.
So
are
there
studies
that
are
looking
at
characterizing
the
impacts
and
then
taking
that
approach
to
determine
how
sophisticated
or
impactful
that
attack
is
and
then
building
agent-based
models
off
of
that.
C
I
haven't
seen
that
I've
thought
about
how
to
do
that,
and
I
would
do
it
with
the
mission
assurance
categories
so
it
it
might
be
out
there
and
it's
super
interesting
because
you
know
obviously
something
like
the
colonial
pipeline.
I
would
love
to
know
what
level
of
effort
went
into
that.
So
you
know
you
could
look
at
it
like
how
much
effort
is
expended
to
make
something
that
literally
affects
the
globe
and
definitely
afflicts
that
region
more
severely.
C
So
it'd
be
interesting
to
you
know.
Do
some
kind
of
analysis
like
that,
like
I
said,
I
haven't
come
across
that,
but
within
the
within
cyber
fit,
I
I
do
have
everything
as
a
mission
assurance
category.
That's
something
that's
well
known
in
the
air
force
where
systems
are
are
given
a
mission
assurance
category.
Obviously,
like
you
know,
the
system
that
gives
out
towels
at
the
fitness
center
is
the
level
one
and
the
system
that's
scheduling
fighter
jets
is
a
level
three.
C
C
Okay,
so
they're
not
they're
not
doing
that
now,
but
I
think
that's
an
interesting.
You
know
you
could
look
at
that
and
then
you
could
have.
The
attackers
also
seek
higher
mission
insurance
categories,
so
they
could,
you
know
they'd
have
to
decide.
Do
I
want
to
spend
more
time
on
higher
value
targets
that
are
harder
to
break
into
or
less
time
on,
lower
value
targets?
You
know
where
does
that
like?
How
does
that
play
out
almost
like
a
game
theory
right.
A
Yeah
I
was
listening
to
npr
last
year
evening
and
there
was
a
professor
from
harvard
who
introduced
this
concept
of
bullwhip
in
the
supply
chain
industry
and
any
little
perturbations
in
the
supply
chain.
Industry
causes
massive
ripple
effects
upwards
and
downwards
to
the
whole
supply
chain
infrastructure
from
the
producer
down
to
the
retailers
and
then
the
customers.
They
took
an
example
of
beer
and
it
was
very
fascinating.
It's
it's
a
game
that
they
have
developed
and
you
may
want
to
search
about
that,
because
it's
a
very
fascinating
example.
A
C
Yeah
I
I've
thought
about
that
too.
I
have
I
haven't,
come
across
a
military,
I
I
tend
to
search
an
assertion
like
you
know,
naval
postgraduate
school.
They
do
a
lot
of
agent-based
modeling
for
military
research
interests
and
I
haven't
come
across
one
that
kind
of
like
differentiates
the
supply
chain.
I
have
some
thoughts
about
that.
I
think
it'd
be
really
you
know.
There's
there's
definitely
a
lot
of
really
cool
projects
to
do
in
that
space.
A
C
C
Yeah,
I
don't
see
why
not,
I
think
you
know
you
could
you
could
have
like
a
chain
of
networks
where,
just
like
information
flow
you
could
you
could
measure
the
information
flow
through
various
networks.
So,
like
that
picture
I
had
on
my
screen.
You
can
imagine,
that's
that's
one
network
that
information
has
to
get
in
there
and
then
out
of
there
and
then
you
know,
connect
to
the
next
one
connect
to
the
next
one
and
slow
down.
C
A
C
Yeah
the
one
question
shannon
left,
but
he
was
saying
about
the
if
the
metrics
are
dual
in
nature
or
you
just
like
switch
the
sign
or
I
think
all
of
the
measures
are
defensive
team
based
so
but
in
a
way
you
could
just
you
know,
switch
the
sign
when
it
comes
to
change
or
rate
and
stuff
like
that,
and
not
maybe
not
just
not
so
much
switches
time,
but
also
min
or
max
right.
So
the
attacker
wants
to
maximize
terrain
vulnerability.
A
C
I
had
a
couple
of
questions
said
something
to
the
extent
of
in
the
event
that
you've
experienced
a
cyber
attack.
Did
you
interact
more
or
less
with
your
team
or
the
same?
So
that's
like
an
example
question
and
one
of
my
advisors
said:
don't
write
cyber
attack
because
that'll
cause
people
to
go.
Am
I
allowed
to
say
that
was
an
attack
or
you
know
that
I
mean
that's
like
that's.
C
That's
a
super,
interesting
and
controversial
question,
because
if
because
it
essentially
comes
down
to
whether
or
not
a
a
commander
at
a
certain
level
declares
that
that
action
was
an
attack
and
if
it
was
an
attack,
there's
like
these
different
titles
in
the
u.s
code
that
then
you
know
legally
justify
certain
reactions
and
what
commanders
can
take
action
right
away
versus
what
they
would
have
to
get
presidential
or
secretary
of
defense
or
congress
to
approve.
So
that's
something
I
don't
know
actually
don't
even
really
know
a
lot
about.