►
From YouTube: IETF104-SMART-20190325-1610
Description
SMART meeting session at IETF104
2019/03/25 1610
https://datatracker.ietf.org/meeting/104/proceedings/
A
B
B
See
what
happens
as
a
result?
So
thank
you
and
for
anyone
who's
in
Bangkok.
We
just
wanted
to
briefly
recap
what
we
talked
about
in
the
planning
meeting
for
smart.
So
before
we
get
started.
Itf
note
well,
please
note
well
I'd,
like
this
research
group
or
whatever
it
ends
up
becoming
to
be
friendly.
So
please
respect
speakers
and
for
this
session,
if
you
could
hold
all
your
questions
to
the
end
of
the
speaker's
talk,
that
would
be
appreciated
so
for
the
agenda
today.
B
This
is
what
we've
got
so
we're
starting
off
with
some
chair
time,
our
Charter
and
the
group
set
up
and
then
we've
got
sorry.
I
think
this
is
actually
wrong
version,
but
then
we've
got
Symantec
talking
about
the
threat
landscape,
that's
our
No
and
then
David
McGrew
will
be
talking
about
privacy
and
cyber
security.
B
Arno
is
presenting
on
class,
which
is
a
draft
that
is
uploaded
and
on
capabilities
and
limitations
of
endpoint
security
solutions,
and
then
we
have
our
tech
director
from
NCI,
see
Ian
Levy
talking
at
the
end
and
I've
just
caught
his
presentation,
one
snake,
because
it
doesn't
really
have
a
good
title
and
and
then
we'll
be
finishing
off
with
some
more
chair
time.
So
the
up
to
date,
agenda
is
on
is
on
the
session,
but
that's
what
we've
got
planned
for
the
next
two
hours,
so
smart.
This
is
our
headlines
and
what
we
discussed
in
Bangkok.
B
We
want
to
be
at
the
advisory
research
group
on
attack-defense
in
the
IETF
IRT
F.
So
these
are
our
three
main
goals
to
research
methods
to
efficiently
and
effectively
detect,
mitigate,
prevent
or
even
eliminate
threats
to
guide
ITF
protocol
development
and
with
a
view
putting
the
cyber
defense
viewpoint
across
and
to
become
the
authority
on
attack,
defense
and
prevention
in
the
IETF
IRT
F.
So
as
a
headline,
that's
what
we're
aiming
to
do.
B
We're
looking
at
case
studies
of
previous
incidents
and
attacks
we're
looking
at
best
practice.
We
want
to
stimulate
new
research
into
how
we
can
find
new
methods
to
prevent,
detect
and
mitigate.
We
want
to
look
at
reports
and
statistics
on
the
threat
landscape,
so
we
know
what
we're
up
against,
and
these
are
just
some
research
projects
that
we
think
would
be
good,
including
threat,
detection,
unencrypted
traffic
and
there's
a
whole
set.
B
B
So
we're
wanting
to
get
to
the
point
where
we
have
a
good
mixture
of
expertise
in
this
room
that
can
go
to
the
IETF,
go
to
IRT
F
and
bring
that
knowledge
in,
so
that
when
protocols
are
being
designed
and
implementers
designers
and
use
we'll
be
better
informed
about
the
potential
impact
and
positive
or
negative
of
their
new
protocols
on
attack
prevention
and
mitigation.
So
our
draft
Charter
is
on
github.
It's
been
there
for
quite
a
while.
Please
do
read
it.
Give
us
feedback
on
it
on
the
list.
B
It's
still
in
draft,
and
but
this
is
what
we're
proposing
to
do
and
then
because
it's
quite
hard
to
find
some
stuff
on
smarts
and
we've
got
a
wiki
page.
We
have
got
a
mailing
list.
We've
got
a
data
tracker,
a
github
and
we've
also
got
a
page
on
Keros,
to
which
Kathleen
will
talk
about
more
in
a
minute.
But
these
are
all
of
the
places
where
you
can
find
out
more
about
what
this
group
is
trying
to
do
and
what
it
wants
to
be
so
I
just
put
them
all
in
one
page.
A
So,
let's
see
about
two
weeks
ago,
on
February
28th
and
March
1st
we
held
a
workshop
I
saw
sponsored
workshop
on
coordinating
attack
response
at
internet
scale.
This
was
a
follow-on
workshop
to
one
we
had
done
initially
four
years
ago,
that
was
IAD
and
I
socks
sponsored,
and
there
was
there
was
more
research
that
was
presented
here.
This
is
these
slides
are
just
a
snippet
of
what
happened
over
those
two
days.
A
They
were
packed
with
brainstorming
activities
to
generate
additional
research
ideas
in
on
top
of
what
was
presented
from
papers,
brought
forth
to
attend
the
workshop,
and
so
these
were
just
some
of
the
highlights.
Open
situ
was
discussed,
and
that's
for
exchanging
information
on
indicators,
response
and
I'm,
sorry,
indicators
of
compromise
and
and
how
you
respond
to
those.
So
what
actions
you
might
take
n
ICT
from
Japan
a
cup
well,
and
they
had
two
different
papers
presenting
work.
They
have
done
extensive
work,
they've
done
on
security,
automation
and
visualization.
A
I
can't
do
that
justice
in
the
very
short
time
we
have
here,
but
that
work
was
quite
impressive
and
their
learnings
from
that
and
deficiencies
in
adoption
of
standards.
And
and
how
can
we
tackle
those
types
of
problems
to
further
the
ability
to
automate
right?
So
there
was
key
areas
where
they
were
hindered
in
their
abilities.
A
That
would
generate
research.
That
would
be
useful
and,
let's
see
Incident
Response
Coordination.
We
had
several
groups
represented
that
do:
Incident,
Response
Coordination,
the
forum
for
incident
response
and
security
teams.
The
chair
was
present
and
he
discussed
what
they've
seen
from
from
their
perspective,
TF
cert
some
community-based
ones
that
were
driven
out
of
specific
attacks
like
the
the
Mira
attack.
A
I
think
is
critical
towards
getting
away
from
that
arms
race,
and
so
that
was
highlighted
just
in
terms
of
them
well
the
work
itself,
but
also
the
thinking
process.
There
were
many
different
styles
and
the
the
breakouts.
This
one
was
on
incident
response
teams
to
see
what
could
be
improved.
So
how
many
people
here
have
are
currently
working
and
incident
incident
response
team
or
have
in
the
past,
oh
great,
more
than
I
would
have
expected.
So
that's
that's
pretty
cool.
A
A
So
they
have
huge
problems.
Incident
Response
right
now
is
isolated
into
small
groups,
trust
groups
and
how
they
exchange
data
they
sometimes
so.
If
you're
at
a
corporation,
you
typically
have
like
eight
feeds
of
data
that
are
mostly
overlapping,
but
you
get
each
of
these
feeds
so
that
you
get
that
one
nugget
of
information
that
might
come
on
one
feed,
not
the
others.
It's
a
lot
to
manage,
and
it's
really
aimed
at
large
companies
right
now.
A
In
my
opinion,
we're
never
gonna
fill
that
gap
with
training,
which
is
a
lot
of
the
efforts
towards
trying
to
to
build
a
gap
in
information.
Perfect,
a
security
professional,
so
I
just
don't
see
that
happening.
So
we'll
have
a
really
nice
summary
that
I
think
could
lead
to
productive
research
and
understanding
the
problems
they
face
and
how
they
work
and
subsequent
workshops
on
this
can
help
as
well
and
actually
be.
A
So
fingerprinting,
then
there
was
snark
had
a
lot
of
positive
perception
to
use
trust
indicators
to
make
decisions
about
good
and
bad
actors,
so
that
one
was
received
really
well
I.
Think
a
number
of
people
want
it
to
help
work
on
that,
so
I
I
would
hope
to
see
that
one
go
forward,
attack,
coordination,
solutions
and
automation.
That's
you
know
always
an
ongoing
theme,
but
we
haven't
done
well
on
it
yet,
and
we
did
compare
a
bunch
of
the
efforts
out
there.
What
works?
What
doesn't
work,
why?
Why
is
there
adoption
for
some?
A
Why
not
so
that
it?
You
know
if
we
go
to
design
something
we
are
thoughtfully
thinking
on
on
those
types
of
aspects
as
well:
cryptographic,
rendezvous
and
then
problems
with
LT
discovery
with
some
other
protocols
and
and
information
revealing.
And
how
do
we
bake
in
more
security
into
existing
protocols
and
and
what
kind
of
concerns
should
we
be
highlighting
to
improve
protocols
elsewhere
right?
So
if
that
advice,
piece
Kirsty
was
talking
about,
ok,
I
think,
that's
it
and
a
full
report
will
be
forthcoming.
B
C
C
D
Yeah
I'll
answer
good
sorry,
I,
don't
know
if
you
wanted
to
have
questions
on
it
sounded
like
you
wanted
them
after
each
presentation-
and
you
just
finished
one
so
I
figured
this
was
not
the
time
so
going
forward
to
it
for
the
rest
of
the
session.
For
me,
it
would
be
helpful
to
understand
what
kind
of
group
you
want
to
be
because
I'm
confused,
we
we.
B
A
So
my
opinion:
well,
it's
it's
been
a
difficult
process
to
get
to
this
point
right,
so
I
think
we're
just
a
little
confused
as
well
are
so
if
we
had
our
choice,
I
think
it
would
be
to
be
a
research
group
and
similar
to
see
FRG.
There
might
be
areas
where
there
could
be
advice
or
there
could
be
work.
That
comes
this
way.
That
really
should
get
put
into
like
just
okay.
F
D
G
H
Good
good
afternoon,
so
my
name
is
ánotá
dei.
I
work
at
initech,
I'm,
Swiss
and
French.
So
sorry
for
my
French
accent.
So
today,
I'm
going
to
speak
about
the
threat
landscape
update
from
the
I
STR.
That's
a
very
well
known
report
from
my
company.
We
issue
it
every
year
as
the
volume
24
I'm,
not
one
of
the
authors
of
this.
My
good
friend
from
Zurich
Candide
West
is
one
of
the
other,
so
the
person
who
is
rewriting
it.
H
So
you
try
to
take
questions,
but
so
for
those
who
don't
know
the
a
STR
is
a
big
reports
around
this
year.
Sixty
pages,
it's
full
of
numbers.
You
have
many
breakdowns
by
country
by
tiles
by
a
lot
of
various
ways,
but
we
try
to
simply
give
back
what
we
learned
from
the
last
year.
So
this
is
the
report
from
2018
all
right.
H
H
Right
so
I'm
not
going
to
take
you
to
you
all
afternoon
on
numbers,
but
just
try
to
summarize
a
few
trends
that
you
observed
the
first,
the
first
four
categories
are
about
focused
on
profits.
That
means
that
the
cybercrimes
tried
to
get
money
out
of
the
attack.
So
the
newcomer
is
here
will
deform
jacking,
so
from
jacking
is
based
on
JavaScript
attacks
and
it's
really
about
a
transparent
attack
on
where
you
do
payments
and
so
on,
to
get
your
information,
your
PA
and
so
on.
So
we
could
clearly
see
a
trend.
H
The
error,
with
around
nearly
5,000
websites,
were
compromised
by
from
jackin
attacks
every
month.
We
blood
to
run
for
me
and
from
getting
attacks
last
year
on
from
endpoint
devices,
and
the
problem
is
that
it's
a
difficult
one
to
detect
by
end
users.
So
that's
that's.
That's
a
painful
one.
That's
that
that
arrived
on
the
market
of
the
cybercrime,
the
other
one
is
crypto
jacking.
So
this
one
is
knowing
there
is
nothing
new,
but,
as
you
can
see,
overall,
it
declines
all
right,
so
it
decreased
because
the
value
of
the
crypto
currency
has
decreased.
H
So
you
can
clearly
see
the
correlation
between
attacks
and
profit,
so
it
remain
at
high
level
so
especially
in
December
2018.
The
other
piece
which
is
interesting
is
that
in
in
2017
we
got
one
malicious
URL
over
16,
and
now
we
are
down
to
one
per
10
URLs,
so
there
is
case
on
patterns
there
all
right.
Thank
you.
Next
slide.
H
Okay,
so
two
other
forms
of
profits,
attacks
we're
on
some
way,
I'm
device,
of
course.
So
when
some
way
we
could
clearly
feel
already
in
2017
that
it
would
decline
and
it
declined
by
the
twenty
persons.
Now,
it's
not
the
only
important
number.
What
is
interesting
here
is
D.
Is
the
attack
moved
to
enterprise
on
somewhere,
so
a
lot
more
companies
got
attacked
on
on
one
somewhere
and
especially
your
mobile
one,
somewhere
infections,
where
episode
33
persons?
H
The
other
piece
that
starts
to
be
concerning
is
the
increase
on
social
media,
especially
on
fake
news.
I
think
this
is
certainly
to
a
peak
that
will
increase
year-over-year,
so
we
expect
to
have
more
problems
through
fake
news
and
propaganda
next
year.
So
that's
for
photo
categories
on
profits,
so
I
think
the
first
takeaway
is
that
from
jacking
appears
the
three
others
interesting.
They
have
declined,
but
there
are
some
specifics
in
in
each
other.
Ok,
let's
continue.
H
So
now,
let's
move
to
two
types
of
attacks,
so
here
we
have
something
that
is
not
new
attacks
on
Internet
of
Things
are
not
you.
We
can
see
that
we
have
75
persons
of
compromised
device
where
routers
that's
problem.
Cameras
are
following
with
15
persons.
We
all
remember
what
happened
with
Mirai
and
what
happened
to
a
number
of
parties
like
Deutsche
Telekom
on
others
that
been
very
painful,
and
we
see
that
is.
The
pattern
is
basically
staying
what
it
is,
of
course,
the
the
major
concern
is
with
5g
coming.
H
We
are
going
to
have
many
devices
that
we
stay
connected
all
the
time,
so
there
you
have
direct
connection,
so
we
expect
that
is
going
to
worsen.
The
typical
problem
is
weak.
Passwords
on
device
exploit
that's
two
principal
causes,
so
so
providers
or
vendors
are
not
making
enough
efforts
to
to
get
it
just
correct
which
which
could
inform
us
on
Internet
of
Things
actions
and
what
it
is
used
for.
Well,
no
surprise
distributed
an
Arab,
Service,
crypto
tracking
at
frozen
and
other
methods.
So
so
we
expect
this
to
continue
next
next
year.
H
The
new
comer
around
these
categories
is
cloud
care.
Environments
are
now
becoming
more
and
more
on
their
attack.
We
could
see
that
on
AWS
as
your
community
stalkers
are
less
applications.
So,
of
course
we
we
know
that
AWS
s3
buckets
got
big
leak,
but
we
have
seen
a
lot
of
data
breaches
going
through
cloud
environments.
H
One
of
the
issue
that's
very
concerning
is
Venera
will
never
abilities
in
Hardware
chips.
We
all
know
of
well
down
and
spectra,
but
there
are
others
that
happen.
I
think
the
concerning
one
is
actually
Sdn
exploits
that's
going
to
to
be
interesting
with
many
operators
and
other
bodies
building
massive
infrastructure.
H
We
are
going
to
have
an
attack
surface.
That's
going
to
increase
on.
Of
course,
people
will
be
try
to
attack
there.
The
the
most
concerning
one
will
be
a
silent
attack.
You
are
bridge
and
you
don't
know
that
that
would
be
terrible.
Okay,
let's
continue
we
finish
with
this
one.
So
we
finish
with
two
categories.
So
targeted
attacks
is
not
new.
We
know
we
have
it
since
quite
some
time
now.
H
In
fact,
we
debated,
if
is
run
somewhere
or
targeted
attack
or
not.
We
could
because
that
would
be
good
topic
for
smart.
Actually,
of
course,
they
remain
undetected
as
long
as
possible.
They
are
long
attacks.
Spearfishing
remains
a
primary
vector,
so
in
65
percent,
but
the
interesting
is
that
the
main
motive
is
intelligence.
Gathering,
so
96%
by
logic
and
equity
groups
are
going.
You
know
after
ICS
and
IOT.
The
other
observation
is
that
they
are
not
using
as
many
zero
days
winner
abilities
as
before.
I
think
it
declined
to
23%.
H
And
finally,
we
have
living
of
the
long
types
of
attacks
and
supply
chain
attacks.
Here,
the
most
surprising
one
was
the
use
of
malicious
PowerShell
scripts
that
increased
by
1000%,
so
they
are
really
insisting
on
that
and
the
other
concerning
numbers
are
office,
files,
48
persons
of
malicious
email,
attachments
from
5%
in
2017
and
so
pension
attacks
with
by
17%
in
2018,
and
that's
it
for
now.
Thank
you
very
much.
E
H
With
attacks
on
the
supply
chain
in
the
supply
chain,
we
know
we're
talking
about
a
hardware
supply
chain
software
supply
chain.
All
of
the
it's
a
composite
number
I
mean
the
the
the
report
is
pretty
big
on
the
breakdown,
so
we
would
need
to
I
mean
I'm
happy
to
discuss
with
each
of
you
applying
for
each
of
the
numbers,
but
it's
60
pages
and
each
page
contains.
E
E
I
Yes,
my
name
is
Karim
from
the
Center
for
Internet
and
Society.
I
have
two
questions,
one
about
the
form
jacking
attacks,
which
vulnerability
of
lower
the
exploiting.
When
you
know
inserting
the
scripts
were.
Was
it
a
cross-site?
Scripting
I
mean
cross-site
script
injection
or
did
they
have
other
access
to
the
server?
Somehow
that's
the
first
question,
and
the
second
question
is
why
the
fake
new
slash
misinformation
bit
was
included
as
a
trend
under
smartphones.
H
Came
it
also,
the
second
one?
You
are
white,
it's
difficult
to
put
60
pages
into
orchid
categories,
I.
Think
the
deponent
social
media
is
easy
to
an
attack
of
the
deep
on
the
device
or
is
it
on
a
second
brain
I
would
argue
it's
an
attack
on
the
brain.
In
this
case
it
should
be
a
separate
category,
but
we
just
wanted
to
be
mindful
that
this
is
concerning
and
growing,
and
so
we
just
put
it
in
a
category.
H
That's
that's
all
what
it
meant:
there's
nothing
wrong
and
the
foundation
question
so
for
the
phone
jacking
question
its
if
I
remember
correctly
in
the
report,
these
are
attack
led
by
match
cards
or
something
ladies
I
forgot.
The
attack,
the
attack
type
so
I
would
have
to
really
look
with
this.
It's
JavaScript
based
most
real-time,
but
on
the
cross,
you
guys
from
the
cross-site
right
yeah
I
would
have
to
really
check
what
it
what
it
really
means.
If
you
have
captured
that
or
not
okay
and.
A
J
J
H
There
is
a
lot
of
constituencies
around
us
governments
and
surprise
consumers
that
have
all
sorts
of
concerns,
and
so
we
prepare
this
document
to
make
sure
at
least
the
understand
which
trends
we
are
so
example
run
somewhere
is
now
the
decline,
and
so
we
are
trying
to
give
trends.
We
train
12
people
to
understand
if.
A
B
H
K
Here
and
I
think
the
co-chair
just
actually
touched
on
my
question,
which
is
these
are
interesting
numbers
and
the
question
is:
what
should
a
research
group
here
do
with
them?
What
is
you
have
this
gentleman
presents
so
now
the
question
is,
yes,
I
think
it
was
motivating,
but
what
is
what
is
it
we're
supposed
to
take
from
this?
As
his
motivation.
A
First,
that
was
Elliot
Lear,
oh,
he
did
I
I
did
not
hear
him
say
his
name.
Sorry,
it
could
be
used
by
people
who
are
unaware
of
the
threat
space
to
look
into
something
that
might
be
a
rising
threat
which
Kirsty
it
said
so
I
think
their
team
uses
it.
Their
research
team
uses
it
in
that
regard,
but
others
could
as
well.
This.
L
Is
no
hats
here,
I
think
that
there's
a
research
area
and
also
trying
to
decide
how
to
manage
resources
for
security
for
dealing
with
security
problems
and
there's
you
know,
actually
you
could
get
some
of
the
folks
who
work
on
that
and
work
on
how
you?
How
do
you
think
about
it?
Because
you
have
you
know
all
sorts
of
game,
theoretic
and
other
issues
about?
Is
this?
The
one
I
need
to
put
my
my
resources
in
or
another,
and
how
do
they.
A
Interact
great
and
that's
thank
you.
That's
actually
one
of
the
things
we'd
like
to
hit
on
this
is:
how
do
we
make
information
security
management,
more
efficient
and
researching
that
aspect
right?
So
that's
one
of
the
reasons
for
Kharis
and
that's
one
of
the
reasons
we'd
like
to
push
this
in
the
research
area
because
of
the
2
million
person
deficit
in
professionals
that
I,
don't
think
I'll
ever
be
filled
right,
so
I
think
we
have
to
do
things
differently
and
I
think
it.
A
F
M
F
A
N
O
So
it's
hard
to
read
this
slide
right,
because
I
wanted
to
put
this
beautiful
graphic
of
the
astronomical
clock.
It's
really
cool!
You
should
go
see
it
if
you
haven't
seen
it
yet.
My
presentation
is
entitled
malicious
uses
of
evasive
communication
and
the
threats
they
posed
a
privacy
right.
So
the
important
thing
about
this
is
so
I'm
going
to
be
talking
about
ways
that
evasive
communication
is
used
by
malware
and
I
want
to
explain
the
motivations
of
the
malware
operators.
O
I
want
to
explain
their
goals,
explain
how
they're
using
evasive
communication
and
try
to
provide
some
of
the
perspective
of
say,
an
incident
responder
or
somebody
trying
to
defend
a
lot
of
data,
so
I
want
to
start
off
with
what
I
think
is
a
really
important
statement
here.
Right,
I'm,
not
I'm,
not
dissing,
evasive,
communication
I
want
to
emphasize
right.
You
know.
Privacy
is
a
human
right.
Encryption
is
a
cornerstone
of
modern
society
right,
so
we
should
all
use
encryption
to
provide
privacy.
O
It's
essential
because
of
you
know:
criminals,
corporations,
governments
and
individuals
that
pose
a
threat
to
privacy
and
there's
a
lot
of
you
know
good
and
krypton
protocols
and
evasive
communication
protocols
who
goal
is
to
protect
privacy
and
that's
a
really
good
thing.
So
my
focus
is
going
to
be
not
on
the
protection
of
the
the
data
in
motion.
You
know
say
in
it
in
a
single
session,
which
is
the
focus
of
a
lot
of
ITR
protocols.
O
Instead,
I'm
going
to
be
talking
about
attacks
that
are
more
on
the
information
systems
being
run
by
the
the
client
or
on
the
server
side
by
the
organization
running
all
the
servers
right
so
I'll
call
that
client,
side
and
server
side
in
the
presentation
here
so
I
wanted
to
find
evasive
communication
by
its
technical
goals
right.
So
this
would
be
in
in
in
the
security
sense
right
in
the
security
considerations
section
or
the
research
analysis.
You'd
want
to
define
these
goals
right,
so
data
confidentiality,
you
know
through
encryption,
is
used
to
provide
data
privacy.
O
That's
a
common
placement
on
goal
that
we
all
rely
on
everyday
there's,
also
the
goal
of
evading
blocking.
So
this
is
used
in
censorship,
circumvention
right,
which
is
the
popular,
benign
use,
there's
the
ability
to
visit
a
targeted
site
without
detection
right.
So
in
the
research
community,
people
use
the
phrase
targeted
site,
and
this
is
the
closed
world
model
of
website
fingerprinting.
O
For
example,
if
you
have
your
your
your
in
a
country
that
doesn't
want
you
to
visit
certain
places
on
the
internet
and
they're
wondering
if
you
actually
visit
them
or
not
right,
so
that's
what
I
mean
by
targeted
site
here.
So
the
benign
use
of
this
is
obviously
privacy
and
the
hardest
goal
of
all
in
this
space
is
communication
without
detection.
O
In
other
words,
I
want
to
go
to
a
country
where
they
don't
want
me
to
visit
someplace
on
the
Internet
I'm
going
to
visit
that
place
and
I
don't
want
them
to
even
know
that
I'm
visiting
I'm
using
any
evasive
communication
at
all,
so
that
that's
the
goal
is
to
hide
sore
convention,
and
so
that's
that's
actually
a
very
hard
goal
to
achieve.
So
you
know
these
goals
might
be
applicable
to
a
bunch
of
work
at
the
ITF
domain.
O
Fronting
do
HD
OTE
encrypted
SMI
and
you
know
a
bunch
of
other
things
right
and
maybe
not
all
these
goals
apply,
but
there
are
plenty
of
good,
benign
uses,
and
you
know
we
want
these
capabilities
and
our
protocols
next
slide.
Please
so,
but
unfortunately,
you
know
bad
guys
have
similar
goals
right.
So
you
know
bad
guys
want
to
use
data
confidentiality
to
hide
their
activities
from
the
people
defending
the
information
systems
they're
attacking
they
want
to
evade
blocking.
O
So
the
malware
can
carry
out
its
fundamental
mission,
which
would
be
the
infection
command
and
control
and
data
exfiltration
most
often,
and
they
want
to
be
able
to
visited
target
sites
without
detection,
because
this
enables
them
to
minimize
what
are
called
the
indicators
of
compromised
right.
So
an
IOC
is,
you
know,
for
example,
there's
there's
lists
of
IP
addresses
that
are
known
to
be
compromised
and
and
or
hosting
malware
servers
right.
O
So
there's
literally
millions
of
those
things,
but
there's
millions
of
infected
sites
right,
but
the
you
know,
tens
of
thousands
of
known,
active
bad
servers.
You
can
download
from
several
security
vendors
that
sort
of
thing
so
malware
wants
to
be
able
to
avoid
that
right.
They
don't
want
to
show
up
on
an
IP
watch
list
and
then,
of
course,
the
major
goal
for
any
you
know
really
serious
attacker
is
hide
the
fact
that
you've
infected
your
victim.
So
you
know
there
are
malicious
users
for
these
goals.
O
So
let
me
talk
a
bit
about
client-side
attacks
on
privacy.
Now
I
want
to
focus
entirely
on
privacy.
Okay,
you
know
we
heard
earlier
about
you
know
some
security
concerns
and
I
don't
want
to
I,
don't
want
to
think
about
security.
I
want
to
focus
on
privacy
of
individuals
right
now.
Security
technologies
are
used
to
protect
data
and
including
personally
identifiable
information,
but
on
the
client
side,
it's
a
there's,
a
more
direct
threat
to
privacy.
O
Rats
provide
the
ability
to
remotely
survey
the
electronic
activities
of
a
victim
by
keylogging
remote,
desktop
viewing
webcam,
spying,
audio
eavesdropping
data
filtration
and
more
okay.
But
please
note:
this
quote:
is
from
a
citizen
lab
report
on
how
citizens
were
targeted
by
a
repressive
government
right.
This
is
not
some
kind
of
claptrap
from
some
vendor
giving
a
pitch
at
RSA
right.
This
is
from
this
quote
about
the
capabilities
of
this
tool
is
specifically
about
somebody
trying
to
defend
privacy
next
slide.
Please
so
Syria
provides
an
unfortunate,
in-depth
look
at
this
sort
of
thing.
O
O
It's
a
you
know
provides
like
a
VPN
service,
so
that
thing,
an
effective
free
gate,
client
was
propagated,
I
think
other
evasive.
Communication
applications
were
used
in
this
sort
of
attack,
and
so
it's
really
well
documented
for
Syria,
but
you
can
see
the
same
thing
against
activists,
lawyers
and
journalists
in
countries
like
Mexico
are
sorry,
Jean,
Egypt,
UAE
and
other
places
in
some
cases
I'm.
O
You
know
these
are
where
one
country
wants
to
spy
on
its
neighbors
or
something
like
that,
but
these
are
all
serious
threats
to
privacy,
and
so
the
important
thing
is
here
that
evasive
communication
is
critical
to
how
well
rap
operates
right.
So
if
you
want
to
prevent
this
sort
of
thing,
you
need
somebody
to
be
watching
for
this
sort
of
exfiltration,
and
you
know
somebody
doesn't
have
to
be
the
network.
It
could
be
if
you
trust
your
OS
vendor,
if
you
trust
some
other
application
provider.
O
You
know
if
you
trust
your
ISP
or
you
know
it's
some
third
party
or
whatever
right,
but
you
know
somebody's
somebody's
got
to
be
paying
attention
next
time
so
to
provide
some
documentation.
You
know
you
know,
I,
encourage
you
to
look
up
these
articles
if
you're
interested.
Frankly,
it's
really
depressing
to
research,
this
kind
of
thing,
but
there's
a
lot
of
good
documentation,
and
so
the
boy
next
door,
who
accidentally
built
a
Syrian
spy
tool,
is
a
really
good
read
if
you
haven't
seen
that
one
and
maybe
I'll
just
move
on
to
the
next
slide.
O
So,
let's,
let's
forget
about
the
the
client
site
for
now
and
think
about
the
server
side.
When
I
say
server,
side
I,
don't
just
mean
the
fit.
You
know
a
single
server
right,
I'm
talking
about
the
data
center.
Probably
that's
holding
you
know
a
lot
of
data.
It
might
have
you
know
three
tiers
in
it.
You
know
it's
going
to
be
complex.
It's
gonna
have
a
lot
of
devices
there.
O
It
might
be
powering
a
cloud
provider,
it
might
be
in
a
content,
distribution
network.
It
might
be
in
an
enterprise
right,
but
this
is
what
I
mean
by
server
side
right.
There's,
there's
big
complex
systems
that
that
hold
personally
identifiable
data
and
other
privacy
relevant
data,
so
I've
got
a
quote
here:
I
want
to
share
with
you
first.
Let
me
explain:
this
is
from
Rob
Joyce
of
NSA's
tailored
access
operations
unit
right.
O
So,
if
you're
paranoid
about
the
NSA-
and
you
don't
know
what
CIO
is,
then
you
should
learn
about
them
and
then
you
should
be
even
more
paranoid.
So
this
is
a
quote
from
a
great
wired
article.
They
covered
his
his
talk
at
I
forget
if
it
was
blackhat
I
think
it
was
blackhat
a
couple
years
ago.
So
what
he
said
was
with
any
large
network.
I
will
tell
you
that
persistence
and
focus
will
get
you
in
will
achieve
that
exploitation
without
0
days.
O
There
are
so
many
vectors
that
are
easier
or
less
risky
and
often
more
productive
than
going
down
the
zero
day
route.
So
here's
somebody
telling
us
you
don't
even
need
the
sophistication
of
zero
day.
You
just
need
to
be
persistent.
You
need
to
understand
your
victims
network
better
than
the
victim
does
right.
That's
that's
actually
what
he
says
in
that
article,
so
we
should
really
be
concerned
about
this
right.
O
We
should
be
concerned
that
it's
hard
to
defend
something
like
a
data
center
and
the
confidence
that
this
gentleman
has
and
in
how
easy
it
would
be,
and
of
course
he
wants
the
world
to
improve
its
security
posture.
So
it
was
nice
that
he
gave
that
time.
Okay,
so
I
don't
have
a
lot
of
detailed
references
about
the
server-side
threats
to
privacy,
but
let's
see
so
here
instead
I
have
a
graphic
there.
There.
O
It's
too
difficult
to
count.
This
is
actually
from
I.
Have
the
citation
on
the
slide?
I
think
it's
just
not
legible
right
here,
but
it's
it's
an
infographic
that
represents
breaches
due
to
hacks.
In
other
words,
there
was
actual
malware
involved
as
opposed
to
just
some
sort
of
accident
and
to
emphasize
you
know
what
I
want
to
provide
a
concrete
example
that
relates
this
to
privacy.
So
there
were
three
major
hacks
of
dating
sites,
including
the
friend
finder
Network
and
Ashley
Madison
and
another
one
that
exposed
hundreds
of
millions
of
people.
O
Homosexuality
is
illegal
in
70
countries
and
the
the
personal
impact
of
being
one
on
one
of
those
accounts
that
was
hacked.
What
would
be
a
there
would
be
a
serious
privacy
violation,
and
so
you
know
I
think
we
need
to
think
about
the
privacy
issues
here,
not
in
terms
of
there's
some
big,
dumb
vendor
that
that
values,
security
and
not
privacy.
No
there's
there
are
big,
dumb
vendors
that
want
to
use
security
to
defend
personally
identifiable
information,
so
next
slide.
O
So
here's
a
quick
slide
illustrating
how
important
communication
is
when
somebody
wants
to
attack
a
something
like
a
data
center
there's,
usually
an
initial
infection
that
involves
communication,
the
secondary
download
of
the
tools
actually
used
in
the
exploit
like
a
rootkit
or
whatever,
there's
command
and
control.
That's
ongoing
data
exfiltration
and
then
there's
often
some
sort
of
exploitation
of
the
data.
Like
you
learned,
somebody's
account
information
next
slide
and
stolen
passwords
are
a
really
big
deal,
of
course,
in
privacy
and
in
in
server-side
security
right.
O
So,
if
you
haven't
looked
yourself
up
on,
have
I
been
poned,
you
know
you
know,
please
do
so.
So
let
me
show
some
data
on
malware's
use
of
evasive
communication.
So
this
is
the
the
original
content
and
I'm
gonna
credit.
My
colleague
blake
anderson
for
this
work,
so
here
we're
reporting
data
that
so
we've
worked
with
people
who
operate
a
commercial
malware
sandbox.
We
have
something
like
six
months
of
data
where
malware
completed
some
just
south
of
two
million
TLS
sessions.
So
a
a
malware
sandbox
is
a
place
where
samples
are
submitted.
O
Where
somebody
thinks
something
is
malicious,
the
sandbox
will
let
it
run
and
basically
wants
it
to.
You
know:
try
to
engage
in
malicious
activity,
that
activity
is
observed
and
and
of
course
it's
observing
in
a
fishbowls.
So
you
can
see
exactly
what
sort
of
network
activity
it
does
you
can
you
can
get
a
strong
conviction
that
it
is
actually
malicious
in
this
case
we're
excluding
adware
right.
So
if
you
actually
look
at
so-called
malware
a
lot
of
it's
actually
adware,
so
it's
you
know.
Is
that
really
malicious
right?
O
It
just
wants
to
sell
somebody's
clicks,
so
we're
excluding
adware
and
we're
only
reporting
malware
that
has
very
strong
convictions
on
it.
In
other
words,
you
multiple
a
be
signature
hits.
So
what
we
observed
is
that
something
less
than
10%
of
these
TLS
sessions
are
actually
using
some
kind
of
dedicated
evasive,
communication
technology
right
so
like
tor,
ultrasurf,
psyphon,
so
on,
there's
also
non-standard
ports,
right,
which
you
know
you
might
consider
that
evasive.
O
O
Malicious
actors
are
absolutely
using
evasive
communication
technologies
and,
if
you're
in
the
business
of
hunting
malware,
you
would
look
at
this
and
say
you
know,
there's
these
things
that
somebody's
going
to
the
trouble
of
using
some
phaser
communication
technology-
probably
more
interesting,
even
because
it's
rare
right,
but
it's
less
than
10
percent.
So
you
know
malware
most
malware.
Authors
don't
feel
the
need
to
do
this,
but
certainly
there's
some
awareness
of
how
to
engage
in
this
sort
of
activity
so
and
just
for
completeness.
O
O
So
I
want
to
kind
of
sum
all
this
up
with
a
with
a
scenario:
I
want
to
characterize
this
as
blowback
against
privacy
right
so
blowback
in
in,
like
the
military
intelligence
term
means
when
you
know
like
when
you
use
a
weapon
and
then
like
the
there's,
something
like
in
this:
the
discharge
of
gases
that
blows
back
on
on
the
person
who
pulled
the
trigger
and-
and
that
has
the
some
harm
right.
So
in
other
words,
it's
an
unintended
consequences
that
that
gives
you
the
opposite
of
what
you
wanted
right.
O
So
suppose
we
have
somebody
who's,
an
individual
who
uses
a
privacy
network
to
visit
some
place
on
the
Internet,
evading
censorship
and,
and
then
somebody
uses
the
privacy
network
to
exfiltrate
data
from
the
data
center
right.
So
they
get
there.
Some
bad
person
gets
a
hold
of
the
PII
using
some
sort
of
evasive
communication
right
so
I
mean
this
literally
could
use
the
same
privacy
network.
O
So
that's
an
unintended
consequence
and
the
you
know
our
li
er
used
the
homosexuality
example
and
it
would
be
easy
to
come
up
with
new
stories
about
religious
minorities,
democracy
advocates
and
journalists.
So,
regardless
of
what
your
you
know,
political
leanings
are:
there's,
there's
a
victim
out
there
that
could
fit
this
scenario.
So
I
wanted
to
highlight
research,
questions
right
because
I
think
you
know
the
IRT
F
is
a
great
institution
and
in
the
context
of
you
know,
what
could
we
bring
to
the
IRT
F
here?
Are
some
research
questions?
O
O
So
can
evasive
technologies
prevent
malware
from
utilizing
their
services
without
reducing
privacy
for
other
users
right
in
some
ways
this
this
is
it
right.
I
mean
that's
exactly
what
we
want.
We
want
there
to
be
no
blowback
when
we
help
individuals
achieve
privacy,
can
an
evasive,
client
or
os
provide
a
strong
assurance
of
the
absence
of
malicious
or
unauthorized
communication
to
its
user
or
administrator.
So
this
this
exactly
would
help
those
those
people
targeted
in
the
Syrian
news
reports
right.
O
C
P
Stephan
Pyles
take
the
place
of
presentation.
If
you
go
back
to
your
research
questions,
so
I
can
only
imagine
if
there
are
answers
to
some
of
these.
They
might
be
answered
on
different
time
scales.
So
some,
for
example,
you
know
I
mean
I,
think
the
answer
to
the
first
one.
That
seems
to
be
yes,
that
there's
work
that
could
be
done
there
I
clears
the
answer,
but
the
second
one
isn't
no,
but
if
it
is
not
know,
then
I
suspect
that
may
take
a
long
time
for
that.
P
O
O
N
C
E
Thanks
for
the
presentation
for
the
research
that
went
into
it,
I
would
be
curious.
It's
follow
up
with
you
later
about
some
of
the
details
of
the
numbers
that
you
got
and
where
that
one
of
my
big
concerns
here
is
how
we,
as
network
designers,
determine
malicious
uses
from
beneficial
uses,
because
I
want
to
recognize
that
one.
E
You
know
one
person's
subversive
terrorism
is
another
person's
seeking
in
no
democracy,
advocacy
and
I.
Think
that's
a
very
difficult
challenge
to
try
to
address
here.
I
fully
agree
with
you
that
that
the
massive
amounts
of
data
stored
on
cloud
servers
on
behalf
of
people
is
represents
real
privacy
risks,
but
I'm
not
convinced
that
the
network
protocols
are
the
way
that
to
like
that.
The
network
protocols
can
distinguish
between
those
cases,
and
so
I
I'm
happy
to
continue
talking
about
this
and
trying
to
figure
out
how
we
can
distinguish
between
those
cases.
O
So
I
think
you
don't
want
one
question
I
put
forward
would
be
if
you're
defending
a
data
center
that
has
a
lot
of
personally
identifiable
information
on
it.
Would
you
allow
unrestricted
unmonitored
tor
communication
from
the
entire
data
center
out
to
the
Internet,
and
if
the
answer
would
be
no,
you
know
you
know,
do
we
want
to
have
a?
How
would
you
want
to
enable
that
and,
like
you
say,
we
should
follow
up?
Maybe.
B
Q
Okay,
hi
everyone.
My
name
is
snoop
my
name's
Simon
Edwards
I'm
from
SC
labs,
but
also
the
anti-malware
testing
standards
organization.
Very
briefly,
what
that
is
is
all
of
the
antivirus
companies
you've
heard
of
I
get
together
with
people
like
me
who
test
their
products
and
massive
the
argue
about
how
we
test
their
products,
we're
a
London
testing
lab.
That's
the
building
we're
in
currently,
and
we
work
with
those
vendors,
but
also
those
vendors
customers.
Q
So
if
you're,
a
very
big
company,
you
might
say
well,
why
would
I
spend
a
million
dollars
on
silence
or
Symantec
or
whatever,
and
they
talk
to
us
would
help
them.
Do
some
proof
of
concept
testing
that
kind
of
thing
we'll
look
at
endpoint
software,
so
anti-malware
EDR
that
kind
of
thing,
but
also
cloud
services.
So,
if
you're
going
to
protect
your
organization
by
putting
an
email
gateway
in,
for
example,
maybe
in
the
cloud
maybe
on
Prem,
we
can
look
at
that
kind
of
stuff,
too,
and
also
big
Network
boxes
as
well.
Q
The
reason
that
we're
doing
this
work
is
because
of
marketing,
basically
there's
no
philosophy
around.
It
was
bad
in
the
90s
and
the
2000s.
It
got
worse
when
the
next-gen
antivirus
guys
came
along
and
said
that
AV
is
dead.
What
you
really
want
is
machine
learning,
artificial
intelligence
which
has
been
around
for
ages
anyway,
and
when
people
charging
millions
of
dollars
for
those
kinds
of
products,
we
think
it's
fair
to
actually
test
them
and
see
if
they
do
what
they
claim
to
do.
Q
One
really
important
thing
about
testing
from
in
my
opinion
is:
it
has
to
be
realistic
if
you
just
grab
a
bunch
of
malware
samples
from
virustotal
or
even
if
you
write
your
own,
that's
not
necessarily
what
the
real
customers
are
seeing
in
the
real
world.
So
there's
lots
of
reasons.
I
could
spend
half
an
hour
talking
about
virustotal
and
why
you
shouldn't
use
it
for
testing
so
I
won't.
Q
What
I
would
say
is
the
most
important
point
to
get
across
is
to
do
the
full
attack
chain
when
you
do
a
test,
if
you
just
run
a
VM
and
drag
a
piece
of
malware
onto
the
desktop
of
that
virtual
machine
and
your
next-gen
center
one
or
whatever
detects
it.
That's
not
very
realistic,
because
victims
in
the
real
world
aren't
doing
that.
Well,
they're
doing
a
downloading
emails
or
visiting
malicious
websites.
Different
vectors
are
available
for
malware
and
other
types
of
attacks.
Q
We
should
use
those
when
we
do
testing
and
what
we
shouldn't
do
is
to
trust
what
the
products
say.
So,
if
I'm
doing
a
test
and
Symantec
endpoint
protection
says
I've
detected
a
threat.
Well,
it's
detected
a
threat
great,
but
then
it
might
claim
this
protect
it
against
that
threat.
But
we'd
have
to
use
forensics
to
prove
that
because
we
can't
trust
the
products,
and
so
we
do.
We
use
forensics
digital
forensics.
Q
Well
that
we'll
analyze
memory
we'll
look
at
the
changes
on
the
file
system,
network
traffic,
that
kind
of
thing
and
once
we've
identified
a
problem,
we
don't
just
say
bad
luck,
we're
gonna,
write
a
mean
report
about
you
or
well
done.
Here's
a
nice
badge
which
we
do
do
as
well.
We
will
also
provide
evidence
the
goal
being
to
improve
the
product.
If
we
found
problems,
those
problems
should
be
solved.
So
it's
on
us
to
provide
enough
evidence
to
prove
we're
not
making
it
up
and
enough
information,
so
they
can
fix
the
issue.
Q
Q
If
I
think
it
should
be
a
signature
based,
detection
thing
and
I
ignore
the
URLs,
then
your
product
will
fail
hard,
even
though
in
the
real
world
who
would
do
very
well
so
a
test
has
to
be
flexible
enough
to
take
into
account
all
the
possible
approaches
and
that
a
vendor
might
make
or
take
which
sounds
impossible
until
you
think.
Well.
Why
don't
we
just
act
like
real
attackers
and
then
all
that
problem
problematic
stuff
goes
away.
Q
You
could,
for
example,
do
a
test
with
millions
of
virus
samples
dating
back
to
1983
I,
don't
know
how
useful
that
would
be
most
of
those
wouldn't
execute
on
Windows
10
anyway.
So
why
don't
we
just
take
a
few
targeted
attacks
that
we
roll
up
in
the
last
year
or
so
and
see
how
they
do
because,
after
all,
that's
what
this
products
say,
they're
going
to
protect
you
against
I
think
prevalence.
Q
That's
a
really
important
if
I
go
and
find
all
sorts
of
edgy,
weird
things
that
doesn't
really
tell
you
how
product
is
going
to
protect
customers,
but
if
I
read
the
internet
threat
reports
from
Symantec
and
other
vendors.
If
my
team
go
out
and
find
stuff,
that's
affecting
customers
in
real
time,
which
is
exactly
what
we
do,
then
we
can
see
what
happens
when
we
expose
the
products
to
the
threats,
but
what
we
shouldn't
do
and
no
one
should
do
this-
is
to
take
a
feed
of
threats
from
any
vendor.
Q
So,
for
example,
if
I'm
going
to
compare
some
I'm
gonna
pick
on
Symantec
again
Symantec
with
Kaspersky
Lab,
only
because
I've,
two
guys
in
the
room
from
Symantec,
if
I'm
going
to
compare
those
two
companies,
products
and
I
take
samples
from
Symantec.
Clearly,
the
test
is
biased
if
I
take
them
from
Kaspersky,
it's
biased
if
I
take
it
from
both
of
them.
It's
biased
because
we're
testing
an
industry
on
what
it
already
knows,
and
that's
no
good
well
how
about
I
take
a
feed
from
McAfee
because
they're
not
in
this
test?
Q
Well
the
problem
there
is
that
all
of
these
companies
share
threats.
They
might
not
share
all
of
the
threats
with
each
other,
but
they
do
share
threats,
so
maybe
mcafee
and
kaspersky
has
a
really
nice
sharing
relationship.
So
now
the
test
is
biased
in
favor
of
Kaspersky,
it's
impossible
to
know
about
all
those
background
sharing
relationships,
which
is
why
it's
not
sensible
to
take
a
feed
from
vendors
and
it's
not
necessary
either
advanced
persistent
threats
are
really
interesting.
Targeted
attacks
are
really
interesting.
You
saw
earlier,
though,
as
I
mention
of
PowerShell.
Q
We
see
loads
of
PowerShell
stuff
at
the
moment
and
lateral
movements
important
as
well.
If
you're
testing
a
product
like
CrowdStrike
or
silence,
they
might
not
detect
the
malware.
That
first
hits
the
system.
But
when
you
start
misbehaving,
remember
full
attack
chain,
malwa
comes
along
bad
things
happen.
I
guess
a
reverse
shell
I
start
dumping
hashes,
whatever
the
minute
I,
maybe
pivot,
to
a
different
system.
That's
when
undetected,
but
if
you
just
stop
the
test,
when
you
get
the
reverse
shell
and
say
oh
well,
I
could
do
something
bad,
so
the
products
failed.
Q
That's
not
really
fair!
That's
when
the
test
starts
not
when
it
finishes.
I'm
just
gonna
skip
the
next
bit
so
move
on
from
that.
Please
don't
worry
about
that
next.
Thank
you.
So
is
it
possible
for
someone
like
me
or
someone
like
you
to
like
an
apt
fire,
I
didn't
used
to
think
so
they
said
you
have
to
actually
be
a
cyber
criminal
or
a
nation-state
organization.
To
be
an
apt,
that's
nonsense!
Q
There
are
lots
of
different
ways
to
be
an
apt
and
lots
of
different
types,
because
not
every
government
or
not
every
nation
state
has
the
same
level
of
resources.
If
you
want
to
hit
Mossad
against
Hamas,
they
have
different
levels
of
resources,
Mossad,
perhaps
less
than
Hamas
no
other
way
around
other
way
around.
Q
We
already
heard
that
people
are
using
less
zero-days.
There
is
that's
because
there's
no
need
to-
and
we
already
heard
that
quote
about
Rob
Joyce-
about
the
NSA
I'm,
using
persistence
rather
than
zero
days.
There
are
actually
reasons
not
to
use
zero
days.
I
know
a
guy
that
used
to
run
a
UK
intelligence
agency
who
up
and
asked
him.
Do
you
guys
use
Metasploit
when
you
hack
other
countries,
he
said?
Well,
we
don't
pack
other
countries,
obviously,
but
if
we
did,
we
would
use
these
frameworks
because
everyone
knows
how
to
use
them.
Q
So
if
you
see
strange
Chinese
characters
or
if
you
see
the
CNCs
are
being
run
during
Philippine
working
times,
all
these
little
clues
can
give
you
away
if
you
download,
Metasploit
or
just
the
Kali
Linux
distribution
or
you
can
do
loads
without
having
to
get
all
Stuxnet
on
everyone's
ass
I
might
sort
of
skip
this
bit,
because
we've
already
done
that
right.
Ok
bit
rude
this
one
hacking
team,
Italian
company,
bit
offensive
literally
in
many
ways
they
sell
malware
to
governments
for
surveillance
purposes.
Q
They
claim
only
to
work
for
nice
governments,
whatever
that
means,
and
so
what
they
did
was
sell
nasty
stuff
to
not
very
nice.
Governments,
I
think
we'd,
probably
agree
the
ones
that
kind
of
torture,
distance
and
things
like
that.
So
what
was
I
going
to
say
about
that
yeah?
So
this
guy
or
girl,
Phineas
Fisher,
hacked
hacking
team
and
leaked
all
of
the
information.
So
you
can
see
invoices
you
can
see
how
much
it
costs
you
see
how
thoroughly
incompetent
the
different
government
employees
were
when
it
came
to
troubleshooting
the
stuff.
Q
The
support,
if
you
want
the
laughs
download
it,
because
the
support
tickets
are
really
funny,
but
anyway,
hacking
exposed
came
out
that
second
edition
in
2002
and
fifteen
years
later
finished,
Fisher
hacks,
hacking,
team
and
a
year
later
he
or
she
published
how
they
did
it
and
it
might
as
well
just
be
from
the
last
page
of
hacking
exposes
an
absolute
playbook
of
how
to
do
it.
It's
just
hacking.
Anyone
can
do.
It
is
quite
easy.
Q
This
is
a
concept
that
we
came
up
with
a
long
time
ago
to
try
and
fend
off
fire
eyes
view
that
you
can
turn
be
an
apt.
You
could
have
someone
down
that
end.
Who
has
zero
resources
and
someone
at
the
other
ends
someone
who's
capable
of
writing
a
Stuxnet
type
malware
framework
and
people
in
between,
but
actually
the
reality
is
even
if
you
can
make
Stuxnet,
you
will
be
doing
all
of
these
other
things
as
well.
Q
Q
Mentioned
earlier
about
how
you
actually
have
to
do
the
bad
behavior?
That's
because
a
breach
is
a
process,
it's
not
just
a
malware
infection.
So
if
I
can
install
malware
on
your
laptop,
that's
not
a
breach,
because
maybe
I
can't
connect
to
your
laptop.
Maybe
when
I
do
connect
to
your
laptops
and
software
will
stop
me
escalating
privileges
which
I
need
to
do
if
I'm
going
to
steal
your
encrypted
passwords
or
put
a
key
logger
on
that
kind
of
thing.
So
you
have
to
go
through
all
of
those
stages
of
the
attack.
Q
The
full
attack
chain
be
like
a
real
attacker
times
troubling
or
five.
Okay,
rootkits
are
annoying
next,
please.
If
you
want
to
test
for
rootkit
and
removal,
you're
gonna
have
to
dump
memory,
there's
a
$10,000
way
of
doing
it
and
there's
three
to
three
ways
of
doing
it.
We
do
the
free
ones.
You
probably
want
to
use
both,
because
when
you
have
an
infected
system,
one
or
two
of
these
will
just
break.
Q
So
if
you
just
click
once
please,
what
you
can
see
there
is
a
file
was
added
to
the
system
in
the
local
folder
called
servitor
XE
in
pls
a
run
entry
was
made,
which
meant
that
that
program
would
run
every
time
the
Windows
PC
was
rebooted.
And
finally,
we
have
a
firewall
rule
that
will
allow
an
incoming
connection
to
serve
it
and
it
really
was
called
server
dot.
Xe
I
mean
usually
they
kind
of
hide
a
little
bit
and
call
it
else
ass,
but
ok
whatever
next,
please.
We
can
skip
that.
Q
This
is
a
memory
dump
from
that
system.
So
when
they
said
oh
well,
we
weren't
infected
what
you
can
see
there.
Those
two
green
lines,
those
that's
the
anti-malware
products
running
silence
by
the
way
and
the
yellow
thing
at
the
bottom
is
the
memory
dumping
tool
that
we
were
running
at
the
same
time
and
if
you
just
click
once
please
that's
the
malware
running.
That's
the
evidence
that
it
really
was
running
yeah.
Q
We
you
have
to
test
to
this
extent
and
generate
this
much
evidence
for
people
to
actually
believe
that
their
god-given
products
aren't
a
hundred
percent
perfect
skip
please
and
again:
ok,
no,
let's
not
do
the
rootkit
stuff,
let's
just
get
through
this!
Please
thank
you.
Keep
going
this
bits
really
interesting!
Sorry,
ok,
so
yeah
we
would
use
memory,
dumping
techniques
and
all
sorts
of
other
bits
and
bobs,
regardless
of
whether
there
are
workers
involved
or
not
simply
because
we
don't
always
know
if
they're
going
to
be
rootkits
involved
or
not
you
can
analyze
offline.
Q
The
problem
of
rootkits
by
the
way
is
the
whole
system
is
compromised.
So
if
you
start
analyzing
on
that
system,
well,
if
the
operating
system
doesn't
know
that
there's
malware
on
it,
how
are
you
going
to
know
and
how
your
tools
going
to
know,
so
you
have
to
take
it
offline
and
test
and
check
the
data
somewhere
else.
Q
Here's
an
example
of
a
targeted
attack
and
I,
probably
don't
have
time
for
the
demo
doing
so
we'll
just
you
have
to
use
your
imaginations
right
so
at
the
very
beginning,
if
we
could
just
come
back
a
little,
please
we've
got
an
attacker
and
he's
generated
a
malicious
PDF
which
is
going
to
be
sensed
by
a
realistic
vector,
in
this
case
an
email
to
the
recipient.
Now,
if
you
have
an
email
gateway
at
that
stage,
that
thing
might
enter
sandbox
of
some
sort.
Q
It
might
see
some
bad
behaviors
and
it
might
be
blocked
to
not
get
through,
but
unfortunately,
in
many
cases
as
we
notice
we
test
email
gateways,
it
will
get
as
far
as
the
user.
At
this
stage,
you're
hoping
he
or
she's
trained
well
enough,
not
to
open
it,
but
most
of
us
believe
that
people
want
to
do
their
job
properly,
so
the
person
will
open
it
because
he
wants
to
get
stuff
done
and
then
the
raccoon,
for
some
reason,
that's
supposed
to
be
an
exploit.
Q
Q
So
at
that
stage
we
can
detect
you,
but
that
I
mean
that's
right
at
the
end
of
the
chain.
There
are
so
many
opportunities
along
there
for
different
products
to
have
stopped
the
threat
and,
if
you're,
relying
on
the
endpoint
to
do
it,
that
really
is
like
the
last
ditch
the
demo
that
we're
not
going
to
show
you
but
I'll.
Just
very
quickly
tell
you
what
happens.
Imagine
that
attack
against
Windows
10
for
the
updated,
Windows
Defender,
fully
updated.
It
didn't
register
anything.
Q
We
completely
hacked
the
system
and
right
at
the
end,
all
the
logs
are
there,
though
in
Event
Viewer,
you
just
run
clear
event
and
they
all
just
disappear
and
you're
left
with
one
event
which
says
events
were
cleared,
which
is
really
cool.
So
if
we
just
quickly
skip
again
so
I
can't
show
you
that
when
we
did
that
testing
lots
of
next-gen
products
didn't
detect
it
lots
of
traditional
ones,
didn't
detect
it.
Some
did
and
yeah
it's
just
not
very,
very
healthful
ready.
Q
So
we
went
back
to
all
of
those
companies
and
said:
look
you're
missing
all
these
things.
Please,
please
fix
it
and
they're
all,
as
you
can
imagine
initially
very
angry
and
then
very
grateful,
testings
really
useful.
You
need
to
know
when
you
buy
the
stuff,
if
it's
any
good
or
not
it's
very
expensive,
and
you
know
if
you're,
if
you're
a
large
mining
company
and
you're
going
to
switch
from
one
vendor
to
another
you're
spending,
at
least
a
million
dollars
and
I'm
spending
a
few
pounds
to
work
out.
Q
A
Q
The
main
message
I'm
trying
to
get
across
is
that
testing
exists
and
diligent
testing
exists
and
if
you
do
the
full
attacked
chain,
when
you
do
a
test,
you
may
I'd
be
able
to
identify,
you
may
or
I
may
be
able
to
identify
areas
in
which
protocols
and
other
things
can
be
improved.
But
my
my
expertise
is
in
being
naughty,
not
in
fixing
stuff.
So
if
I
can
help
by
being
naughty,
then
I'll
do
that.
R
Time
is
done.
You
mentioned
something
very
interesting
in
your
presentation
and
I
wanted
to
go
back
to
pick
up
on
that.
So
you
mentioned
that
you
work
with
different
companies
and
I
know
that
one
of
the
biggest
problems
that
I
observed
the
last
few
years
is
lack
of
collaboration.
They
don't
change
with
themselves.
You
are
in
a
certain
position,
I,
don't
know:
what's
your
contract
with
these
companies
actually
to
unify
this,
and
actually
you
find
from
different,
they
may
not
have
the
protection,
but
you
may
kind
of
open
source
some
ideas.
Q
We
know
about
we're
not
bound
by
any
particular
contracts.
Don't
we
we
often
reuse
data
as
well,
but
there
are
other
things
to
take
into
account.
So
imagine
this
firm,
a
big
mining
company
said
we
want
to
buy
a
web
gateway,
we're
thinking
about
Fortinet
and
we're
thinking
about
Cisco,
and
we
happen
to
know
that
all
of
their
team
are
already
trained
in
fortinet
and
when
we
don't
do
a
test,
there's
not
a
great
deal
of
difference
between
the
two
products.
Q
Our
recommendation
would
probably
be
to
stick
with
what
you've
got,
because
even
if
the
alternative
is
slightly
better,
if
your
entire
team
needs
to
be
retrained,
might
not
do
it
better.
So
it's
more
than
just
plugging
products
in
this.
Well,
it's
quite
complicated
but
yeah
very
happy
to
share
any
knowledge
on
how
different
products
compare.
In
fact,
a
lot
of
our
reports
are
available
for
free
off
our
website.
So
if
you
go
to
su
labs
dot
uk',
you
can
download
the
endpoint
stuff
for
free.
R
Is
it
any
way
that
you
can
actually,
when
you
find
these
things
that
you,
for
example,
are
attack
and
saw
that
you
couldn't
properly
prevent,
but
you
can
explain
to
everybody,
so
the
next
generation
or
next
product
their
product
will
have
that
in
yes,
also
a
very
interesting
service
from
your
your
cos,
you
know
and
understand
your
business,
but
I
think
that
it's
something
that
is
busy
I
was
advocating
for
this.
For
a
few
years,
the
collaboration
I
never
saw
any
real,
so
I
think
you
have
the
potential
company
like
yours.
We.
Q
S
Mcafee
I
have
a
question
on
Windows
10
testing
that
you
had
done
it.
Windows
10
is
using
application
control,
but
the
sandboxing
application
so
that,
even
if
you
download
an
understood
app,
it's
not
gonna,
read
you
or
other
files
and
what
kind
of
attacks
did
you
see
in
that
more?
What
kinds
of
attacks
did
you
seen
that
more
Windows.
Q
What
I
would
say
is
that
two
years
ago
they
turned
on
the
machine
learning
properly,
so
we
were
able
to
get
past
Windows
10
Defender,
with
almost
any
kind
of
targeted
attack.
It
really
didn't
matter
Metasploit
without
evasions.
That
kind
of
thing
it's
still
possible
to
do
it's
much
harder
and
escalating
privileges
is
much
harder
now
than
it
used
to
be
so.
The
video
I
was
going
to
show
you
is
from
two
years
ago,
probably
wouldn't
work
now,
to
be
honest,
I
mean
you
can
do
bits
and
bobs
I'm,
not
sure.
T
T
T
The
one
issue
with
would
you
be
hydrating
is
that
the
public
key
infrastructure
which
supports
the
operations
of
HTTPS
and
TLS
protocols,
is
that
it
depends
itself
on
the
routing
to
be
done
properly
next
slide,
please
and
here's
a
recent
story.
I've
done
that
I've
told
that's
already
in
the
presentation.
Nano
but
I'll
repeat
a
few
slides
in
that.
So
in
April,
2018,
a
cryptocurrency
changed
name
whether
wallets
recorded
an
incident
of
a
hijacking
attack.
T
The
incident
lasted
for
two
hours
until
someone
finally
noticed
the
issue
and
reached
out
to
ice
peace
to
fix
it,
here's
how
basically
basically
worked
yeah
next
slide,
please.
So
you
know,
under
normal
conditions,
browser
starts
a
session
with
an
rb2
website
with
Edina
scuri.
Next,
an
authoritative
DNS
server,
which,
in
case
of
my
other
bullet,
come
used
to
be
Amazon
route.
53
responds
when
I
pee
address
software
requested
domain.
Next
and
after
that,
the
browser
sent
the
HTTP
request
or
better
files
affairs
up
a
TLS
session.
T
With
that
IP
address,
then
the
server
again
posted
in
our
case
in
Amazon
responds
with
that
each
TLS
certificate.
The
signature
of
the
letter
then
being
checked.
So
what
attackers
did
in
that
case
was
a
BGP
hijack
of
route,
53,
IP
prefix,
which
hosted
the
authoritative
server,
and
an
initial
query
would
then
go
to
rock
DNS
server,
which
responded
with
an
IP
address
of
the
server
controlled
by
the
attacker.
After
that,
the
browser
sent
TLS
hello.
T
We
did
which
didn't
go
to
Amazon
this
time,
but
instead
went
to
effect
a
web
browser
in
Russia,
while
all
the
traffic
was
being
sent
to
the
beers.
All
the
requests
were
seeing
it
here,
a
self-signed
TLS
certificate
on
the
remote
side,
so
the
browser
must
have
issued
the
serious
care
warning.
Yet
the
attackers
were
able
to
withdraw
some
creepy
stuff
because
users,
happily
click
through
that
warning
anyway.
T
What's
important
here,
is
that
any
clear
text,
communications
in
this
case
would
have
been
intercepted
and
during
their
HTTP
certificate,
issuance
phase
certificate
authorities
such
as,
let's
encrypt,
usually
communicate
to
their
websites
in
clear
text
there
as
next
slide.
Please
there's
a
comparatively
long
list
of
publications
about
intercepting
HTTPS
with
which
you
be
hijacking
it's
longer
than
that.
Let's
encrypt
has
since
written
a
post
stating
that
they
now
verify
the
website
from
multiple
sources,
multiple
verifying
agents
across
the
Internet
to
address
exactly
the
hijacking
problem.
T
However,
placing
multiple
verifying
agents
on
the
Internet
proves
hard,
because
even
different
IP
networks
in
the
internet
could
have
similar
similar
routing
coverage,
as
if
now
there
was
no
attempt
at
verifying
how
much
of
the
issue
is
gone
now
with,
let's
encrypt,
what's
still
exposed
and
how
the
consensus
is
reached.
In
fact,
a
research
on,
what's
the
the
most
optimal
way
to
place,
those
verifying
agents
might
be
would
be
very
interesting
to
read.
T
The
case
of
my
other
wallet
demonstrates
also
how
the
DNS
infrastructure
is
at
risk,
because
of
hijacking
and
what's
important
is
that
today's
average
website
is
a
very
complicated
piece
of
technology,
relying
on
Citians
databases
as
a
services,
several
less
functions
and
other
hipster
stuff
with
complicated
mechanisms
of
discovery
and
communicates
with
them
via
the
Internet.
At
the
very
least,
you
can
discover
all
those
adjacent
services
through
hijacking,
it's
not
quite
clear
what
could
be
done
at
worst
now,
especially
where
each
server
less
endpoint
authenticates
itself
using
web.
T
Bigeye,
which
is
already
demonstrated
to
be
at
risk
next
slide,
please
what
the
consequences
for
other
particles
could
be.
Here's
a
convenient
way
to
think
of
it.
Imagine
a
global
routing
is
completely
under
the
attackers
control.
What
could
possibly
go
wrong
for
any
mechanism
and
any
piece
of
infrastructure
you
happen
to
run
into
one
example
of
such
mechanism
is
spin
bit
in
quick.
It's
an
effective
way
to
measure
the
latency
between
two
key
or
HTTP
3
beers.
Before
adoption
alone,
discussion
took
place
about
what
the
spin
bit
could
mean
for
privacy.
T
Let's
say:
there's
a
person
behind
the
VPN
who
prefers
not
to
disclose
their
exact
location,
it's
quite
possible,
but
not
entirely
clear.
For
now
that,
with
BGP
hijacking,
to
a
certain
extent,
the
location
of
a
user
could
be
triangulated.
This
area
probably
needs
more
search,
doesn't
make
work.
Okay,
yeah
good
just
in
case
just
in
case
yeah.
So
wherever
a
practical
design
or
an
algorithm
depends
on
a
global
network
to
perform
its
functions.
There's
potential
for
additional
security
analysis
next
question:
this
is
to
address
our
question
in
advance
about
the
authentication.
T
So
next
slide
please
another
yeah,
but
that's
about
the
route.
The
route
objects
next
slide.
Please
yeah!
It
is
a
recent
story
about
disco.
Disco
is
an
experiment
which
was
held
in
January
this.
This
is
an
ongoing
research
on
how
you
can
change
the
bgp
protocol
itself
to
reduce
the
exposure
to
hijacks.
It
was
led
by
researchers
from
Columbia,
Fraunhofer
and
other
universities.
The
ultimate
goal
was
to
use
a
valid
standards-compliant
unassigned,
which
we
attribute
to
enhance
route
origin
validation.
T
The
experiment
proved
itself
very
successful
in
a
way
it
didn't
succeed
in
doing
the
validation
better.
However,
however,
it
has
found
an
open
vulnerability
in
every
router
used
by
some
in
production
that
causes
a
bgp
session
shut
down,
let's
mount
a
CV
advisory
and
the
fix
so
I
personally
declare
that
a
success.
T
A
such
thing,
though,
is
that
apparently
it's
hard
to
conduct
academic
bgp
experiments
in
well
nature,
because
which
should
be
software
present
on
the
internet,
it's
sometimes
old,
sometimes
buggy,
sometimes
old
and
vulnerable,
and
it
which
really
takes
years
to
update
whatever
routing
related
piece
of
code.
There
is
a
good
idea,
then,
to
set
up
a
huge
testbed
with
the
most
common
which
we
implemented,
where
researchers
could
test
their
findings
or
even
a
restricted
model
of
Internet
as
a
sandbox
to
play
with
to
date.
T
T
So
another
piece
of
research
related
to
BGP
hijacking
is
you
know
gal
Rexford
model
of
s
relations.
There
is
a
link
to
the
paper
for
for
anyone
to
look
through
next.
Please,
and
this
is
the
basis
of
work
being
done
currently
in
idea
and
cider
ops.
So
does
anyone
remember
that
XD
strip
about
the
competing
standards
there
are?
There
were
two
competing
drafts
on
the
subject.
Now
there
are
three
and
there's
also
a
draft
offline
in
the
discussion
that
happens
between
the
authors,
so
independent
analysis
of
this
work,
along
with
suggestions,
could
be
available.
B
And
then,
after
that,
we
hope
to
have
some
time
for
discussion
and
so
I'd
like
to
welcome
Ian
Levy,
to
talk
he's
the
tech
director
for
NCSC
and,
oh
sorry,
wrong
I
was
wrong.
It's
it's!
It's
all
No!
Oh,
no!
Sorry,
it's
a
penultimate
speaker
that
was
just
a
dry
run
and
please
don't
suck
me
technically
my
boss.
Yes,
so
I
wanted
to
talk
about
his
draft,
which
you
may
see
in
a
class.
H
So
hi
again
right
so
so
I
would
like
here
to
just
present
an
ID
that
we
prepared
it's
very
early
times
for
this
internet
draft
and
in
fact
we
need
help
and
feedback
to
actually
progress.
So
we
we
got
a
lot
of
feedback
over
the
past
year
and
we
learned
our
lessons
and
listened
and
understood
in
fact
that
we
realized.
Are
we
on
the
same
page
when
we,
when
we
speak
about
endpoint
security,
I'm
very
happy
to
see
your
David's
foundation
Simon's
rotation
before,
because
that
that
actually
shows
some
of
the
things
here?
H
So
we
try
to
ask
ourselves
what
are
the
capabilities
and
limitations
of
endpoint
security
solutions,
high
quality
classes?
We
did
that
with
a
few
colleagues
from
various
places,
so
today,
I
just
want
to
introduce
this
IDE
I,
published
it
very
late,
so
I
don't
expect
the
people
who
actually
is
there.
Anyone
who
read
this
ID,
perhaps
in
the
room,
see
you
ends,
ok,
good!
Thank
you
very
much
so
so
the
the
reason
why
we
came
here
is
how
will
you
light
so
when
we
started
to
look
at
endpoint
security?
H
H
So
when
this
is
about
well
in
the
long
term,
we
would
like
to
in
fact
feed
this
gap
and
create
a
full
review
of
endpoint
security
in
all
its
dimensions.
So
at
this
stage
we
we
started
with
endpoint
models.
That
was
an
interesting
problem.
What
is
an
endpoint?
Is
it
just
one
side
of
the
communication,
the
true
size
of
the
community
communications?
We
decided
to
go
for
the
two
sides.
If
you
go
for
the
two
sides,
we
have
a
problem
in
uniformly
describe
the
endpoint
first
problem.
Second
problem
was
the
threat
landscape.
H
That
was
a
bad
surprise.
We
thought
that
they
will
be
a
threat
landscape
that
was
agreed
outside
of
in
the
in
the
community
and
we
could
not
find
the
closest
was
mitre
attack,
which
is
good,
but
it's
not
complete.
We
we
found
difficulties
to
work
with
it,
so
we
decided
not
to
dive
too
much
on
it.
But,
as
a
question
seems
a
good
question
for
smart
x-ray.
Should
we
do
something
specific
here,
even
on
endpoint
security
capabilities,
we
could
not
find
something
complete
either.
So
we
have
a
lot
on.
You
know.
H
Gardener
and
others
speak
about
endpoint
security
in
terms
of
what
companies
do
like
a
party
add-on
and
concentrating
something
that
you
put
on
top
of
an
endpoint,
but
not
about
your
picture.
There
are
already
in
the
hardware
and
the
operating
system
in
application.
There
are
already
security
capability
so
which
are
theirs
and
is
it?
Is
it
what's
the
balance
between
the
intrinsic
varieties
and
the
know
increasing
capabilities
as
well?
H
We
also
says
if
we
would
have
an
int
feed
bandwidth
of
engineering
money
and
everything
what
it
would
look
like
to
have
a
an
ideal
and
possibly
what
does
it
mean
and
of
course
the
endpoint
is
not
alone,
we've
seen
in
many
examples,
so
on
the
old
chain
there,
it
made
a
very
good
chain
off
from
one
side
to
the
other,
one
all
the
adversaries
on
the
chain.
So
what?
What
is
the
endpoints
position
between
those
those
two
situations
and
and
what
what
we
learn
from
that
and
what
could
mean
a
difference
in
depth?
H
So
in
other
words,
can
we
share
their
information
from
other
sources?
To
add
to
add
the
difference,
then
we
looked
at
the
endpoint
security
limitations.
So,
of
course,
because
we
don't
have
a
threat.
Landscape
is
difficult
to
organize
systematically
the
the
research,
but
that's
fine.
This
is
a
good
outcome.
From
this
this
work
and-
and
we
found
many-
we
found
many
things.
We
tried
to
organize
them
and
categorize
them
the
best.
H
The
best
way
we
could-
and
we
found
the
areas
where
for
sure
the
endpoint
cannot
see
the
attack
or
some
gray
areas,
where
is,
is
the
endpoint
the
best
place
to
run
the
defense
right
or
what
else
we
could
learn
from
other
places?
And
finally,
we
wanted
to
really
prove
ourselves
that
we
had
some
some
real
case,
so
we
we
looked
at
our
and
for
the
system
and
we
managed
security
service.
That
is
very
well
known.
H
We
ran
on
read
of
customers
and-
and
we
found
we
found
actually
a
lot
of
data,
so
we
run
that
over
three
months
we
had
hundreds
of
thousands
of
events
on
hundreds
of
customers,
and
we
found
examples
of
detections
that
we
are
only
done
by
endpoints
and
those
that
were
only
doing
by
by
network.
But
this
was
not
good
enough.
We
wanted
to
say,
let's
really
look
at
it
completely,
so
we
start
to
look
at
regulatory
aspects.
So
what
are
the
other
constraints
we
get
from
certification
regulations
and
so
on?
So
once
the
status?
H
It's
it's
a
very
early
draft.
We
we
seek
feedback
and
we
look
for
future
collaboration.
I
have
already
two
good
areas
here
to
do
two
good
persons
to
discuss
with.
So
how
does
it
look
like
if
you
go
to
the
next
page,
so
in
fact,
I
have
now
updated.
The
should've
dated
the
URL,
so
I
published
it
on
on
as
a
proper
ID,
and
it
is
on
the
smart
meaning
is
we
can
find
it
there.
H
So
it's
very
so
this
is
just
the
table
of
contents
that
map's
what
I
have
explained
before,
and
we
we
put
some
Chris
holders,
for
example,
likely
if
we
cover
regulatory
considerations,
I
would
like
to
to
cover
you
mine,
wise
consideration.
That's
why
I'm
very
interested
by
what
David
is
doing
because
we're
going
to
learn
all
it's,
not
my
area,
but
I
would
be
really
happy
to
learn
what
else
we
should
do
from
from
this
consideration
here,
but
back
to
what
Simon
explained.
For
example,
it's
very
interesting.
H
Are
we
sharing
the
same
language
when
we
speak
about
endpoint
capabilities
and
possibility
capabilities
or
threats?
Are
we
really
talking
about
the
same
thing?
It's
actually
not
obvious
at
all.
I
could
see
that
within
my
company
see
that
with
others,
and
even
there
are
other
organizations
like
the
cyber
threat-
Alliance,
that's
led
by
Michael
Daniel,
which
is
hosted
in
the
US,
which
is
a
group
of
industry
players
that
try
to
share.
You
were
absolutely
right,
Simon
that
that
share
what
they
can
for
all.
H
The
reasons
was
because
all
the
party
started
to
realize
it's
a
busy
world.
We
have
500
security,
pure
players
on
the
planet,
from
big
ones
to
startups,
actually,
the
more
they
are
startups,
the
more
it
creates
a
problem
to
education
back
to
the
points
of
I.
Don't
know
how
many
people
we
miss
as
cyber
security
specialists,
but
the
more
you
add
a
new
technology,
the
harder
it
is
to
defend,
not
the
simpler,
so
why
we
need
to
have
evidence
numbers
I.
Have
some
people
complain
on
numbers,
but
we
need
to
get
trends
to
prioritize.
H
What
of
that
so
a
good
party.
So
we
we
created
that
this
way
nothing
is
a
perfect
or
in
the
stone.
We
are
completely
happy
to
get
any
feedback.
Let's
continue,
so
we
learn
a
few
lessons.
It
was
much
harder
than
than
we
thought
initially
from
his
beginning.
I
thought
this
would
exist
already
and
we
would
not
add
value,
but
the
good
news
and
good
news
for
smart
actually
is
that
I
think
it
could
generate
some
some
good
research,
especially
in
the
threat
landscape.
We
are
very
surprised
not
to
find
an
agreed,
stable,
uniform
approach.
H
True
to
that
even
on
the
capability
list,
for
example
on
intrinsic
capabilities.
This
is
not
my
area
I'm
sure
many
people
could
help
us
to
understand
what
what
happens,
what
they
do
on
operating
system-level,
what
they
do
on
our
way
and
so
on
that
would
be
cool.
The
production
data
on
from
managed
security
services
was
was
interesting.
We
we,
we
found
around
225,
distinct
categories
of
events
and
out
of
those
32
were
only
detected
by
Network.
H
The
rest
were
found
by
by
endpoints,
but
in
the
32
we
found
a
bunch
of
critical,
so
we
think
that
we
can
update
that
away,
so
we
can
continue
and
so
questions
for
the
future.
So
what
I
would
like
is
is
reused
a
little
bit.
Perhaps
the
approach
that
our
colleagues
in
tip-in
don't
to
to
to
normalize
and
unify
the
the
the
endpoint
between
the
two
sides
of
the
communication
like
to
find
better
uniformity.
H
So
again,
do
we
want
to
align
on
mitre
attack
we,
where
about
to
really
do
it,
but
is
it
the
right
way
or
the
right
thing?
Should
it
be
done
in
this
idea?
Actually,
is
it
something
that
that
smoke
should
or
other
people
would
like
to
detection,
give
us
a
full
framework
for
other
work?
It's
not,
but
it
could
be
an
idea
again.
H
It's
an
inventory
job.
So
it's
wrong.
You
have
to
search
for
a
lot
of
a
lot
of
information
from
s
where
another
aspects
I
spoke
about
human
rights
sections.
It's
not
my
area
would
be
happy
to
get
there,
but
actually
get
natural
balance
between
regulation,
human
rights
or
industry
and
other
considerations,
if
others
are
real
production
data
would
be
more
than
happy
and
as
well,
I
think
sure,
Simon
or
David
make
the
point,
but
should
we
have
an
economic
section?
H
I
mean
all
of
this
cost
money
to
build,
to
test,
to
to,
to
install
to
debug,
to
to
operate
and
so
on,
and
it's
not
coming
from
from
from
the
brew.
So
so,
and
this
is
about
showing
you,
you
can't
think
about
creating
a
neighbor
computer
on
the
endpoint,
and
you
have
to
find
some
balance
as
well
on
on
other
aspects.
So
do
we
have
an
economic
section
and
and
I
think
that's
about
it?
H
B
You,
okay,
thank
you,
so
we'll
hold
questions
to
the
end
and
just
in
the
interest
of
time.
Thank
you.
That's
really
interesting.
I
do
encourage
everyone
in
the
room
to
go
and
read
the
draft,
and
that
is
on
the
taste
tracker,
and
so
we
will
now
actually
have
our
final
speaker.
Sorry
Ian
I
am.
This
is
Ian
levy,
tech,
director
for
insisting
I.
U
Hard
orience
right.
This
is
why
it's
called
the
one
snake
presentation
come
on.
There's
got
to
be
a
laugh
there
somewhere,
please
dying.
This
is
the
problem
with
cybersecurity
right.
It's
all
snake
oil
and
fear,
and
one
returned
over
the
next
15
20
minutes
is
tell
you
what
we're
trying
to
do
in
the
UK
government
to
try
and
fix
some
of
that
at
the
end,
I
will
come
to
why
smart
is
really
important
to
us
big
red
circle
at
the
top
part
of
GCHQ
hello
spoon.
U
Please
save
your
ideological
beatings
until
I
have
a
beer
I
only
cut
only
plea,
so
I'mma
showed
a
picture
of
his
building.
This
is
my
building.
We
have
two
floors
in
that
building.
It's
very
cyber
and
the
reason
for
showing
you
that
is
not
to
show
off
it's
to
say
we're
trying
to
be
different,
so
it's
commercial
building.
So
if
any
of
you
in
London
what
I
come
see
us
come
see
us
you
do
not
have
to
go
through
a
body
cavity
search
to
get
in.
It's
really
really
important.
I'll
go
through.
U
Why,
in
a
minute
when
you
set
for
new
organization
we're
only
2
years
old
as
an
organization,
you
need
an
early
win.
This
was
our
early
win.
The
carbuncle
cup
is
the
architectural
award
for
the
worst
architecture
in
the
UK.
We
want
it
right.
This
is
not
bad.
This
is
the
bad
bit
that
came
second,
ok
now
into
the
serious
stuff,
so
governments
are
good
at
a
few
things.
One
of
them
is
setting
policy
right.
Cybersecurity
is
the
only
part
of
government
policy
globally.
U
The
thing
that
I
have
learned
over
the
last
15
years
are
trying
to
do
this?
Is
people
like
us
don't
fix
the
problems?
We
need
everybody
in
the
UK
worldwide
to
be
able
to
understand
cyber
security
to
be
able
to
to
enact
it
because
in
the
end,
it's
risk
management
right.
Everybody
just
risk
management.
Every
day
of
the
week.
We
just
need
to
make
internalize
it,
so
people
can
do
it
better.
I
hate
the
c-word
as
well,
but
I'm
stuck
with
it.
It's
a
part,
the
organisation
name.
U
Sorry,
it's
an
example
password,
so
we
give
to
monumentally
stupid
pieces
of
advice.
I
have
a
different
password
for
every
service
and
change
it
every
28.6
days,
because
it
sounds
like
we
calculated
it
and
don't
click
on
links
or
open
attachments
in
emails,
unless
you
trust
them
right,
look-ins,
both
those
so
the
first
one
is
about
reducing
the
burden
on
people.
So
we
published
this
in
2017
and
it
said
people
should
be
able
to
use
password
Edge's
because
they're
better,
they
are
easier
to
use.
They
are
better
for
people.
U
Basically,
this
this
was
calling
out
the
financial
services
industry,
so
the
banks,
because
they
all
suck
I,
was
saying
you
keep
telling
us
a
security
reason
that
people
can't
use
password
managers
on
their
retail
banking
sites.
Could
somebody
please
explain
to
us
what
that
security
reason
is
because
we
couldn't
work
it
out,
nobody's
ever
done
that
that's
what
this
is
July
last
year,
and
this
has
got
nothing
to
do
with
us.
U
This
is
people
on
the
interwebs
on
Twitter,
complaining
about
Barclaycard
business
and
saying
why
can't
use
a
password
manager,
business
coming
back
and
saying:
oh
well,
we've
got
our
own
special
security,
you
don't
even
of
those
and
then
it
descends
into
the
typical
Twitter
flame
war.
Two
days
later,
Berkeley
can't
change
their
policy.
Alright,
that's
about
trying
to
get
the
UK
populace
to
make
security
better.
It's
not
going
to
be
us
and
then
there's
the
same
example
with
Troy
on
have
I
been
poned.
U
Some
you
mentioned
it
before,
quoting
our
guidance
to
nationwide
a
UK,
Building
Society
same
thing:
how
do
we
get
people?
It's
a
better
use
cybersecurity,
actually
don't
open
a
link
or
attachment
unless
you
trust
the
email.
So
it's
anybody
know
a
guy
called
sin
on
reborn
he's
internet
pranks
that
he's
the
guy
that
went
to
rinse
previous
and
Jared
Krishna
in
the
White
House
made
them
say,
really
stupid
things
over
email
and
then
published
them
all
he's
awesome.
He
had
a
go
at
me
and
I
was
really
really
really
lucky.
U
I
mean
really
I
was
on
the
Eurostar.
I've
had
two
balls
of
white
little
bottles
of
wine,
not
big
balls
of
white.
Just
for
clarity
and
those
two
emails
turned
up
chest.
Their
screenshots
on
my
phone
Paul
Church
Esther
is
our
director
of
arms
right
I.
Was
that
close
to
replying
to
it
and
I
didn't,
but
I
ended
up
talking
to
the
guy
over
email
me
in
the
writing,
this
blog
together,
which
is
him
as
the
attacker
saying,
here's
what
I
was
trying
to
achieve
and
me
as
the
victim
going.
U
Here's
what
it
felt
like
and
saying.
I
was
really
really
lucky
because
those
you
exes
suck
there
is
nothing
there
that
the
average
person
can
use
as
a
cue
to
go.
Hang
on.
This
is
a
bit
dodgy.
Nothing
and
this
is
making
up
for
problems
with
protocols
that
we
designed
20
years
ago.
A
the
fact
that
the
content
and
the
envelope
are
not
cryptographically
bound
is
a
pain
in
the
backside.
U
Somebody
mentioned
IOT
before
it's
one
of
those
things.
It's
like
cyber,
it's
an
evil
word
and
we
should
ban
it,
but
we
have
it
we've
published
this
last
year.
This
is
the
UK's
view
on
cyber
security
for
IOT
devices.
So
this
is
things
that
vendors
have
to
sign
up
to
what
should
sign
up
to
it's
really
complicated
things
like
shouldn't:
have
any
non
changeable
credentials.
U
Things
like
you
should
patch
the
damn
thing
and
tell
people
how
long
you're
going
to
do
that
for
things
like
you
should
have
a
vulnerability
disclosure
policy
all
the
way
down
to
no
external
DMA
capable
interfaces.
So
we
published
that
a
load
of
vendors
have
gone.
Ok,
consumers
have
got
a
clue
what
it's
about,
but
we're
going
to
do
that.
U
So
you
you
can
choose
food
based
on
whether
it's
going
to
kill
you
or
taste
nice
right,
we're
going
to
do
the
same
for
IOT
devices,
red
or
green,
so
three
things
to
start
with
red
or
green
and
get
retailers
to
only
stock
things
that
I've
got
three
greens
so
build
a
market
around
cyber
security
because
there
isn't
one.
Today
we
care
about
two
broad
classes
of
things
we
care
about
nation-state
attacks
when
we
care
about
the
66
million
people
who
live
in
the
UK,
we're
from
the
government
we're
here
to
help.
U
This
is
what
I'm
trying
to
do
for
the
66
million
people
in
the
UK
right,
protect
them
the
majority,
the
time
from
the
jority
harmful
jority
attacks.
Thanks
back
in
2017,
we
publish
this
there's
a
big
blog.
That
goes.
It's
all
thought
this
is
all
on
the
NCSC
website.
By
the
way
the
problem
is
going
last.
Is
people
have
stolen
on
my
thunder
so
BGP?
U
Yes,
BGP
sucks
it
sucks
in
a
way
that
people
don't
understand
it's
worse
than
we
just
heard
much
much
worse,
so
we're
building
a
national
scale,
BGP
platform
where
all
the
ISPs
compare
and
we
can
spot
bad
stuff
and
stop
it
taking
effect
and
we've
done
that
the
denial
for
experiment
seems
to
work
we're
going
to
scale
it
if
it
works,
we'll
publish.
We
are
dns
for
public
sector,
public
sector
being
central
government,
local
government,
some
other
bits,
not
people
clear,
we'll
get
to
it
in
a
minute.
Will
you
take
downs?
We
do
Demark.
U
We
do
all
the
things
that
have
been
around
for
years.
I've
never
been
done
its
scale.
I'm
gonna
talk
about
some
of
the
things
we've
found
over
the
last
couple
years.
Oh
yeah,
we
publish
all
of
this.
So
that's
the
70
odd
page
report
with
all
the
data
behind
what
we
found
in
the
first
year.
The
next
version
should
be
out
in
a
couple
of
weeks.
Hopefully
when
I
finish,
writing
it.
But
the
idea
for
this
is
put
some
objective,
unbiased
data
out
there,
so
that
people
can
do
proper
scientific
research
on
this.
U
So
Demark
comes
with
you
know
about
d
mark,
so
this
is
about
allowing
domain
owners
to
take
control
over
how
that
how
their
mail
is
delivered
or
not.
So
we've
been
saying
for
years,
people
should
do
D
mark.
However,
he
goes
it's
really
hard.
Hey
we're
government
we're
stupid,
I've
got
six
thousand
hundred
and
seventy
four
government
domains
registered
on
my
central
d
mark
processing
platform.
Thirteen
hundred
and
sixty
nine
are
on
P
equals
quarantine,
five
and
Senator
on
people's
reject.
That's
after
eighteen
months,
we
started
off
at
four
can't
be
that
hard.
U
If
government
can
do
that,
we
came
up
with
a
way
of
synthesizing
D
mark
records
for
non-existent
sub
domains,
so
Ian
levy
govt
UK
in
the
three
months
there
four
ones
there
we
generated
four
hundred
thirty
thousand
synthetic
Demark
because
we're
stopping
people
take
use
the
brands
that
government
owns
like
HMRC,
which
is
our
tax
authority
and
stopping
them
using
that
brand
in
lots
of
different
ways
which
I'll
get
to
so
they're
going
to
other
things.
So,
instead
of
going
for
tax
refund
at
UK,
they
move
to
tax
refund
that
tax
refund
doctor
F
dot.
U
U
K,
which
doesn't
exist,
and
so
we
had
to
work
out
how
to
protect
that
and
stop
people
abusing
our
domains
and
things
that
look
like
our
domains.
Just
proof.
People
HMRC's
ranked
in
terms
of
global
fishing
when
we
started
two
years
ago,
was
sixteenth.
So
is
the
16th
most
fished
brand
in
the
world
and
it's
a
tax
office.
They
generally
take
money
instead
of
giving
it
but
hey
when
out
146.
U
Okay,
so
with
some
intervention
you
can
have
an
effect,
simple
stuff
than
a
scale
and
have
an
effect
on
christine's
and
we
do
takedowns.
So
we
do
takedowns
by
the
very,
very
complicated
cyber
thing
of
emailing
the
provider
and
asking
them
to
take
it
down,
because
it's
pretty
their
own
terms
and
conditions.
This
is
not
rappelling
down
the
outside
of
a
data
center
in
black.
U
This
is
just
enforcing
to
provide
his
own
t's
and
c's
and
again
this
will
all
be
in
it's
all
on
the
website,
all
in
the
report
sheets
in
next
one
somebody
I
mentioned
JavaScript
schemers
yeah.
So
we
did
a
thing
in
the
UK
where
we
looked
for
mage
Magento,
ecommerce
sites
that
were
installed
and
had
been
compromised
by
mage
carts,
so
they
had
some
JavaScript
stuck
on
the
checkout
page,
just
copies
all
the
stuff
off
to
a
third
party-
and
this
is
one
of
the
notifications.
U
We
give
it's
about
two
and
a
half
thousand
in
three
months.
I'll
guess
the
numbers
in
a
minute,
but
these
are
all
small
businesses
and
somebody
asks
how
did
they
get
almost
across
site?
Scripting
know
they
just
didn't
patch
the
thing
because
they
bought
it
from
their
web
host
er.
You
gave
it
as
a
free
package,
but
not
with
any
support
all
right.
These
are
the
source
things
we
need
to
change
X.
U
These
are
what
I
would
call
deceptive
domains,
so
these
are
domains
that
have
been
registered
purely
to
look
like
something
that
I
care
about,
so
something
to
do.
The
UK
government,
every
single
one
of
these
had
an
SSL
certificate.
Let's
encrypt
cPanel,
usually
one
of
those
two
okay
and
we've
spent
the
last
20
years.
Training
the
populace
to
go
padlock
means
safe.
Does
it
bug
going
right?
U
We
have
and
don't
start
training
me
about
domain
validated
versus
organizational
validated
versus
excellent
validation,
because
66
million
people
in
the
UK
don't
and
don't
know
how
to
check,
and
we
trying
to
explain
it
to
them
is
not
the
right
way
to
fix
this.
I've
broken
it,
your
back.
No,
no!
Okay!
U
That
had
goes
the
next
one.
This
is
our
public
sector
DNS.
So
this
is
central
government
local
government
things
like
that.
So
in
2018
we
did
a
bout
with
processing
about
one
and
a
half
million
public
sector
employees.
So
remember
this
gun,
nothing
sue
the
public
public
sector,
we
blocked
fifty
seven
and
a
half
million
queries
out
68
billion.
Your
piece
says
all
the
ISPs
in
the
room,
but
twenty
eight
million
queries
relating
to
DJ's,
including
conficker
2007,
and
it's
still
running
somewhere
in
the
UK
public
sector.
U
U
For
minutes,
okay,
next
one
we
take
stuff
down
that
pretends
to
be
UK
government.
Those
are
the
numbers
and
obviously
the
slides
on
data
tracker.
So
you
can
look
yourself.
We
take
down
a
lot
of
stuff
that
pretends
to
be
UK
government.
Next,
if
you
host
anything
bad
in
the
UK,
bad
means
malware.
Kinda
control,
phishing,
anything
like
that.
We
will
go
and
ask
the
providers
to
take
it
down
before
we
started
the
median
uptime
for
those
sites
was
27
hours.
Now,
it's
45
minutes
doesn't
make
any
difference.
U
U
Next,
if
you
do
2333
credit
card
skimmers
in
three
months,
that
was
our
major
car
stuff
next
scaling,
so
we're
asking
ISPs
in
the
UK
to
protect
their
customers
by
default
for
free
okay.
Now
you
can
opt
out
and
that's
fine.
This
is
not
our
data.
This
is
commercial
data,
but
BT
of
announce.
They've
got
six
million
residential
customers
and
they
protect
it.
Well,
they
stopped
235
million
malware
seats
in
a
month
right.
This
problem
is
not
going
away
and
we've
dealt
with
1500
more
than
1500
national
scale
incidents.
U
In
two
years
next
I
was
gonna
talk
about
Russians
attacking
telcos,
but
I
won't
next.
Next,
it's
on
the
website.
You
can
read
it.
This
is
an
interesting
infographic,
so
in
the
sort
of
nine
months
there
these
are
the
sectors
that
were
attacked
and
the
roughly,
how
often
so
these
are
number
of
incidents
that
we
have
done
next.
Oh
that's!
What
Rob
Joyce
said:
it's
not
zero-day
wielding
nation-states.
We've
done
root
cause
analysis
on
most
of
those,
not
all
of
them.
There
are
three
things
that
matter:
okay,
the
three
root
causes
for
those
incidents.
U
That's
number
one
yeah
admins
browsing
the
web
with
their
admin
account
sorry
too
stupid
to
help.
Now,
thanks
what
have
you
got
connected
to
the
Internet?
It's
actually
a
really
hard
question
to
answer.
Everybody
thinks
it's
easy.
It
really
isn't
credentials
matter.
If
somebody
fishing
you
and
getting
your
password
is
the
key
to
your
entire
network
security.
Probably
doing
it
wrong,
but
it's
not
simple
to
do
2fa
everywhere.
U
So
why
smart
that's
starting
to
make
security
into
protocols
in
a
way
that
was
never
done
before
and
designs?
Those
protocols
need
to
be
cognizant
of
an
actual
attacks
on
the
way
attackers
actually
work.
Next
people
apart
of
protocols
I
know
you
guys
don't
want
to
do
UX,
but
you
have
to
find
ways
of
giving
UX
designers
really
high
quality
information
to
present
to
people,
so
they
can
make
better
decisions.
Just
go
back
to
my
iPhone's
screenshots
I.
U
It
is
impossible
for
the
majority
of
people
to
make
sensible
decisions
today,
always
always
always
remember
bad
guys
use
your
shiny
right
just
because
you
think
it's
great
yeah
TLS
is
a
great
example
right
certificates,
yeah,
let's
encrypt
great,
but
guys
are
the
main
proponents
of
those
always
remember.
The
bad
guys,
use
the
shiny
and
remember
adversaries
are
not
passive,
so
you
do
something
they
will
respond
and
we've
got
a
bunch
of
stuff
that
we've
shown
where
we've
done
something
out
of
sirs
adversaries
have
responded,
sometimes
in
a
good
way.
U
Ie
they've
gone
to
France,
sorry,
I
know
sometimes
in
a
pack.
Ok,
then
we
get
a
battle
off
them
upset.
Sometimes,
in
a
bad
way,
because
they
move
to
someone,
that's
harder
for
us
to
manage,
but
adversaries
are
never
passive.
Ok
is
the
only
time
gonna
say
encryption.
Security
is
not
the
same
as
encryption.
Security
is
much
much
much
much
bigger
than
encryption
course.
Encryption
is
important,
but
encrypting
something
does
not
make
it
secure
some
of
the
side
effects
of
changing
how
protocols
works
scale
really
badly.
U
From
the
security
point
of
view,
security,
privacy
and
resilience
are
all
totally
different.
Things
and
again
is
my
last
point:
the
stuff
that's
built
in
the
ietf
because
of
convergence,
it's
starting
to
underpin
critical
infrastructure
in
a
way
it
never
did
before,
and
so
resilience
becomes
really
really
really
important,
and
if
we
do
it
badly,
security
and
resilience
are
incompatible,
but
they
didn't
need
to
be,
and
that's
why
you
guys
need
to
help.
That's
it.
A
T
C
B
E
U
E
E
Well,
we
need
to
actually,
even
if
we
don't
design,
UI
and
I,
don't
design
your
I.
We
need
to
be
thinking
about
what
our
protocols
do
expose,
what
kind
of
signals
they
can
expose
to
the
UI
layer,
which
means
mean
that
we
need
to
have
UI
people
come
here
and
tell
us
what
signals
they
need
or
what
signals
they
think
they
can
work
with.
E
So
I
think
that's
a
really
that's
a
really
critical
part,
and
the
second
thing
that
you
said
that
I
want
to
really
highlight
and
boost
is
remember
that
bad
guys
use
the
shiny
to
write,
and
that
goes
for
I
think,
unfortunately,
almost
all
of
the
types
of
systems
that
you've
described
that
you're
using
so
one
of
the
concerns
that
I
have
with
the
systems
that
say,
oh
yeah.
Well,
you
know
we
just
call
up.
No,
we
have
our
friendly
arrangements
with
the
various
big
providers
and
we
call
them
up
and
say:
hey.
E
You
should
take
this
thing
down.
That's
a
shiny
and
the
bad
guys
use
that
shiny
tube.
They
say
hey.
Wouldn't
it
be
a
shame
if
you
had
to
stop
doing
business
in
our
country.
If
you
know
that
would
be
shame.
But
if
you
take
this
thing
down
or
you
D
host
them,
then
we're
happy
to
like
continue
to.
Let
you
let
you
work
in
the
network
so
so
of
the
types
of
mechanisms
that
you're
using
I
appreciate
the
amount
of
security
that
you've
done,
but
the.
E
U
Does
it
stop
and
what
does
it
say,
the
bottom
net
craft?
So
we
don't
serve
legal
order
on
people
we
use
net
craft
we've
been
doing
this
for
30
years.
You
have
brand
protection,
so
governments,
just
a
brand
UKIP
space
is
just
a
brand
cool
use,
the
well-established
things
cuz
government's
sticking
their
oar
in.
It's
probably
not
great
idea
at
this.
E
Point
right
so
so
the
combination
of
those
two
points,
though
I
realize
you
didn't
present
on
this
here,
but
you
and
I
think
Robinson
presented
this
proposal.
That's
now
known
as
the
ghost
proposal
yeah,
and
that
proposal
seems
to
itself
be
a
way
to
say
that
we
should
make
the
user
interface
less
responsive
and
by
not
showing
users
who's
involved
in
the
encrypted
communication
session,
no
I
just.
U
Talking
to
bar
okay
for
those
that
don't
know
what
we're
talking
about,
it's
a
law,
fair
article
that
I
published
last
year
about
end-to-end
encryption
and
law
enforcement
access
and
the
point
of
the
article
wasn't
to
say
we
should
do
that
thing.
You
say
we
need
to
have
a
conversation
about
how
we
do
these
things,
but
that
thing
is
definitely.
E
A
A
V
V
Elissa
Cooper,
so
my
question
as
the
for
the
chairs
is
two
questions.
Actually
one
is
if
you
were
to
go
forward
as
a
proposed
research
group.
Would
you
expect
sort
of
the
scope
of
contributions
and
work
to
be
similar
to
what
it
was?
What
we
saw
here
today
in
terms
of
kind
of
the
mix
of
everything
presented
and
then
second
question
is,
if
your
expectation,
what
would
your
expectation
be
around
the
kind
of
closeness
of
the
relationship
between
work?
V
A
So
I
think
what
came
up
through
some
of
the
presentations
is
the
time
scale
with
the
research.
Some
of
these
would
be
further
out,
so
we
wouldn't
expect
to
see
some
of
the
area's
come
in
too
quickly.
I
think
we
need
to
see
what
comes
in
in
terms
of
actual
research.
The
drafts
from
Symantec
is
a
really
good
starting
point
with
the
increased
use
of
encryption
so
that
we
actually
could
ensure
the
deployment
of
more
secure
technologies.
A
How
do
we
do
better
endpoint
detection,
so
I
think
that's
a
really
valuable
contribution
and
very
timely
I
think
we
have
to
see
what
comes
in
and
what
people
are
interested
to
work
on,
and
if
it's
research,
then
it
would
be
here
if
it's
more
tactical
than
IETF
and
figure
that
out
with
the
security,
security
area,
directors
and
SEC
dispatch
and
and
the
available
tools
at
hand
and
guidance.
So
we're
obviously
open
to
guidance
to
make
this
successful
and
scoping
is
usually
important
right
so
and
also.
B
Tomoe,
like
mentioned
at
the
end
of
his
talk
and
drafts,
that
the
group
could
look
at
and
bring
a
cyber
defense
perspective
to
reviewing
those
protocols,
so
that
is
very
tied
to
IETF,
so
I
think
there's
quite
a
spread
in
terms
of
the
work
that
could
be
done,
but
the
tie
back
to
protocols.
This
is
the
reason
that
smart
is
hoping
to
be
in
the
IRT
F.
Ok,.
V
K
Sorry,
so
Elliot
leer
here
saying
that
one
of
the
biggest
challenges
we
have
are
bringing
the
is
bringing
the
researchers
to
us
and
here
a
couple
of
different
conferences
that
but
really
I
think
angle
in
this
direction
very
closely.
One
of
them
is
Weiss.
The
workshop
on
the
economics
of
information
security,
security
of
human
behavior
is
another
one.
We
have
a
couple
like
mbss
that
I
saw
hosts
that
I
think
also
intersects
is,
and
so
one
of
the
things
I
think
that
the
chair
should
think
about
IRS.
You
should
think
about
the
new.
K
The
IRT
f
chair
should
think
about,
is
how
do
you
gonna
bring
those
people
here?
You
have
people
like
Angela
sass
in
London,
for
instance,
you
know
all
folks,
all
over
the
you
know
that
that
all
over
the
country
don't
hear
about
focus
on
user
interface.
Economists
who
look
these
problems
very
the
Gordon
Loeb
model,
for
instance,
how
do
you
bring
these
people
into
the
room
and
then
how
can
they
deliver
into
the
IETF
in
terms
of
meaningful
things
that
we
could
do
later
on?
Thank.
B
You
so
a
couple
of
points
on
that
safe
se
and
speaking
with,
and
the
chairs
of
mahadji
they've
noted
that
some
talks
that
have
gone
in
there
previously
would
have
perhaps
been
research
that
should
or
would
have
come
to
smart
if
smart
existed
so
there's.
Definitely
research,
that's
already
coming
in
and
I've
had
quite
a
few
calls
with
various
academic
universities
in
America
and
the
UK
who
are
doing
research
that
we
think
would
be
interesting
to
bring
here
so
and
we
managed
to
get
them
interested
on
the
basis
of
just
having
a
mailing
list.
B
U
So
you
mentioned
Angela,
so
so
he
was
the
chair
of
our
first
Research
Institute
that
we
set
up
five
six
years
ago
on
the
science
of
cybersecurity.
So
if
there
is,
if
this
group
exists,
I
am
more
than
happy
to
say
that
our
research
institutes,
which
are
independent
of
those
were
funded,
will
contribute,
because
you
know
there
five
things
we
really
care
about.
How
do
we
get
the
stuff
out
that.
W
Bret
Jordan
and
I
just
want
to
thank
all
the
presenters
today.
I
think
this
was
brilliant
and
by
the
numbers
of
people
that
are
here,
I
think
that's
clearly
evident
that
this
is
of
use
and
I
think
those
people
of
value.
My
question
is
to
the
chairs:
what
is
the
next
step
and
how
do
we
actually
get
ietf,
I,
RTF
or
whoever
to
approve
this
moving
forward
because
it
seems
like
there
is
enough
people
here
of
interest.
B
X
Morton
two
things:
one!
Thank
you
for
your
apology,
a
quick
and
effective
about
the
ASCII
art
number
two
in
both
the
chairs
defense
I,
downloaded
that
deck
of
slides
I
looked
at
it
in
power,
port,
editing
mode,
and
that
picture
was
not
evident.
So,
if
you
guys
did
the
same
thing,
you
were
to
Mystic
to
and
I
missed
it
in
time
to
tell
you
so
don't
feel
bad
about
that.
Thanks
for
the
apology,
thank.