►
From YouTube: How GitHub uses GitHub to secure GitHub - Universe 2022
Description
One of the best parts about working at GitHub is using GitHub to build GitHub—and not only for software development. GitHub is central to how the GitHub Security team drives and delivers secure operations, infrastructure, and products. In this session, Greg Ose will walk through how GitHub uses the platform to enable security workflows that bring the expertise and processes GitHub needs to manage security risk where GitHub engineers and partners are across the business work.
As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub
Thanks!
Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github
About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com
A
A
A
Thanks
for
joining
us
today,
we're
going
to
switch
in
and
out
and
talk
a
little
bit
about
how
we
use
GitHub
to
secure
GitHub.
So
it's
probably
no
surprise
to
any
of
you.
Who've
used
GitHub,
but
at
GitHub,
We,
Believe,
really
deeply
in
using
the
product
to
build
the
product
and
so
really
across
all
of
the
teams,
both
Technical
and
non-technical.
At
the
company.
A
A
Our
customers
are
also
the
developers
in
our
engineering
team
and
our
product
team,
and
so
when
we
think
about
identity
and
access
management
inside
of
the
company,
we
wanted
a
capability
that
allowed
us
to
keep
our
developers
in
the
flow
and
be
able
to
kind
of
do
the
things
that
we
need
them
to
do,
but
also
meet
all
the
needs
we
have
and
so
we're
thinking
about
this.
We
we
thought
like
what
are
the
requirements?
How
are
we
going
to
approach
this?
We
need
to
be
able
to
show
compliance
and
audits.
A
We
need
to
be
able
to
handle
things
like
reorgs.
We
need
transparency
and
scalability,
but
we
also
need
to
handle
risk.
So
not
all
the
things
that
we're
providing
access
to
are
the
same
risk
level,
and
so
we
landed
on
being
able
to
do
this
inside
of
get,
and
so
we
use
repository
based
access
control
system,
effectively
get
Ops
to
do
this
for
the
entire
company,
and
this
is
how
we
manage
our
I
am
at
the
company.
This
is
a
program
we
call
entitlements,
so
we're
going
to
talk
a
little
bit
about
that.
A
So,
what's
it
look
like
in
practice,
so
let's
say
that
you
switch
jobs
inside
the
company
and
you
need
access
to
a
new,
let's
say
Splunk
in
your
new
team,
and
maybe
you
didn't
have
access
to
that
before.
Well,
we
just
opened
a
pull
request,
so
you
would
open
a
pull
request
in
the
repository
where
we
manage
our
entitlements.
A
You
would
find
the
right
file
whatever
the
Splunk
access
you
need
in
this
case
is,
and
then
you
open,
a
PR,
add
your
handle
to
it,
and
then
you
wait
on
the
approvals
and
then
that's
where
our
automation
kicks
in
and
depending
on
what
you're
trying
to
do,
depending
on
the
sensitivity
of
the
actual
capability,
it
will
require
certain
levels
of
approval.
Some
of
these
are
app
level.
So
in
the
case
of
Splunk,
maybe
the
team
that
owns
a
manage
this
belong.
Maybe
they
have
to
approve
it
as
well
as
your
manager.
A
A
How
do
we
handle
things
that
are
a
little
more
sensitive,
so
this
kind
of
brings
us
to
the
strictly
audited
entitlements,
and
so
this
is
a
capability
where,
let's
say
it's
a
production,
VPN
or
let's
say
it's
super
admin
on
one
of
our
SAS
tools
that
we
use.
We
really
want
to
control
that.
That's
not
something
we
just
kind
of
want.
A
That
I
mean
we
want
the
self-service
capability,
but
we
also
need
to
make
sure
that
we've
put
increased
kind
of
rigor
around
that,
and
so
we
have
a
set
of
entitlements
that
are
strictly
audited
and
what
this
happened.
What
this
enforces
is
a
periodic
review,
usually
it's
six
months
and
an
automatic
automatic
expiration,
and
so
what
happens
is
every
six
months
it
kind
of
comes
back
up
and
says:
hey.
We
need
you
and
your
manager
to
reaffirm
that
you
need
this
access.
If
it
doesn't,
then
it's
just
going
to
go
away.
A
We
also
kick
some
of
those
things
in
when
we
have
a
reorg
automatically
as
well.
So
this
is
the
entitlements
capability
is
open
sourced.
We
open
sourced
this
a
few
months
ago
and
click
the
there
we
go
and
so
it's
available
as
a
kind
of
core
capability.
A
couple
of
things
we
open
sourced
as
well
were
plugins
that
work
with
external
features.
So
you
wouldn't
won't
be
surprised.
We
open
sourced
a
plug-in
for
GitHub,
so
kind
of,
like
from
the
workflow
perspective,
how
this
works
is
whatever
your
source
of
Truth
is
as
an
org
chart.
A
So
on
to
number
two
so
security
findings,
I
know:
we've
had
some
really
good
conversations
with
with
several
of
you
this
week
about
security
findings,
every
company
and
every
kind
of
team.
That's
dealing
with
security
I
think
has
this
challenge
of.
Where
do
we
do?
Where
do
we
deal
with,
and
how
do
we
aggregate
all
the
things
that
are
coming
into
the
security
team
right?
So
it's
not
just
cves.
It's
also
things
like
bug,
Bounty
reports
and
pirs
from
incidents
or
availability
incidents,
security
incidents.
A
It
could
be
things
from
the
red
team,
and
so
how
do
we
think
about
that?
How
do
we
handle
it?
How
do
we
do
that?
Well,
at
GitHub
we
have
Gaz
and
we
use
gas
internally,
so
GitHub
Advanced
security,
so
we
use
all
those
features
and
all
those
tools,
one
of
the
things
we
love
about
that
and
developers
love
about
that
it's
integrated
into
how
they're
doing
their
work
in
the
repositories
every
day.
A
But
how
do
we
do
that
with
all
the
other
stuff
that
we
have
coming
in,
and
so
we
have
built
an
internal
tool
that
we're
using
called
security
findings
that
really
emulates
that
flow
for
the
things
that
aren't
part
of
advanced
security,
and
so
this
could
be
things
like
a
bug,
Bounty
report.
It
could
be
something
from
our
Cloud
security
posture
management,
tooling.
It
could
be
something
from
a
red
team.
A
Our
red
team
is
actively
always
probing
the
infrastructure
in
the
product
looking
for
gaps,
and
so
we
funnel
all
that
into
security
findings,
and
we
pair
that
up
with
our
fundamentals
program,
which
is
an
internal
program
where
we
track
things
like
durable
ownership
and
it's
kind
of
the
Baseline
requirements
for
the
company
for
availability
for
performance
and
security.
We
marry
all
that
up
and
then
we
put
it
where
developers
do
their
work,
which
is
GitHub
issues
and
for
us
we
use
slack
a
lot.
A
So
it
might
show
up
in
a
slack
message
depending
on
how
urgent
it
is
and
how
we
need
to
track
it
and
remediate
it.
And
then
that
gives
us
that
accountability
and
visibility
we
need
for
SLA
management
and
being
able
to
essentially
make
sure
that
the
resources
are
in
the
right
place,
but
we're
also
meeting
our
internal
needs,
our
stakeholder
needs
and
our
customer
expectations.
So
security
findings
is
how
we're
approaching
that
problem.
A
Sorry
I'm
going
to
put
this
down
and
just
use
the
clicker,
so
just
to
give
you
a
sense
of
the
scale
here
last
year
and
these
numbers
are
notional,
but
just
to
give
you
a
scale,
a
sense
of
scale
we
had
in
the
millions
of
detections
that
come
in
every
year
and
we
have
to
triage
those,
and
so
some
a
lot
of
that
triage
is
automated.
We
do
have
some
humans
doing
some
of
that
work
as
well.
That
might
result
in
some
findings.
A
A
One
of
the
things
that
we're
working
on
now
is
a
UI
that
goes
with
that,
and
so
the
reason
for
this
is
sometimes
we
want
to
be
able
to
provide
more
data.
It's
not
required
we're
still
going
to
open
an
issue
where
developers
work
in
the
repo
that
they're
doing
their
work,
but
sometimes
a
cve
can
be
complicated
or
security.
Finding
can
have
multiple
CDs
and
so,
instead
of
in
some
cases,
opening
up
hundreds
of
issues
on
a
developer
repository.
A
What
we'll
do
is
open
an
aggregate
issue
and
then
point
them
back
to
the
security
findings
UI,
where
we're
providing
data,
we're
providing
things
like
the
life
cycle,
more
data
on
the
actual
vulnerability
and
Remediation
status
and
history
of
remediation.
So
you
know
it
was
fixed
by
the
developer
and
the
team.
Then
it
was
reverted
back
and
then
it
was
fixed
again
and
we
can
see
all
that
in
the
flow.
A
B
Thank
you
Jacob
and
today,
I
want
to
kind
of
follow
up
on
what
Jacob
was
just
talking
about
about
how
we
track
findings,
how
we
track
vulnerabilities
and
really
focus
on
one
specific
type
of
vulnerability
and
work
through
some
of
those
workflows.
So,
like
I
mentioned,
I
run
our
product
security
engineering
team,
where
we're
working
to
prevent
and
identify
vulnerabilities.
B
You
know
a
bit
of
a
life
cycle
for
everything
we
get
in,
and
we
want
to
make
sure
that
you
know
we
have
great
ways
to
intake
so
make
sure
we're
aware
of
vulnerabilities
that
are
coming
through.
We
also
want
to
make
sure
that
we're
doing
the
right
things
when
we're
aware
of
that
to
make
sure
it's
fixed.
You
know
where
we
know
it
is
where
it's
fixed
places.
B
We
we
move
all
of
this
into
issues
and
work
with
our
Engineers,
where
they're
working
for
any
other
bug
fix
so
kind
of
the
first
step
is
looking
at
the
intake.
We
have
a
bunch
of
disparate
sources
where
we
might
become
aware
of
a
vulnerability
in
our
product.
We
run
a
very
popular
bug,
Bounty
program,
we're
also
now
rolling
out
our
usage
of
private
vulnerability,
reportings
on
GitHub
for
some
of
our
open
source,
libraries
so
another
way
to
get
in
vulnerabilities
reported
in
our
software.
B
We
also
have
some
more
traditional
means,
like
a
security
at
email
address
and
we're
monitoring
a
bunch
of
things
if
someone
reaches
out
out
into
Twitter,
not
the
best
way
to
get
a.
You
know,
information
about
a
vulnerability,
but
we're
still
monitoring
those
things.
So
we
have
a
really
good
awareness
of
all
these
external
sources
of
vulnerabilities
and
we
also
have
internal
sources
of
vulnerabilities.
Maybe
an
engineer
is
working
on
a
feature
they're
doing
some
development.
B
Looking
at
things,
I
have
an
assessment
team,
that's
redoing
code
review,
doing
security,
pen
testing,
and
all
of
that,
so
we
want
to
handle
all
these
vulnerabilities
in
a
really
consistent
way
and
how
we
do
that
is
all
this
intake
is
kind
of
coordinated
by
an
incident
responder.
So
one
of
the
teams,
underneath
in
PSE
product
security
engineering,
is
our
pcert
team
and
what
they
do
is
they
monitor
all
these
sources?
So
we
have
awareness
of
all
of
these
different
places
for
intake
and
to
make
sure
that
we're
actually
routing
them
to
the
right
teams.
B
B
We
know
you
know
it's
an
issue,
it's
a
vulnerability
in
our
code,
let's
open
that
up
where,
where
our
Engineers
are
doing
all
their
other
development
right
alongside
other
bug
reports,
so
they
can
properly
prioritize
and
track
that
work
just
like
they
would
any
any
other
engineering
work.
So
our
product
security
incident
responder,
then
opens
up
an
issue,
and
we
can
see
here
kind
of
an
example
of
something
that
came
in
through
a
bug.
Bounty
report,
a
specific
vulnerability
and
kind
of
the
different
things
that
we
add
from
that.
B
So
we're
pulling
in
all
of
the
context
of
you
know:
what's
the
issue?
Is
there
proof
of
concept
any
additional
analysis
that
our
triage
team
might
have
done
to
validate
that
or
ask
additional
questions
from
the
reporter
to
really
get
all
that
context?
So
we
can
present
that
to
the
engineering
teams
when
they're
starting
their
fix,
we
also
give
it
a
severity,
and
this
this
really
helps
set
expectations
and
slas
and
time
frames
to
help
Engineers
prioritize.
This
work,
alongside
all
the
other
engineering
work
they're
doing.
B
B
A
pull
request
gives
us
that
great
opportunity
to
pull
in
some
security
Engineers
to
make
sure
the
fix
that's
proposed,
is
complete,
there's
not
any
weird
edge
cases
or
bypasses,
and
we're
really
thinking
about
you
know
is
this
a
you
know
a
robust
fix
against
the
vulnerability
that
was
identified,
making
sure
we
have
test
cases.
All
of
that
making
sure
all
that
goes
into
the
pull
request
that
we
have
and
getting
that
really
deployed
to
where
the
vulnerabilities
at
you
can
see
here,
even
after
the
vulnerability
has
been
fixed
in
one
place.
B
B
That's
a
different
deployment
model
where
we
then
need
to
get
that
into
GitHub
Enterprise
server,
and
we
also
follow
up
if
it
came
in
through
the
bug,
Bounty
saying:
what's
the
severity,
what's
the
payment
for
that
and
bring
that
into
the
issue
just
so,
we
have
a
really
clear
and
transparent
decision
making
making
sure
that
we're
coming
up
with
you
know:
payment
amounts
through
our
bug,
Bounty
program
that
are
Equitable
and
in
line
with
what
we've
paid
before
and
meet
our
severity
guidelines,
foreign.
So
I
mentioned.
B
We
fixed
this
in
the
code
that
we
knew
it
impacted.
One
of
the
big
steps
that
we
always
do
is
where
could
have
this
vulnerability,
or
this
type
of
vulnerability
affected
other
code,
and
this
might
just
be
in
the
same
code
base,
but
this
might
also
be
across
our
code
bases
right
if
we
have
some
weird
Edge
case
of
a
SQL
injection
right,
we're
going
to
want
to
look
for
that
across
all
of
our
code
base.
B
B
You
know
across
all
of
our
repositories
now
I'm
going
to
say
I'm
in
a
very
lucky
spot,
because
I
know
all
of
our
code
is
on
GitHub
GitHub
developers
develop
on
GitHub,
so
I
have
a
pretty
holistic
view
across
organization
in
where
code
lives
and
the
ability
to
search
through
that
I
think
the
cool
thing
is
is
outside
a
code
search.
We
can
even
do
better.
We
can
use
codeql
to
really
model
these
vulnerabilities
to
to
look
at
our
our
code.
Paths
and
say:
is
this
actually
taking
in
user
input
and
build
these
code?
B
Ql
rules
and
the
cool
thing
about
this
is
I.
Have
a
team,
our
code
analysis
team?
That's
looking
at
these
vulnerabilities
we
have
coming
in.
Seeing
can
we
write
a
code
ql
rule
for
this
and
then
running
that
code,
ql
rule
across
all
of
our
code
bases
that
we
have
code
scanning
enabled,
and
it
really
allows
us
to
give
us
that
holistic
view
of
other
variants
of
this
vulnerability
that
might
impact
our
code.
B
And
finally,
I
think
the
next
step
and
the
final
step
like
I
mentioned,
is
we
fix
this?
We
feel
good
that
this
type
of
vulnerability.
You
know
we
have
great
coverage.
We've
we've
looked
across
our
code
bases.
We
need
to
make
sure
our
our
customers
and
the
users
of
that
code
are
aware.
If
there's
any
actions,
they
need
to
take
to
actually
mitigate
that
for
open
source.
This
is
great.
We
use
GitHub
security
advisories
if
it's
an
open
source
library
or
vulnerability
in
one
of
those.
B
We
go
through
the
GitHub
security
advisory
process,
to
cut
a
cve
to
get
that
CBE
published
and
then
dependabot
does
its
thing
and
you
know,
alerts
all
the
users
of
that
vulnerability.
It's
really
great
for
things
that
impact
our
products,
we're
actually
a
CNA,
a
certified
number
Authority
for
our
products
and
largely
this
looks
like
GitHub
Enterprise
server,
and
since
this
isn't
an
open
source
product,
we
run
our
own
process
to
get
cves
published
for
GitHub
Enterprise
server.
B
To
do
this,
we
loved
the
flow
of
GitHub
security,
advisories
and
the
collaboration
it
brings,
and
we
have
our
own
process
that
kind
of
mimics
that
internally.
So
if
we've
decided
a
CV
needs
to
be
issued
for
GitHub
Enterprise
server,
we
go
through
kind
of
this
templated
issue
and
to
submit
a
cve
to
miter.
B
You
need
to
have
a
very
specific
Json
formatted
file
that
then
you
send
to
them,
and
we
have
a
process
that
goes
through
where
we
can
collaborate
on
a
pull
request
very
very
nicely
in
a
yaml
formatted
file,
and
then
we
have
GitHub
actions
run
that
actually
then
converts
that
into
the
schema.
The
Json
schema
that's
required
by
miter.
B
So
after
that's
done,
we
can
actually
create
a
pull
request
to
miter's
repository
with
that
cve
under
the
block
for
our
CNA
and
then
get
an
advisory
published
for
GitHub
Enterprise
server
to
make
sure
that
customers
are
aware,
through
their
own
vulnerability
scanning
of,
if
they're
running,
an
outdated
version
of
GitHub
Enterprise
server.
What
the
risk
is
and
allows
us
to
really
go
through
that
whole
notification
process
and
make
sure
they're
aware.
B
So
one
last
item
I
wanted
to
touch
on
is
what
we're
using
our
product
for
to
to
do
our
security
reviews
and
security
views
are
a
way
for
security,
to
have
awareness,
to
give
input
and
to
do
any
of
the
downstream
reviews
and
assessments
and
Analysis
of
the
products
and
features
we're
shipping
in
a
centralized
manner.
So
engineering
teams
and
product
teams
know
how
to
engage
us
and
one
thing
that
we're
utilizing
for
this
is
again
it's
on
GitHub
and
it's
in
an
issue.
B
But
what
we're
using
is
our
issue
template
forms,
and
what
that
allows
us
to
do
is
really
gather
structured
information
about
a
product
or
feature
that's
shipping,
both
just
general
information.
So
we
have
the
context
of
what
it's
doing.
You
know
what
concerns
what
risks
there
might
be
who
to
contact
about
it
and
things
like
this,
but
it
also
allows
us
to
gather
really
structured
information
like
multiple
choice,
answers
that
help
us
figure
out.
You
know
what
the
risk
of
something
is
help
us
get.
B
An
idea
of
you
know
maybe
some
of
the
concerns
we
might
have,
because
really
what
we're
looking
to
do
in
this
process
is,
as
you've
seen
today.
Github
ships
a
lot
of
things,
and
how
do
we
prioritize
our
security
resources
to
really
focus
on
the
most
important
things
or
the
riskiest
things,
and
by
collecting
this
information,
and
then
we
run
GitHub
actions
against
these
issues
to
pull
out
this
information
and
calculate
a
risk
score.
So
we
can
use
that
in
our
conversation
in
our
own
prioritization
and
structuring
so
again,
here
we
can
see
you
know.
B
A
team
came
in
opened
up
an
issue
filled
out
their
information
about
some
of
the
features
of
their
new
product,
new
new
feature
area,
and
then
we
we
go
through
and
calculate
risk
score,
and
you
can
see
here.
You
know
this
is
a
medium
rare
score
and
that's
going
to
play
into
how
we
prioritize
those
Downstream
reviews.
What
we're
looking
at
are
we
doing
you
know
a
code
review
of
this?
A
So
one
of
the
things
that
is
Corda
kind
of
how
we
think
about
work
every
day.
I
know
a
lot
of
you
can
identify.
This
is
never
force
a
human
to
do
what
a
robot
can
I
think
that
was
pretty
true
in
some
of
the
announcements
you
saw
earlier
today
as
well.
So
a
couple
ways
we
do.
This
is
through
issues
and
projects,
and
so
this
is
very
similar
to
what
we've
talked
about
already.
But
it's
really
focused
on.
How
do
we
run
our
programs?
A
A
A
Is
there
a
blocker
or
have
we
shipped
it,
and
it
can
update
that
all
automatically?
So
we
don't
have
to
go
into
multiple
places,
and
so
this
is
another
way
we
do
it
kind
of
in
sort
of
after
what
Greg
talked
about
in
terms
of
the
security
review
is
a
Security
review
gets
submitted?
One
of
the
other
teams
may
want
to
know:
does
their
team
need
to
look
at
it
based
on
the
risk
score
that
was
calculated
by
the
action
on
the
intake?
A
Well,
we
also
have
actions
that
will
pull
those
In
from
from
the
security
review,
repo
kick
off
actions
and
then
add
them
to
internal
tracking
boards,
so
that
individual
teams-
maybe
it's
the
security
operations
team.
Maybe
it's
the
security
architecture
team.
They
need
to
look
at
those
in
more
in
depth,
syncs
all
those
together
and
and
creates
that
automation,
so
that
folks
don't
have
to
go
in
and
triage.
A
Essentially,
every
single
thing
that
comes
in
only
the
ones
that
apply
to
their
space,
so
just
to
wrap
up
here,
I
think:
we've
we've
talked
about
this
a
bit,
but
we're
globally
distributed
working
team
and
working
at
async
for
us
as
imperative.
Github
has
that
built
in
and
that's
why
we
use
it
for
so
many
things
from
what
the
security
team
is
doing
every
day
to
what
our
managers
are
doing
and
that's
just
not
that's
not
just
how
our
managers
work
in
security.
A
This
is
how
our
managers
work
across
the
company
as
well,
and
we
have
a
diverse
set
of
employees,
and-
and
this
is
how
we've
kind
of
chosen
to
lean
into
building
the
product
product
using
the
product
to
build
the
product,
then
when
things
aren't
working
the
way
we
expect
we
give
that
feedback
back,
and
we
we
have
that
tight,
iteration,
Loop
and
so
really
appreciate
the
time
today
and
I
think
we've
got
some
time
for
questions.
Yes,.
A
C
A
So
I'll
let
Greg
talk
about
this
a
little
bit
as
well.
I
mean
the
we
have
security
developers,
kind
of
split
all
around
the
company,
and
so
it's
not
like
a
direct
like
number.
So
we
have
security,
Engineers
focused
on
the
product
and
building
the
product
that
ships
we
have
a
security
team.
That's
that's
focused
on
the
securing
the
actual
kind
of
back
end
of
the
infrastructure
and
then
doing
the
reviews,
but
they
also
in
Greg's
team.
D
I've
got
one
back
here:
hey
thanks,
you
guys
started
out
by
talking
about
I
am
a
little
bit
and
how
you
work
with
that
with
GitHub.
One
of
the
challenges
we
have
faced
is
in
automation
when
we're
automating
things
with
GitHub.
The
lack
of
service
accounts
is
definitely
something
we've
we've
had
to.
You
know,
do
some
weirder
things
and
I'm
curious
how
you
guys
have
solved
that
problem
and
how
you
guys,
control
automated
access
to
resources
in
GitHub.
A
That's
a
good
question,
so
the
short
answer
is
we
can
spend
some
time
after
this,
and
maybe
we
can
get
a
whiteboard
because
it
is
complicated
space
right.
We
have
a
lot
of
things
that
go
into
that
in
terms
of
how
we
control
the
service
accounts,
some
of
it
is
through
entitlements
and
how
we
leverage
that
and
then
some
of
it
is
it
kind
of
comes
down
to
the
roles
and
the
access
that
we
Grant
into
those
things.
A
So
we
use
a
lot
of
the
internal
tools
that
are
built
into
GitHub,
so
we
use
the
paths
we
use
like.
We
limit
the
access
to
things,
but
we
also
have
kind
of
sensitive
things
like
in
the
entitlements.
We'll
only
tag
certain
teams
or
like
certain
things
that
need
access,
will
require
a
VP
to
approve,
depending
on
how
sensitive
it
is
and
so
depend.
C
A
That's
a
really
good
question:
it's
both
really
so
it
depends
on
how
complicated
it
is.
So
if
it's
an
action
that
is
going
to
have
significant
access
to
our
infrastructure
or
our
SAS
platforms
or
things
that
we're
really
integrating
in
it's
going
to
have
a
more
rigorous
review
through
our
Security
review
process.
If
it's
just
wiring
up,
you
know
maybe
a
repository
to
a
project
board
and
there's
no
actual
sensitive
access
at
play.
It
may
just
be
something
we
would
ship
and
iterate
really
quickly
in
an
afternoon.
A
Some
of
it
is
we're
leveraging
open
source
packages
that
are
in
the
marketplace
as
well.
One
of
the
things
that
we
do
work
very
closely
on
with
our
developers
is
when
using
something
like
that.
We
want
to
make
sure
that
we
pin
to
a
sha
and
not
just
a
tag
so
that,
when
we
kind
of
do
the
initial
review
and
we're
happy
with
the
security
of
it,
we
kind
of
pinned
that
so
that
if
it
gets
updated
at
a
band,
we
don't
get
surprised
by
what
gets
put
in
there.
E
B
I
think
we're
we're
always
recalibrating
it
right,
I
think
when
you're,
when
you're
looking
at
what
information
can
we
collect
and
what
signals
are
we
getting
out
of
there
I
think
it's
always
an
iterative
process
where
you
know
keeping
an
eye
on
what
that
looks.
Like
you
know
there
very
much
might
be.
You
know
a
new
area
of
risk
that
we've
seen
frequently
it's
like
okay,
we
should
really
start
making
sure
we're
aware
of
the
the
features
or
products
that
are
shipping
with
that
and
hold
on
that.
B
So
I
think
it's
always
an
iterative
process
on
that
I
wouldn't
say
we're
always
looking
at
that.
How
we
can
make
improvements,
how
we
can
get
better
signal
out
of
it,
because
you
know
as
we
we
ship.
The
first
version
of
it
was
very
much
of
like.
Let's
throw
this
out,
there
see
what
it
looks
like
and
see
what
we
need
to
iterate
on
and
change
on
that
to
get
to
get
better
signals
or
different
signals
that
we
hadn't
thought
of
yet.