►
From YouTube: QnA with Brendan Dolan Gavitt on AI Code Suggestions
Description
This is a QnA with the author of Fauxpilot where Gitlab Product , Engineering and Incubation team asks question to understand further on AI code suggestion , future of Smart Code as well as large scale models
A
B
A
B
Sure,
yeah
so
I
guess
full
pilot
kind
of
came
about
because
so
I
had
been
using
GitHub,
copilot
and
thought
it
was.
You
know
very
helpful
in
doing
my
own
programming,
but
I
guess
as
researcher
we
wanted.
You
know
my
lab
wanted
a
way
to
be
able
to
do
things
like
you
know
if
we
train
our
own
model
or
fine-tune
our
own
model
for
different
purposes,
we'll
want
some
way
to.
B
Actually
you
know
use
that
in
something
that
looks
like
a
GitHub,
copilot
kind
of
use
case
and
at
the
same
time,
I
guess
you
know.
Whenever
I
saw
people
discussing
you
know
things
like
co-pilot
online,
one
of
the
things
that
they
were
sort
of
concerned
about.
Is
this
the
fact
that
you
have
to
sort
of
send
your
code
up
to
some
remote
server,
hosted
by
GitHub
to
actually
get
suggestions
right
because
it
sends
sort
of
the
code
that
you're
currently
working
on
and
maybe
code,
that's
working.
B
B
You
know:
sort
of
standard
code
benchmarks
like
human
eval
or
apps
for
solving
programming
tasks
so
yeah.
That
was
essentially
it
you
know.
So
a
lot
of
the
research
I'm
doing
these
days
focuses
on
like
okay,
you
know
here
are
these
code
models.
It
seems
like
a
lot
of
people
are
going
to
be
using
them
over
the
next
few
years.
B
B
We've
already
done
one
kind
of
user
study
where
we
had
gave
half
of
the
users
access
to
the
code
model
and
half
the
users
wrote
Things
by
hand,
and
we
didn't
use
fopilot
for
that
one
because
we
ran
it
before
if
a
pilot
existed
but
we're
planning
on
doing
more
of
those
in
the
future
and
we're
going
to
try
and
use
those
with
low
pilot,
because
that
really
gives
us
very
kind
of
fine-grained
control.
Over
the
we
know
the
exact
model
that
they
used.
We
know
what
it
was
trained
on.
B
We
can
use
models
that
were
fine-tuned
on
particular
code
bases,
or
you
know
only
trained
on
code
that
we've
maybe
tried
to
scan
for
security
vulnerabilities
or
things
like
that.
So
yeah
that
was,
that
was
kind
of
the
the
main
thing
I
guess
I
also
don't
want
to
necessarily
oversell
it.
It
is
a
project
that
I
kind
of
put
together
over
the
course
of
a
maybe
a
month
or
two
over
the
summer.
When
I
was,
you
know
not
didn't,
have
to
teach
and
the
main
components
are.
B
A
A
Yeah,
we
can
come
back
into
that
as
well.
We
can
skip
to
probably
buy,
which
would
be
my
question
and
we've
spoken
about
it,
but
I
think
we
are
looking
into
building
out
of
a
POC
using
full
palette.
So
what
would
you
think
we
would
need
to
consider
in
actually
using
full
pilot
and
testing
it
with,
let's
say
even
internally,
with
our
gitlab
audience
of
developers
with
writing
in
different
languages
with
RoR,
BJs
Python
and
all
of
that
and
also
keep
in
mind
compute
storing
what?
B
You
know,
batching
is
a
little
bit
tricky
because
you
sort
of
have
to
have
enough
people
using
it
that
it
makes
sense
to
combine
multiple
requests,
because
if
you
have
to
wait,
you
know
30
seconds
for
the
next
request
to
come
in
before
you
try
to
batch
them
together.
Then,
all
of
a
sudden,
your
latency,
is
30
seconds
for
that
first
user.
But
assuming
you
have
a
lot
of
requests
coming
in
simultaneously,
then
batching
together
makes
a
lot
of
sense.
B
B
Think
that
that's
something
that
you
can
do
pretty
trivially
by
just
running
multiple
servers
and
you
know
having
either
a
front-end
proxy
that
redirects
to
one
of
them
or
by
you
know
changing
the
little
flask
app
to
redirect
one
of
the
inference
servers
you
know
either
of
those
should
work
pretty
well,
the
models
don't
have,
don't
have
to
really
store
any
state
right,
they're,
all
kind
of
one
shot.
You
give
them
the
context,
and
then
they
produce
a
suggestion.
B
So
there's
not
anything
where
like.
If
the
user
was
making
multiple
requests,
they
would
always
have
to
go
to
the
same
server.
So
it's
really
a
very
nice
use
case
for
load.
Balancing
there
going
beyond
that
and
starting
to
think
about.
You
know
what
can
be
done
to
make
things
faster
and
more
performant
and
hopefully
have
a
lower
resource
requirements.
D
B
A
B
Yeah
so
I
don't
know
if
I
can
do
yeah
I
think
I
can
do
screen.
Sharing
and
I
can
just
show
you
the
traffic
page
both
for
the
GitHub
repository,
but
also
for
the
hugging
face.
That
is
the
wrong
one.
Sorry
I
think
it's
this
one.
There
we
go
so
this
is
the
page
on
hugging
face
where
the
actual
models
that
it
downloads
are
stored,
and
this
is
a
pretty
decent
proxy
for
whether
people
are
actually
using
it.
B
B
Almost
no
one
is
using
the
natural
language
versions
of
these,
because
I
think
they're
not
even
exposed
in
the
setup
script,
and
there
aren't
very
many
people
who
are
trying
to
use
faux
pilot
to
help
them
write
English
over
on
the
traffic
page.
So
this
it's,
you
know
it
was
a
little
bit
annoying.
That
GitHub
only
gives
you
the
last
month
of
usage
stats
here,
but
it
tends.
E
B
B
You
know
for
free,
so
that
was
kind
of
very
attractive
as
far
as
like
what
I
think
of
them
as
a
company,
you
know
I
I
like
a
lot
of
the
things
they
do.
They
have
made
it
a
lot
easier
to
train
models
and
to
host
a
bunch
of
fairly
different
models
for
inference,
yeah,
so
I,
you
know
I,
think
I,
guess
I'm
a
little
bit
still
confused
by
if
they're
a
company
how
they
are
planning
to
make
money.
B
Yeah
I
think
that
plausibly,
what
they're
doing
is
something
like
you
know:
they
have
a
bunch
of
people
who
know
a
lot
about
Ai
and
building
and
training
models,
and
they
may
be
doing
a
sort
of
Contracting
thing
where
they
say.
Oh,
you
can
hire
us
to.
You
know,
help
you
train
or
deploy
some
models
that
you
have
or
are
interested
in
having
and
they
may
be
able
to.
B
A
A
Obviously
we
have
four
so
all
the
way
to
the
32
gigabyte
on
and
some
more
insight
as
to
where
the
model
Works,
which
in
in
what
areas,
what
areas
it
wouldn't
and
then
I
think
Alex
and
I
have
a
few
more
questions
just
on
how
we
would
go
in
if
we
had
to
take
that
model
and
optimize
it.
What
would
that
look
like
so
yeah
sure.
B
B
B
But
that
one
was
trained,
I
think
on
a
larger
set
of
languages
and
uses
gbt,
Neo
X,
and
so
that
might
be
another
one
that
could
easily
be
added,
because
again
it
uses
faster
Transformer.
You
can
convert
their
model
into
faster,
Transformer
and
use
it,
and
so
that
one
might
be
a
good
one
to
add
in
the
very
near
future.
B
That
would
let
you
then
run
any
of
those
models
as
well.
Those
would
include
things
like
Facebook.
Has
this
one
called
encoder
hugging
face
itself,
has
one
called
code
parrot
which
is
focused
on
Python,
and
they
are
current.
What's
that
yeah
polycutor
is
the
one
yeah
yeah
a
polycoder
is
the
GPT
neox
one
and
it
is
going
and
I
think
that
could
be.
A
A
D
B
B
B
If
you
see
this
and
the
output
then
you're
done
generating
and
you
can
return
immediately
and
so
the
way
they
do
it
is
most
of
the
time
it
operates
in
a
kind
of
one
line
at
a
time
time
mode
where
the
stop
sequence
is
just
a
new
line.
So
as
you're
typing
an
individual
line,
maybe
it
only
has
to
generate
like
four
tokens
to
be
able
to
complete
that
line
and
then
I
think
if
you
sort
of
stop
and
wait
a
little
bit,
it
will
then
kick
into
this.
B
Certainly
that
makes
a
lot
of
sense,
particularly
in
terms
of
performance,
because
at
least
most
of
I
think
most
much
of
the
benefit
that
I
get
from
copilot
is
the
sort
of
one-line
completions
as
I'm
typing,
and
you
want
those
to
be
very,
very
fast
so
that
you
know
you
can
even
see
them
like,
as
you
type
one
character
at
a
time
it
will
go
through
different
completions
and
so
I
think
that
that
does
tend
to
work
quite
well.
If
you
are
just
trying
to
generate
one
line
it
really.
D
B
Yeah,
so,
generally
speaking,
you
just
pass
it
the
code
you
already
have
and
if
users
want
to
kind
of
instruct
it
in
English
the
way
you
do
it
is
by
just
writing
a
comment
and
I
have
actually
found
this.
It
may
be
a
little
bit
surprising.
Is
that
a
lot
of
the
you
know?
My
code
is
much
better
commented
now,
because
I
write
a
comment
saying
what
I
want
the
model
to
fill
in
and
then
it
fills
in
based
on
that
comment.
B
D
B
Right
and
so
I
think
the
I
mean
the
main
limitation
here
is
that
there's
the
model
supports
2048
tokens
a
time,
and
that
is
includes
you
know
the
input
plus
whatever
it
needs
to
generate,
and
so
that
means,
if
you
are
asking
it
to
generate
up
to
512
tokens,
then
as
that
gives
you,
you
know
around
1500
tokens
of
input
and
so
often,
particularly
in
a
larger
source
code
file.
You're
not
going
to
be
able
to
fit
the
whole
thing
into
the.
B
And
so
you
know
the
simplest
strategy.
Is
you
just
give
it
the
most
recent
1500
tokens?
You
can
get
more
sophisticated
about
this,
because
sometimes
there
might
be
extra
context
that
you
want
to
include.
So
you
know
you
could
do
something
like
say,
assuming
that
you
know
a
little
bit
about
what
the
language
they're
using
is.
B
You
can
like
go
up
to
the
top
of
the
file
and
look
for
import
statements
and
then
have
the
prompt,
be
here's
my
import
statements
and
then
here's
as
much
of
the
code
I
can
as
I
can
fit
so
that
it
has
things
like
Library
definitions
or
structs,
or
things
like
that,
and
then
you
can
even
extend
this
by
saying.
Oh
okay,
you
know
I'm
writing
C
and
they
include
food.h
I'll,
go
to
foo.h
and
try
to
pull
in
some.
D
B
Right
I
mean
so
I
think
this
is
something
that
for
now,
I
have
not
done
very
much
on
kind
of
the
client
side,
so
this
would
all
be
sort
of
done
within.
Like
a
you
know,
a
visual
studio
code,
Plugin
or
an
IDE,
plugin
and
I
have
not
done
very
much
with
that.
I
know
that
the
GitHub
copilot
plugin
does
do
a
lot
of
this,
because
I
see
it
pulling
in
pieces
of
other
libraries
and
other
files
that
I
have
open.
B
You
know
there's
a
it's
a
decent
bit
of
kind
of
implementation
work,
particularly
if
you
want
to
support
a
bunch
of
languages,
because
you
have
to
know
that,
like
okay
in
PHP,
it's
required
and
in
Python
it's
import
and
in
see,
it's
include
and
then
have
to.
You
know
know
where
to
find
these
dependencies.
D
B
You
know,
aside
from
that,
I
guess
just
sort
of
using
it
interactively
I've
I
found
that
it's
it's
pretty
decent,
it's
very
it's
much
better
with
python
than
it
is
with,
like
writing.
C
code,
for
example,
and
so
I
think
this
makes
sense.
Just
given
how
much
python
code
there
is
out
there,
but
yeah
I
think
it
would
be
really
nice
to
have
like
some.
You
know
automated
harnesses
for
saying:
okay
I
mean
even
just
you
know:
does
the
model
still
work
to
generate
code
at
all?
B
A
B
You
know
really
speaking,
it's
not
going
to
make
the
kinds
of
mistakes
like
forgetting
a
semicolon,
or
you
know
doing
something.
Obviously
syntactically
wrong.
The
mistakes
really
are
more
of
well.
You
know
it
doesn't
have
any
context.
So
it
doesn't
know
what
you
know.
The
field
name
of
this
data
structure
is
so
it'll
just
make
one
up,
and
sometimes
he
guesses
right
and
sometimes
it
guesses
wrong.
B
A
B
Yep
so
I
think
certainly
Coppell
is
currently
better
at
producing
mode.
Some
of
that
comes
from
the
fact
that
I
think
they
are
using
a
larger
model.
It
might
be
using
a
larger
model
but
they're,
certainly
using
what's
trained
on
more
data,
so
I
think
that
they
have
trained
it
on
basically
all
the
code
they
could
get
their
hands
on
and,
as
a
result,
it
is
pretty
good
yeah
generating
suggestions
there.
B
That
said,
you
know
I
think
it
has
been
a
it
seemed.
It
has
been
sort
of
perfectly
fine
for
kind
of
writing.
The
kind
of
code
that
I
usually
write,
which
is
like
okay
I'm
in
Python
I,
want
to
like
read
in
a
bunch
of
data
from
somewhere.
Do
some
like
analysis
on
it,
create
some
visualizations
and
graphs.
B
A
B
B
B
Are
these
in
the
16
billion
parameter
model
on
a
data
set
of
verilog
code,
which
is
used
for
like
a
CPU
design
and
Hardware
design,
and
that
was
a
400
Meg
code
data
set
and
it
took
three
a100
gpus
each
with
80
gigabytes
of
vram
about
six
days
to
to
fine-tuning
on
that
data
set
so
yeah
the
the
computational
requirements
are
definitely
not
like
I
can
take.
You
know
my
even
like
my
souped
up
gaming
desktop
and
fine-tune
a
big
model
on
it.
E
E
B
Yeah
I
mean
so
I
think
that
it
is
only
going
to
get
bigger
the
other
one
that
is
available.
Now.
Is
this
Amazon
code
Whisperer,
which
is
yet
another
which
I
have
not
even
had
a
chance
to
play
with
it
all
yet
so
I've
just
you
know
like
read
a
blog
post
about
it,
but
and
we
do
actually
note
the
tab
9
folks
pretty
well,
we've
been
talking
a
lot
with
them
about.
B
You
know
collaborating
on
some
things,
like
you
know,
user
studies
with
their
users
and
figuring
out.
You
know
if
there
are
ways
that
we
can
say
if
we've
trained
like
the
models
that
tries
to
produce
more
secure
code,
can
we
try
having
it
deployed
with
them
to
see?
If
you
know
it
helps,
helps
their
users
as
well.
B
B
I
should
have
heard
that
the
quality
of
output
it
gives
you
is
not
up
to
what
copilot
currently
does
and
I
can
believe
that,
because
the
Codex
model
is
very,
very
good,
so
yeah,
you
know
I
definitely
am
not
the
person
to
ask
about
business
questions
because
you
know
I
I
know
nothing
about
it
at
La
will
say
is.
It
seems
like
it
is
going
to
be
very,
very
popular
over
the
next
few
years,
because
when
it
works,
it
works
really
really
well.
E
Awesome
thanks
for
that
on
to
the
next
question,
and
this
is
actually
kind
of
funny,
as
I
was
looking
at
the
articles
that
I've
been
researching,
you
were
actually
referenced
in
a
number
of
them.
There's
been
a
lot
of
research
done
on
co-pilot
and
detection
of
you
know
bad
insecure,
malicious
verbatim
code.
B
Yeah,
so
this
is
a
great
question,
so
we
I'm
I
was
I've
kind
of
gone
back
and
forth
on
this.
So
initially
we
did
this
big
evaluation
of
copilot
on
a
whole
bunch
of
different
classes
of
vulnerabilities,
where
we
made
these
like
little
toy
scenarios
like
I'm
about
to
insert
some
data
into
a
database
copilot.
B
So
we
actually
did
do
a
user
study
and
this
paper
we
actually
just
submitted
two
days
ago
so
I'm
happy
to
share
a
copy
of
that.
But
we
did
I
use
it.
We
did
this
user
study,
comparing
the
functionality
and
security
of
code
produced
either
by
hand
or
with
the
assistance
of
codex
and
somewhat
surprisingly
to
us.
B
But
our
kind
of
working
hypothesis
at
the
moment
is
that,
because
the
models
were
trained
not
to
write
good
code
but
to
very
accurately
predict
what
would
come
next
in
a
source
code
file.
If
the
code
that
you
write
is
not
very
secure
or
not
very
high
quality,
it
will
very
Faithfully
write
insecure
and
low
quality
code
for
you,
and
so
it
did
tend
to
kind
of
match
the
quality
of
code
written
by
the
user.
B
B
B
This
is
how
I
produce
like
high
quality
code
and
then,
when
you
are
trying
to
generate
user
like
actual
suggestions
for
a
user,
you
would
always
include
the
high
quality
tag,
so
the
model
is
strongly
influenced
to
produce
higher
quality
code,
even
if,
what's
already
there
is
not
as
good
so.
Something
like
that
could
work
We've
also
been
looking
at
ways
of
doing
this
without
having
to
do
any
additional
training.
B
B
B
Unfortunately,
I
mean
the
recommendations
are
mostly
the
same
recommendations
you
would
give
for
people
who
are
writing
code
by
hand
like
you
should
test
this.
You
should
use
security
scanning
tools.
You
should
maybe
try
fuzzing
your
software
and
I
guess
pessimistically
I
expect
that
advice
to
be
taken
up
about
as
well
as
it
is
for
human
written
code,
foreign.
E
B
B
B
A
A
B
So
it's
it
was
sort
of
shocking
to
me
that
you
could
take
a
language
model
and
compress
it
down
to
just
using
four
bits
per
parameter
without
really
hurting
its
performance
and
I.
Think
that
that
trend
is
probably
I
mean
it
has
to
stop
somewhere,
because
you
know
I,
you
can't
represent.
You
can't
use
zero
bits
per
parameter,
but
I
think
that
we,
it
will
still
be
able
to
go
down
even
a
little
bit
further.
B
So
I
think
that
that's
going
to
be
really
nice,
so
quantization
is
going
to
be
a
big
area
and
it's
going
to
be
right.
Now,
it's
sort
of
only
a
few
models
have
been
quantized,
but
I
think
many
more
could
be
and
then,
as
far
as
just
pain,
speeding
up
inference.
This
is
something
we
actually
have
a
couple
students
looking
at
right
now
and.
C
B
How
do
you
speed
up
inference
in
the
case
where
you
know,
maybe
you've
got
a
data
center
with
some
fixed
capacity.
You've
got
lots
of
requests
coming
in
and
you
know
at
some
point.
Maybe
you
start
getting
overloaded
and
so
like.
Is
there
like
a
small
reduction
in
quality?
You
would
be
willing
to
sacrifice
in
favor
of
keeping
inference
time
very
low
and
there's
some
kind
of
more.
A
B
Yeah
I'm
trying
to
think
so
I
mean
the
main
thing
is
really
just
you
know.
Are
there
things
that
you've
found
kind
of
about
the
way
things
are
currently
implemented
that
you
know
you
say?
Oh
gosh,
the
you
know
it
would
be
much
better
if
it
were
done
this
way.
You
know
this
is
clearly
not
something
we
could
deploy,
because
you
know
it's
something.
That's
fundamentally
broken
here
or
something
like
that.
B
I
know
that
Fred
has
been
very
kind
of
providing
pull
requests
to
help
improve
it
in
many
ways.
So
far
and
I've
been
trying
to
make
sure
I
give
some
attention
to
his
code
suggestions
but
yeah
you
know:
are
there
things
that
yeah?
Basically,
how
can
I
and
I
will
caveat
this
by
saying
that
I
don't
have
tons
of
time,
but
are
there
ways
that
I
can
help
make
Profile
better
for
the
use
cases
that
you
have
in
mind.
C
And
I
think
the
you
are
right
on
the
money
with
prompt
engineering,
I
think
that's
going
to
be
like
the
major
differentiator,
because
I've
been
playing
around
with
it,
and
it
really
greatly
depends
on
what
you
give
at
the
prompt
or
what
kind
of
return
you're
going
to
get.
That's
actually
meaningful,
so
I've
been
currently
working
on
authentication
so
because
we
don't
want
anyone
to
be
using
the
computer.
We're
gonna
host,
but
I've
also
been
working
on
the
vs
code.
Extension
for
the
gitlab
official
workflow
I.
Think
that
will
be
pretty
awesome.
C
If
we
can
like
also
kind
of
promote
that
within
the
project
that
we
can
get
contributions
there,
because
I
think
there
are
a
lot
of
people
with
really
clever
ideas
about
prompt
engineering,
because
right
now
it's
just
the
most
simplistic
thing
there
is
I.
Think
it
just
takes
the
the
last
I.
Don't
know
how
many
tokens
it
takes,
but
it
just
gives
it
and
yeah
it's
not
optimal.
C
B
I've
I've
definitely
wanted
to
be
able
to
point
users
to
an
extension
that
works
well,
particularly
with
faux
pilot,
because
right
now
there's
some
kind
of
you
know
you
can
hack
up
the
GitHub
copilot
plugin
in
various
ways
to
make
it
talk
to
Pho
pilot,
but
a
lot
of
things
might
be
a
little
bit
broken
and
it'd
be
really
really
nice
to
be
able
to
say,
hey,
look:
here's
a
actual
Visual
Studio
code
extension
that
you
can
install
that
will
work
with
copilot,
specifically
and
I.
Then
I'd
definitely
be
very
happy
to
start.
C
B
C
That
will
be
awesome,
yeah,
yeah
and
I.
Think
we
all
also
discussed
is
like
cicd.
That's
probably
something
that's
going
to
be
quite
crucial,
so
right
now
I'm
hosting
it
for
on
gitlab
that
does
have
cicd
but
yeah.
Maybe
we
could
have
a
follow-up
conversation
on
that
on
how
to
make
that
more
publicly
available.
A
D
A
B
Sure,
yeah
and
I
guess
we're
a
little
bit
over
time.
So
I
don't
want
to
keep
everyone,
but
thanks
very
much.
You
know
it's
great
to
talk
with
folks
and
I'm
very
excited
to
hopefully
be
able
to
make
full
pilot
a
lot
better
and
you
know
have
a
lot
more
people
actually
being
able
to
use
it
to
public,
build.