►
From YouTube: Open source AI for code - Universe 2022
Description
Presented by: Vincent Hellendoorn
Following the release of GitHub Copilot, GitHub’s developer community quickly responded by building new open source neural code models and tools built on top of GitHub Copilot. In this talk, Vincent will review several of these models and discuss their design, impact, and trade-offs compared with GitHub Copilot. Vincent will then take a broader look at the emerging trends in this space and discuss the opportunities and challenges for open source AI tool builders.
As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub
Thanks!
Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github
About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com
A
A
Hi,
my
name
is
Vincent
I'm,
a
professor
at
Carnegie
Mellon
University,
where
I
study
how
AI
can
help
software
Developers
I
chose
this
direction,
because
I
firmly
believe
that
AI
can
help
make
programming
easier,
safer
and
more
accessible,
I'm,
also
a
programmer
and,
as
such,
I've
always
been
passionate
about
the
transformative
power
of
Open
Source
Code.
In
the
past
few
years,
something
amazing
has
happened.
These
two
interests
have
started
to
overlap
with
the
arrival
of
tools
like
copilot
AI
is
both
learning
from
and
serving
open
source
Developers.
A
So
it's
perhaps
no
surprise
that
I
spent
part
of
my
time
as
a
visiting
researcher
at
GitHub
working
to
improve
AI
solutions
for
programmers
by
trying
to
better
understand
the
needs
of
Open
Source
Developers.
Based
on
this
experience,
here's
my
pitch
AI
for
code
needs
open
source.
I,
don't
mean
for
data.
I
mean
that
open
source
is
a
key
part
of
the
future
of
AI
I,
actually
think
this
is
true
for
all
of
AI,
but
I
will
focus
on
code,
because
in
this
talk,
because
that
is
the
domain
I
know
best.
A
If
you
spend
time
on
the
eyesight
of
GitHub,
you've
probably
already
noticed
this
when
new
models
are
proposed
in
the
research
Community
developers
on
GitHub
quickly
provide
implementations
and
come
up
with
improvements.
For
example,
if
you
search
for
a
Cutting
Edge
machine
learning
model,
I
can
almost
guarantee
that
a
nice
implementation
by
atlucid
rains
will
come
up.
There
are
now
more
than
a
million
GitHub
repositories
related
to
Ai,
and
this
number
is
climbing
fast.
So
far
this
year,
more
than
a
thousand
such
repositories
have
come
alive
every
day.
A
This
helps
me,
as
a
researcher
I
often
prefer
to
read
and
run
the
code
over
equations
in
a
paper,
and
it
helps
the
community
to
create
new
variations
of
proposed
models.
Models
themselves
are
also
more
and
more
often
released
in
platforms
like
hugging
face
developers
are
integrating
these
into
new
smart
tools
and
calibrating
them
for
new
scenarios.
A
So
the
open
source
Community
is
accelerating
the
improvements
in
Ai
and
is
making
AI
more
broadly
accessible
and
useful.
I
will
share
some
examples
in
this
talk
specifically
I'm
going
to
do
three
things.
First
I'm
going
to
do
a
bit
of
teaching
I'm
sorry,
but
I
cannot
help
it
and
explain
the
very
basics
of
what
happens
behind
the
scenes
in
tools
like
copilot
that
generate
code,
for
you,
I
will
tell
you
about
what
language
models
are
and
what
has
changed
in
the
last
few
years
that
makes
them
so
powerful.
A
A
Finally,
I
will
talk
about
the
open
challenges
that
remain
not
just
to
make
better
models,
but
to
build
tools
that
are
more
accessible
and
useful
to
developers
when
explaining
how
AI
for
code
works,
I
always
like
to
start
with
a
code
snippet
any
developer.
Looking
at
this
will
recognize
that
we
are
about
to
start
writing
a
for
Loop.
That
is
we're
about
to
iterate
over
something
we
can
tell,
because
we've
probably
read
and
written
many
lines
of
code
with
this
keyword.
A
So
when
we
see
the
font
in
the
blinking
cursor,
we
know
what's
up,
we
don't
quite
know
what
comes
next.
If
you
are
a
Java
or
a
sea
developer,
you
probably
expect
an
open
parenthesis,
since
that
is
the
syntax
used
there
I
ride
more
python
than
anything
else
these
days.
So
my
first
response
is
to
expect
something
like
this.
A
Now,
maybe
focusing
on
python
would
have
lost
me
half
the
audience,
but
that
is
actually
part
of
the
challenge.
If
you
want
to
build
a
tool
that
supports
programmers,
it
has
to
work
for
people
doing
very
different
things.
There
are
hundreds
of
programming
languages
that
are
actively
used
in
GitHub
right
now.
If
I
read
on
and
see
these
next
few
tokens,
I
am
pretty
confident
that
we
are
looking
at
python
code
and
I
might
think
further
ahead.
I
have
more
context
now.
A
I
know
that
we're
in
the
for
Loop
and
the
loop
variable
is
called
I,
so
odds
are
that
We're
looping
over
a
range
of
integers
I'd
expect
the
next
token
to
be
range,
and
maybe
the
ones
after
that
to
specify
a
number.
This
is
what
language
models
do
in
a
typical
language
model.
The
goal
is
to
generate
text
or
code
from
its
context.
Let's
see
that
in
action.
In
the
previous
example,
we
first
saw
four
and
based
on
that
we
might
have
predicted
I
to
follow
once
we
confirm
that
that
is
the
next
token.
A
We
can
then
extend
our
context
with
it
and
query
the
model
again
to
predict
another
token
in
Python.
There
will
always
be
in
so
that's
easy.
We
repeat
this
process
expanding
our
code
until
we've
hit
some
stopping
points
like
the
end
of
a
line
now,
obviously
the
second
token
won't
always
be
I.
Code
is
creative
like
other
languages,
so
the
model
can
be
wrong
or
it
can
generate
a
correct
implementation
that
is
different
in
style
from
what
a
developer
needs.
We
train
these
models
by
encouraging
them
to
write
code.
A
That's
more
like
what
we
might
find
in
public
code
on
websites
like
GitHub.
Now,
in
practice,
the
model
isn't
a
simple
black
box.
It's
a
complex
neural
network
called
a
Transformer
that
contains
many
parameters.
What
has
changed
since
the
ancient
days
of
2020
is
that
this
many
has
started
being
measured
into
billions
instead
of
millions,
and
so
has
the
number
of
lines
of
code
that
these
models
are
trained
on.
A
That
might
not
seem
like
a
big
shift,
but
trust
me
when
I
say
it
is
these
many
billion
parameter
or
as
we
commonly
call
them
large
language
models
are
much
more
expensive
to
train,
but
they
are
also
far
better
at
predicting
source
code.
Let's
zoom
in
a
bit
on
one
of
these
models
here
called
codex
codex
was
the
first
large
language
model
of
code.
It
is
also
the
model
power
covering
github's
co-pilot.
It
arrives
most
of
its
strength
from
training
a
fairly
simple
but
very
large
model
on
a
lot
of
code.
A
The
result
is
a
model
that
can
quite
regularly
write
entire
methods
for
you,
given
a
reasonable
prompt
just
based
on
the
millions
of
patterns
it
has
seen.
One
of
the
things
this
work
proved
is
that
training
large
models
is
pretty
much
a
requirement
for
coming
anywhere
near
the
capabilities
of
programmers.
Programming
is
objectively
hard
and
small
models.
Just
don't
do
well
at
generating
correct
code
for
difficult
tasks.
That's
bad
news
in
a
way,
since
these
large
models
are
both
really
expensive
to
train
and
quite
expensive
to
query.
A
It
takes
a
lot
of
dedicated
Hardware
to
make
this
happen,
but
going
back
to
discharge
you'll
notice,
there
are
quite
a
few
other
large
language
models,
many
of
them
in
Orange,
which
means
they
are
publicly
available
right
now.
In
fact,
several
more
open
source
models
have
come
out
since
this
chart
was
made.
Most
of
these
models
were
trained
after
the
release
of
codecs,
in
part,
to
better
understand
the
challenge
of
training
such
large
models
for
code
and
in
part
to
meet
the
needs
of
the
open
source
Community
for
freely
accessible
models.
A
Let's
take
a
look
at
three
of
these.
The
first
model
was
code
parrot.
This
was
a
1.5
billion
parameter
language
model,
trained
on
50
gigabytes
of
python
code
from
GitHub.
That
is
available
right
now.
In
hugging
phase,
it
comes
with
a
helpful
blog
post
and
what
it
takes
to
collect
this
data
and
train
such
a
model,
and
it
has
released
all
its
training
data.
In
case
you
want
to
train
your
own.
When
code
paired
came
out,
it
inspired
me
and
several
collaborators
at
CMU
to
train
a
separate
model
that
we
nicknamed
polycoder.
A
The
main
reason
we
did,
that
was
to
support
other
programming
languages
besides
python.
So
correspondingly
we
trained
a
somewhat
larger
model
to
make
sure
it
had
the
capacity
to
learn
patterns
that
span
across
many
programming
languages.
Our
model
supports
12
languages
in
one
and
is
quite
strong
on
some
of
these
for
a
while,
it
was
even
slightly
better
at
predicting
C
code
and
codex,
though,
of
course
the
latter
has
evolved.
A
Since
then,
a
few
months
later,
a
series
of
models
called
Cogen
came
out
and
knocked
us
off
the
largest
open
source
model
of
code
thrown
with
models
going
up
to
16
billion
parameters.
These
include
both
models
with
multilingual
and
with
python.
Only
support
a
niche
related
project
is
faux
pilot
built
by
a
professor
at
NYU.
A
Faux
pilot
sets
up
a
server
around
to
code
gen
models
that
lets
you
host
a
neural
code,
completion
service
on
your
own
Hardware.
You
can
actually
point
codepilot
to
this
servers
and
it
will
serve
up
completions
from
your
own
models
using
the
same
plugin.
One
thing
that
this
highlights
is
that
copilot
isn't
a
model,
it's
a
tool,
just
one
that
talks
to
codecs
by
default.
A
It
also
helps
us
understand
some
of
the
challenges
for
imitating
this
with
open
source
models
of
code.
Let
me
explain
that
by
means
of
a
graph
in
this
plant,
I'm
going
to
compare
the
performance
of
a
whole
bunch
of
models
on
the
x-axis
I
will
show
the
performance
of
each
model
expressed
in
terms
of
how
good
it
is
at
implementing
an
algorithm
from
a
detailed
specification.
Think
of
the
algorithms
as
typical
lead
code
problems.
This
reflects
the
model's
ability
to
write,
challenging
programs
on
the
y-axis.
A
So
what
this
performance
reflects
is
How
likely
is
the
model
to
be
able
to
implement
an
entire
function
in
a
real
project
correctly
in
one
go.
I
ran
the
models
on
functions
from
about
100
random
python
projects.
So
let's
look
at
the
results.
The
first
two
models
I
talked
about
were
code
parrot
and
polycoder
code
pair.
It
is
the
smaller
of
the
two
and
was
the
very
first
open
source
model,
so
it
makes
sense
that
it
doesn't
quite
do
as
well.
A
Polycoder
is
a
bit
better
at
algorithmic
tasks,
but
especially
performs
a
good
deal
better
on
real
functions.
Solving
about
one
in
five
problems
in
a
single
guess
now,
you
might
have
noticed
that
the
two
circles
are
a
different
size
I'm
using
size
to
indicate
how
fast
each
model
is
in
terms
of
how
many
tokens
it
can
generate
per.
Second.
A
Smaller
models
are
faster
to
query,
which
matters
for
some
applications,
I'm,
also
using
a
different
outline
for
polycoder
to
highlight
that
it
is
multilingual.
Next
came
to
various
code,
gen
models,
these
actually
come
in
four
sizes,
and
you
can
see
the
progression
from
left
to
right
as
the
models
get
bigger
and
better.
The
final
model
can
predict
over
30
percent
of
real
world
functions
correctly
in
one
guess,
one
big
advantage
that
these
models
have
over
the
previous
models
that
makes
them
better
is
that
they
were
trained
for
much
much
longer.
A
They
are
also
all
monolingual
for
comparison.
The
multilingual
version
of
the
largest
model
is
quite
a
bit
weaker,
scoring
somewhere
around
a
monolingual
model
about
1
8
its
size.
An
important
thing
to
notice
is
that
throughput
goes
way
down
when
models
get
this
big
again.
Looking
at
the
size
of
the
circles,
the
largest
model
takes
quite
a
while
to
generate
longer
functions
even
on
high-end
Hardware.
It
also
comes
with
a
significant
memory
footprint,
so
most
laptops
and
desktops
can't
realistically
run
this
model.
For
this
reason,
many
developers
may
prefer
to
use
a
smaller,
faster
model.
A
All
that
said,
codex
is
quite
a
bit
ahead.
It
solves
about
half
of
each
type
of
challenge
function
in
a
single
guess.
It's
also
multilingual,
and
it
is
really
fast
to
query,
because
it
is
hosted
in
the
cloud
where
it
can
take
advantage
of
much
more
powerful
resources,
and
that
makes
sense.
Codex
was
the
first
Model
to
come
out
and
as
a
commercial
product
that
has
different
constraints
from
open
source.
Solutions,
open
source
models
need
to
be
lightweight
and
faster
query,
which
naturally
comes
with
a
performance
penalty.
Open
source
models
also
have
their
advantages.
A
A
So,
for
the
time
being,
the
story
to
me
is
this:
open
source
models
have
gotten
quite
a
bit
better
in
a
very
short
period,
but
they
do
operate
in
a
different
performance
domain
from
copilot
The
Optimist
in
me
says
hey.
That
means
there's
clearly
quite
a
bit
of
room
for
improvement
ahead.
So
let's
talk
about
what
is
ahead,
both
how
we
might
navigate
the
challenges
facing
current
open
source
Solutions
and
what
I
think
are
promising
opportunities
in
this
space.
A
Based
on
these
results,
I
just
presented
three
targets
seem
worth
pursuing
to
make
open
source
models
more
powerful
and
accessible
to
regular
Developers
one
to
work
in
a
typical
software
development.
Environment,
open
source
models
need
to
be
light
and
fast,
there's
quite
a
bit
of
unexplored
territory
here,
including
model
quantization
that
has
shown
really
promising
results
in
other
fields.
I'm
very
optimistic
about
this.
Looking
at
image
generation,
another
AI
for
creativity
domain,
the
open
source
Community
was
incredibly
quick
to
create
and
accelerate
open
source.
Diffusion-Based
image
generators
after
Dali
2
came
out.
A
I
wouldn't
be
surprised
if
models
of
code
can
also
become
two
to
four
times
as
small
and
fast
as
they
are
today
without
performance
loss.
In
the
next
year,
two
developers
rarely
write
in
a
single
programming
language,
so
models
that
support
multiple
languages
are
going
to
be
more
useful.
However,
as
we
saw
previously,
this
decreases
model
performance
compared
to
a
model
trained
for
one
language.
Only
one
solution
is
to
build
a
catalog
of
small
language,
specific
models
for
a
range
of
programming
languages,
perhaps
all
built
in
the
same
shared
base
model.
A
This
would
lend
itself
really
well
to
the
kind
of
distributed
setup
at
which
the
open
source,
Community
excels
and,
finally,
there's
always
room
for
improvement,
and
we
clearly
aren't
close
to
Perfection
yet,
fortunately,
improving
model
quality
isn't
just
a
topic
for
academic
research
anymore.
These
days,
I
get
as
many
of
my
insights
and
ideas
from
open
source
projects
on
GitHub
as
anywhere
else.
A
If
you
want
to
contribute,
there
are
several
concerted
efforts,
such
as
big
Sciences,
big
code
project,
which
aims
to
train
a
15
billion
parameter
model
of
Code
by
bringing
together
many
experts
working
on
different
components
in
parallel.
Overall
I
am
excited
for
the
future.
Models
like
the
ones
that
have
emerged
in
the
last
two
years
have
the
potential
to
save
developers
many
hours
of
tediously
writing
code,
which
frees
us
all
up
to
focus
on
the
more
creative
big
picture
aspects
of
software
engineering.