►
Description
Presented by
Benedetta Dal Canton, Co-founder & Developer, Ponicode
Hamza Sayah, Data Scientist, Ponicode
In this talk, Ponicode's Cofounder, Benedetta Dal Canton, will share a roundup of the latest concepts in natural language processing that researchers and tech startups love. She will then chat with Ponicode's Innovation Lead, Hamza Sayah, about his work in NLP and beyond. This session is fully dedicated to developers who want to wrap their head around what the fuss is all about.
For more from GitHub Universe 2020, visit https://githubuniverse.com
As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub
Thanks!
Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github
About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com
A
As
you
can
imagine,
machine
learning
and
nlp
in
particular,
are
very
important
topics
for
us
at
pony
code.
So
today,
I'm
here
because
I'd
like
to
explain
some
of
the
key
concepts
from
these
topics.
This
is
meant
to
be
a
very
much
of
an
introduction,
so
it's
suitable
for
people
who
have
absolutely
no
pre-existing
knowledge
of
both
ml
and
nlp.
A
But
if
you
do
know
lots
about
these
topics,
it
might
still
be
worth
staying
with
us,
because
I
can
guarantee
that
it
will
be
a
little
bit
different
from
other
introductions
to
ml
that
you
might
have
seen,
because
to
make
it
a
little
bit
less
dull.
I've
decided
to
present
it
to
you
in
the
form
of
a
story.
A
They
were
all
really
talented.
So
this
is
terrible
news
for
for
the
company,
obviously,
and
the
hr
department
realizes
that
they
absolutely
need
to
do
something
about
it.
They
need
a
way
to
predict
if
the
developers
are
happy
or
unhappy
so
that
they
can
act
before
they
receive
a
letter
of
resignation.
A
So
what
they
do
is
that
they
go
to
the
city
of
the
company
and
they
ask
him
to
help
them
and
the
city
of
being
a
massive
machine
learning
fan.
He
decides
that
he's
going
to
take
this
opportunity
to
practice
his
skills
a
little
bit
and
he
decides
he's
going
to
provide
the
hr
department
with
a
model
that
can
predict
which
developers
are
happy
and
which
developers
are
unhappy.
A
So
he
starts
thinking.
He
knows
that
the
first
step
in
any
machine
learning
application
is
deciding
how
to
represent
your
data
points.
So
in
this
case
the
data
points
are
the
developers,
and
so
he
says,
okay,
I'm
going
to
keep
it
simple
and
use
just
two
parameters,
so
he
starts
thinking.
What
could
I
use,
but
maybe
beauty
and
charisma-
maybe
you
know
they're
not
happy
because
they're
not
very
beautiful
and
not
very
charismatic,
but
then
he
looks
like
he's
developers
and
he
realizes
that.
A
Surely
this
is
not
a
good
metric
in
his
case,
because
all
his
developers
are
really
beautiful
and
really
charismatic,
like
most
developers,
are
really.
We
all
know
that,
and
so
he
starts
thinking
again.
Maybe
I
can
find
something
better.
Maybe
I
could
look
at
their
code
quality.
So
if
I
take
the
coverage,
the
test
coverage
and
the
documentation
coverage
for
each
developer,
maybe
I
could
find
a
pattern
there.
So
he
looks
he
looks
at
the
developers
and
realizes
that.
A
At
this
point
he
can
feed
all
this
data
to
a
machine
learning
model.
Finally-
and
in
this
case,
he
uses
a
classification
model
and
decides
to
to
train
it
now,
the
job
of
the
classification
model
is
to
divide
the
representation
space
in
a
way
that
maximizes
the
number
of
developers
that
are
classified
correctly
for
our
training
set.
So
in
our
case,
which
is
very,
very
simple,
we
see
that
this
is
simply
a
straight
line,
but
of
course
it
becomes,
it
can
be
much
more
complex
in
reality.
A
In
other
examples,
so
using
using
this
system,
the
hr
department
can
predict
which
developers
are
happy
and
which
developers
are
unhappy
and
overall
it
does
a
pretty
good
job
now
I
know
I
know
what
you're
going
to
say
that
of
course,
they're
going
to
be
mistakes.
Of
course,
there's
going
to
be
some
developers
that
are
sad,
even
if
their
code
coverage,
the
test
coverage
and
documentation
coverage
is,
is
good
and
at
the
same
time,
there's
going
to
be
some
developers
that
are
happy,
even
though
their
code
quality
is
very
low.
A
But
this
is
the
thing
about
machine
learning
models.
They
are
at
the
end
of
the
day,
just
statistical
models,
so
we
can't
expect
them
to
be
right
all
the
time
as
long
as
they're
right
most
of
the
time.
So
I'd
say
that,
overall,
this
is
quite
a
good
thing
for
the
hr
department,
but
then
a
month
comes
a
month
goes
and
the
cto
receives
another
call
and
it's
a
hr
again
and
they
say.
Actually
you
know
we're
not
happy
with
this.
It's
it's
not
enough.
A
It's
too
black
or
white
this
system,
and
we
just
end
up
having
developers
put
in
boxes
where
we
know
that
they're
either
really
happy
or
or
really
unhappy,
but
in
reality
some
developers
just
lie
in
between
and
okay.
They
might
not
love
their
job
with
a
passion
but
they're
not
completely
unhappy
and
they're,
definitely
not
close
to
resignation.
A
So
they
suggest
why?
Don't
you
give
us
a
happiness
score
instead,
so
each
developer
would
be
assigned
a
score
from
a
0
to
100
of
happiness,
and
hopefully
this
means
that
hr
has
a
bit
of
a
more
nuanced
vision
into
how
close
a
developer
is
to
leaving
the
company
and
and
they
cannot
adjust
on
the
cases
that
are
very
urgent
and
so
the
the
cto
starts.
Looking
at
his
developers
and
said,
okay,
I
think
I
can
do
this.
I
think
I
just
need
a
different
model,
so
I'm
going
to
use
a
regression
model.
A
A
A
This
is
unacceptable
and
because
they're
french,
they
work
in
a
french
company,
they
decide
to
go
on
strike,
say
we're
not
coming
back
until
you
find
a
more
human
way
to
find
out
about
our
well-being
and
their
representative
proposes
that
maybe
they
should
be
able
to
send
emails
to
the
hr
department
to
tell
them
how
they're
feeling
and
the
hr
should
use
these
emails
to
make
their
judgments
and
act
when
it's
necessary
immediately.
Hr
is
inundated
by
emails
and
they
realize
there
is
no
way
they
will
be
able
to
do
this
job
manually.
A
So
once
again,
they
go
to
the
cto
and
ask
him
for
a
way
to
to
to
help
them
in
the
process.
The
cto
says
no
problem.
Okay,
but
at
this
point,
he's
become
so
obsessed
with
the
fact
that
code
quality
equals
happiness
at
work
that
he
uses
this
criteria
to
label
all
the
emails
that
the
hr
department
starts
receiving.
A
So
his
training
set,
he
labels
all
the
ones
that
talk
about
testing
code,
with
various
degrees
of
happiness,
so
kind
of,
like
let's
say,
50
to
100
and
instead
all
the
ones
that
you
know
where
the
developers
confess
that
they
are
not
big
fans
of
testing.
He
says:
okay,
this
must
be
zero
percent
to
fifty
percent
and
he
has
the
problem
now.
What
is
his
data
set?
A
So
his
data
set
is
very
different
from
what
it
was
before
before
each
developer
had
a
vector
with
a
score
of
documentation,
coverage
and
test
coverage.
But
now
all
he
has
for
each
developers
is
an
email
which
is
essentially
natural
language,
so
he
realizes
that
he
needs
to
use
nlp
and
he's
very,
very
happy
because
he's
never
used
it
before.
So
it's
his
first
time
he
decides
to
apply
techniques
of
sentiment,
analysis
to
detect
if
each
developer
is
happy
or
unlikely
or
unhappy.
A
Sorry,
based
on
on
his
email
now
before
you
comment
that
sentiment,
analysis
does
not
cover
the
whole
of
nlp.
This
is
absolutely
true.
It's
only
one
of
many
tasks
that
nlp
can
do,
but
I
decided
to
use
this
as
an
example,
because
it's
for
the
sake
of
simplification,
but
also
because
most
like
many
techniques
used
in
sentiment,
analysis
are
actually
the
root
of
nlp
as
a
whole.
So
let's
get
back
to
our
story
and
actually
my
screen
froze
for
a
second.
A
A
Sorry
now
the
cto
notices
immediately
that
sometimes
there
are
small
mistakes
or
differences
in
how
a
word
is
spelled
and
differences
between
uppercase
and
lowercase,
and
he
realizes
it's
in
his
interest
to
remove
these
discrepancies
because
it
doesn't
look
like
they
add
much
value
to
the
meaning
of
the
sentence.
So
he
decides
to
do
a
little
bit
of
preprocessing
and
he
transforms
everything
into
lowercase,
which
means
that
we
end
up
with
a
slightly
more
limited
number
of
unique
tokens,
and
we
can
identify
the
same
word
even
when
it's
capitalized
in
a
different
way.
A
Then
he
notices
that
there
are
some
words
that
are
used
in
different,
maybe
some
verbs
that
are
used
in
different
sentences
in
different
sentence,
different
tenses
in
different
sentences.
Sorry
or
some
adjectives
that
I
used
sometimes
are
singular,
sometimes
in
a
plural,
and
it
would
make
sense
to
reduce
those
just
to
the
essence
of
what
it
is.
So
what
he
does
is
he
breaks
down
all
these
words
where
he
notices
this
behavior
and
he
breaks
them
into
some
words.
A
A
A
Now
the
question
is:
what
do
we
do
with
these
words?
So
when
we
had
code
coverage
and
and
documentation
coverage,
it
was
quite
easy
to
picture
how
to
represent
each
other
space.
But
how
do
we
do
this
for
words?
So,
basically,
we
need
to
go
from
words
to
numbers
if
we
eventually
want
to
represent
a
whole
email
with
numbers
with
a
series
of
numbers.
A
A
Let
me
explain
a
bit
more,
so
if
we
take
an
example,
if
I
notice
that
the
word
bug
appears
very
often
near
the
word
sad
and
almost
never
near
the
word
happy,
then
I
can
deduce
that
basically,
they
belong
to
the
same
family
of
meaning,
and
we
have
some
very
powerful
algorithms
in
nlp
that
can
exploit
this
information
and
they
create
a
numerical
representation
for
words
and
they're
called
embeddings.
Now
embeddings
help
us
obtain
a
space
that
looks
a
little
bit
like
this,
where
we
can
find.
A
In
our
example
positive
words
all
cornered
on
the
top
right
and
negative
words
are
all
on
the
bottom
left
and
all
neutral
words
are
in
the
middle.
So
this
is
good,
so
we
have
a
representation
space
for
all
of
the
words
that
can
appear
in
any
email.
But
what
does
the
cto
do
at
this
point?
So
he
needs
to
exploit
this
mechanism
one
moment.
A
So,
for
example,
he
can
take
each
of
the
words
that
make
up
an
animal
and
take
the
vector
that
represents
them
and
make
the
mathematical
mean
between
them.
So
if
we
look
at
it
in
our
embedding
space,
so
we
have
lots
of
individual
vectors
and
we
end
up
having
only
one
vector
that
represents
the
whole
sentence,
that's
kind
of
like
in
the
center
between
all
of
them,
and
now
we
can
repeat
this
for
all
of
my
emails
and
we
end
up
having
something
that
looks
like
this,
which
is
very
similar.
A
A
A
So
the
output
of
this
model
is
that
the
hr
department
can
predict
the
happiness
score
for
each
new
email
that
comes
in
pretty
much
in
the
same
way
that
they
did
before,
with
with
the
regression
model
that
we
looked
at
before,
and
this
looks
fantastic
at
first
sight.
But
if
we're
honest,
this
is
completely
based
on
how
good
the
mapping
for
the
words
is,
and
it
means
that
it's
also
only
based
on
semantics.
A
So
we
reconduce
those
two
here.
We
see
that
it
seems
to
have
quite
positive
meaning,
and
we
know
that
this
is
absolutely
impossible-
a
developer
that
never
tests
their
code
cannot
be
happy
really.
So
it
looks
like
there
is
a
problem
and
the
cto
needs
to
fix
it.
So
what
does
he
do?
He
looks
up
his
machine
learning,
notes
and
decides
to
add
another
ml
piece
to
his
to
help
his
embeddings,
and
this
new
piece
is
called
recurrence
now
our
embeddings,
so
the
red
piece
didn't
really
look
at
the
relationship
between
words.
A
A
So
basically,
when
we
move
from
one
word
to
another,
it
keeps
some
sort
of
shorter
memory
and
it
can
alter
the
meaning
of
a
word
based
on
what
came
before,
and
especially
it's
able
to
look
at
words
that
are
present
in
many
different
semantic
contexts
and
don't
seem
to
have
a
specific
value
of
their
own,
and
these
are
words
like
very
some,
not
etc
and
recognize
that
actually
they're
not
useless.
They
do
have
some
utility
because
they
react
to
the
semantic
value
of
other
words
in
the
same
sentence.
A
Let
me
see
okay,
yes,
so
he
puts
together
the
two
pieces
and
looks
at
what
happens.
So
the
new
model
starts
looking
at
the
tokens
in
order
so
takes
them
one
by
one.
So
first
we
see
the
word
I
for
the
sentence.
I
really
never
test
and
I
is
quite
neutral,
so
nobody
says
anything,
but
then
we
get
to
the
word
really
and
really
has
has
an
amplifying
meaning.
A
A
At
the
same
time,
yellow
peace
keeps
saying
amplify,
negate
amplifying
the
gate
and
because
we're
at
the
end
of
our
sentence,
once
we
put
the
two
together,
we
have
the
embedding.
That
tells
us
that
it's
a
happy
email
and
the
recurrence
that
tells
us
to
amplify
and
negate
whatever
the
embedding
said.
So
in
the
end,
we
found
out
that
the
developer,
who
sent
the
email
is
very,
not
happy,
which
is
good.
It's
what
we
wanted.
So
cto
is
super
happy.
A
He
sends
a
new
model
to
hr
and
says
use
this
for
all
your
future
predictions,
but
we
have
another
problem.
We
know
that
chatty
developers
do
exist,
they're
quite
rare,
but
they
exist.
We
all
know
some
of
them
and
one
day
hr
receives
a
very,
very
long
email
in
which
this
developer
decides
to
be
oversharing.
A
He
talks
a
lot
about
his
life,
not
just
a
professional
side.
He
talks
about.
You
know
his
wonderful
wife,
his
children,
where
he's
going
on
holiday,
his
friends,
etc
and
somewhere
in
there.
He
also
says,
but
I
never
test
my
coat
but
overall
he's
conveying
like
enthusiasm
about
the
rest
of
his
life
so
having
some
problems
with
slides
again,
okay.
A
So
this
means
that
when
the
nlp
algorithm
that
we've
prepared
processes
this,
it
will
make
a
prediction.
The
developer.
The
developer
is
really
really
happy,
but
the
cto
looks
at
this
and
says:
oh
my
god
not
again.
This
is
not
possible.
It
failed
because
if
this
developer
doesn't
test
this
code,
there's
absolutely
no
chance
that
he's
really
happy.
There
must
be
a
mistake.
So
all
this
chatting
about
wives
and
children's
and
holidays
has
confused
my
algorithms.
I
need
to
do
something
about
it.
A
So
once
again
he
decides
to
add
a
new
piece
to
the
puzzle.
The
new
piece
is
called
attention.
Attention
is
a
mechanism
that
gives
way
to
important
things
and
ignores
the
things
are
less
important.
So
the
concept
is
that
what
is
the
concept
of
what
is
important
is
completely
dependent
on
on
our
objective.
So
in
our
case,
the
attention
mechanism
will
prioritize
information
related
to
developers,
happiness
at
work,
and
it
will
give
less
weight
to
things
that
belong
to
different
semantic
areas.
A
So
we
introduce
this
piece
and
we
try
them
all
together
and
what
happens
is
that
when
the
the
developer
talks
about
personal
stuff,
the
attention
piece
is
kind
of
distracting
the
other
two
and
says:
don't
pay
attention
at
all.
It
prevents
them
from
doing
their
job,
but
when
we
get
to
something
that
talks
about
code
quality
which
we
know
is
important,
it
shuts
up
and
forces
them
to
do
what
they
have
to
do.
So
they
do.
A
The
usual
so
recurrence
identifies
that
he
has
to
negate,
and
then
the
embedding
piece
recognizes
when
there
is
a
polarizing
word
like
test,
and
that
is
happening
and
finally
about
the
rest
of
the
sentence.
Pretty
initial
and
finally,
when
we
get
back
to
talking
about
holism
friends,
attention
once
again
says:
no,
no,
no,
don't
listen,
don't
pay
attention.
A
When
we
get
to
the
end,
we
put
together
everything
that
we
have
so
embedding
and
recurrence
what
they
were
able
to
guess
in
the
moments
that
they
were
paying
attention,
and
we
realized
that
the
developer
was
definitely
not
happy,
which
is
the
correct
conclusion
for
the
cto.
Cto
is
very,
very
proud
of
himself
and
he
presents
a
new
model
to
hr
and
everybody
is
happy-
and
this
brings
us
to
the
end
of
this
presentation
now.
I
guess
the
meaning
wait
one
second,
once
again,
it
froze
okay.
Yes,
the
the
message.
A
I
want
you
guys
to
keep
from
this.
Apart
from
the
story,
is
that
all
impressive
neural
networks
of
nlp
really
are
just
very
smart
combinations
of
these
mechanisms?
So
we
have
semantic
mechanisms
that
find
meaning
in
words
and
sub
words.
We
have
recurrent
mechanisms
that
capture
the
order,
dimension
of
sentences
and
attention
mechanisms
that
select
interesting
parts
from
text
depending
on
what
our
objective
is,
and
these
are
you
know
some
of
the
the
most
relevant
mechanisms
from
nlp
they're,
not
the
only
ones
but
I'll.
A
I
really
hope
you
enjoyed
it
and
that
you
learned
something
from
it
now
at
the
start
of
this,
I
also
told
you
that
I
am
a
developer,
I'm
not
a
data
scientist,
so
why
is
it
me
presenting
all
of
this
to
you
considering
I
work
in
a
company
where
there
are
many
great
data
scientists,
but
the
reason,
let
me
introduce
the
reason
for
you.
His
name
is
hamsa
saya
and
he's
joining
me
now
on
this
virtual
stage,
hello,
hamza.
A
Hi
around
eight
months
ago,
hamza
introduced
to
pony
code
a
new
format
that
he
called
ml
inspiration.
So
every
week
hamza
holds
a
30
minutes
presentation
about
a
machine
learning
topic
of
his
choice
for
the
benefit
of
the
whole
team,
so
developers,
data
scientists,
marketing
team
sales,
anything
and
that's
20,
minutes
of
presentation,
plus
10
minutes
of
q
a-
and
I
am
always
there.
A
So
we
actually
thought
it
would
be
cool
for
me
to
try
and
give
back
and
make
a
talk,
inspired
by
hamza's
presentation
and
explain
kind
of
the
concepts
that
I
learned
from
him.
But
machine
learning's
inspiration
is
so
loved
at
pony
code
that
we
thought
we'd
actually
share
a
little
bit
more
about
the
format
that
might
inspire
some
of
you
to
do
the
same,
and
there
is
no
better
person
to
do
so
than
hamza
himself.
B
B
I
figured
that
we
need
to
reduce
this
gap
in
order
to
work
together
well
and
know,
and
I
know
that
my
machine
learning
knowledge
comes
from
years
of
study,
so
I
shouldn't
really
expect
others
to
understand
immediately.
When
I
speak
about
something
technical,
I
watch
a
lot
of
videos
on
youtube
and
I'm
a
big
fan
of
scientific
vulgarization.
B
So
so
I
do
this
often,
and
I
know
the
points
where
people
block
directly
and
I
reflect
on
it
a
lot
and
try
to
find
analogies
and
tangible
examples
that
can
make
the
concept
more
clear.
So
in
reality
the
larger
part
of
my
preparation
is
done
in
this
informal
context
with
my
friends
and
people
I
and
people
I
know.
A
B
I'd
say
there
were
three
effects.
I
noticed
first
inspiration
which
I
guess
was
the
main
goal
of
the
the
given
the
name
I
have
had
people
come
in
to
me
with
new
ideas
inspired
by
my
presentations.
Second,
communication
between
devs
and
data
scientists
has
been
more
fluid.
Many
cases
we
know
can
use
words
from
the
machine
learning
dictionary
without
fear
of
being
misinterpreted,
and
it
happened
a
lot
less
to
waste
time
debating
semantics
like
before.