►
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from April 17-21, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Training AI To Code Using the Largest Code Dataset - Tommy Li & Animesh Singh, IBM
Speakers: Animesh Singh, Tommy Li
Project CodeNet is a large dataset of 14 million code samples totaling 500 million lines of code in 55 programming languages. It enables machine learning for code, like finding code similarity, extracting semantic context, and even translating between different programming languages. Using the Machine Learning Exchange (MLX), a Linux Foundation for AI & Data Sandbox Project, we demonstrate how Project CodeNet can be leveraged to classify code and analyze code complexity in three steps. Using DataShim we turn domain specific subsets of the data into Kubernetes Custom Resources. Running Jupyter notebooks on Kubernetes we use the datasets to train deep learning models. The models are then served for inferencing as Kubernetes Custom Resources using KServe. For each of these steps, MLX generates Kubeflow Pipelines on Tekton so data scientists are not required to write Kubernetes specific code.
A
Thanks
everyone
for
coming
to
the
last
session
of
the
day,
my
name
is
Tommy
I'm,
a
senior
software
developer
at
IBM's,
and
today
we
are
going
to
talk
about
how
to
training
like
AI
to
call
using
the
largest
code
data
set
called
konet
that
will
release,
and
in
the
past
few
decades
we
have,
you
know
focused
a
lot
on.
You
know:
training
AI
on
natural
languages,
but
there's
also
like
different
kind
of
other
form
of
languages
that
we
use
in
a
day-to-day
basis,
such
as
in
the
molecules.
A
We
have
different
kind
of
representations
for
molecules
stories,
imagery
and
especially
for
coding
as
well,
and
today
we
want
to
focus
on
how
we
actually
want
to
emphasize
on
building
AI
for
code
and
as
a
power
initiative
in
IBM
research.
We
have
this
new.
You
know
program
called
afo
code,
which
we
want
to
focus
on
teaching
the
machine,
the
language
of
machines,
and
it
is
very
new
in
terms
of
like
fields
in
computer
science,
and
we
leverage
a
lot
of
the
you
know.
A
You
know,
tools
to
help
your
software
developer
to
build
your
practical
tasks
such
as
you
know,
code
search,
summarizations,
completions
and
protocol
translations,
and
with
all
these
tools
right,
we
are
hoping
to
able
to
like
accelerate
the
output
to
you,
know,
modernized
magazine
or
software,
and
how
to
migrate
their
model
ethics
software
into
like
a
new.
You
know
microservice
architectures,
and
you
look
in
the
past
right
when
Engine
imagenes
just
announced
they
first
have
like
three
million
image,
and
you
know
5000
class
and
just
put
in
three
years
of
spam.
A
As
we
see
that
a
lot
of
the
governments
and
banking
automobiles
software
they're
still
using
like
you
know,
legacy
codes
like
covers
right,
and
these
are
more
than
220
billion
lines
of
you
know,
Legacy
Global
codes
and
as
we
were
like,
we
don't
have
that
many.
You
know
code
developers
they're
able
to
translate
this
into
modern
Technologies.
A
You
know
with
deep
learning
and
GPU
accelerations
and
all
the
research
we're
able
to
achieve
a
human
level
accuracy
just
within
six
years
and
even
with
IBM,
will
be
able
to
like
produce
a
lot
of
NLP
based.
You
know,
services
and
document
understanding
using
this
new
like
AI
Technologies.
So
we
want
to
focus
the
same
Technologies
on
core
as
well,
so
it
helps
us
to
do
nlpn
MLS,
for
you
know,
software
artifacts,
other
main
reasoning
and
decision
making
and
furthermore,
is
one
able
to
explain
like
how
this
code's
been
generated.
A
So
we
create
an
explainable
AI
as
well
and
with
all
this,
we
need
to
create
a
new
data
set
right,
just
similar
to
image
in
open
source.
So
you
know,
developer
can
use
this
to
create.
You
know
like
code,
language,
translation
code,
search
code,
similarities,
code,
performance,
Improvement
and
core
memory,
improvement
and
code
classifications
or
examples
and
algorithms.
So
we
got
to
use
this
and
put
into
like
more
specific
use
case
for
each
Developers,
and
this
is
why
we
want
to
announce
the
project
co-net.
We
have
released
the
protocol
net.
B
A
Set
to
the
open
source
last
year,
this
data
set
is
actually
a
high
quality
code
data
set
of
augmented,
Innovation
and
benchmarking.
It
is
very
large
scale,
so
you
can
see,
there's
40
millions
of
code
samples
right
more
than
4
000
code
problem
is
out
there
and
it
covers
55
different
languages
from
like
Global
ancient
Legacy
language
to
the
latest.
Modern
C,
plus,
plus
Java
Python
language
as
well
and
Optical
are
actually
well
tested,
and
it
also
provided
a
various
test
case.
A
So
you
can
choose
training
on
top
of
the
different
classes
and
make
sure
that
you
could
understand
how
this
code
are
being
ran
and
how
this
code
has
been
failed
as
well
and
from
some
of
the
examples
we
have
seen
like
the
potential
use
case
is
actually
modernized.
You
know
legacy
codes,
so
one
of
the
kinds
we
have
is
the
model
Automotive
clients
they
have
very
old.
A
You
know
like
stack
of
java
legacy
codes,
multiple
version
monologue,
applications
more
than
3500
driver
files
and
more
than
really
nice
of
calls-
and
we
want
to
you
know,
might
we
even
migrate
them
into
the
modern
architectures
and
in
our
initial
effort,
when
we
actually
estimate
how
many
therapists
and
how
many
effort
we
need
to
spend
yeah?
Actually,
at
least
it
needs
to
take
us
a
year
to
migrate.
All
these,
you
know
code,
internet,
modern,
Java
versions.
A
A
A
A
A
What
was
all
this
data
set
like
that's?
This
is
how
we
kind
of
collect
all
these
data,
so
we
actually,
like
you,
know,
talk
to
the
Azure
online
church
and
at
code
and
help
help
them
ask
them
to
able
to
collect
those
data
for
us.
This
actually
contains
like
more
than
four
thousand
problems
and
have
more
than
30
million
some
submissions
right
and,
as
we
described
before,
like
more
than
half
is
actually
accepted.
A
So,
let's
dive
into
a
little
bit
on
like
what
kind
of
metadata
is
actually
provides
so
for
each
of
those
problems
that
we
have
defines
like
kind
of
like
ID
names.
You
know,
time
limit
memory,
limits
and
complexity,
that's
required
for
these
kind
of
problems
and
when
it
comes
to
submission
levels
for
each
of
these
problems
right,
each
submissions
is
able
to
identify
how
many
CPU
times
memory
times.
A
Accuracy
has
been
produced
right
because
we
have
a
tons
of
you
know,
test
cases
available
to
calculate
how
how
much
accuracy
you
could
satisfy
with
these
Solutions
and
once
the
submission
is
done,
what
whether
or
not
it
pass
a
no
pass
will
output
a
status.
So
this
is
actually
how
you
filter
out
like
what
kind
of
problems
that
you
want
to
like
categorize.
So
not
only
does
a
problem
could
be
just
passed
or
fails,
but
you
also
click
like
determine
whether
or
not
this
problem
is
like.
A
And
furthermore,
so
with
all
this
kind
of
information
we
have,
we
also
provide
like
different
tools
and
examples
on
how
you
get
how
you
can
get
started.
With
this.
You
know
data
sets,
so
the
tool
we
provide
is
like
statistic
from
the
data
set.
That
is
the
stuff
you
have
seen
before.
So
we
have
I
told
you,
how
do
you
join
a
different
scientific?
A
You
could
get
subset
of
the
data
set
as
well
using
our
tools
and
also
like
you
could
convert
this
kind
of
data
set
into
like
different
kind
of
data
formats
right
so
by
default.
It's
just
like
you
know,
code
files,
but
you
can
also
convert
them
into
like
a
stack
of
files
or
you
know,
put
them
in
the
text
file
as
well,
and
we
also
provide
different
kind
of
pre-processing.
A
You
know
source
files
to
data,
depending
on
what
kind
of
you
know
model
you
want
to
train,
you
might
want
to
do
different
kind
of
pre-processing
and
we
provide
like
some
simple
tools
like
a
tokenizer
to
generate
estimate
tokens.
You
want
to
build
like
traditional,
you
know
abstract
syntax
tree.
We
could
have
like
AST
generation
as
well,
and
you
want
to
do
like
just
control
and
the
data
flow
graph,
and
we
also
have
like
cool
analysis.
A
B
A
A
You
know
guide
you
how
to
do
that
from
pre-processing
to
actually
producing
your
models
and
test
it
out
on
the
mass
language
model
and
language
classification
that
we
have
fit
and
with
all
this
kind
of
information
and
Tool,
we
are
actually
aiming
to
expose
the
potential
use
case
right
with
these
data
sets,
because
we
can
see
that
afo
code
could
help.
You
know
developers
do
you
know
closed
classification
to
know
what
language
the
code
is,
how
to
do
like
code,
summarization
search.
A
So
when
you
want
to
identify
a
particular
problem,
you
could
just
you
know,
search
it
based
on
the
topics
and
then
you
could
do
the
social
source
code
translation.
So
when
you
migrate
from
Legacy
code,
older
version
of
the
code,
it
could
help
you
like
automate
some
of
those
process,
so
the
Republicans
just
spend
less
time
just
on
you
know,
migrated
man,
earnings
and
lastly,
also,
more
importantly,
is
I
want
to
help
deal
with
the
right,
much
medical
and
faster
goals.
A
So
some
of
the
kind
of
like
Technique
we
have
the
other
is
actually
using
natural
language
to
generate
code.
So
when
you
actually
type
down,
let's
say
I
want
to
like
run
an
array
from
two
to
five
and
you
could
generate
a
function
for
me
to
do
that
and
also
you
know
it
improved
like
existing.
You
know
code
performance
and
memory
Footprints
so
and
it
will
analyze.
A
You
know
code
and
tell
you
the
runtime
complexity
and
able
to
provide
you
a
better
Solutions
and
then,
of
course,
more
importantly,
is
like
the
key
for
this
stuff
help
us,
you
know,
find
different
error
and
debugging
the
existing
code
and
create
you
know.
Code,
test
generation
so
make
the
course
more
robust
than
in
the
long
terms,
and
when
we
see
out
there,
we
have
multiple
in
existing
application.
B
A
A
You
can
just
plain
English,
so
it's
a
use
case
where
you
just
provide
simple
English
on
contacts
and
create
in
the
automations
and
infrastructure
as
code
and
the
the
main
focus
for
us
right
now
is
actually
just
aim
to
build
their
Foundation
models
right
and
with,
while
keeping
the
accuracy
High.
We
want
to
figure
out
how
we
can
reduce
the
number
of
parameters,
so
we
like
this
new
model,
is
able
to
compute
and
we
train
like
in
a
decent
amount
of
times.
A
As
you
can
see
in
like
this.
Like
page,
when
someone
kind
of
like
put
down
a
simple
text
on
installing
in
nginx
and
node.js
12
package
right
on
red
hat,
it
was
able
to
generate
you
something
like
scripts
right.
Some
infrastructure,
that's
called
in
ansible
to
help
you
like
just
run
the
install
package.
So
this
is
kind
of
like
very
useful
tools
for
anyone
who
just
want
to
like
do
simple
task,
but
they
don't
have
any
knowledge
of
infrastructure
in
sascript.
A
So
some
of
the
high
levels
on
what
the
machine
learning
exchange
is
actually
contains.
So
mainly
this
machine
has
changed.
The
main
purpose
is
actually
showing
all
these
catalogs,
so
we
have
different
kind
of
like
data
assets
such
as
data
pipelines,
components,
models,
data
sets
and
notebooks.
Those
are
like
the
core
components
that
other
data
scientists
have
need
to
build
and
share
within
different
teams
and
I.
A
So,
for
you
know,
pipeline
engine
will
actually
like
Leverage,
you
know
Q4
pipelines
and
because
we
actually
run
it
on
openshift.
So
we
actually
also
use
the
Techno
version
of
people
pipeline,
so
we
could
actually
run
the
red
hat.
The
openshift
approved
version
of
the
textile
runtimes
and
for
serving
engine
we
chose
k-serv,
or
we
also
have
an
option
for
you
to
just
Deploy
on
playing
kubernetes
deployments.
A
So
k-stop
is
actually
a
very
popular
project
for
deploying
you
know,
serverless
models
on
top
of
kubernetes
and
then
for
data
sets.
We
used
a
project
called
Data
stream,
where
I
could
to
help
you.
You
know
like
host
all
your
data
set
into
your
local
cluster
and
use
them
within
your
custom
nodes
and,
lastly,
with
all
this
kind
of
execution
on
engine,
we're
actually
able
to
fine-tune
our
data
and
AI
catalog
metadata
and
review
those
metadata
based
on
a
spec
called
ml
spec
in
open
source
as
well.
A
A
A
You're
able
to
see
a
list
of
like
catalog
hosts
on
machine
learning
Exchange.
So
this
is
just
only
a
catalog
page
because
on
on
the
private
server,
we
don't
have.
We
don't
Leverage
The
runtimes
so,
but
you
could
see
it.
What
kind
of
like
data
is
that
you
could
download
and
what
kind
of
model
you
could
use.
So
we
look
at
like
the
list
of
data
sets.
A
You
could
see
like
project
co-net,
different
kind
of
sub
step,
project
code
and
language
classifier,
and
we
click
on
like
let's
say
the
project
code
in
Australia
I
actually
have
like
different
selections
of
the
BSN.
Let's
say
the
full
data
sets
right
like
it
might
contains.
Like
several
gigabytes
of
files,
you
might
not
put
the
downloading
in
one
fly.
We
also
have
different
kind
of
data
selection
where
you
want
to
just
view
the
metadata
or
you
only
need
a
benchmark
for
python
Java
FC
plus.
A
A
And
similarly
with
the
models,
so
we
also
have
like
different
kind
of
models
that
you
could
write
in
containers,
so
just
upload
as
a
model
files.
So,
for
example,
we
have
this
like
co-net
language
classifiers.
Let's
click
into
this,
so
with
this
we
actually
have
determined
like
what
kind
of
like
data
set
is
used.
So
with
this
data
set
we're
using
like
Covenant,
so
it's
under
the
ctla
permission
V2,
but
also
with
the
model
weights
and
the
model
code.
We
actually
make
sure
it's
under
Apache
2.0
license.
A
And
similarly,
with
the
pipelines,
this
is
like
we
leverage
IQ
for
product
behind
the
scenes
where
you
can
actually
see
how
you
could
build
a
simple
partner,
just
leverage,
multiple,
you
know,
tasks
and
join
it
together.
So
when
the
data
scientists
kind
of
build
like
a
full
bloom
pipeline
that
do
like
different
kind
of
data,
pre-processing
different
kind
of
like
data
training
with
how
you
distributed
the
training
process,
this
could
also
built
into
a
single
Pipelines,
and
you
know
upload
it
over
here
and
share
with
different.
A
Data
science
teams
and
each
of
those
you
know,
pipeline
components
also
could
be
shared
into
the
components
categories.
So
this
is
like
the
category
how
each
component
is
being
built
and
you
could
connect
them
into
a
whole
pipelines
and,
lastly,
with
The
Notebook
is
where,
if
you
wanted
to
see
some
examples
and
how
these
examples
has
been
executed,
let's
say,
for
example,
you
could
pick
like
the
project
called
net
Mass
language
models.
A
You
could
see
like
how,
where
is
this
data?
Where
is
this
notebook?
It's
been
host
right
on
githubs,
but
you
could
also
like
Leverage
our
internal
backend
engine,
where
we
actually
help
you
to
rather
than
available
to
preview
on
this
page.
You
just
want
to
feel
like
just
render
and
stay
on
this
page
to
see
how
you
you
know,
let's
say
for
these
examples
for
the
mass
language
models.
A
You
could
see
like
how
you
know
you
could
just
take
a
subset
of
the
model
from
conet
prepare
the
data
right,
do
some
kind
of
tokenization
and
create
a
you
know,
available
models
and
at
the
very
ends
right
like
it's
able
to
train
in
just
using
CPUs
within
an
hours
and
with
for
this
example,
will
make
it
very
small.
So
we
just
train
for
five
parts
and
just
show
you
some
of
the
evaluation
on.
A
A
So
with
this,
if
you
actually
Wonder,
like
you,
know,
find
out
like
what
kind
of
models
or
like
data
set
you
want
to
book
on,
and
you
just
happened
more
than
I
try
out
on
what
should
I
exchange
you
guys
already
Deploy
on
your
own
cluster
and
run
on
the
cluster,
using
our
integrated
runtimes.
So,
for
example,
I
have
one
instance
that
is
deployed
on
my
development
clusters,
so
I
have
imported
the
same
kind
of
like
data
categories
here.
A
So
let's
say,
for
example,
if
I
go
to
decoder
language
classified
constriction
models
in
my
own
cluster,
enable
like
the
execution
run,
transfer
so
I
could
actually
able
to
launch
these
models
right
on
on
my
cupidity
and
try
out
so
let's
say:
I
just
want
to
launch
this
model
as
a
plane.
You
know
deployments
container
and
you
test
it
out.
I
will
just
simply
do
like
launch
and
behind
the
scenes
right,
depending
how
you
actually
train
the
model
or
certain
models.
A
So
once
the
pipeline
is
finished,
it
will
just
show
us
like
whether
the
department
is
available
and
for
this
pipeline
I
actually
configured
to
deploy
on
no
ports
around
kubernetes.
So
with
like
100
IP
and
an
open,
we're
able
to
like
just
try
it
out
so
with
the
note
put,
we
have
is
3
11
74.,
so
I
just
do
here
so
with
this
way
behind
us
in
this
pipeline,
just
kind
of
expose
these
models
as
a
rest,
API
server.
A
So
you
can
actually
leverage
currently
this
Swagger
uis
and
this
model
is
extremely
simple,
so
you
could
see
like
this.
Apis
only
have
like
two
method
get
and
post
and
you
could
just
try
out
with
this
model.
Just
by
sending
you
know
some
files,
for
example,
let's
say
we
submit
like
python
files
here
and
we
could
try
out
these
examples
able
to
produce
us
the
top
three.
A
B
A
B
A
B
A
A
Yes,
so
I
would
say
like
some
of
the
kind
of
like
public
announcements
we
have
I.
Think
one
of
the
dealers
advancement
we
have
is
the
project
wisdom.
So
we
actually
work
with,
like
red
hat
team,
to
attribute
new
tools
to
do
like
code
generation
for
other
Automation
and
infrastructure
as
code,
so
we
hopefully
to
help
like
help
developers
to
able
to
do
automation
by
just
you
know
putting
like
a
plain
English
text
right,
so
this
does
describe
your
workloads
and
build
a
answerable
pipeline
for
you.
A
This
is
kind
of
like
one
of
the
use
case
and
other
customer
use
case.
We
have
done
it's
actually
like
help
us
modernize.
You
know
old
Java
code
into
new
Java
codes,
so
those
are
like
kind
of
the
use
case
we
currently
have,
but
we're
also
like
Envision
like
more
use
cases
in
our
open
source
and
hopefully
like
people,
will
get
feedbacks
and
we're
able
to
use
those
feedback
to
enhance
and
improve
this
data
set
over
time.
C
A
Yeah,
let
me
repeat
the
question
so
I
think
the
question
is
around
like
how
we
got
the
approach.
You
know
different
institutions
like
Bank,
you
know
to
migrate.
You
know
old
Legacy
code
like
Kobo
to
like
the
market,
Technologies,
so
I
think
at
this
moment
for
the
research
and
open
source
team.
We
actually
just
want
to
leverage
with
a
kind
of
different
use
case
and.
B
A
Yeah
I
think
the
question
is
like
how
we
envision
you
know
this
data
set
to
plug
into
attribute
like
proposition
and
values
to
real
customers
right
so
I
think
it.
We
are
still
kind
of
in
an
initial
stage,
I
think,
as
we
kind
of
see
it
in
like
different
kind
of
AI
road
maps.
When
you
know
undue
data
set
is
being
created,
new
method
has
been
created.
It
usually
takes
several
years
to
actually
able
to
achieve
like
like
more
real
world
or
like
more
sustainable
use
case.
A
We
actually
want
the
community
to
show
us
right
how
the
different
research
paper
and
different
like
open
source
use
case
that
have
you
know,
kind
of,
like
just
posted
on,
like
developer
blocks
and
use
those
feedback
to
help
us.
You
know
improve
and
build
a
new
kind
of
models
that
could
leverage
data
sets.
C
C
A
Right
I,
I
think
like
right
now
we
do
have
like
55
different
language
right
for
our
initial
kind
of
data
sets.
So
as
you
can
see
that
even
for
the
cobles
right,
like
we
kind
of
describe
over
here
and
with
the
code,
was
like,
we
at
least
have
like
100
solutions
for
them
for
80
percent
of
the
problem.
D
A
I
was
saying
more
or
less
similar
yeah
like
I,
think
I
would
open
AI
for
it,
as
you
mentioned
like
how
openly
have
I
kind
of
Leverage,
similar
Concepts,
because
openly
I
kind
of
like
proprietary,
we
don't
know
what
kind
of
core
models
they
use
behind
the
scenes.
So
we
cannot
really
say,
but
is
the
concept
similar?
A
Just
like
imagine
that
you
could
create
like
a
benchmark
on
other,
let's
say
models
you
build
and
just
to
make
sure
like
when
you
create
a
new
use
case
like
you
could
use
this
data
as
a
benchmark
to
test
whether
or
not
your
generated,
let's
say
code
is
actually
matching
the
one
we
have
in
our
data
set
and
use
that
to
improve
your
model.
Accuracies.