►
Description
Jun Duan (1:30) gives a demo on ArgoCD and the bottlenecks that occur as users approach 10,000 managed applications. We relate this information back to how we might overcome these challenges using KCP-Edge in the future working together with benchmarking efforts in the ArgoCD community. Kevin Roche (19:40) gives a demo on using one of OpenAI's GPT models to adjust deployment parameters inside a Kubernetes deployment manifest. This work demonstrates why it is important to implement some level of automation into the placement and workload definition process when trying to configure large numbers of heterogenous edge locations. The KCP-Edge community is actively exploring what it will take to incorporate any cloud-native project/product to automate and influence the placement and workload definitions using consistent interfaces and a modular integration strategy.
A
Hi
everybody
and
welcome
this
is
The
kcp
Edge
demo
day
today
is
March,
2nd
2023.
We
have
two
items
on
the
agenda
today
to
talk
about
June
dwan
who's,
an
IBM
researcher
we'll
be
showing
a
presentation
with
regard
to
scalability
challenges
for
Argo
CD
and
secondly,
we
will
have
Kevin
Roche
who's,
also
an
IBM
researcher
who
will
be
showing
us
an
Exemplar
workload
with
using
the
opening
openai
GPT
to
fuel
the
changes
in
declarations
for
workload,
placement
across
a
kubernetes
network,
so
without
further
Ado
we'll
get
started.
A
June
is
going
to
go
first,
so
this
this
demo
day
is
representative
of
works
that
are
being
done
by
The
kcp
Edge
Community
kcp,
Edge
Community.
You
can
find
at
kcp-edge.io
if
you
want
to
join
us
in
our
quest
to
create
a
state-based
management,
offering
platform
that
allows
you
for
workload,
placement
using
filtering
and
allows
you
for
rollout
control
and
do
this
all
consistency
consistently
across
heterogeneous
Edge
locations
in
a
scalable
manner,
so
without
further
Ado
June.
Please.
B
We
recently
have
some
quantitative
austerity
on
University
scalability,
so
we
are
happy
today
to
share
what
we
had
with
the
audience
just
some
short
introductions.
My
name
is
Jun,
and
this
work
is
done
together
in
team
with
follow
and
Andy,
and
we
are
from
IBM
research
at
Yorktown
lab,
which
is
shown
as
my
background.
B
First
I
want
to
talk
a
little
bit
about
what
motivates
our
study.
We
are
working
in
the
context
of
ads
Computing,
where
we
often
have
to
manage
a
lot
of
edge
locations
in
a
large
number
of
edge
locations,
because
the
number
is
large.
We
have
to
consider
the
scalability
issues
and
also
the
ad
location
is
also
applications,
often,
if
not
always
run
customized
workloads
and
this
customization
imposes
further
computational
requirements
on
the
automations.
B
Another
city
is
one
of
the
admissions
must
take
scalability
into
the
consideration,
and
that's
why
we
are
doing
these
experiments
here
in
this
abstract
page
shows
the
summary
of
our
work
in
this
work.
We
study
how
to
handle
as
many
as
I
could
see
the
applications
with
reasonable
performance
in
terms
of
performance.
We
care
about
one
single
one,
single
parameter,
which
is
the
thinking
Delay
from
the
changes
at
the
Upstream
git
repository
to
the
downstream
artillery
resources,
which
is
deployed
biography
at
the
downstream
kubernetes
clusters.
B
We
have
several
findings.
First,
we
have
seen
that
with
thousands
of
applications,
the
CPU
of
the
of
the
machine
which
hosts
argue
CD
easily
becomes
a
performance
bottleneck
and
we
tried
several
ways
to
to
to
handle
this
issue.
We
try
to
reduce
our
computational
requirements.
We
try
to
provision
more
computing
power
and
we
also
change
our
expectation
on
how
long
we
want
the
delay
to
be
in
our
experiments.
We
finally
managed
to
sync
10
000
applications
in
less
than
40
minutes.
B
This
picture
shows
the
just
the
block
diagram
of
our
experiment.
We
host
our
kick
repositories
on
GitHub
and
we
run
our
good
city
as
well
as
Downstream
kubernetes
clusters
on
AWS.
We
use
cluster
loader,
which
is
the
community
tool
to
load
our
large
number
of
applications
of
pursuit
applications
into
your
acrosity
server.
We
use
Prometheus
and
profana
to
monitor
The
Matrix.
B
B
C
B
Verify
that
by
take
a
close
look
at
the
CPU
usage
of
each
replica
of
the
reference
server
here,
we
can
see
that
each
replica
of
the
wrapper
server
roughly
uses
about
400
mini
calls,
but
in
the
first
scenario
the
one
single
replica
use
more
than
800
mini
costs,
because
in
the
second
scenario
the
replicas
couldn't
use
as
much
of
as
many
of
mini
course
as
the
in
the
first
scenario,
because
this
small
virtual
machine
only
have
two
mini
cups.
I
only
have
two
vcpus,
that's
why
we
haven't.
B
We
couldn't
reduce
time
to
one
third,
comparing
to
the
first
scenario.
We
check
the
dashboard
and
find
that
CPU
is
almost
saturated,
so
this
is
the
first
bottleneck
we
hit
on
in
our
experiment
in
order
to
handle
that
we
do
two
things.
First,
we
just
switch
to
a
more
powerful
motion
which
has
more
CPUs,
and
then
we
identified
that
are
the
customization
inside
our
arcosv
plugin
consumes
a
lot
of
CPUs.
We
have
the
switch.
We
have
a
binary
switch
in
that
plugin
to
disable
or
enable
our
customization.
B
We
try
to
disable
that
and
then
we
observed
a
considerable
decrease
in
CPU
usage.
So
we
also
did
that
in
order
to
in
order
to
handle
this
bottleneck
issue,
that's
the
end
of
the
first
story,
and
then
let's
talk
about
the
Second
Story.
The
second
story
is
about
the
default
resync
period
of
argosity.
We
know
that
argosity
has
a
default.
Three
minutes
we
think
period
for
every
three
minutes
ago.
City
tries
to
make
sure
that
the
git
repository
is
in
upstream,
and
the
resources
in
the
downstream
are
consistent.
B
B
In
this
scenario,
in
this,
in
our
story
too,
we're
owning
want
to
check
the
racing
period
to
make
it
simple.
We
actually
didn't
introduce
any
change
to
the
Upstream
git
repositories.
That
means
in
this
story
The
scenarios
in
this
Story
2.
We
have
only
one
work
queue,
which
is
application,
reconciliation
queue.
That
is
about
the
consistency
between
the
Upstream
kit
and
the
local
Repository
local
local
cash.
We
will
talk
about
the
second
queue
in
in
story.
B
3.,
so
we
also
have
several
scenarios
in
this
story,
and
if
we
compare
the
first
two
story,
we
can
see.
The
only
difference
between
the
scenario
is
the
number
of
applications.
In
the
first
scenario,
we
have
two
thousand
applications
and
in
the
second
scenario,
we
have
6K
applications
and
if
we
look
at
the
right
hand,
side
look
at
the
graphs,
we
can
observed
two
things.
First,
we
can
see
that
the
height
of
the
peak
equals
to
a
number
of
applications,
which
is
pretty
straightforward.
B
Actually,
the
two
things
between
these
two
scenarios.
The
first
thing,
the
first
difference
is
that
we
increase
the
number
of
applications
from
8K
to
10K.
The
second
thing
is
we
increase.
We
change
this
the
workload
of
the
applications.
In
this
scenario,
we
have
the
workload
of
the
applications
roughly
two
kilobytes,
but
in
the
next
scenario
we
have
some
of
the
applications
have
a
heavier
workload
it.
By
making
these
two
differences,
we
can
observe
that
the
controllers
were
not
able
to
even
empty
the
work
cues,
so
that
means
we
have
hit.
B
The
default
recent
period
hit
the
bottleneck
of
the
default
releasing
period,
so
the
solution
of
that
is,
we
just
increase
the
recent
period
from
3
minutes
to
6
minutes
right.
That's
the
only
thing
we
change
between
these
two
scenarios
and
after
that
change,
we
observed
that
the
controllers
were
able
to
periodically
empty
the
work
Cube
again
so
bringing
the
work
you
get
Kingdom.
D
B
B
D
B
B
D
B
B
So
I
want
to
share
this
story
with,
with
the
audience
so
recall
that
in
the
Story
2
we
only
have
one
word
queue,
which
is
the
work
queue,
which
is
the
work
queue
for
checking
the
inconsistent
inconsistency
between
the
option,
git
and
the
local
cache,
and
now
in
scenario.
Three.
We
want
to
be
more
realistic
by
simulating.
B
B
B
We
can
see
that
the
time
taken
being
increased
to
roughly
30
minutes
and
also
we
observe
that
the
work
here
is
flatly
emptied,
because
we
only
use
three
minutes
now
boosting
period
and
then
the
last
scenario
we
use
six
minutes.
As
the
recent
period
we
use
10K
applications.
We
use
the
mixed
workload,
we
introduce
changes
to
Upstream
simultaneously
and
finally,
we
got
all
the
applications
synced
in
less
than
40
minutes.
B
B
A
Andy
your
presentation
appreciate
it
yeah.
So
if
you
scan
the
QR
code
there
in
the
upper
right
hand,
side
you'll
get
to
our
repository,
where
our
community
is
interacting
and
working
on
issues
and
pull
requests
related
to
things
like
placement,
scheduling,
customization
of
workloads
and
status
and
Status
summary
to
work
at
scale
across
a
distributed
State
management
system.
A
A
So
we
have
Kevin
roach,
who
is
also
from
IBM
research
in
California
almondine's
office,
the
Almaden
office
and
he's
joined
the
community
and
has
has
donated
a
sample
workload
that
he'll
describe
and
shows
how
it
can
be
customized
to
be
deployed,
different
regions,
different
locations
based
on
parameters
that
can
be
configured
using
of
all
things,
something
that
we've
seen
in
the
headlines
recently
chat,
GPT,
the
underlying
technology,
the
open
AI
GPT
model,
and
so
one
of
the
models
that
they
have,
and
so
with
that
I'll.
Let
Kevin
take
it
from
here.
C
C
All
right,
so
thank
you
for
the
introduction
Andy.
As
he
mentioned,
I'm
a
researcher
out
at
our
Almaden
lab
in
San
Jose
I
am
most
the
time
on
Material
Science,
doing
laboratory
automation,
which
means
I
have
far
too
much
experience.
Making
disparate
interfaces
talk
to
each
other
and
along
the
way,
while
I
was
learning
about
Cloud
programming
as
part
of
a
coursework.
Last
year,
I
wrote
a
toy
application
which
we
thought
would
be
a
great
demo
workload
to
use
for
these
large-scale
deployments.
C
C
What
made
this
different
was
that
the
whole
thing
was
then
packaged
to
run
inside
a
microshift
cluster
running
in
Centos
on
a
Raspberry
Pi,
and
that
was
my
big
demo
app
for
the
end
of
my
coursework,
and
it
occurred
to
us
that
this
is
exactly
the
sort
of
thing
that's
a
real
app.
That
would
be
interesting
to
deploy
across
a
large
number
of
at
locate
Edge
locations.
C
If
you
don't
have
the
sensors
or
the
camera,
it
can't
run
the
Pod
that
talks
to
the
camera
will
crash
over
and
over
and
over
we'll
run
on
a
an
Intel
processor,
and
in
this
case
it
can
cope
with
not
having
sensors
there's
actually
code
that
will
emulate
the
sensorboard,
but
without
a
camera
again
the
camera
pod
will
keep
crashing,
and
in
this
case
the
problem
is
that
the
time
zones
don't
match.
So
there
are
ways
to
fix
this
and
yeah.
C
First
of
all,
I
can
make
the
the
code
actually
multi
art
multi-architecture
compatible.
So
I
was
able
to
do
that
that
solves
the
problem
of
not
running
on
a
Raspberry
Pi,
but
it
doesn't
solve
this
camera
problem
and
at
the
time
we
read
some
interesting
discussions
about
what
some
of
the
large
language
modules
models
at
open,
AI
could
do
and
I
was
still
learning
how
to
use
how
to
write
manifests.
So
the
biggest
problem
was
I.
C
Couldn't
put
any
logic
in
my
manifest
to
decide
which
parts
it
might
system
to
turn
on
and
off.
Customize
can
do
it,
but
you
still
have
to
learn
how
to
write
the
templates
for
customized
I
managed
to
do
all
that,
but
I
wondered
if
there's
a
way
to
use
a
gpt3
model
and
foundational
AI
models
to
help
do
that
adaptation.
C
So
I'm
talking
about
what
I'm
calling
a
placement
configuration,
it's
deliberately
a
little
bit
fuzzy,
but
that
means
it
includes
the
architecture
what
peripherals
are
installed
and
the
environment,
so
time
zone
or
anything
else
or
a
placement
Target
system.
So
my
idea,
my
goal
was
to
write
a
simple
text:
description
of
a
whole
bunch
of
systems
of
placement
targets
and
then
run
it
through
open
ai's
codex
model
and
generate
custom
manifest.
C
So
I
didn't
have
to
worry
about
it
anymore,
and
you
know
it's
a
little
bit
of
a
simple
experiment,
but
it
was
really
interesting
to
do
and
in
fact
it
worked
really
well.
So
this
video
is
slightly
time
compressed
because
the
Codex
large
language
model,
open
area
was
in
beta
M
had
rate
restrictions,
so
I
could
only
send
a
request
about
every
10
seconds,
as
it
reads
in
my
text.
Description
turns
that
into
a
table
and
then
starts
working
through
the
different
placement
descriptions.
The
first
one.
C
The
hardware
is
great,
but
it's
in
the
wrong
time
zone,
the
second
one.
It's
just
needs
to
confirm
the
time
zone.
Third,
one
doesn't
have
any
hardware,
so
it's
gonna
have
to
take
out
the
camera
pod.
It's
going
to
have
to
activate
the
replacement
for
the
camera
and
it's
going
to
have
to
update
the
time
zone
and
it
just
keeps
chugging
through
it's.
You
know
not
a
complicated
program.
It's
only
190
lines
of
python
and
it
just
did
the
work
now.
C
I
had
to
do
some
advanced
work
in
the
sandbox
for
the
model,
so
I
could
learn
how
to
make
the
queries
that
would
do
these
changes
when
it
was
all
done.
I
could
take
the
Manifest
and
actually
test
run
them
on
model
on
systems
that
matched
the
placement
configuration
I
described
and
was
very
pleased
to
discover
that
in
fact,
they
all
ran
the
way
they
were
supposed
to
and
I'm
not
going
to
make
it
go
all
the
way
to
the
end,
because
it's
another
two
minutes
of
Our
Lives
I'd
rather
give
back
to
you.
A
D
Yeah
Kevin
one
question:
first,
it's
a
very,
very,
very
impressive
demonstration
and
the
second
question
is:
do
you
envision
all
the
purpose
was
to
continue
using.
You
know,
AI
for
any
deployment,
or
it's
just
some
initial
configuration
once
you
generated
with
the
AI
configuration
for
a
specific
kind
of
conditions.
You
store
it
somewhere
and
you
will
use
that
one
for
if
that
request
will
come.
What
is
the
vision
here.
C
However,
what
I
would
picture
is
as
we're
building
kcp
Edge
we
talk
about
locations,
and
so
a
location
would
include
some
of
this
kind
of
information
so
as
we're
using
our
whatever
form
our
control
plane
looks
like
it
would
take
advantage
of
that
information
to
make
these
mods
before
it
starts,
actually
placing
the
workload
on
those
locations
and,
as
I
said,
some
things
you
can
solve
by
building
multi-architecture
packages.
In
other
cases
you
do
have
to
worry
about
what
is
the
physical
Hardware.
C
So
if
your
location
record
can
include
that
information,
so
the
control
plane
can
make
these
customizations
to
turn
things
on
and
off.
Sometimes
you
have
to
take
things
out
of
the
Manifest.
In
other
cases,
you
can
solve
it
by
setting
an
environment
variable
properly,
but
you
need
to
know
to
do
that
before
any
manifest
is
actually
used
to
deploy
the
actual
workloads
to
the
Target.
D
So
so
one
interesting
question
not
related
at
all
to
cloud
or
anything,
but
actually
it's
for
personal
from
personal
kind
of
curiosity
about
you,
know,
jgpt
and
open
AI.
Did
you
encounter
with
errors?
Because
we
heard
a
lot
about
you
know,
mistakes
and
even
quote
unquote
lies
technology
is
kind
of
I.
I
actually
saw
a
question
where
GPT
invent.
You
know
a
kind
of
invented,
the
a
character
that
wasn't
in
the
book
at
all.
So
did
you
encounter
issues
like
that?
I.
C
Did
not-
and
there
are
some
really
important
reasons
I
did
that
that's
a
great
question.
Actually,
the
the
first
is
I.
This
was
not
chat
gbt.
This
was
using
the
da
Vinci
text
and
DaVinci
codex
foundational
models,
rather
than
a
chat
model.
I
started
with
a
master
copy
of
a
working
manifest
that
worked
on
a
fully
equipped
Raspberry
Pi
in
my
in
my
local
time
zone
and
then
used
the
Codex
model
to
modify
that.
C
Usually
that
meant
it
might
mean
removing
the
deployment
and
service
for
the
camera
module
the
camera
pod,
because
if
you
don't
and
it
can't
run,
it
will
crash
and
restart
repeatedly
and
eventually
it'll
take
down
your
your
cluster,
because
the
the
resource
load
of
restarting
it
over
and
over
and
over
again
eventually
will
crash
the
cluster
on
a
VM.
That's
annoying
on
a
Raspberry
Pi.
C
That's
a
horror
show
because
it
takes
so
long
for
a
pie
to
reboot
the.
So
it
was
starting
with
known
good
code
and
I
was
really
really
thrilled
at
how
well
the
Codex
model,
understood,
yaml
and
manifest
structure,
so
yeah
I
would
say,
remove
the
the
camera
deployment
section
and
boom,
it
was
gone,
say,
remove
the
camera
service.
It
was
gone
and.
E
Just
to
be
clear,
right
and
I
think
maybe
to
improve
your
presentation,
maybe
kind
of
a
a
block
diagram
of
you
know,
training
and
inferencing,
and
what
information
comes
from.
Where
would
help
make
this
clear
but
I
think
what
you're
saying
Kevin
if
I
understood
correctly,
you
wrote
some
python
that
tells
the
model
or
it
says
what
changes
to
make
to
the
to
the
yaml
right
in
in
kind
of
language
form.
Yes,.
E
So
my
point
is
the
knowledge
that
relates
the
characterization
characteristics
of
the
location,
to
what
should
be
done
to
the
yaml.
That's
in
Python
code
that
you
wrote
you're
using
the
the
AI
technology
to
implement
the
plain
English
instructions
for
how
to
edit
the
animal
and
and
actually
do
the
editing
of
the
ammo.
C
C
Yeah
they're
they're
the
way
that
this
model
works.
You
have
to
provide
a
basic
instruction
to
start
with
and
then
a
modifier.
So
in
my
case
it
was
oh
okay,
edit
this
manifest,
which
has
these
separators
in
it.
I
was
using
the
standard
row
of
hyphens
as
separators
and
sections
for
a
for
a
kubernetes
deployment,
so
that
wasn't
that
totally
what
it
was
looking
at
and
then
I
could
say,
remove
this
remove
this
deployment.
C
Remove
this
service
update
all
of
the
time
zone,
environment,
variables
and
and
those
then
it
knew
it
was
a
kubernetes
style
manifest,
and
it
knew
what
I
wanted
to
do
and
he
would
go
through
and
find
it,
and
so
it
was.
You
know
this
was
a
a
limited
experiment,
but
I
was
surprised
really
at
how
well
it
worked
foreign
there
is.
There
are
more
details:
I
wrote
a
medium
blog
post
about
this
yeah.
C
C
A
All
right,
thank
you,
Kevin
very
much.
So
what
you
can
see
by
what
Kevin
just
showed
you
right
is
this.
This
ability
to
use
consistent
declaration
have
it
modified
somewhat
automatically
and
to
cover
a
heterogeneous
set
of
deployments
or
heterogeneous
set
of
Edge
Edge
locations
right
with
different
capacities
capabilities
time
zones.
Maybe
even
sovereignty
comes
into
play
and
do
this
at
scale,
and
one
of
the
reasons
why
this
is
so
important
is
that
once
we
get
to
a
certain
level
of
scale,
people
don't
want
to
manage
these.