►
From YouTube: 2. CNCLinz - Denis Jannot & János Pásztor
Description
https://www.meetup.com/Cloud-Native-Computing-Linz/events/273726815/
--------- The Speakers ---------
- Denis Jannot is the Director of Field Engineering at Solo.io, a company building application networking solutions for the edge and service mesh. Denis is a passionate engineer who has spent his career in technical roles working directly with customers and users in architecting and adopting technologies like Object Storage, Big Data, Containerization, Service Mesh into their infrastructure. He enjoys sharing what he learns with the community and can be found creating demos, writing blogs, and speaking at events.
- János Pásztor is a software engineer who loves writing. János have been in the industry for over 10 years, and likes to share what he have learned. He also lectures at FH Campus Wien, being one of the responsible of FH Cloud Computing Fall 2020 (https://fh-cloud-computing.github.io/). If you want to know more about him visit https://pasztor.at
--------- The Talks ---------
Denis Jannot - The challenges of exposing and connecting microservices
In this presentation, Denis will cover the common challenges people are facing when they expose and connect micro services.
He will review the different options to expose services running on Kubernetes (Ingress vs API gateway) and to manage service to service communications (API gateway vs Service Mesh).
He will also discuss the additional complexity introduced when services are spread across multiple environments (multiple Kubernetes clusters, Service Meshes, ...).
Finally, Denis will introduce several Open Source projects (Gloo, Service Mesh Hub, Web Assembly, ...) which they are working on at Solo.io to tackle those challenges.
---------
János Pásztor - Custom Autoscaling with Prometheus, Grafana, and a little bit of code
Some cloud providers give you the ability to automatically scale up or down based on simple metrics like CPU or memory usage. What happens if you need a complex logic? What if you need custom factors like combining CPU, RAM, and disk usage? Or application metrics?
Prometheus gives us a wonderful way to gather metrics from a varying number of machines using service discovery, and Grafana an easy to use way to create graphs and alerts. Let's combine these to build a custom, self contained and testable autoscaler for our cloud platform.
---------
Agenda:
5pm - Welcome
5.10pm - Denis Jannot
6.pm - János Pásztor
6.50pm - Closing remarks
7pm - End
Meetup will be streamed on Youtube and Q&A will be done via discord.
Youtube channel: https://www.youtube.com/channel/UCmSELZztLD8TeEEKSKWFp-A
Discord: https://discord.gg/MgM5BGV
https://www.meetup.com/Cloud-Native-Computing-Linz/events/273726815/
A
A
A
C
A
A
That's
the
interesting
part
with
all
these
virtual
meetups
that
people
are
joining
from
all
over
I
in
the
afternoon
I
did
a
meet
up
for
cloud
native
group
in
jakarta
in
indonesia,
and
they
were
actually
also
they
had
people
from
all
the
different
regions
joining
so
dennis.
You
heard
my
talk
last
week.
D
A
Using
stream
yards
so
very,
very
similar
setup
and
they
had
quite
a
couple
of
questions
directly
from
youtube
coming
in.
So
if
there
are
any
questions,
I
will
also
watch
the
youtube
comments
when
they're
coming,
I
will
ask
them,
I
will
ask
them
to
you
later
or
I
will
just
with
streamyard
it's
it's
pretty
easy
to
directly
show
them
to
the
speaker.
So
yeah.
B
Yeah,
so
it's
actually
it's
it's
funny,
because
lots
of
remote
conferences
happening
and
make
every
made
everything
a
little
easier,
because
now
you
don't
have
to
travel
to
places
and
it's
it's
become
acceptable
to
to
speak
at
a
conference
which
is
halfway
around
the
world
and
also
as
a
as
a
visitor.
So
when
you're,
you
know
like
traveling
to
conferences
and
buying
conference
tickets
was
quite
expensive.
B
Yeah
ticket
is
like
600
and
then
the
flight
in
the
hotel
and
everything.
So
that's
like
it's
it's
you
could
say
it's
the
new
normal
and
and
it's
really
hard
for
for
some
people,
but
I
think,
like
the
the
inner
nerd
who
just
likes
to
sit
at
home
in
a
hoodie
is,
is
very
thankful
to
what
happened.
Even
though
it's
not
I
mean
it's
not
without
controversy,
to
say
that.
D
B
C
So
if
you
are
on
twitter,
please
follow
us
cloudnativelinks
and
if
you
are
watching
just
if
you
would
like
to
comment
and
everything
just
tag
us
and
help
us
to
promote
the
group-
and
you
can
also
join
the
community.cncf.io-
we
are
part
of
the
cncf
community
group
and
yeah.
That's
it
so
before
starting
here's
the
agenda.
So
I'm
starting
with
the
welcome
message.
Then
we
will
start
with
dennis
jannat
from
soloyo
and
then
at
six
around
six
we
passed
by
to
janus
bastard.
C
Well,
we
just
found
out
that
he's
joining
red
hat
soon.
So
that's
really
nice
and
then
we
finish
with
closing
remarks
and
at
seven
we
are
coming
to
an
end.
Of
course
that's
not
strict
and
we
can
go
a
little
bit
over
and
or
shorter
in.
Sometimes
then
we
follow
the
berlin
code
of
conduct.
If
you
would
like
to
know
more
about
it,
you
can
take
a
look
into
berlincodeofconduct.org,
but
mainly
we
welcome
everyone
and
it
doesn't
matter
the
gender
or
the
skills
level
or
whatever.
C
C
The
the
event
is
organized
by
myself
and
jurgen.
I'm
a
soft
engineer
at
dynatrace.
Here's
my
twitter
handle.
If
you
would
like
to
to
get
in
touch
with
us
in
the
page
of
the
cncf
meetup,
you
can
also
register
for
a
talk
and
to
sponsor
our
event
I'll
tell
about
that
in
a
second.
But
if
you
want
to
get
in
touch
directly
with
us
here,
are
our
twitter
handles
you're
gonna?
Do
you
wanna
say
something
just.
A
C
So
I
would
like
to
thank
dynatrace
for
sponsoring
this
streaming,
so
we
are
using
a
paid
streaming
platform
and
if
you
would
like
to
stream
one,
we
would
like
to
sponsor
one
of
our
next
streamings.
Please
get
in
touch
with
us
and
we
can
mention
your
company,
your
brand,
and
you
will
help
the
community
and
us
to
have
a
better
streaming
experience.
C
That's
a
new
part
of
our
community
group
where
we
share
some
community
news,
and
here
we
have
some
workshops
on
the
cloud
native
cloud
native
technologies,
part
by
microsoft,
austria
and
if
you
would
like,
I
think
jurgen
will
paste
this
link
in
the
in
the
youtube
channel
afterwards.
So
you
can
take
a
look
into
the
to
those
workshops
and
if
you
have
something
to
share
and
you're
part
of
our
community,
we'll
be
happy
to
share
with
everyone
as
well
so
get
in
touch
with
us.
C
If
you
want
to
continue
the
discussion
and
raise
questions
during
the
presentation
and
everything,
of
course
we
are
watching,
we
are
following
the
youtube
channel
as
well,
so
you
can
post
your
questions
over
there,
but
if
you
would
like
to
continue
the
discussion
afterwards
and
keep
engaged
with
the
community
join
us
on
our
cnc
liens
discord
channel
and
that's
something
that
we
are
trying
to
have
to
replace
the
the
the
talks
that
we
have
after
the
meetups.
So
we
were
mentioning
before
that.
C
C
A
Screen
perfect-
and
I
just
want
to
add:
if
there
are
any
questions,
please
ask
them
early,
ask
them
on
discord
or
on
youtube,
because
dennis
has
to
leave
quite
on
time
for
another
presentation.
He
has
to
do
so
thanks
for
being
with
us
and
we
hope
we
can
still
have
a
great
presentation
and
hopefully
a
little
bit
of
q
a
afterwards.
So
I
will
give
you
the
screen
and
thanks
so
much.
D
A
D
Good
good
thanks.
So
thanks
for
introduction
so
yeah
I
am
like
you
said,
working
at
solo.I,
o
and
being
a
director
of
sales
engineering
covering
like
emea,
and
you
have
like
some
of
my
details
here.
If
you
you
want
to
contact
me,
but
the
idea
of
this
talk
is
really
to
go
through
the
common
challenges
that
people
face
when
they
try
to
expose
services
to
the
outside
world
or
when
they
want
to
improve
the
way
they
and
their
service
to
service
communications.
D
D
So
you
you
can
then
define
these
ingress
objects
to
to
expose
your
apps
to
the
outside
world.
And
you
can
you
have
a
few
options
there,
like
you
can
define
which
certificate
you
want
to
use.
If
you
want
to
you
to
expose
them
with
https,
you
can
have
some
basic
routing
capability,
but
it's
very
quickly
limited
and
when
you
start
to
when
you
start
to
need
more
more
capabilities,
and
especially
because
a
lot
of
people
when
they
write
new
application,
they
don't
want
to
like
handle
like
the
authentication
on
each
application
separately.
D
D
You
can
expose
your
grpc
service
as
an
http
application,
and
it
also
allows
you
to
expose
applications
that
are
not
running
on
kubernetes
so
that
you
can
expose
some
application
running,
for
example,
on
on
vmware
some
other
application
that
are
just
like
lambda
functions
on
aws
and
so
on.
So
that's
really
the
benefits
you
get
when
I
use
like
a
true
api
gateway
versus
like
an
ingress
controller,
and
then,
when
you
have
secured
the
edge,
then
the
next
step
is.
How
do
I
get
smarter
ways
of,
except
for
service
to
service
communication?
D
How
I
can
secure
this
communication,
how
I
can
get
more
insight
about?
What's
going
on
and
basically
the
requirements,
there
are
many
requirements,
the,
but
the
most
important
ones
are
the
one
that
I've
listed
here.
So
you
need
to
manage
identity
so
that
when
a
request
comes
from
one
service
to
another,
then
the
targeted
service
knows.
Who
is
the
the
service
that
requests
this
information
you
want
to
enter
encryption
between
these
services?
You
want
to
manage
certificates,
especially
the
rotating
certificate,
is
not
something
easy
to.
A
D
D
If
you
just
use
the
traditional
kubernetes
approach,
where
you
expose
the
the
targeted
microservice
with
a
kubernetes
service,
and
then
you
don't
get
much,
I
mean
you
can
obviously
use
labels
so
that
you
can
do
a
little
bit
of
traffic
management.
That
way,
you
have
a
little
bit
of
else
check,
capabilities
there
as
well,
but
for
example,
if
you
want
to
encrypt
the
traffic,
then
it
has
to
be
done
at
the
application
level.
D
So
the
application
has
to
agree
that
the
source
micro
service
has
to
agree
with
the
target
microservice
to
on
the
way
they
will
encrypt
the
communication
together,
and
you
need
a
lot
of
third-party
software
on
top
of
your
kubernetes
cluster,
like
some
for
access
control,
some
for
like
identity
management,
some
for
managing
certificates
and
so
on.
So
you
don't
get
much
by
just
using
the
traditional
approach
on
kubernetes.
D
Another
way
is
to
use
like
an
api
gateway
and
that's
what
has
been
done
for
some
time
on
I
mean
it
has
been
quite
common
to
have
communication
between
micro
services
going
through
an
api
gateway,
and
it
solves
a
lot
of
the
problems
that
I
described
before
a
lot
of
the
challenges
you
can
manage
access
control.
That
way
you
can,
you
know,
transform
your
request.
D
You
can
get
insight
about
what's
going
on,
but
you
still
need
to
manage
your
encryption
at
the
application
level
and
you
you
don't
have
everything
that
I
listed
before,
but
the
the
biggest
issue
with
this
approach
and
oh,
okay,
that's
a
first
time.
I
get
that
on
google
slides,
that's
okay,
it
should
be
okay.
If
I
present
again,
oh
okay,
it
doesn't
like
this
slide
anyway.
I'll
just
use
this
presenter
mod
here.
D
It's
really
the
proxies
that
are
doing
this
encryption,
but
also
because
all
the
communication
goes
through
the
proxy.
Then
this
proxy
collects
some
telemetry
data
that
you
can
use
to
know.
What's
going
on
in
your
mesh
and
as
you
can
see
here
this,
you
probably
recognize
this
logo,
which
is
envoy,
and
it's
not
a
hazard.
Why
I
put
this
logo
is
because,
even
if
there
are
many
different
options
in
terms
of
service
mesh,
most
of
them
now
are
like
leveraging
a
sidecar
approach,
and
most
of
them
are
like
based
on
envoy.
D
So
you
have
different
solutions
like
if
you
look
at
console
connect,
for
example,
instead
of
having
a
single
control
plane,
you
would
have
like
a
control
plane
as
a
demand
set
on
each
node,
but
they
will
still
have
a
sidecar
approach
and
even
if
you
can
use
something
different
than
envoy
envoy
is
one
of
the
option
there.
If
you
look
at
like
linker
d,
if
you
look
at
other
like
istio,
obviously
like
they
they're
all
based
on
on
envoy.
A
A
D
Yeah,
so
they
are
all
based
on
on
android,
like
I
said,
and
it's
not
the
other,
I
mean
first
of
all,
it's
like
part
of
the
cncf
which
make
it
like
neutral
and
it's
it
really
provides
high
performance.
You
can
really
under
like
thousands
of
requests
per
second
with
just
a
single
instance,
and
it
has
really
been
developed
in
a
way
that
is
driven
by
api.
D
So
it's
not
like
a
lot
of
traditional
proxy
technologies
where
you
were
like
creating
a
configuration
file
that
was
loaded
by
the
proxy
when
it
was
starting
to
define
the
rules
here,
it's
really
designed
to
be
driven
by
an
api.
So
it's
really
designed
to
be
driven
by
a
control
plane
which
makes
it
really
suitable
for
service
mesh.
Obviously,
and
the
next
step
after
people
start
to
adopt
service
mesh
is
quickly
becoming
like.
D
How
do
I
allow
services
running
on
different
clusters
or
different
environments
to
communicate
together,
and
that's
one
of
the
nice
thing
you
can
do
with
a
service
mesh,
especially
with
a
istio,
but
it's
not
straightforward.
You
have
several
challenges
there.
I
won't
go
through
these
challenges
now
because
I'm
covering
that
later
in
the
presentation,
but
that's
definitely
becoming
really
popular-
to
to
use
service
mesh
to
allow
these
cross
cluster
communications.
D
D
D
But
right
now.
What
I
want
to
highlight
is
the
fact
that,
when
you
deploy
glue,
basically
glue
comes
with
a
set
of
filters
already
built
in,
and
some
of
them
communicate
with
other
components
running
in
in
glue
like
external
detection,
server
for
oidc
or
rate
limiting
server.
That
is
using
like
radius
as
a
back
end,
so
that
when
a
request
comes
in,
it
can
send
a
request
to
this
external
dot
exchange
server
to
decide.
D
D
Another
thing
we
we've
released
quite
recently
is
called
like
the
glue
federation,
which
is
a
way
to
manage
multiple
glue
instances
in
a
in
a
centralized
way.
So
you
deploy
blue
federation
somewhere.
It
can
be
in
a
cluster
where
you
have
blue
or
on
a
different
cluster,
and
you
can
then
define
all
your.
D
You
know
routing
policies
and
things
like
that
in
this
cluster
and
you
can
define
which
instance
it
should
target,
and
it
really
allows
you
in
this
case
to
have
like
a
central
repository
with
all
your
rules
and
it's
easier
to
then
maintain
them,
and
you
can
have
like
a
dashboard
that
shows
you
like
what
rules
you
have.
You
have
defined
on
the
different
places,
but
it's
also
provides
some
nice
functionality
like
this
one
that
I
will
demo
quickly,
which
is
like
a
failover
capability.
So
you
can.
D
You
can
have
like
requests
coming
for
this
application
running
here
and
going
through
this
glue
instance,
and
you
can
configure
glue
in
a
way
that
if
it
is
not
able
to
reach
the
service
locally,
it
can
reach
the
service
running
on
another
cluster,
going
through
another
glue
instance.
So,
like
this
one
running
here,
so
I'm
going
to
do
a
quick
demo
of
that
I'm
going
to
do
that
for
the
different
projects.
D
Obviously
I
could
spend
like
an
hour
on
each
topic,
but
I
just
want
to
give
you
like
a
quick
introduction
to
these
different
projects
and
obviously
slides
are
nice
nice
to
to
set
the
the
scene,
but
it's
always
easier,
I
think,
to
to
get
the
value
when,
when
you
use
your
demo,
so
I'm
trying
to
do
like
many
small
demos
so
that
you
get
a
good
idea
of
these
different
projects.
So,
let's
start
with
the
the
glue
federation,
like
I
said
I
unite.
D
D
This
one
is
a
service
where
I
have
a
failover
configured
and
basically
the
way
I've
deployed
it
is
that
it's
it's
the
same
service
that
runs
on
both
sides,
but
in
one
case
it
returns
blue
and
in
the
other
case
it
just
returns
green.
So
that's
the
only
difference,
and
I
can
show
you
that
here
so
you
see
it's
a
blue
green
deployment
and
the
text
is
blue
on
the
local
site
and
the
text
is
green
on
the
remote
side.
D
So
if
I
do
a
curl
now
you
can
see
I
get
blue
because
on
the
local
side,
I
I
have
like
this
photo
running
here
now.
If
I
scale
down
to
zero
replica,
what
will
happen
is
that
glue
is
configured
here
so
that
it
has
this
failover
set
so
that
it
knows
that
it
can
reach
this
other
service
on
cluster
2
when
it's
not
available
on
cluster
1.?
D
D
D
So
in
the
service
mesh.
Basically,
you
have
like
a
certificate
authority
running
and
you
you
get
a
root
ca
and
basically
each
pod
has
its
own
identity
and
this
pod.
When
it
communicates
with
this
pod,
they
can
do
like
a
mutual
tls
because
they
are
able
to
validate
the
identity
of
each
other
because
they
have
the
same
the
certificates
that
they
use
come
from
the
same
ca.
D
But
if
this
one
want
to
talk
to
this
service
here,
then
they
have
different
ca
so
that
they
cannot
really
validate
their
identity.
So
before
you
can
allow
cross-cluster
communication
in
a
secure
way,
you
need
to
federate
the
identity.
So
that
means
that
you
need
to
create
a
common
road
certificate
that
will
be
used
to
create,
like
a
interval
will
be
used
to
sign
the
intermediate
certificates
that
are
different
on
the
two
clusters.
D
So
you
need
to
have
a
way
to
make
this
cluster
aware
of
these
services
and
also
to
indicate
how
they
can
communicate
with
them
and
basically
it's
using
like
the
edge
gateway
and
with
these
two
it's
the
steel
ingress
gateway,
but
you
need
to
configure
those
things
to
make
that
happen,
and
you
need
to
also
then
have
a
way
to
manage
access
control
globally,
because
now
it's
not
about
just
like
one
mesh
isolated
from
another.
It's
it's
really
like
a
lot
of
different
meshes
like
that.
Can
talk
together.
D
So
you
need
to
have
this
manage
this
global
access
control
and
you
need
many
other
things.
But
what's
important
is
that
each
one
of
them
is
highly
complex
and
that's
really
what
service
mesh
does
it
simplifies
everything.
So,
let's
take
an
example
for
the
quick
demo,
so
we
have
this
for
people
who
are
familiar
with
istio.
We
have
this
book
info
application
that
is
used
for
demo
purpose
where
you
have
this
product
page,
which
is
a
micro
service.
D
That,
basically,
is
the
front
end,
and
this
front
end
talks
to
different
back-end
services
to
get
reviews
and
ratings
and
details
and
so
on.
And
here
what
we
are
going
to
do
is
we
are
going
to
distribute
the
request
coming
from
the
product
page
to
the
reviews
so
that
it
sends
some
of
the
requests
on
the
reviews
locally
and
some
of
the
requests
on
the
reviews.
D
Basically
what
I
say
that
when
I
send
a
request
to
the
review
service
on
cluster
one,
I
want
to
distribute
this
request
and
that
way,
so
I
want
to
do
like
75
percent
of
the
request
going
to
the
version
3
on
the
cluster
2
15
on
v1
on
cluster
1
and
ten
percent
on
v2
and
cluster
one.
So
it's
very
straightforward
to
describe
what
you
want
to
achieve
and
just
like.
Creating
this
object
basically
should
make
it
happen.
D
D
So
again,
just
like
a
very
quick
overview
of
of
what
you
can
do
with
with
service
mesh
herb.
You
can
do
much
more,
but
what
I
really
want
to
highlight
is
that
by
creating
this
traffic
policy
here,
basically
service
mesh
hub
has
translated
that
in
multiple
objects,
multiple
istio
objects.
So
if
I
would
do
that
manually,
I
would
have
to
create,
like
a
virtual
service,
a
istio
virtual
service
that
is
quite
similar
to
my
policy
that
I've
just
created.
D
Then
I
should
create
a
destination
rule
for
each
label
for
each
version
that
for
each
subset
that
I
define
here,
then
I
would
have
to
create
a
service
entry
to
tell
istio
how
to
reach
this
service.
On
the
other
side.
With
you
know
the
ip
of
the
hto
ingress
gateway
under
the
site
and
so
on,
then
I
would
have
to
have
like
an
android
filter
to
rewrite
my
dot
kind.
Three
dot
global
by
you,
know
cluster.local
and
then
destination
wall
to
reach
this
service
locally,
and
so
on.
D
So
again,
it's
not
important
to
really
understand
all
these
different
objects,
but
just
like
one
very
simple
to
write
objects
is
creating
all
these
history
objects
to
make
the
magic
happen
and
what
I
I've
shown
you
how
to
do
that
for
like
multi-cluster
traffic,
but
you
can
also
do
that
for
like
service
to
service
failover
a
little
bit.
What
similar
to
what
I
described
before
when
when
I
went
through
this
glue
federation,
but
for
for
a
steel
you
can
do
much
more.
D
You
can
also
combine
them
together,
so
you
can
have
like
glue
as
a
replacement
of
the
steel
ingress
gateway
for
the
request
coming
from
the
outside
world.
If
you
want
to
take
advantage
of
some
features
you
find
in
glue,
but
you
don't
find
in
the
studio
in
this
gateway,
and
and
that's
it-
I
mean
for
the
for
the
service
mesh
project.
D
Another
one
project
I
want
to
introduce
you
to
is
like
what
we
call
a
developer
portal
and
what's
interesting
with
the
developer
portal,
is
that
you
can
use
it
with
or
without
a
mesh,
and
you
get
exactly
the
same
experience.
So
if
you
you
don't
have
a
mesh,
then
it
will
enforce
the
rules
on
glue.
If
you
have
a
mesh,
you
can
just
configure
the
portal
so
that
it
will
enforce
the
rules
on
the
steering,
wheels
gateway
and
in
both
cases
it
works
exactly
the
same
way.
D
So
I'm
going
to
show
you
the
developer
portal
on
istio,
because
that's
what
I
have
in
the
environment
running
now,
but
that
would
be
exactly
the
same
if
I
don't
have
a
mesh-
and
I
just
want
to
use
it
with
glue.
So
again,
I'm
going
to
do
a
demo.
It's
a
lot
easier
for
you
to
understand
the
benefit
of
the
portal
than
if
I,
if
I
just
like
use,
slides
so
going
back
to
my
environment.
Here
I
have
a
developer
portal
and
this
is
the
admin
view
of
it.
D
So,
basically,
when
I
come
here,
I
can
see.
I
have
exposed
my
api
with
one
portal
and
the
way
you
do
it
is
that
you
first
of
all
create
what
you
call
api
docs.
So
an
api
dock
basically
is
like
an
open
api
specification
that
or
swagger
specs
that
you
either
you
upload
these
specs
or
you
just
like
paste
a
url,
and
it
will
use
these
specs
to
define
what
functions
are
available
in
the
api.
D
D
They
have
the
same
concept
where
you
can
define
the
configuration
with
crds
and
then
you
can
use
like
a
ui
and
dashboard
to
to
see
what
what
you
you,
what
you
build
and
and
what's
going
on
in
the
system.
So
here,
instead
of
going
through
the
ui
I
could
just
create-
and
this
is
what
I
did
to
prepare
the
demo
like
just
creating
an
api.crd.
D
So
that's
create
this
api
dock
in
my
in
my
portal.
But
then-
and
that's
the
api
lock
here,
but
then
what
you
want
is
that
you
want
to
expose
this
api
to
the
outside
world
and
sometimes
what
you
want
to
expose
is
an
api
that
is
based
on
multiple
micro
services
so
that
each
micro
service
will
handle
a
part
of
the
api
or
you
want
to
expose
just
a
subset
of
the
api
that
is
running
on
this
micro
service.
So
that's
really
the
purpose
of
an
api
product,
so
basically
an
api
product.
D
They
can
only
send
five
requests
per
minute
and
then
you
will
be
able
to
expose
this
api
dock
in
the
portal.
So
one
one
portal
can
be
used
to
expose
one
or
several
api
docs
and
api
products.
Sorry
and
each
api
product
can
be
composed
of
one
or
several
api
docs
and,
as
I
said,
you
can
enforce
access.
You
know
who
can
access
it?
You
know
with
oidc
or
basic
authentication
and
as
soon
as
you
check
this
box
like
root
enabled
what
it
does
is
that
it
creates
automatically
the
routing
for
you.
D
D
You
you
associate
your
api
product
to
portals,
and
you,
when
you
define
your
portal,
you
can
you
can
brand
them
the
way
you
want.
You
can
change
some
of
the
colors.
You
can
even
like
completely
override
the
the
css
style
sheet
so
that
it
really
looks
like
you
want
it
to
look.
You
can
add
some
pages
to
your
portal
that
can
be
static
pages,
but
even
dynamic
pages
that
load
in
the
context
of
your
user
or
the
context
of
your
api
and
and
then
you
can
define
also
access
control
at
that
level.
D
So
you
can
have
different
level
of
of
access
controls
and
the
the
user
experience
is
that.
Then
you
have
this
portal
that
can
be
accessed
by
your
end.
Users
and
your
end
user
can
log
in
with
either
oidc
or
local
user.
So
here
I
just
like
have
like
a
local
user.
I
have
to
remember
the
password
I
put,
which
is
not
that.
A
D
D
D
And
now,
if
I
go
to
the
api
that
are
available-
and
I
want
I
can
try
them
out
so
I
can
just
say:
oh
there
is
this:
like
api
call
for
like
getting
the
products
and
if
I
click
on
trying
out,
I
can
click
on
execute.
And
here
I
see
I
get
like
a
401
response
code
because
I
didn't
specify
my
my
key
so
to
do
that.
D
Basically,
I
click
on
autorize
and
I
can
paste
my
key
here
and
now,
if
I
run
it
again,
it
should
be
a
load
because
I
have
my
key
and
you
see,
I
get
my
200
response
codes
and
you
remember,
I
have
a
plan.
I
can
just
do
five
per
minute.
So
if
I
click
on
execute
you
know
several
times,
then
I
should
get
like
you
see,
429,
which
is
like
too
many
requests,
because
it's
height
limited.
D
D
Portal
and
basically
I
spoke
about
our
current
project
that
people
use
in
production.
Now,
let's
speak
about
the
future,
so
the
future
is
really
web
assembly
and
not
only
web
assembly,
but
I
should
even
not
call
that
that
way,
but
web
assembly
in
general,
so
web
assembly
is
really
the
future
of
that.
D
Will
you
know
the
way
you
will
extend
envoy
in
general
in
the
future
and
why
we
we
need
that,
because,
currently
you
remember
when
I
discussed
about
glue,
I
I
said
that
we
have
this
filter
chain
and
basically
the
way
we
build
these
filters
at
solo.io
that
we
use,
like
c
plus
plus,
to
create
these
filters,
and
then
we
basically
need
to
statically
link
it
to
a
voice.
So
we
compile
everything
together
and
we
ship
a
new
version
of
glue
with
all
these
filters.
D
So
it's
not
easy
because
obviously
c
plus
plus
is
not
really
the
language
of
choice
for
many
people,
and
also
it's
not
dynamic.
You
know
you
cannot
just
load
a
new
filter
when
you
need
it,
and
the
good
news
is
that
the
android
community
has
has
worked
hard
and
we
participated
to
this
effort
to
to
allow
people
to
run
web
assembly
filters
in
envoy
and
webassembly
has
been
designed
at
the
beginning
for
the
browser
as
a
way
to
run
some
codes
from
javascript.
D
That
is
not
written
in
javascript
and
that's
used
now
in
envoy,
and
basically
you
have
some
sdks
that
exist
in
multiple
different
languages,
like
you
have
one
in
in
assembly
script,
one
in
rest,
one
in
tiny
go
and
you
can
use
this
sdk
to
build
your
filter
in
a
long
way
that
is
more
accessible
than
c,
plus
plus
and
then
what
we
did
at
solo.iota.
We
created
this
web
assembly
hub.
So
it's
really
the
same
idea
as
a
docker
web.
So
you
you
you,
you
use
like
a
cli
that
we
provide.
D
D
You
have
multiple
choices
there
as
soon
as
it's
envoy
behind
the
scene.
So
it's
it's
really
the
future.
It's
not
really
ready
for
production
now,
and
I
I
shared
some
links
there
so
that
you
can.
You
can
see
and
learn
more
about
it.
My
colleague
christian
has
written
this
blog
post.
That
is
really
useful
if
you
want
to
understand
the
current
status
and
also
like
what
you
can
really
get
from
it
today
and
you
know
where
it
is
going
so
far.
D
D
So
a
lot
of
of
reasons
why
you
cannot
really
use
them
in
pollution
now,
but
it's
coming
very
soon,
and-
and
here
you
can-
you
can
get
a
lot
of
information
about
it.
So
I
really
encourage
you
to
to
read
that
basically,
the
one
of
the
nice
thing
with
it
is
that,
because
you
can
apply
this
filter
anywhere,
you
can
apply
that
in
glue
and
then
that
will
be
a
new
filter
in
your
filter
chain
for
the
request
coming
from
the
outside
world.
D
But
if
you
apply
that
on
istio,
you
can
apply
that
on
the
hto
ingress
gateway.
If
you
want
to
also,
you
know,
modify
the
request
coming
from
the
outside
world,
but
you
can
also
use
that
for
enforcing
some
rules
in
your
east-west
traffic.
So
I
have
my
micro
service,
a
that
talks
to
microservice
b
in
my
mesh,
and
I
can
now
modify
this
request,
basically
using
an
a
filter
that
I
create
on
on
using
webassembly.
D
D
D
So
if
you,
if
you
are
like
fluent
in
english
and
kubernetes,
which
is
probably
the
the
the
fact
for
all
of
you
and
you-
you
have
some
knowledge
on
you
know
either
like
istio
or
api
gateway
or
android,
like
we
really
look
for
this
profile.
So
please
reach
out
to
me
and
I
I
will
be
very
happy
to
to
talk
with
you.
So
thank
you.
I
think
we
have.
A
By
doing
that,
maybe
that
was
my
mistake.
I
already
edited
unlimited
screens
yeah
thanks
so
much
dennis
for
for
the
presentation
we
do
have.
If
we
still
have
a
couple
of
minutes,
we
we
got
two
questions.
First,
questions
is
all
of
this.
Everything
that
you
presented
is
this
free
to
use
for
the
developers
or
is
it
the?
Is
it
some
some
paid
hit,
plus.
D
The
service
mesh
hub
is
open
source,
so
everything
I
have
shown
you
you've
seen
on
service
mesh
is
open
source,
but
we
have
an
enterprise
version
coming
and
the
dashboard
you've
seen
will
be
part
of
the
enterprise
version
and
you
will
have
some
special
features
there
as
well,
but
there
is
this
open
source
version,
all
the
traffic.
You
know
between
clusters.
All
these
things,
it's
open
source,
all
our
effort
on
the
web
assembly
side
like
web
assembly.
All
this
stuff
are
free
and
open
source.
D
A
D
What
a
typical
deployment
with
glue
would
be
like
you
would
have
like
glue.
Is
the
control
plane.
You
have
one
glue
running
in
in
your
cluster
and
then
you
can
have
multiple
instances
of
android
like
let's
say
a
very,
very
common
way
is
to
have
like
two
of
them
and
so
that
you
get,
but
you
can
scale
to
more
if
you
need
more
performance.
Obviously
it's
not
very
common
because
you
get
a
lot
of
performance
with
with
invoice
with.
You
know
two
two
instances
in
general,
it's
more
than
on
us.
A
D
I
mean
like,
I
would
say
that
you
know
the
the
the
micro
services
decision
is
generally
taken.
For
other
reasons,
like
you
know
you
you
cannot
handle
the
monolith
anymore.
You
want
to
scale
it
a
different
component
differently.
You
want
to
be
more
agile
and
so
on,
but
then,
when
people
took
the
decision
to
go
and
use
micro
services
and
they
took
the
decision
to
go
to
run
on
kubernetes,
then
in
any
case
they
need
to
expose
them
to
the
outside
world
and
they
need
to
think
about
how
they
can
be
smart
about.
D
You
know,
like
I,
don't
know,
argo
cd
or
things
like
that
to
enforce
the
rules
on
on
the
cluster
and
then
basically
so
they
they
handle
that
you
know
with
this
infrastructural
code
approach
and
then
they
they
they
want
to
have
this
dashboard
so
that
they
can
see.
You
know
what
what
they've
configured
was,
the
effect
of
everything
or
the
statues,
and
things
like
that.
B
D
D
Oh
no,
no,
it
is.
We
have
like
a
lot
of
very
small
customer.
We
even
have
a
starter
pack,
startup
pack,
so
that
we
promote
this
usage.
I
mean
like
startups.
Generally,
they
develop
new
applications.
Obviously
they
they
start
with
micro
services
right
away.
You
know
they
start
in
the
cloud.
They
start.
D
You
know
with
kubernetes
containers
so
then,
when
they
have
to
decide
about
an
api
gateway
because
they
don't
want
to
you
know
they
don't
want
to
develop
like
all
the
authentication
layer
in
the
microservices
themselves,
and
they
don't
want
to
do
all
these
things
there.
You
know
they
worry,
then
they
have
to
find.
D
A
Bye,
bye,
thanks,
dennis
see
ya
yeah,
as
I
said,
venice
is
jumping
off
a
little
bit
earlier
is
he's
going
to
present
at
the
cncf
webinar
starting
in
two
minutes,
and
I
think
we're
already
handing
over
to
janos
to
start
with
this
presentation
on
a
topic
around
the
building.
Your
scaling
with
the
promethouse
metrics
right,
something
like
this.
B
B
Yeah,
thank
you
very
much.
So
thanks
for
having
me
first
of
all,
so
this
is
a
very
interesting
topic
that
I
developed
a
while
ago
and
well
as
as
mentioned
earlier,
I'm
currently
working
as
a
lecturer
at
the
effa
campus
win,
which
I
will
continue
doing
so.
This
is
kind
of
like
a
side
gink
and
I'm
starting
with
red
hat
next
week,
and
it
took
the
past
one
and
a
half
months
off
to
to
build
a
couple
of
interesting
things.
B
This
is
this
auto
scaler
is,
is
not
the
only
thing
I
built
recently.
So,
if
you're
interested,
you
could
take
a
look
at
container
ssh,
which
is
a
cloud
native
ssh
server
that
talks
to
the
docker
and
kubernetes
api
to
launch
containers
for
each
session.
It's
it's
completely
open
source
and
we
just
put
in
significant
amounts
of
effort
to
to
getting
it.
Let's
just
say
user-friendly.
There
is
a
very
good
documentation,
take
a
look,
and
if
you
want
to
use
it
contribute,
let
us
know
and
have
fun
with
it.
B
So
this
is
basically
what
I
do
now
why
why
are
we
talking
about
auto
scaling?
After
all,
kubernetes
already
has
a
horizontal
port,
auto
scaler.
It
has
a
cluster
auto
scaler
and
there
is
like
a
five
bazillion
other
solutions
as
well.
Amazo
azure
has
has
their
azure
auto
scale.
Amazon
has
their
aws,
auto
scaling
groups,
etc,
etc.
So
is
there
even
a
need
to
talk
about
auto
skilling
and
one
of
the
things
I
I
I'm
recently
conflicted
with
is
is
it
seems
like
this
is
solving
an
issue
that
we
don't
really
have.
B
After
all,
when
you're
a
small
startup,
then
your
auto
scaling
needs
are
very
modest
and
most
likely
you
can
just
use
whatever
your
cloud
provider
is
providing.
On
the
other
hand,
if
you
are
a
larger
company,
then
you
are
probably
stuck
with
some
sort
of
a
legacy.
Infrastructure
and
scaling
isn't
even
on
the
horizon
for
you
or
if
it
is,
then
you
wouldn't
be
taking
advice
from
me.
B
The
thing
I
found
with
autoscaling
is,
first
of
all,
you
might
be
a
working
in
a
large
company
that
has
an
on-premises
system
and
you
would
kind
of
like
to
go
into
the
direction
of
the
cloud
but
you're
not
quite
ready
to
use
a
public
cloud
provider.
So
in
that
case,
a
little
bit
of
orchestration
around
creating
and
destroying
vms
could
be
actually
quite
helpful.
B
What
I
did
in
the
past
is
work
on
web
hosting,
where
posting
is
kind
of
my
pet
projects
for
a
good,
almost
10
years,
and
the
issue
there
is
that
when
you're
running
web
hosting
for
php
is
php,
it
doesn't
really
have
a
good
metric
in
terms
of
cpu
or
ram.
Rather,
the
php
fpm
starts
one
process
per
user,
and
what
you
run
out
of
very
often
is
the
number
of
processes
that
you
can
run,
so
your
scaling
decision
might
be
based
on
something
very
very
custom.
B
The
other
thing
is
vendor
lock-in.
A
certain
level
of
lock-in
is,
of
course,
to
be
expected
when
you
move
into
the
cloud,
but
you
don't
want
to
go
full
crazy
with
with
using
everything
the
cloud
provider
has,
because
you
will
never
ever
be
able
to
move
if
their
service
offering
is
too
expensive,
or
you
decide
that
you
don't
want
that
anymore.
I
recommend,
looking
at
netflix's
talk
talks
on
how
they
moved
from
using
everything
in
the
cloud
to
building
their
own
edge,
cdn
kind
of
thing.
B
The
other
thing
is,
is
you
will
have
a
monitoring
system
anyway,
and
it's
most
likely
going
to
be
based
on
prometheus
and
or
graffana,
and
it's
a
very
good
piece
of
technology.
So
why
not
use
it
with,
in
conjunction
with
your
application,
so
do
more
with
your
monitoring
system,
and
that
also
leads
into
learning
something,
because
a
lot
of
people
are
using
prometheus
and
don't
necessarily
understand
how
this
works
or
how
to
add
custom
metrics
to
their
applications.
So
this
is
also
what
this
talk
is
about.
B
B
This
is
a
special
http
server
that
I
wrote
that
specifically
generates
a
very
high
cpu
load
as
a
demonstration
of
course,
and
we
are
going
to
be
running
that
and
we
are
going
to
run
the
prometheus
node
exporter
and
the
prometheus
node
exporter
is
the
piece
of
software
that
gives
the
cpu
and
ram
and
whatever
metrics,
to
prometheus
and
we'll
see
in
a
moment
how
that
works.
What
we
will
use
from
our
cloud
provider
is
a
load
balancer.
B
If
you're
running
on-premises
system,
then
you
will
have
to
figure
out
how
to
do
that
as
well,
and
we
will
run
the
whole
thing
in
a
scaling
group.
The
scaling
group
in
this
sense
means
that
whenever
a
new
vm
is
needed,
then
the
scaling
group
will
provision
that
vm
and
if
there
are
too
many
vms,
then
it
will
destroy
that
vm.
If
a
vm
crashes,
it
will
destroy
and
recreate
that
virtual
machine.
We
will
also
install
a
monitoring
server
and
that
monitoring
server
will
of
course,
run
prometheus.
B
Of
course,
the
two
now
has
methods
to
update
the
database,
but
it's
scripting.
It's
hacky,
you
have
to
put
in
a
lot
of
work
and
prometheus
does
most
of
that
itself.
We
will
use
grafana,
you
don't
necessarily
have
to,
but
it
is
a
very
nice
dashboard.
So
why
not
use
that
as
well,
and
we
will
use
a
small
component
that
we
write
ourselves
called
the
autoscaler
that
will
be
triggered
by
grafana
webhooks.
So
this
is
the
setup
that
we
will
be
looking
at
today.
B
Let's
look
at
the
scaling
group
first,
so
the
thing
is
with
within
when
you
look
at
the
cloud.
The
way
that
virtual
machines
are
provisioned
is
from
an
image,
and
this
image
often
contains
a
piece
of
software
called
cloud
init
or
if
you're
running
windows,
then
often
an
alternative
version
from
a
company
called
cloud-based
or
the
cloud-based
init,
and
this
cloud
in
it
is
talking
to
the
metadata
server
of
the
cloud
provider.
This
metadata
server
can
give
you
a
large
number
of
fields.
For
example,
you
can
fetch
your
own
ip
address
on
amazon.
B
You
can
fetch
your
your
your
machine,
specific
api
keys.
You
can
fetch
many
other
things
and
the
most
important
to
us
is
the
user
data
field,
and
this
user
data
field
is
a
yaml
file
or
a
shell
script
can
be
both
in
the
yaml
file.
You
can
do
a
little
more,
but
in
our
example,
we
will
use
a
shell
script
and
this
shell
script
will
be
executed
when
the
machine
first
starts.
B
B
So
the
first
thing
we
are
going
to
do
is,
of
course,
install
docker.
We
are
not
using
kubernetes
in
this
example,
but
again
you
can
extend
this
very
easily
to
kubernetes
as
well.
First
thing
is:
please,
don't
use
this
script
in
production.
I
hope
I
don't
have
to
say
that
we
will
use
the
convenience
script
from
docker
again,
not
recommended
for
production
and
install
docker
using
that
convenience
script.
The
second
thing
we
are
going
to
do
is
install
the
web
server
using
docker.
So
what
we
are
going
to
do
is
do
a
docker
run.
B
B
If
you
don't
know
how
the
docker
commands
work,
then
the
manual
is
very,
very
well
written.
So
please
take
a
look
at
that
and
finally,
we
are
going
to
give
it
the
image
name
and
you
can
use
this
image
name
if
you
ever
need
a
load
generator.
So
this
is
my
nickname,
slash,
http
load,
generator
version
101.
B
The
second
thing
we
are
going
to
install
the
node
exporter
again
with
the
restart
policy
always
and
we
are
going
to
run
it
in
the
host
network,
namespace
the
net
namespaces
in
docker
and
then
containerization
are
a
way
to
isolate
containers
from
each
other
and
from
the
host
operating
system.
So
there
are
multiple
namespaces,
the
network,
namespace,
the
pid,
namespace,
uts,
etc.
I
have
a
whole
talk
on
a
topic.
If
you
want,
you
can
look
it
up
or,
of
course,
if
you
prefer
reading
there
is
a
blog
post
about
that
too.
B
The
main
point
is
that
our
node
exporter
is
going
to
run
unconfined
in
terms
of
networking
and
in
terms
of
reading
processes,
and
this
is
needed
by
the
node
exporter.
We
are
also
going
to
mount
the
file
system
from
the
host
operating
system
into
the
slash
host
folder
again.
This
is
normally
very
dangerous
if
you're
running
some
untrusted
software,
because
the
the
container
would
then
have
access
to
the
whole
host
operating
the
file
system.
B
But
in
our
case
the
prometheus
node
exporter
needs
that
to
be
able
to
read
the
the
metrics
of
the
host
file
system,
so,
for
example,
how
full
the
disk
is
and
then
we
are
going
to
pass
it
the
image
name.
This
is
quay
dot,
io,
slash,
prometheus
node
exporter
and
we're
going
to
pass
it
a
parameter
called
minusminuspath.rootfs
is
located
in
slash
host.
So
that's
it
our
instance
pool
is
now
ready
and
every
time
a
vm
comes
up,
it
will
install
install
the
http
server
and
it
will
install
the
node
exporter.
B
So,
let's
take
a
look
at
the
second
part:
the
monitoring
server.
If
you
look
at
the
second
part
and
specifically
the
service
discovery
and
the
prometheus,
this
looks
as
follows.
First
of
all,
we're
going
to
write
a
prometheus
config.
This
prometheus
config
will
have
a
script
config
the
scrape
config
you
can
provide
multiple
script.
Configs
is
what
tells
prometheus
where
to
fetch
the
metrics
from,
and
in
this
case,
what
we're
doing
is
we're
going
to
provide
it.
A
file
service,
discovery
config.
B
That
means
that
prometheus
will
every
10
seconds,
as
you
can
see,
fetch
this
config.json
file
and
use
the
ip
addresses
in
there.
So
whenever
we
update
the
config.json
prometheus
will
update
its
ip
address
list,
we
start
prometheus
just
again
with
docker
run.
Minus
must
restart
always
expose
port
1990
on
the
monitoring
server
and
we
pass
it.
The
prometheus
yaml
in
the
folder
ect
prometheus
prometheus
yaml
and
we
pass
it
a
special
folder.
This
folder
almost
doesn't
matter
what
it
is
in
our
case.
We're
just
picking
srv
service
discovery
and
we
are
launching
problem.
B
Slash
prometheus,
remember
the
srv
service
discovery.
Slash
config.json
is
the
file
where
we
provide
the
ip
addresses,
so
this
folder
is
important,
the
second
one,
so
we
can
provide
the
ips
now
the
service
discovery
agent
needs
to
provide
the
ips
in
this
format.
So
you
have
a
json
file.
It
has
an
array
in
it
then
an
object
and
then
there
is
a
target
and
then
you
provide
the
ip
addresses
and
ports
of
the
metrics
endpoint.
That
prometheus
is
support
supposed
to
fetch.
B
B
This
is
not
again
something
that
you
should
do
in
production,
but
it
is
a
very
useful
way
to
showcase
what
I'm
doing
in
my
cloud
provider
in
this
case
is
exoskill.
This
is
what
I
built
the
demo
for,
and
I'm
just
listing
the
instance
ids.
Then
I
am
looping
over
these
instance
ids.
So
I'm
running
the
code
between
the
four
and
the
done
for
every
single
instance.
I'm
fetching
the
ip
address
of
that
specific
instance
in
the
instance
pool
and
then
I'm
just
writing
that
to
a
temporary
file.
B
B
Now
you
can,
of
course,
do
this
with
other
cloud
providers.
So
with
aws,
your
aws,
auto
scaling
describe
auto
scaling
groups.
Azure
is
the
az
vmss
list
instance
public
ips
or
in
the
google
cloud
it
would
be.
Google
gcloud
compute
instance
groups
to
list
the
instances
in
an
instance
pool
whatever.
Of
course,
the
name
is
always
different
depending
on
the
cloud
provider.
B
Now,
what?
If
you
don't
want
to
do
the
service
discovery?
You
can
of
course,
look
at
the
prometheus
configuration
and
you
will
see
that
there
are
a
bunch
of
drivers
for
azure,
digitalocean,
docker,
swarm,
amazon,
ec2,
etc,
etc.
The
only
thing
you
have
to
take
care
of,
if
you
have
more
instances
than
just
your
auto
scaling
group,
then
you
will
need
to
label
the
auto-scaling
group
appropriately,
so
it
only
fetches
the
metrics
from
the
auto-scaling
group
and
you
have
the
necessary
labels
to
make
decisions
only
based
on
your
auto-scaling
group.
B
Okay,
let's
take
a
look
at
the
next
part
and
that's
going
to
be
how
prometheus
fetches
the
metrics.
If
we
look
at
prometheus,
we
click
on
status
and
then
targets.
We
will
end
up
on
this
screen
and
this
screen
will
show
us
the
ip
addresses
that
we
have
provided
in
the
config.json,
and
you
can
see
the
url
there,
which
is
slash
metrics,
is
where
the
where
prometheus
fetches,
the
metrics
from.
If
we
open
this
url
in
our
browser,
we
will
see
something
like
this.
B
The
first
part
here
is
the
metric
itself
in
this
case,
go
gc
duration
seconds.
Then
we
can
also
provide
one
or
more
labels.
These
labels
are
basically
filters
and
we
can
do
wonderful
things
with
these.
I
will
show
you
in
a
minute
and
finally,
we
will
provide
the
actual
number
of
that
specific
metric.
There
are
some
other
things
like
which,
what
metric
type
it
is
some
help
text
etc.
But
this
is
the
basic
idea:
you
can
write
your
own
prometheus
exporter
for
your
application.
B
So
if
your
developers
just
want
to
give
you
some
metrics,
then
this
is
the
easiest
way
they
should
just
need
to
provide
a
metrics
endpoint
and
from
that
metrics
endpoint,
you
can
sketch
the
application
specific
metrics
now
in.
If
you
are
really
interested
in
doing
that,
I
would
recommend
reading
this
piece
of
documentation.
This
is
the
exposition
formats
documentation,
and
this
is
now
being
standardized
into
something
that
is
independent
of
prometheus.
So
this
is
a.
B
Now,
if
you
open
prometheus,
we
can
of
course,
then
see
this
very,
very
bare-bones
screen
and
we
can
look
at
the
metrics
up
there.
The
metrics
up
there
in
my
example
are
look
like
this.
This
looks
this
is
in
written
in
a
language
called
problem
ql,
so
the
prometheus
query
language
and
you
can
see
that
we
can
do
a
bunch
of
things.
I'd
like
to
take
one
piece
out.
This
is
the
node
cpu
seconds
total.
B
So,
for
example,
we
take
the
cpu
usage
in
user
mode
and
kernel
and
whatever
not-
and
we
sum
those
up
by
instance
to
get
the
current
cpu
usage
since
the
start
of
the
vm,
and
we
divide
that
by
the
total
number
of
cpu
seconds
since
the
start
of
the
machine
and
that's
how
we
get
the
rate
function,
we
get
the
current
cpu
usage
and
then
we
can
of
course,
average
it
by
instance,
and
that
will
show
us
a
very
nice
average
graph
of
the
cpu
usage
of
each
instance.
So
that's
already
very
helpful.
B
The
important
part
for
us
is
that
we
want
to
have
it
installed,
and
that
is
very,
very
simple
indeed,
and
we
run
docker
run
with
the
restart
policy
of
always.
Of
course,
we
expose
port
3000,
that's
the
graphana
web
server
and
we
pass
the
image
name
and
that's
it,
of
course,
in
production
you
would
need
to
do
some
persistence
for
your
configuration
or
you
would
need
to
provide
the
dashboard
and
the
metrics
as
part
of
the
startup
sequence,
which
is
also
a
possibility
again.
B
I
have
also
written
a
few
blog
posts
on
that
topic,
so
feel
free
to
dig
through
my
blog
when
you
open
the
grafana
dashboard,
then
this
is
what
you
are
greeted
with.
You
can
enter
the
username
and
password
admin
admin.
Of
course
you
should
that
then
immediately
change
it,
and
then
you
see
this
and
you
can
as
a
first
step,
click
on
the
little
cogwheel
icon
and
then
click
on
data
sources
and
in
data
sources.
You
click
the
data
sources
button
and
you
can
see
that
prometheus.
Among
many
other
things
is
already
supported.
B
So
then
you
click
the
select
button
and
you
set
up
a
new
data
source
in
prometheus,
and
this
should
be
on
the
public
ip
address
of
your
monitoring
instance.
So
we
are
basically
telling
grafana
where
to
find
prometheus.
You
can,
of
course,
use
docker
networking
to
do
that
internally.
In
this
case,
it's
a
lot
easier.
B
If
we
do
that
publicly,
so
we
just
enter
the
port
in
1990,
where
prometheus
is
listening
and
then
save
everything
and
then
click
on
the
plus
icon,
the
plus
icon,
will
let
us
create
a
new
dashboard
and
then
we
add
a
new
panel,
and
this
then
greets
us
which
looks
like
it's
a
nasa
dashboard.
And,
oh
my
god,
there
are
a
lot
of
buttons
here.
So
what
we're
doing
here
is
we
click
in
the
metrics
field,
and
you
can
see
already
that
the
default
met.
The
default.
B
Backend
is
the
prometheus
one
and
we
enter
a
prom
ql
query,
so,
let's
copy
in
the
little
metric
that
we
previously
did
and
you
can
see
that
a
green
and
the
yellow
line
is
appearing
in
the
graph
above.
This
is
very
nice,
because
now
we
see
per
instance
what
their
average
cpu
usage
was
over
the
last
minute.
We
can
also
go
to
that
little
legend
field
in
the
bottom
and
enter
a
legend
to
make
the
labels
there
on
the
graph
more
readable.
So
this
is
basically
how
you
configure
grafana.
B
You
can
then
look
at
the
visualization
settings
and
make
the
graph
look
very
very
good.
Indeed,
there
is
a
map
plugin,
so
you
can
really
feel
like
you're
in
some
sort
of
a
james
bond
movie
and
you're
looking
at
the
world
map
where
the
nukes
are
being
launched,
or
what
have
you
so
yeah?
Well,
isn't
that
everybody's
dream
to
have
a
control
panel
like
that
grafana
can
do
it
now?
The
interesting
part
for
us,
though,
is
not
the
let's
feel
like
a
james
bond
villain,
rather
the
alert
section.
B
If
you
go
to
the
alert
section
that
will,
let
us
create
the
webhooks
that
we
so
desire
to
drive
our
autoscaling.
Let's
take
a
look
at
how
that
works.
Specifically,
we
go
to
the
little
bell
icon
and
then
to
the
notification
channels
and
in
the
notification
channels.
We
are
going
to
create
a
new
notification
with
the
name
scale
up
type
webhook
and
in
the
url
we
are
going
to
enter
the
public
ip
again
of
our
monitoring
server.
B
This
time
we
are
choosing
a
port
that
we
did
not
use
before
port
4000
and
we
are
going
to
enter
the
url
up.
The
slash
up,
url
is
going
to
be
the
scaling
up.
Webhook
and
the
slash
down.
Url
is
going
to
be
the
scaling
down,
webhook
we
scroll
down
and
we
need
to
check
a
few
additional
settings.
First
of
all,
we
need
to
disable
the
resolve
message.
B
Otherwise,
if
the
cpu
usage
goes
back
to
normal,
then
grafana
will
send
another
webhook,
which
will
scale
up
our
instance
pool
again
unnecessarily
and
we
will
also
set
the
set
set,
send
reminders
option.
The
send
reminders.
Option
can
send
a
reminder
every
15
minutes
if
the
cpu
usage
is
still
too
high.
So
let's
take
a
look:
how
that
works.
B
So
if
we
go
back
to
graffa,
then
we
can
change
this
little
formula
here.
This
little
prom
ql
query
to
give
us
a
single
metric,
instead
of
using
an
average
by
instance,
we're
using
an
average
over
all
the
instances-
and
this
is
the
metric
we
are
going
to
use
in
terms
of
cpu
usage.
The
alert
section
is
then
set
up
in
such
a
way
that
we
have
an
80
threshold.
So
if
the
cpu
usage
climbs
above
80
percent
over
one
minute,
then
it
will
trigger
the
scale
up
notification.
B
So
this
is
something
we
have
to
set
up.
We
can,
of
course,
then
set
up
an
according
graph
with
the
scaled
down
notification.
If
the
cpu
usage
falls
below
20
and
then
this
is
how
it
should
look
like.
As
you
can
see,
the
graph
on
the
left
is
triggering
the
scaling
up
and
the
graph
on
the
right
is
triggering
the
scaling
down.
B
So,
let's
install
nginx
and
on
the
monitoring
server.
This
is
again
a
very
non-production
ready
solution,
so
apd
install
nginx,
we
are
going
to
install
a
little
extension
libengenix
mod
http
lua.
So
this
lets
us
run
some
lua
scripts
from
nginx
and
we
need
the
jq
utility.
So
let's
write
the
nginx
config,
don't
do
this
in
production,
please
we
are
writing
into
the
file,
etc.
Nginx
site's
enable
default,
create
a
server
block,
listen
on
port
4000
and
create
a
location
handler
for
slash
up
again.
B
This
is
where
the
web
hook
goes
and
we
are
going
to
execute
this.
Shell
script
called
slash,
sre,
slash,
app.sh.
We
are
also
going
to
do
a
scaling
down
equivalent
for
slash
down
just
going
to
launch
the
down.sh.
So
how
does
one
more
thing
we
need
to
do?
Is
we
need
to
set
up
an
api
key
for
these
two
shell
scripts
and
we
need
to
restrict
it
to
only
be
able
to
scale
the
instance
pool
up
and
down.
Otherwise
we
will.
B
We
don't
want
to
give
this
little
script
root,
privileges
that
would
be
overkill
now,
let's
take
a
look
at
this
app.sh.
Don't
use
it
in
production.
Set
minus
e
will
make
sure
that
all
errors
immediately
lead
to
failure.
We
will
declare
a
maximum
size,
for
instance,
pool
after
all,
our
budget
is
limited
and
then
we
will
fetch
the
current
size.
B
I
did
not
detail
the
script
here,
but
basically
this
fetches,
the
number
of
instances
in
your
instance
pool
if
the
number
of
instances
is
smaller
than
the
maximum
size,
then
we
increment
the
size
by
one
and
then
set
the
new
instance
pool
size
and
our
cloud
provider
will
immediately
take
care
of
provisioning.
The
new
instance
that
we
need
again
the
reminders
will
make
sure
that
this
script
is
called
repeatedly
until
the
cpu
usage
falls
below
80.
B
So,
let's
test
it,
we
are
going
to
use
apache
benchmark
and
we
are
going
to
fire
away.
Two
parallel
requests
and
thousands
of
those
we
fire
them
against
the
load.
Balancer
on
slash
load,
slash
load
is
the
special
url
that
generates
the
load,
and
we
can
see
that
our
script
will
be
working.
We
can
see
that
the
instance
pool
is
scaling
up
and
in
our
little
graph
we
can
see
that
initially
it
started
with
two
instances.
B
B
But
before
we
come
to
demo,
let's
talk
about
redundancy,
because
this
is
actually
not
production.
Ready.
As
I
mentioned,
one
of
the
things
with
grafana
is
that
it's
very,
very
beautiful,
it's
very
useful,
but
it
is
kind
of
hard
to
make
redundant
because
you
need
to
provide
a
redundant
database
behind
it
instead
of
kefana.
B
So
a
few
recommendations
on
my
part
again,
please
don't
use
shell
scripts.
Please
learn,
go
or
at
least
learn
python.
Go
is
in
this
sense
preferable
because
it's
much
easier
to
containerize
than
python,
and
it
will
let
you
write
something
like
this
in
a
couple
of
lines
of
code
and
then
now
we
are
switching
to
the
demo
and
I
will
just
change
my
screen
share.
B
In
a
minute,
if
I
can
find
the
appropriate
button
share
my
screen
entire
screen,
there
we
go,
and
now
I
made
the
screen
section
what
I
so
you
can
see
that
I
have
already
set
up
the
infrastructure
here.
My
monitoring
instance
is
running
here
and
my
instance
pool
is
already
running
and
it's
running
one
instance.
This
is
done
using
a
a
terraform
script,
because
I
am
using
this
for
my
fr
campus
win
lectures
as
well,
so
this
is
one
of
the
projects
work
that
the
students
have
to
do.
B
The
terraform
code
will
be
released
mid
december.
Now,
if
we
look
at
the
grafana
section
here,
you
can
see
that
I
have
three
graphs
here.
This
is
the
low
cpu
usage,
the
high
cpu
usage
and
the
instance
pool
size.
If
we
go
into
the
data
sources
section,
you
can
see
that
I
have
prometheus
set
up.
In
my
case
I
did
some
internal
networking,
so
this
is
a
little
different
than
in
the
demo,
but
you
can.
B
Now,
if
I
go
to
my
dashboard,
what
happens
if
I
hit
the
end
point
of
the
network
load
balancer,
I'm
going
to
give
this
ip,
and
this
is
going
to
load
for
a
while
and
then
it's
going
to
generate
some
cpu
usage.
You
can
see
that
the
cpu
graph
is
already
climbing.
Prometheus
has
a
quite
fast,
quite
a
fast
update
rate,
so
we
will
see
that
it's
going
to
be
a
little
bit
of
a
bump
and
then
it's
going
to
go
back
down
nothing
too
serious.
This
is
not
going
to
trigger
a
scaling
event.
B
The
low
cpu
usage
is
now
back
into
green
because
when
it
was
under
the
the
red
line,
it
actually
continuously
triggered
the
scaling
down,
but
the
script
prevented
it
from
scaling
down
below
one
instance,
and
you
will
see
in
a
moment
it
will
this
one.
This
graph
will
climb
into
the
red
area,
which
means
that
first
of
all,
it
will
enter
the
yellow
state.
The
yellow
state
in
grafana
is
the
pending
state
and
in
the
pending
state.
B
It
will
wait
if
it
goes
back
down
and
you
can
configure
how
long
this
pending
state
is
to
make
sure
that
sudden
spikes
don't
trigger
some
erroneous
alerts,
and
if
it
stays
in
the
in
the
red
area
which
it
will
then
it
will
enter
the
alertings
at
the
alerting
state
and
in
the
alerting
state
it
will
actually
send
the
web
hook
to
our
auto
skiller.
It
should
just
take
about
two
minutes
to
happen
in
the
meantime.
If
there
are
any
questions,
I
would
be
more
than
happy
to.
B
B
B
And
it
should
happen
any
second.
Now
there
we
go,
there
is
the
alerting
state,
so
that
means
that
a
web
hook
was
triggered.
We
can
look
at
it
in
the
alert
rules
scale
up
if
this
is
this
is
actually
under
load.
This
is
a
little
slow
to
load.
I
provisioned
a
very
small
instance
for
this,
so
this
might
not
play
as
this
is
not
loading,
but
in
this
screen
there
we
go
there's.
There
are
already
two
instances
running,
so
that
was
quite
quick,
which
is
not
unexpected.
B
B
So
that's
why
I'm
still
using
them.
You
can
see
that
there
are
two
instances
running
now
and
the
cpu
load
should
go
down
once
the
new
instance
is
provisioned
correctly.
This
is
going
to
take
about
another
one
or
two
minutes,
because
the
the
user
data
script
actually
needs
to
execute,
install
the
web
server
and
once
the
web
server
is
installed,
the
network
load.
Balancer
health
check
will
be
green.
B
A
A
B
So
so
the
the
the
the
terraform
code
for
this
will
be
made
open
source
in
mid-december,
because
this
is
a
project
work
that
the
students
need
to
hand
in.
So
that's
why
it's
not
open
source.
Yet
I
can
paste
the
link
the
yes,
you
can
see.
The
cpus
is
now
going
down
because
the
second
instance
kicked
in.
So
there
are
a
couple
of
repositories:
the
fr
cloud
computing
repository.
I
will
share
that
in
a
minute
is
one
of
the
repositories
where
the
learning
material
is.
B
I
can
actually
open
that
real
quick,
so
this
is
f
card
cloud
computing,
dot,
github.io.
This
is
where
the
learning
material
is.
If
you
want
to
go
through
the
exercises
and
the
lectures
yourself-
and
there
is
an
original
article
about
this-
auto
scaling
behavior
on
the
exoskill
blog.
So
you
might
want
to
take
a
look
at
that
as
well.
A
Cool
pretty
amazing
demo
thanks
so
much
you
really
rushed
us
through
the
through
the
whole
setup
at
demo.
My
personal
question
would
be:
how
long
did
it
take
you
to
build
to
build
this?
It
was
for
for
me
was
we
we
see
really
a
lot
of
different
building
blocks
and
all
the
different
parts
you
have
to
be
like
profound
in
it.
So
how
long
did
it
take.
B
To
build
this,
so
I
did
not
start
with
this
setup.
That's
the
that's
the
preface
I
I
did
use
prometheus
for
quite
a
while
on
grafana
as
well.
I
don't
think
that
that
you
can
follow
the
instructions
and
I
had
a
few
people
test
it.
I
had
well.
As
I
said
I
worked
for
for
exoskill
and
one
of
the
new
colleagues
who
joined
straight
out
of
the
fr
technicum
needed
about
based
on
the
instructions
needed
about
two
days
to
automate
this
using
terraform.
B
So
I
don't
think
that
it's
an
impossible
thing
to
build
the
hard
part
about
prometheus
becomes
when
you
want
to
put
it
into
production
because
you
actually
need
to
make
it
redundant.
You
need
to
make
sure
that
the
databases
is
backed
up
properly
and
then
you
can
restore
it.
I
did
have
a
single
prometheus
node
setup,
which
was
continuously
backed
up
to
an
object,
storage
and
then
the
terraform
script
when
it
reprovisioned
the
server
it
just
downloaded
the
data
from
the
object.
B
Storage,
that's
also
a
possibility,
but
it
does
require
a
significant
amount
of
of
engineering.
So
that's
actually
one
of
my
main
misgivings
with
with
the
direction
that
we
are
currently
heading
into
in
terms
of
cloud
native
that
there
are
too
many
tools.
There
are,
of
course,
there's
there
are
these
best
practices
that,
yes,
you
should
have
everything
redundant.
You
should
have
kubernetes
and
and
and
whatever
else
tool
set
on
top
of
it.
But
this
all
requires
orchestration.
B
You
can't
just
go
there
and
then
click
clicky,
install
kubernetes
and
then
expect
everything
to
work
because
nine
months
into
kubernetes
you
will
face
a
version
change
and
then
you
need
to
change
your
manifests
and
and
even
if
it's
a
if
it's
a
managed
service.
So
that's
why
I
sometimes
would
argue
that
that
in
some
cases
it
is
better
to
to
stick
with
what
you
know
and
if
you
really
need
auto
scaling
and
I'm
seriously
saying
need
here
then
look
into
these
things.
B
So
it's
not
something
that
you
would
install
for
four
or
five
machines,
because
the
amount
of
effort
you
have
to
put
in
to
make
something
like
this
happen
outweighs
the
benefits
that
you
have
from
auto
scaling.
And
you
know
things
can
break
so
you
you
have
to.
You
have
to
realize
that
that
is
not.
This
is
not
something
that
you
can
just
leave
unattended
and
be
happy
with
it.
B
The
main
reason,
the
main
benefit
of
building
such
a
project
is
that
you
really
really
get
to
understand
how
prometheus
works.
The
service
discovery,
how
you
can
drive,
auto
scaling,
building
web
books,
etc.
So
I
think
this
is
a
good
learning
project
and,
if
you're
really
hosting,
I
don't
know
two
three
racks
of
servers,
then
it
might
also
be
worth
considering,
potentially
starting
and
shutting
down
servers
based
on
some
sort
of
a
scaling
metric
to
conserve
power,
because
that's
when
it
starts
to
be
economically
viable.
A
C
A
A
B
Set
it
depends
it.
It
depends
because
you
can
get
a
vm,
so
you
can
get
a
vm
for
with
half
a
gig
of
ram
or
gig
of
ram
and
those
are
fairly
cheap,
so
they
cost
five
to
ten
euros
a
month.
That's
not
really
expensive,
but,
yes,
preserving
state
is
hard
and
when
you
go
for
and
again
you
could
do
auto
scaling
of
the
vms
under
a
kubernetes
cluster
as
well.
That's
what
the
cluster
auto
scaler
does
too.
B
They
provide
block
storage,
digitalocean
provides
block
storage,
everyone
else
is
is
kind
of
in
the
in
the
beginning,
and
this
is
because
block
storage
is
hard,
so
the
block
storage
part
is
is,
is
basically
the
the
disk
that
is
not
attached
to
the
physical
hardware
under
it,
and
building
such
a
system
is
is
difficult
because,
for
example,
digital
ocean
uses
ceph,
I
believe,
and
they
did
have
some
issues,
capacity,
issues
and-
and
I
believe
also
some
some
other
issues
with
it.
It's
hard
because
it's
very
hard
to
predict
the
the
actual
load.
B
That's
hitting
your
your
storage
servers
that
are
they're,
providing
this
service.
So
no
matter
what
you
do,
storage
and
persistence
is
hard
if
you
use
it
from
your
cloud
provider
and
it
of
course
becomes
easier
unless
you
run
into
issues
like
we
did
two
years
ago
when
you're
running
out
of
bandwidth.
B
So
so
when
you're
starting
to
hit
certain
invisible
bandwidth
limits
and-
and
things
like
that,
and
that's
why
microservices
are
better
kept
small
because
then
you
can
launch
more
servers
instead
of
using
bigger
servers
with
with
bigger
amounts
of
traffic,
but
these
are
all
hard
problems
to
solve.
So
that's
why
my
previous
question
was
also
about
about
at
what
point
does
it
make
sense
to
go
for
microservices?
Because,
yes,
in
the
beginning,
it
might
look
fancy
it
might
work.
B
But
when
you
actually,
you
know
the
the
application
when
it
survives
day
one
then
suddenly
the
the
real
scaling
problems
start
to
hit
there
and
those
are
the
ones
that
that
you
didn't
think
of,
because
the
scaling
problems
you
think
of
are
not
problems.
The
ones
that
you
didn't
think
of
are
the
ones
that
are
going
to
come
and
haunt
you
and
you
might
need
to
re-architect
your
whole
application
anyway.
So
I
don't
think
that
this
is
something
that
you
would
do
on
day.
One
yeah.
A
D
B
A
A
B
C
B
A
B
The
defaults
are,
I
think,
to
five
minutes
is
above
0.8
oopsies,
so
the
the
default
is
to
for
five
minutes,
and-
and
I
just
scale
that
down
to
to
one
minute-
and
this
is
of
course
automated-
the
way
that
works
is
you
can
set
up.
Basically,
what
this
does
this
whole
dashboard
here
is:
when
I
click
save
it
doesn't.
Let
me
save
because
it's
prevision,
but
this
is
a
json
file,
and
if
this
json
file
you
can
provide
the
grafana
on
startup
and
in
this
json
file.
B
C
C
B
Yeah,
so
so
again
that
depends
on
on
my
general
recommendation.
In
terms
of
of
writing
an
application
and
and
doing
this
is,
is
think
of
what
kind
of
teams
you
have.
If
you
have,
one
team
then
focus
on
the
business
on
the
business
perspective,
so
don't
start
orchestrating
your
own
auto
scaling
unless
you
really
really
really
need
to,
because
you
want
to
focus
on
on
the
thing.
B
And
then
you
know
the
next
person
who
is
going
to
take
that
job
over
from
you
when
you
start
hating,
it
is,
is
not
going
to
hate
your
guts.
It's
just
going
to
be
a
tedium,
but
it's
also.
It
makes
life
so
much
easier,
but
it
requires
an
upfront
investment
in
in
building
things
and
also
requires
that
your
applications
are
actually,
as
jurgen
said,
stateless.