►
From YouTube: Continuous Performance Benchmarking for Vitess - Alkin Tezuysal & Florent Poinsard & Manan Gupta
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
面向 Vitess 的持续业绩基准测试 | Continuous Performance Benchmarking for Vitess - Alkin Tezuysal & Florent Poinsard & Manan Gupta, Planetscale Inc.
Vitess 是围绕 MySQL 构建的可扩展分布式数据库系统。时下，业绩是一个相当重要的市场因素，我们得快速又准确的处理一切事宜。随着京东、Slack 和 GitHub 等大型用户的加入，Vitess 必须确保为其用户提供完美的性能。为满足这一需求，我们为 Vitess 创建了一个基准测试监控工具，即 AreWeFastYet。在本次演讲中，我们将首先介绍 Vitess，然后将讨论我们如何跟踪 Vitess 的性能，最后将概述我们创建的基准测试工具。
Vitess is a scalable distributed database system built around MySQL. Nowadays, performance is a considerable market factor, and things have to go fast and right. With large scale adopters such as JD.com, Slack and GitHub, Vitess has to ensure it’s delivering flawless performance to its users. In order to meet this need, we created AreWeFastYet, a benchmarking monitoring tool for Vitess. In this talk, we will first introduce Vitess and then discuss how we track the performance of Vitess and an overview of the benchmarking tools we created.
A
Welcome
to
continuous
performance
benchmarking
for
vitesse
maintainer
talk,
my
name
is
alkin
and
I'll,
be
your
host
today,
and
today's
agenda
I'll
be
doing
an
introduction
to
with
tess
and
manan
will
do
benchmarking
as
a
product.
Information
and
florent
will
give
you
benchmarking
on
internals
from
the
bts
project.
A
little
bit
about
the
tests
is.
It
is
a
database
clustering
system
for
the
horizontal
scaling
of
mysql.
A
It
is
a
known
cncf
graduate
project,
an
open
source
licensed,
and
we
have
contributors
around
the
world
and
more
on
vtes
is
a
single
illusion
of
a
database,
actually
dedicated
connection
that
can
work
on
microscope,
5.7
or
800,
and
it
is
compatible
with
the
frameworks
like
an
rms
common
frameworks
and
rams,
and
it
is
known
to
be.
Scalable
also
provides
high
availability
and
durability.
Guarantees
of
the
database
very
highly
adopted
and
serves
millions
of
qps
around
the
world
you
might
be
using
day-to-day,
and
mostly
a
very
large
implementations
are,
are
there
to
be
known?
A
Also,
other
implementations
are
in
the
market
right
now.
So,
let's
go
into
the
concepts
of
office
and
with
this
because
it's
a
sharded
system
has
a
key
space
concept
for
a
logical
database
and
provides
a
shard
as
a
logical
database,
and
it
comes
with
a
with
a
concept
called
cell
for
a
failure.
Domain
with
this
architecture
actually
serves
with
a
primary
and
replicas.
It
uses
the
mysql
primary
and
replica
concept
and
mainly
comes
with
a
vt
tablet,
and
a
vt
tablet
is
a
is
a
as
a
side
car
to
a
mysql
d
process.
A
It
usually
sits
next
to
the
mysql
d
and
and
drives
the
test
process
against
the
mysql.
Here's
an
example
of
multiple
clusters
and
you
can
have
be
running
under
with
tests
for
the
large
implementations.
So
we
need
a
concept
like
vtgate.
Vtgate
is
a
stickless
proxy
speaks
to
mysql
protocol
and
impersonates
monolithic,
mysql
server.
This
allows
applications
to
speak
to
vt
tablets
from
the
test
standpoint.
A
How
does
the
application
access
the
sharded
system,
so
in
this
example,
you
will
see
vtgate
routes,
the
traffic
to
the
sharded
clusters?
In
our
example,
we
have
two
shards
in
commerce
database
and
one
internal
chart
on
an
uncharted
space,
and
this
way
you
can
actually
have
an
access
to
the
target
environment
with
the
access
from
the
application.
A
So
in
our
example
over
here
we
have
a
commerce
database
and,
and
then
we
want
to
search
by
the
orders
in
customer
id
and
then
application
sends
the
query.
As
always,
it
does
to
a
regular
database
connection
to,
in
this
case
the
mysql,
but
the
vtgate
understand
and
knows
where,
where
the
customer
id
parts
will
fall
in
and
it
will
route
the
traffic
to
those
in
this
way
you
can
actually
scale
indefinitely.
A
And
also
another
component
from
the
test
architecture.
Important
component
in
the
test
architecture
is
the
topo
topo
is
a
is
a
state
of
where
the
the
mysql
and
database
components
are
as
well
as
the
test
components
sits.
So
there
are
multiple
implementations
of
toppo
that
you
can
choose,
favorite
ones
are
hcd,
zookeeper
and
kubernetes,
and,
and
the
known
console
is
also
a
topo
manager
that
was
used
in
the
past.
A
Vtc
dld
is
another
component
of
vtest,
it's
a
control
daemon
and
it
runs
the
ad
hoc
operations
that
serves
as
an
api
server
and
then
interacts
with
it
with
the
topo.
So
they
all
come
together
in
this
architecture
summary.
So
you
have
an
application.
Server
that
connects
the
load.
Balancer
load
balancer
actually
speaks
to
vtgate,
which
interacts
with
ctld
and
topo
server,
and
and
and
serves
the
incoming
queries
from
the
sharded
clusters
behind
the
scenes.
A
We
test
also
have
very
new
and
upcoming
features.
As
of
today,
we
have
a
12.0
release
in
nga
and
we
support
online
ddl
operations
without
locking
and
bts
comes
with
an
experimental
version
of
the
gen
4
planet,
which
is
a
very
new
and
highly
improved,
and
we
do
continuous
benchmarking,
which
is
the
subject
of
the
talk
today,
and
we
have
ongoing
performance
improvements
and
also
tools
like
vt
admin
and
are
coming
up
in
the
next
releases.
B
Thank
you
elgin.
I
am
vanu
gupta
and
I
will
be
taking
you
through
the
next
part
of
our
presentation.
I
am
going
to
introduce
re
fast
yet,
which
is
the
nightly
benchmarking
tool
for
the
test,
and
then
eventually,
I'm
going
to
show
you
the
website
of
refaster,
which
I
have
linked
over
here
as
well,
and
the
results
that
it
has
produced.
I've
also
linked
the
code
over
here.
Please
go
through
it
after
the
after
the
presentation,
so,
let's
dive
in
first
things.
First,
what
is
benchmarking?
B
Benchmarking
is
a
way
to
measure
and
compare
the
performance
of
one
software
version
against
another.
So
when
we're
building
a
test,
we
have
the
main
branch.
We
have
different
releases
of
data
as
release
12
release.
11.
We
have
different
patch
releases.
We
have
prs
which
are
changing
some
part
of
the
code
base
and
we
want
to
measure
the
performance
of
each
of
these
and
compare
that
which
one
is
better.
B
B
Largely
benchmarks
can
be
broken
into
two
parts:
micro
benchmarks
and
micro
benchmarks.
Let's
go
over
both
of
these,
so
micro
benchmarks,
like
the
name,
suggests
micro
benchmarks.
They
measure
a
small
part
of
the
code
base.
Usually
this
is
done
by
isolating
a
single
function,
call
and
you
doing
it
repeatedly.
What
we
mean
is:
let's
say
that
we
have
a
benchmark
and
we
run
it
for
two
seconds
within
those
two
seconds.
B
Now
we
have
a
wide
array
of
micro
benchmarks
in
vitesse
which
tests
the
planner,
the
parser
rpc
calls,
and
these
are
the
things
that
are
on
the
top
of
my
head,
among
others.
So
we
have
a
very
extensive
code
coverage
and
micro
benchmarks.
If
you
want
to
take
a
look,
you
can
go
over
the
witness
code
base
and
within
the
testing
files
of
go
local
unit,
testing
files
of
go
you'll
find
benchmark
tests,
which
is
what
a
micro
benchmarks
are.
The
macro
benchmarks
measure
the
performance
of
a
code
base
as
a
whole.
B
So
when
a
user
ex,
when
a
user
uses
the
test,
they're
not
going
to
be
able
to
see
what
the?
What
is
the
amount
of
time
that
the
parser
took
or
what
was
the
amount
of
time
that
the
query
was
planned
for
or
what
was
the
execution
type?
What
they
generally
see
is
just
the
latency
of
the
of
the
result
set
that
they
get
back
when
they
query
the
database,
and
this
is
what
we
measure
via
our
macro
benchmarks.
B
We're
not
going
to
go
over
how
we
could
bring
up
that
environment
or
how
this
is
all
done,
because
that
is
something
that
flora
will
cover
in
the
next
part
of
the
code
base
when
he
dives
into
how
re
fast
it
is
built
and
the
way
that
we
bring
up
the
infrastructure
and
how
to
run
the
macro
and
micro
benchmarks,
the
micro
benchmarks
that
we
run.
We
have
two
categories
for
them:
otp
and
tpcc.
Again,
we
won't
go
into
much
detail.
Florian
will
cover
that
later
now,
the.
C
B
B
Taking
that
a
step
further,
we
also
have
slack
integration
so
rv
fast,
yet
once
it
has
run
the
micro
benchmarks
and
the
macro
benchmarks
and
stored
those
results,
we
use
those
results
and
compare
them
against
each
other.
For
example,
we'll
take
the
main
branch
and
we
compare
it
against
the
results
of
the
previous
day
run
of
the
main
branch
same
we'll.
Do
it
for
the
main
branch
against
the
release
and
we'll
do
it
for
prs,
also
against
the
base
of
the
head
of
the
pr.
Once
we
have
these
comparisons.
B
If
there
is
any
regression,
then
we
send
a
slack
notification
on
the
benchmarking
testing
channel
and
then
the
developers
can
look
at
this.
They
can
check
out
what
the
regression
was
when
which
benchmark
the
regression
came,
and
they
can
then
figure
out
what
was
the
change
that
made
the
regression
or
that
caused
the
regression
I'm
going
to
dive
and
share
a
little
bit
and
talk
about
the
general
planner.
So
what
is
the
general
planner?
Gen
4
is
a
new
btg
planner,
which
is
meant
to
succeed
b3.
B
So
this
is
something
that
we've
been
working
on
with
test
lately
and
currently
it
is
an
experimental
feature,
but
eventually
we
are
going
to
make
it
the
default
and
gen4
is
a
much
more
advanced
planner
than
v3.
It
provides
a
larger
query:
support
a
bunch
of
coordinate
sub
queries
will
start
working
with
gen4.
B
But
how
can
we
say
for
certainty
that
the
optimized
plans
are
faster
or
the
plans
that
jennifer
produces
are
actually
better?
How
do
we
gain
confidence
that
gen
4
is
indeed
a
much
as
a
successful
vpn
and
we
can
replace
the
default
on
the
webtest
website.
The
way
that
we
do,
that
is
via
the
benchmarks.
So
we
run
the
benchmarks,
the
macro
benchmarks
for
both
v3
and
gen4,
and
then
we
compare
the
results.
B
We
look
at
the
queries
that
they've
sold
per
second
the
amount
of
user
memory
usages
that
they
have
on
pd
tablet
the
amount
of
cpu
users
that
they
have
on
mitigates
and
all
these
metrics
and
then
we
take
a
look
at,
does
gen4
actually
improve
the
latency
that
the
user
is
going
to
see
enough
talk.
We've
talked
about
what
benchmarks
are
the
different
type
of
benchmarks,
how
gen,
4
and
j4
is
and
how
we're
supposed
to
use
my
benchmarks
for
gen4.
I
think
it's
time
that
we
look
at
the
website.
B
Are
we
fast
yet,
and
we
ought
to
talk
about
what
the
users
are
going
to
gain
from
it,
we've
already
divulged
into
what
the
developers
gained
from
it,
how
pr's
run
against
that
and
it
exp
it
improves
the
developer
experience
it's
time
to
take
a
look
at
what
the
user
end
users
can
gain
from
rb
fast
here.
So
this
is
the
main
home
page
for
rv
fast
here,
and
here
we
have
a
brief
overview
of
what
rb5
kind
of
what
like.
B
I
gave
you,
and
here
we're
also
looking
at
the
server
that
we're
using
actually
to
run
the
macro
benchmarks.
It's
an
equinex,
m2
x,
large
x86
server,
and
for
the
most
more
enthusiasts
of
you
guys,
you
can
actually
take
a
look
at
the
server
specifications
and
see
what
what
hardware
we're
actually
using
to
run
our
macro
benchmarks
and
micro
benchmarks
more
on
this
later,
the
next
page
that
I'm
going
to
show.
You
is
the
front
page
of
of
this
page.
B
So
we
talked
about
prawns
a
little
while
ago,
wherein
we
track
the
main
branch
and
we
run
the
macro
and
micro
benchmarks
for
it
every
day.
If
the
kit
has
changes
and
those
results
for
the
previous
month
are
actually
shown
here
for
the
past
30
days.
So
this
tracks,
the
oltp,
runs
the
transactions
per
second,
the
qps,
the
latency.
All
of
this
are
shown
over
here
over
the
past
month
and
the
same
we
do
for
tpcc
as
well.
B
So
the
next
page
that
I'm
going
to
show
you
is
the
search
page
over
here.
You
can
actually
give
a
very
specific
github
hash
that
you're
looking
for,
so
you
can
take
a
github
hash
from
github
and
look
for
the
for
for
results
for
that
specific
hash.
If
you
have
those
results
for
it,
then
we're
going
to
display
them
over
here
and
you
can
look
at
the
micro
benchmark
results
over
ltp
runs.
B
We
also
track
the
world
cpu
time
individually,
for
what
the
cpu
time
for
vtg
gate
was
really
tablet
was
the
amount
of
alloy
bytes
it
was
it
allocated
on
vtgate.
The
amount
of
bytes
are
allocated
on
the
tablet.
All
of
this
information
is
available
over
here
for
both
micro
and
macro
micro
benchmarks
of
ppc
oltp,
and
we
also
have
a
bunch
of
micro
benchmarks
over
here
which
show
their
runs.
B
The
number
of
iterations
all
the
information
that
you
need
now
I
can
even
go
over
in
one
of
these
micro
benchmarks
and
it
will
show
the
runs
of
the
past
five
days
of
past
few
iterations
of
that
micro
benchmark.
So
these
are
the
past
iterations
of
the
past.
Ten,
ten
or
so
runs
for
this
micro
benchmark
that
we're
looking
at
it
will
also
have
the
start
time
and
exactly
at
what
time
these
were
benchmarked,
the
github
references
for
them.
B
So
you
can
take
a
look
at
this
as
well
now
this
page
for
specifically
catering
to
one
github
hash.
What
if
you
wanted
to
compare
two
git
hashes
together
and
that's
what
this
page
does
so
the
compare
page
of
of
this
website?
You
can
actually
go
over
and
give
two
bit
of
ashes
and
you
will
have
their
comparisons
for
macro
and
micro
benchmarks,
along
with
some
nice
graphs,
which
will
show
you
graphically
of
what
the
differences
are.
So
this
is
actually
a
much
older
commit
and
aa79
is
actually
a
later
coverage.
B
So
this
later
commit
you'll
see
that
actually
there
has
been
some
performance
improvement,
we'll
talk
about
that
a
little
later
as
well,
the
amount
of
cpu
time
that
we
were
using
actually
decreased,
and
so
this
is
where
you
can
see.
The
comparison
between
two
two
commit
hashes,
along
with
this
is
the
comparison
between
their
micro
benchmarks
as
well,
so
like
all
the
information,
but
that
you
would
need
for
comparing
two
two
hashes
is
over
here.
B
so
over
here
in
drop
down
boxes,
you
can
just
select
the
version
that
you're
on
and
select
the
version
that
you
want
to
get
to
or
what
you
want
to
compare
it
against
and
just
go
over
here.
I'm
currently
showing
results
for
the
comparison
between
10
and
version
11,
and
here
you'll,
see
the
micro
macro
benchmark
or
micro
benchmark
results
for
all
of
them.
So
this
interfaces
are
actually
very
similar
that
we
have
for
the
macro
benchmarks
as
well,
where
you
have
the
results
for
two
tags
that
you
can
compare.
B
You
don't
specifically
need
to
search
for
the
github
hash
and
use
those.
You
can
directly
compare
different
tags
here.
I'm
comparing
release
10
versus
release
11.,
and
if
you
look
here,
there
is
actually
a
five
percent
increase
between
between
the
qps
that
vs
11
was
able
to
serve
against
east
end
for
the
oltp
workload.
B
B
Finally,
we've
created
one
special
tab:
the
one
that
I
talked
about
that
gen
4
is
a
new
planet.
We
also
need
to
benchmark
what
that
new
planner
is
doing,
so
we
have
a
specific
page
for
it,
which
is
called
v3
versus
gen4,
and
this
page
shows
the
results
of
the
comparisons
between
v3
and
z4.
Here
as
well,
you
can
select
which
tag
you
want
to
check
the
comparison
for
you
want
to
check
it
on
main
on
some
other
release.
B
For
example,
let's
say
that
you
are
on
release
12
and
you
want
to
look
at
what
the
what
was
the
comparison
between
v3
and
gen4
for
release
12..
You
can
take
a
look
at
that
as
well,
so
here
you
can
see
that
gen4
is
slightly
better
than
v3,
but
most
of
its
capability,
as
of
now
in
gen
4
is
actually
coming
from
the
fact
that
it
can
support
a
lot
more
queries
than
v3.
B
So
here
you
can
compare
the
results
if
you
go
over
to,
if
you
click
here
to
see
actually
see
the
query
plans,
this
is
kind
of
a
beta
feature.
It's
not
entirely
developed
kind
of
in
production
work
in
development,
but
over
here
we
actually
have
all
the
queries.
You
can
look
at
the
queries
that
we're
running
in
the
oltp
workload.
These
are
generally
point
queries
and
you
can
look
at
the
plan
that
we
get
and
we
that
that
v3
and
gen4
fallback
built
for
it.
B
So
this
is
the
query
that
you
got
select
c,
it's
a
normalized
query.
So
all
input
arguments
have
been
converted
to
bind
variables
and
then
you
can
actually
look
at
what
vtgate
is
going
to
do
with
it.
It's
going
to
actually
send
it
to
a
single
chart,
because
it's
a
select
equal,
unique
query,
and
this
is
also
a
place
where
you
can
look
at
the
plants
that
they're
going
to
that.
B
This
is
pretty
much
all
that
is
there
on
the
website,
but
this
allows
the
users
to
actually
know
beforehand,
even
before
they
start
testing
their
own
local
environment,
what
differences
they
can
expect
or
what
performance
improvement
they
can
expect
whether
the
mini
tablet
will
actually
start
using
more
cpu
time.
Will
it
start
allocating
more
bytes
is?
Is
the
gen
4
planner
actually
worth
it?
Will
it
give
you
more?
Will
it
give
you
better
performance?
All
of
that
can
be
answered
and
with
high
confidence
because
of
the
benchmark
tool
that
we
have
in
rb
fast.
B
Yet
before
I
pass
the
bit
on
to
fluoron
there's
just
one
more
thing
I
would
like
to
add.
J4
right
now
is
an
active
development
and
we're
looking
for
ways
to
improve
its
performance.
So
if
any
of
you
guys
are
running
with
us
in
production,
we
we
would
love
if
you
would
be
able
to
share
your
production
queries
with
us
and
we
we
then
optimize
gen4
to
actually
work
as
best
as
possible,
like
produce
as
optimal
plans
as
possible
for
your
specific
workload.
B
C
A
C
First,
with
the
implementation
that
we
use
for
our
fast
yet
so
we
have
two
permanent
server.
The
first
one
is
the
website.
Its
goal
is
to
serve
the
web
ui.
That
manager
showed
you
earlier
and
its
secondary
goal
is
to
handle
and
manage
the
whole
execution
of
benchmark.
When
I
say
the
whole
execution
of
benchmark,
I
mean
the
the
chrome
schedules
and-
and
that's
it
just
the
current
schedules.
C
Secondly,
we
have
the
matrix
server.
This
one
is
scrapping
the
benchmark
results
and
storing
the
data.
Another
key
point
of
our
implementation
is
the
fact
that
we
spawn
a
new
server
for
each
benchmark
that
we
have.
This
allows
us
to
have
more
reliable
results
and
make
sure
that
the
benchmark
is
not
influenced
by
anything
else
when
it
runs.
C
C
Let's
talk
about
the
execution
pipeline,
it's
a
very
important
part
of
of
the
implementation
and
rfs.
Yet
the
goal
of
it
is
to
manage
the
whole
existing
workflow
from
start
to
finish
so
we'll
get
to
what
the
whole
execution
workflow
means
in
a
bit.
The
pipeline
is
configurable
through
a
yaml
file,
meaning
that
we
can
have
tons
of
different
execution.
We
want
to
have
so,
for
example,
micro,
benchmark
and
macro
benchmark,
like
manon
just
explained
before,
and
those
two
different
types
of
benchmark
will
have
different
yaml
files,
so
the
whole
execution
workflow.
C
What
is
it?
What
are
the
responsibilities?
First,
the
creation
and
configuration
of
the
new
server?
This
is
the
very
basic
step
we
want
to
make
sure
that
we
create
a
server.
We
configure
it
based
on
a
very
detailed
and
explicit
configuration,
and
that's
it.
Another
point
is
another
responsibility:
is
the
setup
of
a
test
cluster
depending
on
the
type
of
benchmark
we
might
or
might
not
want
to
have
a
vts
cluster?
C
So
this
is
a
part
of
the
execution
pipeline
responsibilities.
Another
one
is
the
actual
execution
of
a
benchmark.
When
I
say
actual
execution
of
benchmark,
I
mean
running
the
benchmark,
then
we
need
to
store
the
results
and
the
metrics.
And
finally,
we
need
to
upload
and
publish
those
results
and
there's
kind
of
like
a
intermediate
steps
between
those
two
responsibilities,
which
is
aggregating
the
results,
compiling
them
and
checking
if
there
is
any
regression
or
not.
This
is
a
whole
part
like
everything
is
part
of
the
execution
pipeline.
C
C
So
we
feed
the
configuration
and
then
second
step
is
the
provision
we
create
the
execution
server
using
terraform.
So
this
is
just
to
provision
and
and
configure
the
very
basic
steps,
very
basic
things
about
the
server,
and
this
is
provisioning
on
equinix
metal.
Like
I
just
said,
third
step
is
the
configuration
of
the
server.
When
I
say
configuration,
I
mean
the
installation
of
packets
installation
of
of
tools
that
we
need
setting
up
the
network
setting
up
the
disk.
Anything
this
is
done
using
ansible.
C
C
The
fifth
step
is
actually
storing
the
results,
so
this
can
be
done
either
at
the
end
of
the
benchmark
or
during
the
benchmark
like
throughout
the
whole
execution
process
of
a
benchmark.
We
store
the
results
in
different
locations.
Most
basic
one
is
my
sequel.
This
is
gonna,
be
like
very
basic
information,
for
example
the
type
the
type
of
benchmark.
C
I
don't
know
the
idea
of
the
benchmark,
everything
like
that,
like
the
sha,
for
example,
the
git
shop,
then
we
store
information
on
prometheus,
then
the
information
that
we
store
from
it
is
obviously
time
series
data,
and
this
is
all
the
metrics.
For
example,
the
cpu
usage
that
we
that
we
had
for
a
specific
benchmark
at
a
specific
time,
we
store
the
matrix
that
we
have
in
the
front
db
in
information.
Sorry,
we
the
data
that
we
have
there.
We
store
it
in
an
influx
db.
C
Server
later
on,
this
is
done
to
keep
data
a
bit
longer,
keep
as
little
as
possible
inside
from
the
host
then
store
for
longer
term
inside
influx
db,
and
that's
it
sixth
step
is
actually
destroying
tearing
down
the
execution
server.
So
again,
this
is
done
using
terraform
seventh
step
is
the
publishing
like
it's
actually
publishing
the
results
so,
like
I
said
before,
right
before
publishing
the
results,
we
have
to
compile
them,
aggregating
them,
making
sure
there's
a
regression
or
not.
C
C
So
this
is
the
the
architecture
I'm
going
to
move
on
to
how
we
do
a
micro
benchmarks,
a
micro
benchmark.
So
this
is
out
of
the
two
types
of
benchmarks
that
we
have.
This
is
the
simplest
one.
Why,
because
is
coded
in
golden
and
go
has
an
amazing
testing
framework
which
include
a
micro
benchmark
tool.
C
So
you
know
when
you
write
code
and
go
you
can
in
specific,
when
you
write
test
and
go,
you
can
either
have
like
an
actual
unit
test
or
you
can
have
a
benchmark
test
and
we
have
a
bunch
of
those
inside
inside
the
code
of
your
test
and
the
goal
is
just
the
goal
of
the
micro
benchmark.
It
just
is
just
to
execute
all
of
those
benchmark
test.
We
get
an
output
every
time
that
we
execute
one.
C
To
them
during
the
demo
macro
benchmarks,
second
type
of
benchmarks,
that
type
has
a
longer
execution
time.
Why?
Because
it's
well.
First
of
all,
we
have
to
install
a
lot
more
packages,
for
example,
mysql,
etcd,
etc.
We
have
to
install
more
stuff.
We
have
to
set
up
more
stuff,
for
example,
the
vtest
cluster
needs
to
be
created.
It
has
to
be
up
running
so
yeah
longer
execution
time.
C
C
Another
important
part
of
a
micro
benchmark
is
actually
distinguishing
the
different
types
of
benchmark.
That
means
that
we're
going
to
get
a
lot
of
different
results
for
all
of
our
macro
benchmarks,
and
we
have
to
make
sure
that
oh
this
result
is
from
this
type
of
macro
benchmark
this
other
type
of
micro
benchmark.
C
So
you
know,
measuring
the
result
of
a
macro
benchmark
is
easily
done
using
sysbench,
but
it's
usually
not
enough.
We
have
to
get
more
information,
for
example
the
cpu
usage
of
the
host,
while
we
were
in
the
benchmark.
This
is
very
important
for,
like
we
want
to
know
in
the
20
minutes
that
the
benchmark
run,
how
much
like
how
long
the
cpu
was
used
for
this
is
important
to
compare
across
different
versions.
C
Like
I
say
we
set
up
a
vtest
cluster
for
the
micro
benchmarks
and
the
topology
that
we
use,
for
it
is
two
vt
gates,
six
tablets
and
the
topology
server
that
we
have
is
etcd.
This
is
the
configuration
that
we
use
so
far
and
we're
aiming
to
change
it.
We
want
to
make
some
tests
discover
which
topology
might
or
might
not
be
the
best
for
the
benchmarks
which
one
is
giving
us
more
reliable
and
more
accurate
result,
sis
bench
so
I
talked
about
it.
It,
like
I
say,
widely
known
tool.
C
C
C
C
Looking
almost
like
an
actual
like
a
like
an
actual
end-to-end
situation
and
final
step
is
the
execution.
It's
also
called
as
the
run
step
inside
sales
bench,
so
the
execution
is
where
we
actually
send
a
lot
of
queries
to
vtes
and
where
we
measure
the
performance,
we
have
two
custom
forks
of
cisbench,
one
of
which
is
the
first
link
here.
The
plan
skills
is
slash
sysbench.
This
fork
includes
a
tpch
benchmark
and
it
includes
a
different
way
of
formatting.
C
The
results,
sysbench,
usually
format,
the
results
using
text
just
plain
plain
text,
and
what
we
do
instead
is
just
display
them
using
json
and
second
fork
is
just
a
bunch
of
lua
files
for
tpc,
benchmark
tpc
and
tbch
benchmarks
are
two
big
benchmarks
using
the
database
word
they're
part
of
the
tpc
family.
If
you
know
a
bit,
databases
and
performance
of
databases,
you
might
know
them
here-
is
a
sample
of
a
seismic
result
like
we
can
like.
We
can
see
it's
pretty
simple,
pretty
pretty
straightforward.
C
We
just
have
the
time
that
the
benchmark
run
in
that
case
it's
10
seconds,
the
number
of
threads
that
we
used
the
number
of
transactions
per
second
tps,
and
then
we
have
a
block
with
qps
query
per
second.
So
we
have
the
total,
the
number
of
reads
the
number
of
writes
and
then
other,
which
is,
for
example,
begin
commit,
etc.
We
also
have
the
latency
and
the
number
of
errors
per
second,
that
we
had
and
same
thing
for
reconnects.
The
number.
B
C
Reconnection
that
we
had
but
like
we
can
see
it's
not
enough
for
us
to
tell
if
there's
a
regression
between
version
a
and
version
b
of
test.
It's
it's
already
good,
a
lot
of
data,
but
it's
not
enough.
So
we've
added
support
for
matrix.
So
inside
each
inside
each
execution
server,
we
start
a
prometheus
server,
which
will
scrap,
scrape
data
from
the
different
components
of
the
tests
and
the
host
different
components
being
the
vtgate
dvt
tablets,
vtc,
tld,
etc.
C
Those
are
all
the
different
types
of
metrics
that
we
don't
want
to
keep,
and
that
might
be
interesting
if
we
want,
if
it
we
see
a
regression,
so
an
important
part
of
the
matrix
is
obviously
getting
the
metrics
producing
the
metrics,
and
then
we
have
to
scrape
them
and
store
them
so
to
export
the
matrix
from
a
benchmark
server.
We
will
like
we
use
prometheus
so
to
gather
all
the
metrics.
C
This
is
inside
the
execution
server
and
then
because
the
execution
server
is
going
to
eventually
get
killed
like
get
get
destroyed
at
the
end
of
the
benchmark.
We
want
to
make
sure
that
the
matrix
we
collected
stays
for
a
long
time,
so
we
can
use
them.
We
can
analyze
them
in
the
long
run,
and
so
for
that
we
use
the
matrix
server,
which
is
just
we
just
create
and
store.
The
matrix,
like
I
said
before,
metrics
are
going
to
be
duplicated
to
an
influx
db.
C
C
I'm
now
going
to
move
on
to
a
demo
where
I'm
going
to
present
the
execution
of
the
micro
and
micro
and
macro
benchmark
and
and
show
you
a
bit
about
the
graphene
ui
and
the
matrix
all
right.
So
here
we
can
see
a
configuration
file.
This
is
a
yaml
file
and
we're
going
to
use
this
file
to
feed
the
execution
pipeline
and
configure
the
whole
benchmark.
C
Then
we
have
to
feed,
for
example,
the
commit
that
we
want
to
benchmark
the
the
type
the
type
like
the
name
of
the
benchmark,
and
then
we
have
the
database
where
we're
going
to
store
all
the
results.
The
different
time
series
databases
that
we
use
this
one
is
prometheus.
This
one
is
influx
db
and
then
a
little
bit
down,
we
have
all
the
macro
benchmark
configuration.
C
So
all
this
part
here
is
what
we're
going
to
give
to
sysbench
to
configure
it.
So,
for
example,
here
we
want
to
have
50
tables
with
a
certain
size
we
want
to
have.
We
want
to
use
a
hundred
a
hundred
threads,
and
then
we
configure
the
different
steps
of
cisbench
so,
like
I
said
before,
we
have
the
previous
step,
the
warm-up
step
and
then
the
run
step,
all
of
which
are
configured
differently.
C
So
here
we
can
see
that
we
have
six
getaways,
so
six
different
videos,
and
here
we
have
the
same
thing
for
the
tablets,
and
we
can
see
that
we
want
to
define
two
different
vt
tablets
all
right.
So
now
I'm
going
to
use
that
configuration
file
to
actually
start
a
benchmark,
so
I'm
just
going
to
use
the
command
line
here
manually
and
fit
it
feed
the
the
configuration
file
that
we
have.
C
C
C
C
And
here
we
can
see
the
transaction
per
second,
the
latency,
the
number
of
errors,
the
time
it
took
for
the
benchmark,
the
number
of
qps,
the
number
of
reads
per
second
and
rights.
So
all
of
these
results
are
displayed
in
the
website.
That
management
showed
you,
and
now
I'm
going
to
show
you
on
grafana
what
it
looks
like
so
on
grafana
we
can
visualize
all
the
different
metrics
that
we
collected
during
the
benchmark.
C
As
we
can
see
here,
we
have
all
the
host
metrics,
so
the
cpu
usage
memory
network,
etc.
We
can
also
take
a
look
at
the
different
queries,
like
the
queries
in
my
sequel
or
v-test
related
matrix,
for
example,
qps.
The
number
of
such
like
the
success
rate,
the
time
it
takes
for
each
queries,
and
we
can
also
see
like
a
more
general
overview
of
of
the
might
of
the
vtest
cluster
that
we
had
for
the
benchmark.