►
From YouTube: How We Built a Cloud On K3s: The Learnings Of Growing Up Fast - Alex Jones & Anaïs Urlichs, Civo
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How We Built a Cloud On K3s: The Learnings Of Growing Up Fast - Alex Jones & Anaïs Urlichs, Civo
K3s has radically altered the way we think about interacting with Kubernetes by reducing operational knowledge to an API. This promotes a highly compelling experience through its simplicity and speed. At Civo, we have spent the past two years building an ethical cloud computing company around this technology in collaboration with the open-source community. But the journey has been hard... Running hundreds of micro-clusters has had a steep learning curve. Observability, production support and reliability have been dramatically scaled and tested as services have become popular. We have found nuances in K3s, changes within the hardware requirements and had to rebuild server racks. This talk will illustrate our journey of making clusters resilient within a multi-tenant environment and how company culture had to scale with the tech and community to support it.
A
A
I
worked
for
several
years
as
developer
advocate
first
in
the
blockchain
space
and
then
transitioned
into
devops.
I
also
started
a
challenge
called
100
days
of
kubernetes,
where
we
aim
to
learn
something
new
related
to
cubanitas
across
100
days.
If
you
are
curious,
you
can
find
more
resources
on
my
twitter
linked
on
the
slide.
B
Hi
there
I'm
alex
jones,
I'm
the
principal
engineer
here
at
sivo,
I'm
also
a
technical
advisory
group
technical
lead
for
app
delivery
within
the
cncf.
I've
worked.
Companies
such
as
microsoft,
beat
sky
b,
jp
morgan,
american
express
and
many
more,
and
you
can
always
reach
me
on
alex
jones
ax
at
twitter.
If
you
want
to
chat.
A
So
what
is
sievo?
What
are
we
actually
basing
our
experience
that
we
share
with
you
here
in
this
talk
on
xero
is
our
managed
kubernetes
provider,
meaning
you
can
spin
up
kubernetes
clusters
in
90
seconds
or
less?
We
are
highly
community
focused
and
community,
driven
meaning
a
lot
of
our
offering
is
based
on
community
input.
A
A
A
We
will
dive
into
more
detail
into
what
k3s
actually
is
in
a
second
and
lastly,
there
is
a
need
for
a
cloud-native
kubernetes
service
provider.
Sivo
is
cloud
native.
First,
a
lot
of
the
cncf
projects
that
you
will
hear
about
throughout
this
kubecon
are
actually
integrated
with
you
can
spin
them
up
when
you
spin
up
a
kubernetes
cluster
alongside
it.
So
you.
A
So
what
is
k3s
k-3s
is
a
lightweight
cubonetus
distribution
on
upstream
kubernetes.
You
have
a
lot
of
components
added
to
it
that
are
provider
specific.
Now
those
have
been
removed
from
k3s.
You
only
need
about
512
max
of
ram
to
run
on
a
server
meaning.
You
can
run
k3s
on
a
raspberry,
pi
and
other
small
devices,
it's
extremely
lightweight
and
it
comes
packed
as
a
single
binary.
So
you
can
install
it
like
that,
and
the
best
part
is
that
it
comes
with
core
technologies
such
as
cni
and
csi,
and
so
on.
A
A
So
now
that
we've
learned
about
the
value
and
benefits
of
k3s,
let's
connect
it
back
to
the
infrastructure
we
have
at
sivo.
You
can
see
here
on
this
image,
one
of
our
super
clusters
with
the
compute
nodes.
Within
now
our
superclasses,
our
infrastructure
is
open,
compute
compliant
on
the
next
slide.
You
can
see
a
diagram,
an
illustration
of
those
set
super
clusters
we
have
in
each
supercluster,
32
compute
nodes
of
which
four
are
the
control
plane.
A
B
So
let's
have
a
look
at
how
the
tenant
k3's
clusters
actually
look
within
our
supercluster
at
sivo.
So,
first
and
foremost,
we
have
the
k3s
process,
but
we
need
to
run
that
in
a
safe
way.
We
chose
to
use
libert
and
that
with
cube,
vert
allows
us
to
run
a
virtual
machine
within
a
pod.
This
virtual
machine
as
a
pod
is
set
up
with
several
others
to
form
a
k2's
cluster.
B
So
how
does
that
actually
work?
Well,
fundamentally,
when
you
start
up
a
virtual
machine,
it
gets
loaded
from
a
base
image
and
then
libvert
on
the
host.
Supercluster
node
will
create
a
vm
that
is
then
attached
through
a
virtual
machine
instance.
This
is
extremely
powerful
because,
with
hot
plug
technology,
we
can
attach
pvcs
and
allow
storage
to
be
provided
to
that
k-10
cluster.
A
B
Let's
talk
about
networking
for
10
k3
clusters,
as
you
can
see
from
the
diagram,
we
have
a
namespace
for
tenants
and
in
that
main
space
they
can
have
multiple
clusters.
However,
what
they
will
have
is
a
single
router
pod.
This
router
pod
is
an
l3
network
bridge
and
on
that
router
pod
it
will
have
two
nicks.
One
of
those
nicks
will
be
on
the
private
tenant
network
and
one
will
be
on
the
public
network
and
we
have
the
ability
to
control
ingress
and
egress
traffic
with
firewall
rules
and
ip
table
exclusions.
B
This
is
an
exciting
proposition
because
it
keeps
things
simple,
and
on
top
of
that,
because
of
the
lightweight
footprint
of
k3s,
we
can
pack
more
k3s
classes
in
per
physical
compute
node.
That
combined
with
the
fact
that
every
single
node
pool
from
a
tenant
cluster
is
distributed
across.
Our
digital
infrastructure
means
that
reliability
and
resilience
is
built
in.
B
So,
having
described
the
technologies
that
sivo
has
built
and
some
of
the
challenges
around
our
virtualization
and
creating
tenant
clusters
within
our
supercluster,
it's
important
to
reflect
on
what
we
learned
in
our
beta
period.
Once
our
mvp
was
pushed
out
the
door,
it's
been
really
really
insightful
to
learn
about
some
of
the
community-led
feedback
and
also
some
of
the
things
that
we
discovered
along
the
way
that
we
didn't
know.
We
went
into
this.
B
Firstly,
deployment
maturity.
When
you
get
things
started,
it
was
a
mixture
of
a
bunch
of
different
scripts.
We
had
terraform
ansible,
we
had
distributed
deployment
processes,
some
of
which
were
manual,
and
we
had
very
very
little
control
over
how
fine-grained
we
could
be
with
our
deployments.
It
was
at
a
regional
level.
So
if
you
had
something
you
had
to
distribute
to
london,
one
the
entire
region
will
be
updated
through
terraform
or
through
ansible.
This
meant
also
that
it
was
really
constrictive
for
multiple
team
members
to
work
on
the
same
code
base.
B
B
Well,
we
moved
to
a
workspace-based
model.
We
moved
to
dry
configuration
with
terraform
and
terra
grunt,
and
the
idea
that
you
can
apply
micro
changes
that
every
single
region
is
comprised
of
modules
and
you
can
update
those
modules
independently
of
the
work
of
other
team
members.
This
also
was
in
in
conjunction
with
a
large
scale,
change
to
having
gated
pipelines,
so
we
could
deploy
through
our
test
regions
then
to
production
all
the
while
performing
linting
and
testing
along
that
process.
B
This
is
now
matured
to
the
point
where
we
have
automation
to
build
clusters
to
check
that
they
work
and
to
then
continue
on
that
journey
of
continuous
deployment.
The
next
part
was
observability
maturity.
When
we
went
into
this,
we
were
effectively
flying
blind.
We
had
some
raw
prometheus
configuration,
some
very
prometheus
based
alerts
into
the
sense
that
they
were
node,
exporter
alerts
and
kubernetes
default
api
alerts.
We
didn't
really
know
how
civo
stack
was
doing.
We
didn't
really
know
what
our
customers
were
doing.
We
couldn't
answer
basic
questions.
Like
is
a
customer
cluster
working.
B
Are
the
tenant
nodes
working
or
are
they
facing
loads
of
problems
inside
of
their
clusters?
There
was
a
huge
manual
effort
in
having
to
jump
in
and
look
at
that
kubernetes
cluster.
However,
we've
now
moved
more
in
the
direction
of
having
three
different
signal.
Generators
and
collectors.
We've
moved
to
have
aggregated
signal
collection
and
the
ability
to
have
a
holistic
view
over
all
of
our
telemetry
across
our
regions.
B
This
is
in
the
shape
of
having
prometheus
with
distribution
into
that
it's
in
the
shape
of
having
loki
to
collect
our
logs
from
the
syslog
level,
all
the
way
up
to
the
tenant
cluster
level
and
about
having
jaeger
tracing
to
look
at
how
our
own
operators
perform
are
they
hanging?
Are
they
finding
that
there
are
particular
function,
calls
that
are
becoming
unoptimized
as
we're
building
out
this
observability
control
plane
we're
also
starting
to
mature
our
model
of
what
is
the
kind
of
data
that's
meaningful
to
us.
B
We've
mentioned
that
our
cluster
launch
times
are
an
extremely
important
part
of
our
proposition,
and
so
therefore,
we
have
a
big
dashboard
that
displays.
How
are
we
doing
in
performance
across
regions?
How
does
this
particular
container
d
or
storage
layer
change
impact,
our
ability
to
perform
and
our
ability
to
deliver.
B
That,
because
we've
had
this
distributed
approach,
where
we
have
now
federation
of
our
observability,
we
have
much
more
of
a
holistic
viewpoint.
The
most
recent
thing
that
we've
moved
towards
is
having
thanos
to
enable
us
to
see
how
our
clusters
performing
over
time
and
we're
now
starting
to
build
out
forecasting
telemetry
so
that
we
can
detect
problems
before
they
happen.
B
A
lot
of
those
behaviors
are
specific
to
the
individual
and
it's
very
difficult
to
then
transfer
that
up
to
a
new
team
member
in
an
effective
and
a
reproducible
way.
This
was
self-evident
in
the
fact
that
way
incidents
were
handled
earlier
on.
We
didn't
really
have
that
process
of
postmortems
and
learning
from
those
incidents
and
really
trying
to
embrace
the
sre
mindset
of
spending
50
of
our
time,
developing
solutionizing
and
improving
those
systems.
B
Company
makes
sense,
so
we've
started
moving
the
direction
of
having
more
rituals
around
okay.
Let's
look
at
the
backlog,
let's
decide
how
we're
distributing
work.
Let's
look
at
the
rituals
that
don't
make
sense,
such
as
arbitrary
things
in
the
board
like
picking
up
items
that
won't
improve
our
toil
and
let's
look
at
the
things
that
our
active
feedback
mechanisms
are
telling
us
that
early
warning
system
I
mentioned
earlier-
that's
been
a
good
indicator
that
we
have
some
toil
or
some
tech
debt
in
a
certain
area.
B
Is
there
a
particular
type
of
activity
that
customers
are
participating
in
that
we
can't
really
tell?
Are
we
looking
at
billing
on
the
customers
that
we
have
very
poor
granularity
on
what
sort
of
node
sizes
are
more
successful
and
we
should
be
able
to
also
facilitate
the
ability
for
our
development
teams
to
iterate
faster
and
faster,
as
they're
pushing
operators
out
by
version.
B
B
So
we've
tried
to
explain
to
you
in
this
talk
some
of
the
reasons.
We
really
think
that
there's
huge
potential
in
the
market
for
technologies
with
k3s
with
providers
like
sibo
that
are
community
driven,
we've
spoken
about
the
hardware
that
we
use
and
some
of
the
technical
challenges
that
we
had
to
solve
and
some
of
the
interesting
things
that
we
managed
to
accomplish.
On
top
of
that,
we
tried.
B
With
the
learnings
about
things
that
went
really
well
versus
the
things
that
were
a
little
bit
tougher
to
achieve
we're,
not
sure
what
the
future
holds,
but
we're
really
happy
that
we
picked
k3s
as
our
core
technology,
because
we
know
that
we
can
iterate
on
it
really
fast.
We
can
sidestep
a
lot
of
the
more
traditional
kubernetes
challenges
that
you
have
when
you're
trying
to
scale
it
up
really
quickly.
B
This
also
means
that
we
can
stick
to
our
promise
of
sub
90
second
launch
times
with
sub
60
seconds
becoming
tantalizingly
close
and
so
from
sivo's
perspective.
We're
going
to
see
a
lot
more
interaction
with
the
community
and
a
lot
more
excitement
in
terms
of
where
k-3s
is
going
as
an
independent
distribution
of
kubernetes.