►
Description
OpenShift Commons Gathering @ Kubecon/NA San Diego November 18 2019
A
Who's
ready
for
a
week
of
arguing
how
to
pronounce
cube
CTL
I've,
pronounced
it
the
correct
way,
the
one
correct
way
so
y'all
can
head
home
early,
I'm,
Jeremy,
eater,
I
work,
for
you
may
know
me
from
my
past
work
in
the
community,
but
I'm
working
in
Red
Hat
service
delivery.
Now,
which
is
the
team
that's
responsible
for
running
open
shift
as
a
service
today
I'm
going
to
talk
to
you
about
how
we
do
that.
A
Maybe
it
comes
as
a
surprise
that
redhead
has
sre
folks
I
traded,
my
performance
engineering,
shirt
for
site,
reliability,
engineering,
shirt
and
everything
else
remain
the
same
kind
of
we
have
a
team,
and
so
that's
the
first
did
you
know
by
the
way,
who's
old
enough
to
remember
when
ESPN
used
to
have
at
the
end
of
every
SportsCenter
used
to
have
it?
Did
you
know
moment?
Remember,
did
you
know
so?
Did
you
know
that's
the
shtick?
We
got
fully
managed
OpenShift.
A
You
can
choose
right
now,
whether
whether
you
want
sorry
but
the
head
of
lapel
mic,
you
can
choose
whether
you
want
to
run
that
on
AWS,
which
is
called
OpenShift
dedicated,
and
then
we
also
have
an
offering
which
we
run
with
Microsoft
called
a
sure,
Red
Hat,
OpenShift,
who's
heard
of
arrow
yep
and
then
there's
another
flavor
called
OpenShift
dedicated
and
of
course
you
can
manage
it
yourself,
of
course.
So
another
did.
You
know.
A
Openshift
too
then
running
openshift,
three
and
now
running
openshift,
four,
so
there's
a
fair
amount
of
history,
even
though
the
skills
of
sari
OpenShift
has
changed
fundamentally
several
times
as
you're
aware
next
did
you
know
right
now
the
only
place
to
get
open
shift4
in
a
managed
way,
so
who
here
has
any
software
as
a
service
as
part
of
their
portfolio
that
they
run
I
want
to
say:
that's,
that's
over
50%
of
the
group,
so
that's
good
good
to
know
only
place.
Did
you
open
shift
for
right
now?
It's
open
shift
opens.
A
You
have
dedicated
our
teams
not
only
responsible
for
SRA
customer
clusters,
but
we
also
run
some
of
the
some
of
the
fundamental
services
that
underpin
a
lot
of
the
value
that
you
saw.
Derek
and
Clayton
talk
about
this
morning
in
particular
tool
ammeter,
which
backhauls
data
to
us
for
analysis
and
proactive,
proactive,
troubleshooting
and
whatnot.
So
we
run
those.
B
A
Seen
cloud
dot
right,
calm,
a
couple
people
20,
30
%
I'll-
show
it
in
just
a
second.
If
you
haven't
been
there
and
then
maybe
more
importantly,
and
what
I
think
is
kind
of
one
of
the
coolest
things
that
we're
doing
right
now
is
we're
allowing
custom
or
building
a
way
for
customers
to
interact
with
Red
Hat
via
API,
so
show
you
a
little
command-line
utility
that
we
have
to
stand
up,
openshift
clusters
that
are
managed
by
us
and
we
have
a
go
SDK
as
well
for
it.
A
So
what
does
it
take
to
run
OpenShift
as
a
service?
First
of
all,
we
are
foundationally
bought
into
the
fact
that
operators
and
the
design
of
core
OS
Red
Hat
core
OS,
allow
us
to
scale
openshift
as
a
managed
managed
services
business.
So
the
clusters
are
self-driving
self-healing
for
the
most
part.
So
that's
a
very
fundamental
kind
of
bedrock
change.
What
does
it
take
to
actually
run
openshift
for
the
platform
itself
is
pretty.
It
is
pretty
snazzy
day
to
operations
in
operators.
Our
team
has
been
a
bought
into
that
I.
A
We
have
to
harden
our
platform
to
handle
a
cloud
providers
bad
day,
so
API
driven
OpenShift
cluster
management
I'll
show
you
the
guts
of
this
in
just
a
second,
but
here's
how
its
strung
together
currently
and
if
you've
heard,
of
get
ops
based
cluster
management.
You
may
have
heard
of
some
of
that
this
morning
our
team's
been
doing
get
ops
for
many
many
years
now,
and
bringing
that
to
open
shift
would
certainly
lighten
our
load
quite
a
bit.
A
So
anyway,
here's
what
it
looks
like
architectural,
a
cloud
entrada
comm,
slash,
open
shift
is
a
set
of
micro
services.
One
one
handles
the
cluster
is
called
cluster
service
that
guy
handles
standing
up
the
cluster,
the
second
micro
service
AMS
just
handles
your
subscription
stuff,
not
too
glamorous
but
highly
important,
and
what
the
cluster
service
does
is.
It
renders
a
what's
called
a
cluster
deployment,
which
is
a
custom
resource
that
we've
invented
in
a
in
a
set
of
controllers,
called
hive
and
hive
us
how
we
spin
up
how
we
spin
up
open
shift
clusters.
A
Hives
job
is
to
talk
to
the
cloud
provider
API
and
ultimately
run
the
open
shift
installer.
It
can
install
any
version
of
open
shift
four,
so
four,
not
whatever
from
there.
We
have
a
cluster
and
we
can
lay
down.
So
we
don't
do
anything
too
OpenShift
itself
in
the
front
end.
Everything
we
do
is
you
know
principally
OpenShift
wants
you
to
do
everything
opens
your
four
wants.
You
did
do
everything
as
a
date
to
operation
and
that's
what
those
operators
come
in.
So
we
have
a
way
to
create
that
cluster.
A
Wait
till
it's
done
and
then
lay
down
a
bunch
of
operators,
and
then
we
have
our
open
shift,
dedicated
or
managed
open
chef
product.
That's
how
that
works.
We
good
so
far.
I'll
show
you
a
cluster
deployment.
Yama
file
in
just
a
sec
that'll,
hopefully
help
clear
things
up
here,
are
some
of
the
operators
that
that
we've,
the
product
team,
has
written,
open,
chef,
product
team
or
or
the
managed
services.
A
Folks,
on
the
left
hand,
side
of
the
screen
you'll
see
we
have
a
centralized
back-end
that
handles
common
operations
that
are
not
doesn't
have
any
metadata
shared
with
any
particular
cluster.
So
hi,
if
I
mentioned
earlier,
just
bins
runs
the
open
shift
installer
and
we
have
integration
with
pager
Duty
so
that
we
can.
Actually,
you
know,
SLA
these
things.
We
lay
down
certificates
whenever
you
install
open
shift
for
if
you've
gone
to
the
management,
console
or
open
shift.
Three,
you
have
a
self-signed
certificate,
so
we
wrote
an
operator
that
goes
out.
A
Fetches
a
let's
encrypt
certificate,
attaches
it
to
the
ingress
and
to
the
console
and
then
refreshes
that
every
whatever
I
think
it's
45
days.
You
never
have
to
worry
about
certificates.
You
know
expiring.
The
last
one
is
a
bit
of
DNS
stuff
that
we're
doing
so.
If
you've
gone
to,
try
it
out
open
shift,
calm,
you'll
see
it's
gonna,
ask
you
to
create
a
DNS
domain
and
if
you've
ever,
it's
always
DNS
by
the
way,
if
anyone's
ever
been
in
SRA,
I'm
sure
you're.
Well
aware
of
that,
we
currently
use
dine
DNS.
A
But
the
point
is
that
we
take
care
of
that
DNS
juggling
for
you,
so
that
happens
in
the
shared
side.
On
the
right
hand,
side
of
diagram
here
a
set
of
a
handful
of
operators
that
run
on
each
cluster,
one
that
lays
down
security,
one
that
lays
down
some
config
changes
that
we
do
like
we.
We
do
a
couple
of
things
and
then
kind
of
that
gets
folded
back
into
the
product,
or
maybe
it's
too
specific,
so
it
never
gets
folded
back
into
the
product.
A
We
run
back
over
to
vet
CD
using
Blaire
L
if
you're
curious,
and
then
we
have
a
cassette
of
custom
alerts
which
I
mentioned
earlier,
so
that
encompasses
what
you
get
with
OpenShift
dedicated
at
the
moment
when
we
release
arrow
with
version
running
sorry,
arrow
running
OpenShift
version
4,
it
will
look
something
like
this
certainly
will
use
operators
and
maybe
a
little
bit
of
different.
You
know
different
architecture
and
whatnot,
but
ultimately
the
same
kind
of
an
end
goal.
A
A
Ok,
so
I
fetched
this
code
lest
I
be
yelled
at
by
Diane
I
fetched
it
already
and
it's
a
we
have
a
command-line
utility
called
osya
and
what
this
guy
does
it's
in
its
in
github
and
it's
the
open
shift
cluster
manager
CLI
tool.
So
you
can
do
a
bunch
of
things
two
clusters
here
you
can
see.
You
can
look
at
your
account
information.
A
Real
simple,
but
it
may
seem
simple,
but
actually
this
is
this
is
quite
a
big
deal
and
here's
why
interacting
with
a
vendor
by
API
is
is
fairly
commonplace.
These
days,
AWS
GCP
everybody's
got
their
own
SDK
and
here's
an
SDK
that
Red
Hat
can
offer
you
for
managed
services.
So
that's
actually
a
first
and
and
pretty
cool.
Ok,
so
I've
got
three
I've
got
quota,
415,
multi,
a-z
clusters
and
quota
for
well
I've
got
three
running
single
AZ
clusters.
Right
now,
so
I
can
find
out
a
little
bit
of
that
information.
A
Let's
see
here,
I'm
gonna
create
a
cluster
here.
You
know
it
takes
a
half
hour
to
install
it
so,
but
I've
already
I've
already
created
one
super
simple
arguments:
I'm
going
to
create
it
in
u.s.
West
too,
because
my
favorite
least
finicky
AWS
region
is
Oregon
and
if
there
is
still
internet,
this
command
will
return.
A
A
We
couldn't
do
any
of
this,
incidentally,
without
CRTs.
So
this
is
all
extensions
to
kubernetes
itself,
and
so
we
use
kubernetes
or
we
use
open
shift
to
stand
up
open
shift.
It's
quite
a
cycle
clusters
get
labels,
for
example:
are
they
in
production
stage
or
integration?
You
know
we
have
some
knobs,
whether
we
turn
off
page
or
duty.
For
example,
we
don't
want
the
SRA
folks
who
are
on
call
to
get
paged
for
development
clusters,
for
example.
So
there's
a
couple
of
things
in
here
that
are
related
to
the
operators.
A
A
Yeah,
in
this
case,
the
master
has
IO
one
volume
types.
If
you've
seen
some
of
the
best
practices
we
recommend
for
SC
D.
So
that's
the
kind
of
stuff
is
baked
in
not
a
lot
of
rocket
science
going
on
here,
but
this
is
what
it
looks
like
cool.
So
this
is
open
shift
four
to
two
at
the
time,
I
deployed
it
and
I
just
checked
before
getting
up
here
and
it
looks
like
the
SRE
team
has
upgraded
this
cluster
in
the
last
couple
days.
A
A
So
here's
the
cloud
that
reddit.com
OpenShift
cluster
manager
right
now
I've
only
got
two
clusters
stood
up
here.
One
of
them
is
nukes
from
earlier
today
and
there's
the
third
one
actually
as
I'm
speaking
so
you've
got.
I've
got
I've
deployed
it
for
two
earlier
and
then
I've
got
this
J
dieter
Commons,
one
from
which
is
currently
being
deployed.
A
So
this
this
resource
usage
from
the
cluster
is
actually
back
hauled
and
and
all
of
the
details
about
the
cluster
are
back
hauled
from
the
cluster
into
our
two
limiter
system
and
the
one
of
the
micro
services
for
cloud
that
right
at
the
comm
calls
out
and
fishes
that
data
out
of
our
telemetry
system
to
display
it
so
yeah
four
to
four
is
what's
currently
running
so
so
as
a
managed
service.
We
want
to
make
sure
you
have
the
latest
versions
as
well,
so
we're
doing
upgrades
right
now
of
busy
streams
on
a
weekly
basis.
A
So
if
you're
to
purchase
and
OpenShift
dedicated
it,
we
get
upgraded,
every
I,
don't
know
Wednesday
or
something
like
that.
Maybe
later
in
the
week,
we
backhaul
some
of
the
monitoring
data
as
well
no
alerts
firing
on
this
one
status
of
all
the
cluster
operators
and
so
forth.
So
yeah,
all
that
comes
from
some
Flender
system
that
we
pulled
data
back
from
and
here's
the
install.
So
this
is
the
pod
when
I
mentioned
earlier
that
hive
was
going
to
kick
off
the
Installer.
A
The
Installer
runs
in
a
pod
and
these
logs
are
what
the
standardout
was
from
that
pod,
so
towards
the
end
is
like
install
succeeded.
So
that's
what
it
looks
like
and
then,
of
course
we
can
click,
Launch,
console
and
I
end
up
at
a
login
screen
and
I've
configured
github
as
an
authentication
provider
and
I
am
in
too.
A
A
B
A
Had
a
couple
more
I
just
want
to,
let
you
know
some
of
the
other
stuff
that
we
lay
down
on
top
of
OpenShift,
so
things
we've
learned
along
the
way,
a
couple
things
we
have
to
keep
an
eye
on
cloud
providers.
Storage
may
get
wedged.
They
they
have
someone
giggled
cuz,
that's
a
fact.
Life
DNS
for
sure
can
be
problematic.
We
want
to
know
about
those
things
whether
they
resolved
them
on
their
selves
is
something
else
entirely.
Sometimes
they
do
and
then
we
want
to
keep
an
eye
on
something
funny.
We
noticed
it's
like.
A
There
was
no
alert
at
the
time.
There
was
no
alert
for
whether
the
number
of
machines
matched
matched
what
they
operator
thought
there
should
be,
and
when
there's
a
mismatch
near
that's
something
we
need
to
alert
about,
so
we're
currently
carrying
that
and
it'll
eventually
get
back
into
the
product.
A
couple
other
things
that
we're
learning
and
trying
to
feedback
through
to
a
product
team
and
then
to
eventually
to
customers
who
want
to
use
openshift
on
pram
or
or
just
manage
it
themselves.
A
I
mentioned
earlier
we're
doing
our
own
get
ops
thing.
That's
it's
totally
built
in-house,
like
most
of
you,
others
might
have
we're
keeping
an
eye
on
some
of
the
developments
in
the
get
op
CI
CD
for
kubernetes
world
a
lot
of
customers,
so
we
have
a
Prometheus
that
runs
on
every
open
shift,
dedicated
cluster
or
every
era.
Yeah
every
openshift
dedicated
cluster,
eventually
you'll
be
able
to
stand
up
a
second
copy
of
for
me
theist
for
your
own.
A
What's
called
user
workload,
monitoring
we're
keeping
a
really
close
eye
on
that,
because
that's
probably
the
number
one
feature
asked
right
now
is
to
allow
for
you
to
scrape
your
own
prometheus,
metrics
and
that'll
come
into
into
OpenShift
and
one
of
the
upcoming
releases
and
certainly
will
support
it.
We
want
to
make
sure
you
can
customize
your
own
DNS.
A
So
we'd
like
to
see
is
the
control
plane
also
be
a
machine
set
so
that
we
can
do
things
like
change
the
type
of
node
potentially
scale
out
the
number
of
masters,
although
that
would
be
pretty
rare,
so
a
couple
of
tips
on
how
we're
debugging
these
clusters?
How
many,
how
many
of
you,
how
many
of
you
have
seen
the
OC
debug
command?
A
Not
enough!
So
now,
there's
no
excuse,
go
type,
OC,
debug,
somewhere,
OC,
debug,
node,
slash
and
then
the
name
by
which
kubernetes
knows
your
node
and
it
will
launch
a
privileged
pod
on
that
node
that
you
can
do
local
debugging
on
the
system.
So
that's
a
fun
one,
of
course,
I
mentioned
telemetry
earlier
and
then
the
audit
logs
kubernetes
audit
logs,
so
there's
a
ton
of
data
being
emitted
by
the
system
itself.
A
Cool
I
have
just
a
one
or
two
slides
on
some
of
the
roadmap
stuff
that
I
guess
I
picked
off
the
product
matters
to
see
which
kind
of
stuff
I
thought
might
be
interesting
for
this
audience,
not
exposing
the
eight,
the
API
publicly,
so
you'll
have
basically
no
public
publicly
routable
dns
name
for
your
private
cluster
was
what
we're
calling
private
clusters.
I
should
say.
Every
cloud
provider
seems
to
define
these
differently
bring
your
own
clouds.
You
can
give
us
your
AWS
account
and
we
will
provision
all
that
stuff
into
your
account.
A
So
when
we
round
out
that
story,
we'll
have
coverage
across,
at
least
in
the
u.s.,
the
three
major
cloud
providers
when
we
have
already
had
open
shift
OSD,
which
is
open,
shift,
dedicated
running
on
it
on
AWS
and
then
open
shift,
dedicated,
also
running
on
TCP
and
your
front
end
will
look
like
something
like
a
member
I
showed
you
the
ocm
cluster
create
earlier.
Imagine
an
additional
flag
that
says
cloud
provider
equals
GCP,
something
like
that.
A
Put
it
wherever
you'd
like
I,
think
maybe
one
of
the
more
fun
projects
we've
got
going
on
right
now
is
so
right.
Now
you
can
stand
up,
openshift,
imagine
being
able
to
provision
not
only
OpenShift,
dedicated
or
arrow
or
something
else,
but
other
Red
Hat
products
that
were
on
top
or
take
it
the
other
way.
Maybe
you
just
bought
some
middleware
from
Red
Hat
that
runs
on
top
of
OpenShift,
imagine
being
able
to
provision
that
and
you
automatically
get
a
managed
cluster
underneath
it.
A
That
kind
of
stuff
is
being
worked
right
now
on
the
arrow
side,
just
in
the
last
couple
of
weeks,
I
think
at
ignite
last
one
two
weeks
ago,
they
finally
removed
there
a
requirement
for
reserved
instances
which
caused
you
to
pay
for
a
bunch
of
compute
in
advance.
Now
we're
back
to
you
I
think
it's
hourly
billing
we're
tying
in
as
your
log
analytics
to
that
service
and
eventually
sorry.