►
Description
The Oregon Trail to Kubernetes [I] - Joshua Roppo, Lytics
Can a small team operating 2000 CPUs, escape the glorified bash infighting of Configuration Management to homestead the scalable compute plains of Kubernetes? A journey of transitioning from Google’s Compute Engine to the blessed Container Engine.
The route we chose diverged from the never ending landscape of single purpose YAML tutorials and retreading Configuration Management tools with templating. Instead, we chose a mountain pass of defining Kubernetes Resources as Go code for compiled type checking, composability, validation, and potential for extension. The case study of a small team breaking trail through ecosystems of application design, schedule paradigms, deprecation dysentery, and holding legacy together with bailing wire. A retrospective of value added versus time wasted on the path to great opportunities on Kubernetes.
Talk Overview: Lytics Stack and overview(whoami) Loading the Wagon: Design and decision considerations(Read the Borg Paper) Deprecation Dysentery: Wait wait don’t use that. Mirages of disappointment: Systems which couldn’t make it to Kubernetes. Compute Resource Hunting Massacre: Avoiding compute underuse; taking advantage of scheduler. Handyman’s Corner: The bailing wire and zip-tie Kubernetes tools and services built to keep the broken axle(legacy systems) intact through the journey. Blizzards of the Kubernetes: from a user’s perspective who can’t follow every SIG; surviving the avalanche of ecosystem changes. Cascadia found: the wins, plans to rebuild what was abandoned, and breathing the free air. Next: Sim City
About Joshua Roppo
Platform Operations Engineer with a preference to write code over Bash. Managing operational decisions and transitions at Lytics for three years where we turn raw user and event data into actionable personalization APIs for marketing. Pedantic gopher who enjoys the challenges of Operations.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
thanks
for
coming
who
everyone's
had
a
good
coupe
con
when
I
first
proposed
this
talk,
this
was
actually
kind
of
a
little
bit
cynical
and
based
on
the
description
in
the
schedule.
I've
changed
it
a
little
bit,
but
I've
kept
the
parts
that
I
think
are
most
applicable,
the
people
who
are
coming
to
keep
con.
So
what
we're
gonna
cover
our
story,
I
work
for
a
company
called
lytx
and
we
transition
to
kubernetes
from
GCE
and
jumped
on
to
the
gke
system.
So
this
is
kind
of
the
story
of
how
we
went.
A
We
transitioned
fro
from
GCE
to
gke
and
the
stumbling
blocks
that
we
kind
of
ran
into
along
the
way.
As
you
can
guess,
from
the
theme
of
Oregon
Trail,
we
ran
into
some
issues
over
came
from
stuff
and
kind
of
ended
up
an
interesting
place
that
not
a
lot
of
companies
seem
I.
Don't
think
a
lot
of
companies
ended
up
in
our
position,
but
I
think
we
actually
uncovered
some
cool
stuff
along
the
way.
So
Who
am
I.
My
name
is
Josh
rope.
Oh
that's
actually
not
on
the
slide,
which
is
great
I've,
been
doing.
A
It's
a
good
thing.
So
the
lytx
story
we're
a
customer
data
platform
for
marketers
and
web
developers.
We
enable
personalized
web
interactions,
so
the
classical
example
is
the
recommendation
engine
from
Amazon.
So
you
look
at
a
page
and
then
you
see
the
other
items
that
you
might
be
interested
in.
We
enable
you
to
based
on
content,
recommendation
and
machine
learning.
We
enable
you
to
create
a
list
of
from
we.
We
can
tailor
personalized
experiences
to
your
users
based
on
our
generic
engine
and
entity
resolution.
A
A
We
moved
to
Google
cloud
platform
for
because
of
stability
and
performance,
we
started
adopting
a
lot
of
their
technologies
to
replace
things
like
kafka,
so
the
coffee
was
probably
pub/sub
casandra's
place
a
big
table,
it's
been
really
nice
platform
to
build
on
and
as
far
as
the
spectrum
of
monolith
versus
micro-services
we're
kind
of
we're
kind
of
in
the
middle
right
now
we
haven't
gone
full
micro
service
adoption,
but
we're
not
we're
not
tied
down
to
a
single
monolith.
So
we
have
an
API
tier.
A
We
have
a
workflow
management
tier,
which
we
call
a
metaphor,
and
then
we
have
an
event
stream
and
I.
We
have
a
stream
processing
system,
that's
distributed
across
a
number
of
group
of
deployments,
and
that
does
energy
resolution,
most
gene
learning
classification
in
real
time
so,
where
we
came
from,
we
were
originally
on
saltstack
and
that
is
both
powerful
and
terrifying.
It's
a
you
can
take
the
ammo
run
it
through
a
jint
template
and
then
generate
more
gamma
based
on
that.
A
So
it's
a
very
complex
system,
but
it
is
pretty
powerful,
so
obviously
I'm
not
gonna
get
into
wive
kubernetes
we're
at
cube
con.
We
all
know
all
the
great
points
of
what
kubernetes
is
good
at,
but
for
us
it
was
primarily
the
idea
of
going
into
VM
management
is
not
very
appealing
because
you
have
to
build
all
these
heavyweight
tools
around
it
and
bird
deploying
a
single
binary.
A
So
building
all
of
these,
using
all
these
heavy
tools
to
manage
things
like
operating
system
kernels,
managing
all
those
upgrades
it
just
is
over
for
just
trying
to
deploy
something
very
simple.
So
when
we
kubernetes
started
coming
on
the
scene,
we're
like
this
is.
This
is
good
because
we'd
played
around
with
docker
but
and
each
one
he's
tried
to
deploy.
Just
docker
to
production
knows
that
that
doesn't
go
very
well,
so
the
goals
are
moving
to
kubernetes.
We
eventually
want
to
get
everything
in
kubernetes
right
now
we
are
only
about.
A
A
So
one
complication
that
you
do
kind
of
see
and
it's
kind
of
visible
on
the
screen-
is
that
we
have
a
deployment
system
called
link
grid
which
can
be
scaled
from
one
we
we
might
need
to
have
from
anywhere
between
one
to
and
a
number
of
these
deployments
and
up
front.
That
creates
a
little
bit
of
an
issue,
because
that
means
you're
gonna
have
to
define
yamo
or
template.
A
Yamo
declarations
for
these
appointments
from
1
to
n,
and
so
I
wanted
to
come
up
with
a
I
wanted
to
have
a
good
solution
to
manage
that
going
forward
so
and
before
we
could
even
get
to
there.
We
had
to
do
some
application
preparation.
A
lot
of
our
services
were
kind
of
built
in
the
older
model
where
local
state
was
available,
and
it
would
be
there
with
your
surface
crash
and
then
came
back
up.
A
So
we
had
a
few
things
like
both
DB
files
that
were
on
a
server
and
if
something
crashed
it
was
checkpointing
its
last
location
and
but
in
the
communities
world.
If
you
kill
a
pod,
need
to
be
able
to
rescheduled
somewhere
else
and
won't
have
that
state
anymore.
So
you
have
to
build
rebuild
your
application
to
accommodate
that
fact
that
persistent
storage
shouldn't
be
a
thing
and
you
should
always
use
distributed
stores
like
GCS,
maybe
F,
just
something
that's
over
the
network.
We
also
really
utilize
a
lot.
A
We
need
to
be
able
to
let
that
finish
so
the
HP
PT,
stop
hook
on
a
pod
is
actually
super
useful
in
that
case,
and
it's
been
pretty
rock-solid.
So
when
you
request
a
pod
to
terminate
it
actually
does
wait
that
full
time
because
effectively
the
way
we
use
it,
we
use
it
opens
a
connection,
and
then
you
just
hold
that
connection
open
and
until
you're
ready
to
shut
down.
A
A
Sorry
I'm
blanking
right
now
and
we
have
some
clients
which
can
using
this.
We
have
some
clients
which,
using
the
same
workflow
like
an
export
to
Facebook
or
something
might
or
bigquery,
is
a
good
one
will
consume
a
lot
of
resources
on
that
one
pod.
If
it's
a
smaller
account,
it
might
be
burst
for
only
a
little
bit,
but
it's
a
flat
distribution
across
a
set
of
pods.
A
So
if
I
mean
the
VM
system,
where
it's
hard
to
create
and
allocate
VMs
just
because
it's
a
little
bit
ops
intensive
and
something
you
don't
want
to
do
very
often
you
end
up
having
to
buy
more
compute
than
you
actually
need.
So
you
end
up
in
the
situation
where
you're
just
wasting
money,
one
of
the
really
cool
things
about
kubernetes
is
that
there's
a
difference
between
what
you
scheduled,
pods
for
and
what
they
actually
are
limited
to.
A
So
if
a
pod
needs
to
burst
for
a
single
workflow,
he
can
just
for
simplicity's
sake,
on
the
vm
utilization.
Let's
say
that
there
is
one
there's
one
pod
that
actually
is
running
hot
most
of
the
time
and,
let's
just
say,
hypothetically,
that
these
other
ones
are
kind
of
hanging
out
not
doing
a
whole
lot
well
with
kubernetes
scheduler.
If
you
use,
if
you
combine
all
those
and
to
say
like
a
two
core
machine,
you
could
schedule
all
of
these
VM.
A
A
There
will
only
be
limited
to
only
running
on
using
one
core
at
a
time,
but
if
they
all
buffer
up
to
the
point
of
or
if
they're
all
limited
by
CPU,
they
won't
be
killed.
So
they
can
all
fight
over
CPU.
But
chances
are
if
they're
all
running
along
at
this
average
percentage
usage,
they
will
actually
be
able
to
fit
nicely
into
that.
Vm.
A
So,
on
the
trail
of
the
kubernetes,
obviously
yeah
Mille
lots
and
lots
of
yeah
Mille
I'm,
not
the
only
person
who
thinks
this
and
it's
you
know
some
people,
love
yeah,
Mille
I'm,
on
the
opinion
that
coming
from
salt
whitespace
is
difficult
to
think
about
sometimes,
and
the
fact
that
my
animal
doesn't
have
any
termination
characters
means
that
you
can't
format
it
so
there's
a
lot
of
times
when
yeah
mole
just
becomes
more
confusing
than
you
would
like
it
to
be
and
managing
tons
of
the
mo
files
it's
like.
Unless
you
have
scripts
to
I.
A
Getting
I
think
for
static
resources
and
rules
like
your
are
back.
Consider
it
like
your
are
back
settings
and
services.
I
think
it
works
fine,
because
those
things
don't
change
very
often,
but
for
things
that
you
might
want
to
manage
and
update
regularly,
it
seems
like
they're
or
create
that
1
to
n
number
of
I.
Think
it's
a
little
bit
problematic
because
your
have
to
manage
it
somehow.
A
A
That's
out
of
order,
so
another
option
client
go
it's
a
client
library,
it's
extracted
from
the
actual
kubernetes
repo.
So
it's
not
quite
as
much
code
that
you
need
to
import
you've
used
it
before
probably
experience
some
pain.
It's
a
little
bit
tricky
to
get
organized
correctly.
The
first
time
you
set
it
up
because
the
dependencies
are
complex,
but
it
does
afford
some
pretty
cool
stuff,
so
initially
I
jumped
into
it
because
I'm
a
bit
of
I
like
go
more
than
I,
like
gamble,
so
I
was
like
curious
to
see.
Alright.
A
So
at
this
point
it's
like
well,
if
we're
gonna
be
you
have
to
dig
into
the
code.
I,
don't
just
use
the
code
itself
and
one
of
the
cool
things
about.
Actually
writing
the
library
is
that
or
using
client
go.
Is
that
when
you
run
compile,
everything
is
type
checked?
That's
really
nice,
like
you
know,
at
least
when
you
run,
go,
build
that
everything
lines
up
correctly.
So
this
is
kind
of
like
an
equivalency
on
the
bots
on
the
top.
A
There
is
a
camel
definition
and
on
the
bottom
there
is
the
same
thing
written
in
specified
in
the
in
go
and,
as
you
can
see,
the
ml
is
more
a
little
bit
more
concise
and
a
little
bit
more
readable.
But
the
verbosity
of
the
go
example
is
very
it's
nice
and
clear.
We
have
hard
typed
field,
like
name
and
volume
source
which
define
how
you
need
to
build
this
structure
and
at
any
point
you
can
actually
dig
into
these
types
and
actually
see
what
the
documentation
is
and
how
it's
supposed
to
be
used.
A
This
is
an
actual
example
that
I've
been
working
on
for
the
past
couple
days,
trying
to
make
it
an
example
for
people
to
start
using
the
client
I
go,
and
it's
just
like
you
can
just
pick
this
pick
up
this
library
and
start
playing
around
with
it,
because
everything's
already
vendored
everything
is
already
defined
so,
like
you,
can
just
pick
this
up
and
start
playing
around
in
this
packet
in
this
git
repo.
But
in
this
example
down
here
we're
creating
a
v12.
A
We,
an
API
v1
beta
to
deployment,
obviously
still
running
into
the
numerous
number
of
versions
in
here
with
of
kubernetes
deployments.
But
if
we
highlight
this
deployment,
we
can
see
a
nice
little
structure,
definition
pop
up,
and
then
we
can
dive
into
what
the
type
actually
is.
So
now
we're
actually
diving
into
the
go,
definite
or
the
client
go
library,
so
we've
jumped
into
the
vendor
folder
and
now
we're
in
the
type
of
deployment.
A
So
now
we
can
see
that
there's
four
fields
on
this
type
and
these
are
the
specs.
So
this
is
the
deployment
spec
hit
control
dive
into
it.
Now
we
can
see
all
of
the
fields
on
this
and
we
get
nice
I
mean
it's
code.
So
it's
not
like
a
nice
HTML
file.
It's
not
a
nice
HTML
page,
but
this
is
the
documentation
for
what
kubernetes
is
actually
using
under
the
hood.
A
Okay,
so
the
first
thing,
I
kind
of
did,
after
digging
into
the
client
go
library,
was
I
built
a
CLI
tool
just
to
like
kind
of
see
like
does
this
work,
and
initially
it
was
just
like
list,
the
pods
from
some
coop
contacts.
So
the
cool
thing
about
building
a
CLI
tool
is
that
you
use
the
same
cube,
config
file
and
you
just
specify
at
parameter
and
parameters
to
the
client
creation
function
which
cube
context
you
want
to
use.
A
So
if
you're
running
a
mini
cube,
you
would
pass
mini
cube
and
I
have
an
example
of
that
later
so
lytx.
We
actually
started
using
this
to
define
some
of
our
deployment
objects
and
I
actually
created
a
superset
object.
On
top
of
that
called
a
deployment
set,
which
is
effectively
a
specified
set
of
deployments
that
all
share
roughly
the
same
type
but
are
just
like
name,
has
a
separate
names
and
separate
a
few
different
field
like
metadata
tags,
to
define
their
role.
A
A
A
Just
a
few
things
about,
if
you
are
building
a
CLI,
make
sure
you
specify
and
make
sure
that
the
user
is
defining,
which
cube
context
they're,
actually
trying
to
use
coop
CTL
is
actually
kind
of
loose
on
that
which
I
find
surprising,
like
you,
don't
actually
buy
the
context.
It
will
just
use
whatever
you
have
configured
and
your
cube
config,
which
is
a
little
little
risky,
because
kubernetes
has
never
made
it
I
accidentally
the
whole
cluster,
but
and
then
also
December.
All
of
your
all
of
your
deployments.
A
Obviously,
kubernetes
is
moving
really
fast
and
that's
great,
but
it
also
means
from
users.
Perspective
stuff
is
moving
really
fast.
The
things
are
changing
out
from
underneath
eat
regularly
on
the
right.
There
is
a
list
of
SIG's
of
special
interest
groups,
and
this
only
goes
to
see
there's
a
lot
of
SIG's
out
there
and
it's
hard
to
pay
attention
to
all
of
them.
Things
like
replication
controllers,
third-party
resources.
A
Those
were
already
dead
at
this
point
you
shouldn't
be
using
them
and
I
know
that
talking
to
people
here
at
coop
con,
everyone
is
kind
of
DL
des
users
of
kubernetes
are
kind
of
dealing
with
this,
but
everyone
on
the
everyone
who's
actually
building
kubernetes,
wants
to
go
faster,
like
they
want
releases
every
month,
not
every
not
every
few
months.
So
we're
in
this,
like
weird
kind
of
it's
described
as
I
heard,
Tim
Hawkins,
ascribe
it
as
a
u-shape.
A
A
All
of
these
changes,
the
client
go,
library
has
had
a
rough
history,
I
started
using
it
at
the
client,
go
version
1.4
and
it
was
like
this
is
kind
of
cool
and
then
1.5
come
out
and
they
created
another
path
for
the
one
five
client
they
just
shoved
everything
in
there
and
then
they
kind
of
realized.
Oh,
this
was
a
bad
idea
because
we're
just
appending
tons
of
code
to
the
same
git
repo,
so
they
ripped
that
out
in
version
two
and
then
from
then
on
things
got
a
little
weird.
A
I
would
I
hope
that
things
are
stable
now
and
it'll.
Be
them
a
lot
simpler
to
upgrade
so
again
when
you're,
if
you
decide
to
use
client,
go
think
a
bit
about
your
design.
Look
at
how
the
interface
is
use
your
look
at
the
interfaces
of
the
types
you're
going
to
be
interacting
with
and
think
about
it.
A
little
bit
there's
some
chaining
that
goes
on
that
makes
some
of
the
client
go
functions
a
little
bit
hard
to
use
so
think
about
it
before
you
jump
into
it
vendor.
A
Everything,
if,
like
make
sure,
there's
no
code
that
is
coming
out
of
your
go
path,
and
there
are
some
issues
with
depth
right
now
on
Lily's
talk
yesterday,
she
mentioned
that
our
back
has
a
little
few
issues.
So
if
you
try
to
use
our
back
in
client
in
the
latest
version
of
client
go
there,
you
might
run
into
some
issues
with
some
API
versioning
and
that's
kind
of
that's
a
bug
and
depth.
Actually
right
now,
they're
trying
to
sort
that
out
but
be
wary,
but
also
I.
A
Don't
again,
if
your
yam
yam
will
seems
like
the
appropriate
place
to
define
our
back
rules,
don't
ever
mix
any
other
projects
with
the
project
that
you're
trying
to
build.
Your
client
go
code
with
because
kubernetes
moves
so
much
faster
so
like
we
have
like
our
I,
tried,
moving
client
and
go
into
or
the
dependencies
into
our
main
repo,
and
we
use
a
lot
of
Google
libraries
and
use
a
lot
of
G
RPC
libraries,
and
we
also
have
Etsy
D
and
all
those
three
together
just
create
this
massive
storm
justic.
A
You
cannot
find
the
resolution
of
libraries
to
make
all
those
things
work
and
unless
you
can
match
the
velocity
of
kubernetes
project,
you're,
probably
gonna
end
up
caught
in
a
difficult
place
and
it
just
you
won't
be
able
to
resolve
the
different
the
dependency
trees
so
build
your
client
go
library
on
its
own
project.
Don't
try
and
merge
it
in
with
any
other
projects.
A
We
have
a
kebab,
we
use
Kabana
there.
We
you
they
used
to
be
our
only
source
of
our
only
log
storage,
but
there
are
things
like
google
lager
to
our
stack
driver
logging
technically,
and
that
is
one
option,
but
the
search
functionality
is
a
little
bit
underwhelming
and
there
are
some
nice
features
that
people
really
like
about
Cabana
and
I
wanted
to
keep
those.
So
we
actually
use
something
that
glider
Labs
created
and
I
forked
it
and
made
a
few
minor
modifications.
A
If
you
are
interested
I
would
be
happy
to
show
you
how
to
make
those
changes
as
well,
but
effectively
it.
Let
it
create
a
daemon
set,
runs
on
every
single
node
in
your
cluster
reads
the
logs
from
docker
and
then
ships
them
straight
to
your
elk
that
your
elk
cluster.
But
you
don't
need
a
log
stash
in
any
of
this.
A
So
if
you
do
cube
CTL
get
events,
that's
actually
a
useful
stream
of
information
to
know
about
what
your
cluster
is
actually
doing,
created
a
little
tool
just
to
read
that
out
and
write
it
to
logs.
It's
actually
super
useful.
You
can
create
metrics
off
of
that.
So
when
pods
are
dying-
and
you
don't
know
why
or
if
pods
are
dying,
it's
a
good
way
to
create
a
metric
on
it
and
then
notify.
A
A
This
is
kind
of
a
unique
one,
but
if
you
use
something
like
stats,
D
for
metrics
and
graphite
or
carbon
Prometheus
is
well
designed
in
that
you
can
take.
You
can
give
it
a
list
of
pod
names
as
like
part
of
the
part
of
the
path
for
metrics,
and
it
handles
that
fine
like
it's.
It
writes
it
correctly,
but
if
you're
using
an
older
system
like
carbon,
it
writes
every
single
metrics
path
as
a
filesystem
directory.
So
if
you
use
pods
that
will
just
if
so,
if
it's
using
pod
IDs,
that's
just
gonna
explode.
A
So
created
a
distributed
effectively
a
naming
flattener,
so
you
can
effectively
share
different
names,
so
something
that
has
a
very
something
that
as
a
normal
pod
name,
it
has
a
replica
set
and
then
a
unique
identifier.
You
can
squash
that
down
to
just
like
one
and
then
one
two
three
but
the
one
pods
exit,
then
they
will
release
that
and
then
someone
else
can
claim
it.
When
then
another
pod
starts
up,
so
you
effectively
recycle
the
name
set.
A
So
this
was
pretty
useful.
It
provides
nice
visibility
to
our
staging
environment,
so
people
don't
like
stomp
on
each
other
and
it's
a
nice
tool,
but
it
is
again
you're
issuing
commands
it's
not
as
clean
as
like
something
that
is
just
reactive
based
on
your
github,
like
it's,
not
something
you
something.
It
is
still
a
manual
tool
effectively
at
this
point.
So
we
want
to
get
to
the
point
where
we're
using
like
an
operator
pattern,
but
we're
not
quite
there
yet,
but
all
the
types
are
defined.
A
A
A
So
with
that,
I'm
gonna
try
and
give
a
you
a
short
demo
of
actually
using
the
client
code
library
to
the
do
end
and
testing
without
having
to
pass
yam
will
file
it
around
and
cat
scripts.
So,
like
typically,
whenever
I
see
like
some
project,
there's
like
alright
to
do
end-to-end
testing,
you
runs
bash
script,
which
evals
his
other
bash
script
and
then
the
output
of
that-
and
you
know
that
runs
a
bunch
of
coop
CDL
commands
and
applies
them
so
one
of
them.
A
Actually,
let's
go
to
the
actual
test,
so
what
this
test
is
gonna,
do
it's
going
to
communicate
with
my
local
mini
cube,
it's
going
to
use
that
deployment
that
I
showed
you
earlier.
It's
gonna
call.
This
function,
create
the
deployment
structure,
we're
going
to
create
a
connection
to
mini
cube
and
then
we're
going
like.
Actually,
this
is
creating
the
connection
we're
going
to
create
that
deployment,
we're
gonna
validate
that
it
exists,
which
is
not
very
sophisticated
right
now,
but
and
then
we're
gonna
delete
it
afterwards.
A
A
A
I
think
it's
really
useful
I
think
other
people
should
explore
it,
and
especially
now
that
it's
not
there's
not
so
much
wahkeena
swith
versioning,
going
on
I'm
hoping
from
here
on
out.
We
will
have
like
complete
feature
sets
for
every
release,
so
there'll
be
a
new
client
go
version
which
doesn't
line
up
numerically
assembler
wise,
but
hopefully
that
will
tie
very
nicely
to
the
upcoming
kubernetes
releases.
Going
forward.
I've
been
talking
to
a
few
a
few
of
the
people
working
on
client
ochco
and
they
sounded
like
hopefully
from
here
on
out
it'll,
be
solid
GK.
A
We
haven't
really
run
into
any
issues
like
occasionally
a
couplet
will
die
and
in
the
early
days
I
didn't
really
have
to
like
go
kill
them
every
once
in
a
while,
but
it's
been
rock
solid
for
a
long
time.
So,
if
you
are
considering
GK
or
kubernetes
manage
systems,
it's
been
pretty
great
for
us,
so
we
eventually
did
make
it
to
kubernetes.
A
A
A
A
I'm
really
looking
forward
to
service
meshes
and
what
those
will
provide
us
because,
like
being
able,
the
stos,
TLS
automatic
automatic
encryption
of
traffic
is
gonna,
be
a
big
game,
changer
and
really
nice
and
auto
scaling
and
client
and
cluster
resizing
is
gonna,
be
really
nice
going
forward.
So
that's
all
I
got.
Thank
you.