►
Description
Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
CoreDNS: Intro and Deep Dive - John Belamaric, Google & Yong Tang, Ivanti, Inc
Come to learn about CoreDNS and the latest updates to the project and roadmap. Stay to learn about how to write your own CoreDNS plugin!
A
B
Yeah,
my
name
is
I'm
from
ivanti,
I'm
also
one
of
the
maintainers
of
according
us,
okay
yeah.
So
let
me
continue:
okay,
okay
yeah
in
first
of
all,
thanks
for
everyone
for
joining
session.
It's
glad
to
see
everyone
is
coming
here,
and
especially
since
these
kobe
times,
so
I
think
it's
still
tough
life
is
still
tough,
but
I
think
we
are
seeing
the
end
of
the
tunnel
of
this
occult
period.
So
hopefully
everyone
has
a
better
life.
You
know
moving
forward
right
in
today's
meeting.
In
today's
session
we
are
going
to
discuss
about
coding
s.
B
We
are
going
to
do
two
things.
First
of
all,
we
are
going
to
discuss
about
doing
a
little
bit
of
introduction
in
coordinates
about
the
latest
update
in
coordinates
and
several
things
related
to
coordinates
community
and
then
I'm
going
to
hand
over
to
zhang
jiang,
we'll
do
a
deep
dive,
encoding
us
and
some
of
the
things
that
we
want
to
discuss.
B
B
Okay,
so,
first
of
all,
just
in
case
you're
not
very
familiar,
I'm
going
to
give
you
a
little
background
on
coding
s.
So
what
is
coding
as
codings
is
a
flexible,
dns
server.
It's
reading
goal.
Initially
it
started
as
at
the
fork
of
academy
atp,
and
then
I
think
in
in
2016
right.
That's
that's.
When
coordinates
project
was
started
initially
mick
driven.
B
He
contributed
to
the
majority
of
the
original
code
for
coding
us
to
make
the
caddy
server
to
him,
actually
make
a
fork
of
cad
atp
server
and
transform
that
category
server
into
a
dns
server,
and
that's
why
it
was
originally
originally
named
as
the
caddy
dns
and
then
over
the
years
with
different
contributions
from
so
many
contributors,
so
many
community.
So
many
members
from
a
community
coding
has
gradually
evolve
into
one
of
the
best
best
day
and
server
around
the
world,
and
most
notably
coordinators
now
became
the
default
dns
server.
Kubernetes.
B
That's
why
we're
here,
because
that's
a
cool
because
so
we
discussed
about
coordinates
but
at
its
core
coordinates,
is
still
a
dns
server
from
the
beginning
and
the
coordinates
is
very
much
different
from
other
dns
servers.
You
know,
we
all
know
the
bind
like
a
dns
server
bind
under
some
other
dns
servers,
but
the
difference
in
coding
is
another.
Dns
server
is,
according
as
a
focus
on
service
discovery,
and
also
coding,
has
a
very
special
architecture.
That's
that
is
actually
the
it's
a
plugin
based
architecture,
which
means
it
can
be
easily
extended.
B
If
you
want
to,
let's
say
if
you
you
want
to
have
some
features,
and
you
cannot
find
find
the
support
from
codings
along,
as
you
know,
how
to
write
go.
You
can
easily
write
this
feature
for
yourself
and
that,
in
fact,
as
late
later
today,
jiang
will
work
through
some
of
the
plugins,
and
you
can
find
out
how
easy
it
is
to
implement
a
plugin.
Just
for
your
usage,
as
long
as
you
know
how
to
write
and
go
right.
B
The
dns
over
grpc
is
not
a
true
dns
standard,
but
it's
more
like
a
customer
implementation
according
is
also
support
of
a
supported
feature
of
forwarding
to
upstream,
via
dns
or
tos
or
grp
0.
So,
if
you
use
coding
as
to
serve
your
dns
server
to
server
dns
traffic,
which
is
udp-
and
you
want
to
to
use
coding
to
query
to
upstream
dns
server,
you
actually
don't
need
to
always
go
through
dns
or
udp.
You
can
use
other
features.
B
B
Another
thing
with
accordions
is
that
accordions
is
fully
embedded
into
the
cloud
native
ecosystem.
It
has
integration
with
the
promises,
open,
tracing
and
opa.
Both
awesome
clown
native
project,
of
course,
as
I
mentioned
before,
the
biggest
feature
is
coordinates,
is
coding
as
now
is
the
default
dns
server
in
kubernetes?
So
whenever
you
use
kunuti,
you
probably
already
noticed
that
there
is
a
a
power
up
and
running:
that's
the
name
with
the
name
of
coding
as
right:
okay,
so
let's,
let's
okay!
So
let's
get
through
some
of
the
recent
updates
since
last
year.
B
B
The
latest
version
is
1.9.2
has
been
released
just
10
days
ago.
This
month,
10
days
ago,
in
may
2022
over
the
over
the
past
half
a
year
or
so
two
plugins
has
been
added.
One
is
the
grip
plugin
which
allows
you
to
report
where
the
query
comes
from
so,
which
is
very
nice
feature
that's
been
requested
by
by
the
community,
and
we
finally
bring
this
plugin
into
the
default
coordinates
plugin
system.
B
We
also
have
added
another
plugin
called
header
plugin,
which
allows
you
to
fiddle
with
header
bits
of
your
dns
query
message:
the
releases,
including
us
over
the
past
half
year,
also
consists
of
a
couple
of
backward
incompatible
change.
So
if
you
ever
want
to
update
your
dns
server,
you
may
need
to
pay
close
attention.
One
thing
is
incorporated:
we
remove
the
remove
the
wildcard
query
functionality.
B
So
we
finally
decided
to
say:
let's
remove
this,
remove
the
plain
text-
secret
saving
equal
file,
so
that
that
also
means,
if
you
ever
use
this
feature
before
you
have
to
find
some
other
way.
For
example,
you
can
pass
the
sequence
through
the
invited
variable,
which
is
much
safer
from
a
security
standpoint.
B
B
Some
people
may
notice
that
the
golan
1.17.6
actually
consists
of
several
security
vulnerabilities.
Sad
in
impact
a
lot
of
software
not
just
coordinates.
Also,
if
you
ever
use
e-steel
or
ever
use
some
of
the
selfies
built
by
golang
you'll
notice
that
if
you
ever
use
a
vulnerability
scanner,
you
probably
notice
that
you
want
to
scan.
I
just
report
quite
a
few
things
recently
right,
that's
actually
related
to
golan
1.17.6
and
because
of
that
in
including
1.9.1,
which
was
released
just
a
couple
months
ago.
B
B
Okay,
so
we
we
talk
about
the
security
of
course
for
the
past
year,
or
so,
the
security
has
been
the
focus
not
just
for
coding
us,
but
for
the
whole
selfie
industry.
You
know,
especially
if
we
think
about
like
in
early
2021,
people
were
talking
about
ransom
attacks
and
then
later
we
talked
about
log
4g.
Both
both
of
those
events
has
been
like
epic,
in
in
terms
of
news
updates,
like
people
just
receive
those
those
news,
even
from
cnn,
from
some
of
the
news
channels.
B
People
talking
about
the
log4j
ransom
tags,
so
that's
why
the
security
has
been
a
focus
for
for
the
whole
2021.
and
and
for
coding,
as
we
actually
complete
a
security
audit.
This
security
audit
has
been
done
recently
in
march
2022.
The
secret
audit
was
done
by
a
third
party
auditor.
That's
a
trio
of
beats
located
in
new
york.
The
event
the
secretary
event
has
been
sponsored
by
lynch
foundation.
B
So
now,
here
I'm
going
to
to
say:
okay
thanks
a
lot
for
the
support
from
lenin
foundation
and
the
cncf
to
allow
the
coding
as
to
utilize
a
resource
that's
available
to
us
so
which
allows
us
to
making
great
progress
and
also
helping
the
community
community.
That's
using
coordinates
in
this
security
audit
in
conducted
by
trailbeats.
B
B
There
is
also
another
medium,
medium
issue
related
to
the
usage
of
plain
text:
saving
plain
text
in
the
core
files,
but
this
medium
issue
reported
it's
actually
not
so
much
of
critical,
because
again,
it's
possible
to
mitigate
the
issue.
Even
if
you
don't
update
your
server,
but
the
rest
of
the
issues
discovered
by
the
rest
of
the
issue
discovered
by
trailbeats
are
all
related
to
informational
or
low
level
stability.
So
we
feel,
like
coordinates,
is
very
much
a
safe,
dns
server.
So
no
and
all
the
issues
for
now
has
been
resolved.
B
B
B
B
You
will
become
a
contributor
just
by
adding
this
no
entry
right.
We
also
have
a
9200
stars,
so
we
are
hoping
to
reach
10
000,
10,
000
stars
very
soon.
So,
let's
see
when
we
can
get
reach
to
this
goal,
and
also
another
thing
I
want
to
mention-
is
that
for
the
past
five
years
or
so
coordinates
has
been
participating
in
two
programs.
One
is
the
linux
foundation's
ufx
community
program,
which
helps
the
students
to
working
on
open
source
in
return.
B
Students
will
receive
a
small
amount
of
money
in
return
which
is
going
to
help
financially
and
also
we
participate
in
google
somehow
code.
Both
program
has
been
running
for
quite
several
years
and
we
participate
almost
every
year
for
the
past
five
years.
This
year
there
is
another
project,
that's
actually
the
acme
support
for
certificate
management.
This
project
has
been
accepted
by
google
somehow
code.
B
B
C
A
So
we
have
about
20
minutes
left
before
I
jump
into
this.
I
want
to
ask
a
few
questions
know
how
much
time
I
should
spend
on
different
parts
of
this,
so
how
many
of
you
are
using
kubernetes
and
corey,
and
s
and
kubernetes,
probably
almost
all
of
you,
okay,
awesome
and
then
how
many
of
you
are
using
coordinates
without
kubernetes,
just
nothing
to
do
with
kubernetes,
okay,
hey
we've.
A
Awesome
great
well,
so
what
what
I'm
going
to
talk
about
a
little
bit
are
how
you
could
customize
coordinates
now
for
most
of
you
in
the
audience
who
are
using
coordinates
as
part
of
kubernetes
you're.
Probably
not
going
to
want
to
do
this,
but
those
few
of
you
who
raised
your
hands
we'll
talk
about
it,
but
I'll
try
to
be
a
little
bit
brief.
So
we
leave
a
good
amount
of
time
for
q
a
around,
since
most
of
you
in
the
audience
might
not
be
too
keen
on
this
stuff.
A
And
that's
what
we're
talking
about
here
so
there's
three
basic
ways
to
extend
it.
The
simplest
way
is
to
enable
an
external
plug-in.
So
we
have
plug-ins
that
come
with
your
standard
coordinates
that
all
of
you
running
our
income
are
running
in
your
kubernetes,
because
whoever
your
provider
is
built
it
and
or
they
pulled
it
from
our
docker,
our
docker
hub.
But
if
you
wanted
to
do
something
fancier,
like
I
don't
know
back
your
your
core
dns
in
memory
cache
with
a
layer,
2
redis
cache,
we
have
a.
A
We
have
a
an
external
plugin
for
that
or
there's
a
whole
host
of
them,
and
you
don't
even
need
to
know
how
to
use
go
to
do
this.
They
are
written
in
go,
but
it's
actually
super
super
simple.
There
are
a
couple
of
really
important
things,
though,
to
note:
plugins
are
not
loaded.
Dynamically
they're
built
they're
done
at
compile
time
build
time
and
the
plug-in
ordering
is
fixed
at
compile
time
so
that
processing
of
that
request
through
that
pipeline.
A
So
this
is
probably
the
most
accessible
way
to
do
it
and
your
all
your
prerequisites
are,
is
docker
and
a
shell.
So
simple
things
you
clone
it.
You
modify
the
plug-in
cfg
that
tells
it
what
plug-ins
to
compile
into
core
dns
and
in
what
order,
and
then
you
build
it.
So
I'm
not
going
to
step
through
it,
because
we
don't
have
time
for
that,
but
I
I'm
not
going
to
actually
do
it
on
a
shell,
but
it
actually
really
is
quick
and
easy.
So
you
clone
core
dns.
A
Cd
into
that
directory,
open,
plug-in
cfg,
so
what
you'll
see
in
plug-in
cfg
is
colon-delimited
lit
colon
deliminated
delimited
list.
I
can't
speak.
The
first
word.
Is
the
directive?
That's
the
word
that
will
appear
in
your
core
file
when
you
configure
that
plugin
and
then
the
go
module
that
implements
that
plugin.
A
A
All
right
second
way
coordinates
as
a
library.
So
here
what
you're
doing
is
in
instead
of
actually
running
the
coordinates
binary
itself.
You're
embedding
coordinates
in
another
binary.
You
can
use
this
to
strip
out
plugins,
you
don't
care
about.
So
how
many
of
you,
if
you
know,
use
the
node
local
dns
feature
in
kubernetes?
A
Okay,
a
couple
of
you
so
that
that
project
uses
this
technique
so
essentially
the
node
local
dns,
which
all
of
you
should
be
using
it
by
the
way,
because
it's
much
much
better
what
it
does
is.
It
runs
a
little
mini
core
dns
just
for
caching
on
every
node
and
it
redirects
all
the
dns
requests
from
that
node
to
that
local
cache
and
then
for
any
requests
that
need
to
go
to
the
central
cluster
dns.
It
upgrades
the
connection
to
tcp,
which
fixes
some
kernel
bugs
and
issues
and
race
conditions.
A
So
how
many
of
you
know
have
any
idea
about
what
I
just
said
as
far
as
dns
like,
like
you
understand?
Okay,
hopefully,
hopefully
you
can
look
that
up
and
it'll
get
through
but
yeah.
So
essentially,
if
you
have
dns
issues
in
kubernetes,
weird
dns
issues
try
node
local
dns,
because
there's
there's
kernel
issues,
there's
race
conditions,
there's
a
contract
filling
up
with
udp,
there's
all
sorts
of
subtle
things
that
can
happen
under
load
that
that
fixes
anyway.
A
It
uses
this
technique
and
I
have
an
example:
you
can
pull
off
of
github
super
easy
to
to
build.
I,
along
with
I
don't
know.
If
you
know
cricket
lou
is
he
wrote
a
bunch
of
the
dns
books,
dns
and
bind,
and
all
of
these
things
he
and
I
wrote
accordion
s
book
and
we
go
through
this
example
there.
So
you
know
you
can
go
buy
the
book
too.
That
would
be
great
all
right.
A
A
What
do
we
do
with
it?
Well,
we
tend
to
classify
plug-ins
into
three
categories.
This
is
not
strictly
necessary.
This
is
just
sort
of
a
best
practice
we
use
because
what
we
want
like
a
unix
pipeline,
we
want
each
of
those
plugins
to
be
composable.
We
want
you
to
be
able
to
use
them
with
the
other
plugins,
so
we
want
to
kind
of
scope
them
to
some
some
small
piece
of
functionality,
so
you
can
pull
them
into.
You
know
as
an
external
plug
and
pull
it
into
different,
coordinate
instances
and
things.
A
So
when
you
decide
to
write
a
plug-in
you
should
think
about.
Am
I
writing
a
back-end
plug-in
a
back-end
plug-in
means
I'm
a
source
of
data,
the
kubernetes
plug-in,
which
you're
all
most
of
you
are
using,
is
actually
a
back-end
plug-in.
It's
pulling
data
from
the
cube
api
server
and
publishing
it
as
dns
the
cloud
dns,
the
route
531
all
the
same
thing:
they
read
from
those
cloud
provider
apis
and
then
they
present
the
data
as
dns.
A
Those
are
black
back-ends,
there's
also
back-ends,
like
external
back-ends,
for
storing
your
dns
names
in
postgres,
for
example.
Mutators
mutators
are
things
that
muck
with
the
request.
They
do
something
to
it.
They
change
it.
They
deny
it.
So
we
have
a
rewrite
plugin
that
lets
you.
Somebody
queries
for.
You
know.
Well
you
know
some
some
name.
A
We
look
at
that
name
and
we
say
we
don't
really
want
you
to
go
there,
we're
going
to
rewrite
the
query
and
we're
going
to
send
it
to
the
upstream
name
server
as
something
different,
and
then
we're
going
to
reply
with
that
ip
address.
So
that's
what
rewrite
is
for,
and
actually
that
makes
me
something.
I'm
not
sure
if
young
mentioned
core
dns
is
a
authoritative,
dns
server.
A
So
in
dns,
there's
authoritative,
servers
and
recursive
servers
when
you
a
recursive
server
takes
a
dns
request
and
it's
broke
breaks
it
down
into
the
labels
and
it
goes
out
to
each
of
those
other
dns
servers
and
figures
out
which
name
server
owns
that
particular
domain.
So
when
you
look
up
food.google.com
your
local
recursive
server
will
go.
I
don't
know
anything
about
food.google.com,
I'm
not
authoritative.
I
don't
have
those
records,
so
I'm
going
to
ask
I'm
going
to
figure
out
what
name
server
does
have
those
and
you
know
so
it's
going
to
say:
okay.
A
A
Mutators!
That's
where
we
implement
apples,
cache
rewrite
things
like
that.
Finally
configurators,
if
that's
a
word,
are
just
things
that
modify
the
state
of
kubernetes.
Of
course
dns.
Rather
so
the
bind
plug-in
tells
you
which
ip
addresses
to
bind
to
the
log.
Plugin
tells
you
what
kind
of
logs
to
to
do
things
like
that.
So
think
about
your
plugin
that
you
want
to
write
as
one
of
these,
then
you
just
implement
four
functions.
These
are
the
mandatory
ones.
Of
course,
your
logic
is
gonna
have
to
live
in
some
function
somewhere.
A
The
name
function,
super
simple:
the
serve
dns
function,
that's
the
meat
that
takes
the
the
request
in
and
does
something
with
it.
So
our
example.
I've
used
up
all
my
time,
but
I'll
I'll
go
quick
again.
This
is
out
there
on
the
core
dns
organization
in
github.
You
can
step
through
this.
It's
it's
super
easy,
but
basically
there's
a
plug
in
here
that
I've
written
as
an
example
that
will
take
a
response
when
you,
when
you
get
the
response
from
say
the
upstream
name,
server
or
the
the
later
plugin
in
the
chain.
A
So
super
simple,
like
I
said,
name
function,
just
returns.
The
name
setup
registers
the
plugin
with
the
parsing
routines,
so
that
so
that
when
we're
parsing
the
core
file,
we
know
what
you're
referring
to.
When
you
use
that
directive,
I'm
just
gonna.
Well,
oh!
Actually,
this
is
important,
the
it
also
does
this
add
plugin,
and
this
is
what
inserts
it
in
the
chain.
So
in
that
setup
in
that
plug-in
cfg,
basically,
when
we,
when
we.
A
When
we
read
the
core
file,
we're
not
looking
at
the
order,
the
directives
come
in
in
order
to
initialize
all
of
these
chains,
we're
actually
running
through
a
fixed
order.
That's
why
why
you
have
that
issue?
I
talked
about
earlier
all
right
and
I'm
just
gonna,
because
I
want
to
leave
time.
Oh
wait,
sorry
young,
how
about
it
did
come
up?
A
No,
it
didn't
come
up
all
right,
I'm
going
to
skip
showing
you
that
in
the
interest
of
time
and
leaving
time
for
q
a,
but
you
can
check
out
the
serve
dns
on
github,
like
I
said,
super
easy
just
takes
in
a
request.
Modifies
it.
The
one
thing
I
will
tell
you
is
well
how
about
this
I'll
tell
you
contact
me
afterwards.
A
If
you
want
to
do
this
and
I'll
go
through
with
you
in
detail,
because
I
want
to
leave
you
guys
time
to
talk
all
right,
so
some
resources,
then
for
for
any
of
you
out
there
who
are
interested
in
learning
more
about
it
or
diving
into
actually
modifying
coordinates.
This
is
where
you
can
find
us
online.
A
We
have
a
slack
channel,
it's
not
the
kubernetes
slack.
It's
the
cncf
slack.
We
have
a
slack
channel
on
there
and
of
course,
github
is
the
main
place
where
you
can
reach
us.
That's
where
we're
most
active.
There
is
a
mailing
list,
but
we
don't
use
it
almost
at
all.
We
will
get
the
emails
and
reply
to
you,
but
nobody
uses
it.
So
I
don't
even
think
we
put
it
up
there,
but
mostly
slack
and
and
github
all
right.
D
Yeah
my
questions
about
ptr
and
a
records.
I
know
services,
core
dns
is
quite
consistent,
but
pod.
It
seems
that
sometimes
you
get
ptr
records,
reverse
dns
for
pods,
and
sometimes
you
don't
and
I
see
on
github
some
people
saying
that.
Oh
that
should
never
ever
work,
which
I
actually
personally
agree
with,
but
my
my
users
like
it
that
it
works.
A
A
A
A
So,
if
you
think
about
coordinates,
does
in
kubernetes
context,
cordera
sits
and
listens
on
the
cube
api
server,
so
every
time
that
a
service
or
now
in
modern
ones,
the
endpoint
slice
is
actually
what
backs
a
service.
Whenever
an
endpoint
slice,
a
pod
comes
or
goes
from
an
endpoint
slice,
we're
updating
the
cache
within
coordinates,
so
we're
watching
and
we're
seeing
all
these
events
come
down
from
the
api
server.
A
A
I
don't
know
we
can
dig
into
it
later.
You
won't
see
them
for
raw
pots.
It's
got
to
have
a
service
backing
it
in
order
for
it
to
have
any
ptr
record
at
all
now
in
a
stateful
set.
If
these
pods
are
participating
in
a
staple
set,
they're
gonna
they're
gonna
have
that
service
created
automatically.
So
that
might
also
be
where
you
would
see
them.
A
A
E
A
Okay,
there
is
a
an
external
plug-in
called
well
at
one
point.
In
the
kubernetes
plug-in
we
had
a
configurable
option,
for,
I
believe,
naming
specific
namespaces
for
which
we
produce
records,
but
I'm
not
sure
if
we
had
one
for
excluding
them,
we
would
have
to
check,
but
we
do
have
a
policy
external
plug-in.
That's
the
one
I
was
showing.
We
called
it.
Firewall
was
directive,
but
policy
is
the
plugin
that
lets.
You
do
do
a
little
more
with
that,
but
that's
certainly
a
very
feasible
functionality
to
add.
A
C
C
B
B
Okay,
so
first
first,
first
of
all,
I
want
to
mention
one
thing:
you
know
like
dns.
Everyone
know
thing
is
simple.
So,
okay,
it's
dns
protocol,
which
is
udp
what's
the
old
stuff,
but
the
dns
is
one
very
important
feature:
dns
is
massively
scalable.
Your
whole
internet
is
because
it
supports
the
whole
internet
right.
That's
a
lot
of
people
talking
about
distributed
systems.
They
didn't
realize
that
dns
by
itself
is
a
distributed
system.
How
dns
is
able
to
handle
that
dns
when
you
support
the
internet,
it's
through
caching
at
different
levels.
B
A
Yeah
yeah,
no,
no
local
cache
will
probably
help
with
that
because
it
caches
the
negative
responses.
I
think
what
you're
talking
about
is
the
search
list
so
in
in
dns
the
client
side,
resolver
sitting
on
your
node
or
in
your
pod,
when
you,
when
you
ask
for
for
for,
say
google.com,
the
kubernetes
has
configured
a
a
list
of
names
that
it
tries
because
you
didn't
qualify
fully
with
a
dot
on
the
end.
So
the
first
thing
is:
if
you
control
those
names,
just
put
a
dot
on
the
end
and
the
problem
goes
away.
A
Okay,
but
you
can't
really
get
people
to
do
that
because
people
don't
do
it.
So
we
have.
We
actually
implemented
a
feature
a
few
years
ago,
specifically
to
to
address
this
problem.
There's
some
gotchas
with
it,
though.
First
of
all,
it
means
you
need
to
you
need
to
watch
pods,
because
in
order
to
make
it
work,
we
have
to
figure
out
which
namespace
the
pod
making.
The
request
is
in
in
order
to
understand
its
search
path
and
so
and
there's
a
race
condition.
A
If
the
pod
comes
up
and
we
don't
get
notified
fast
enough
about
the
pod,
then
internally,
we
can
run
into
a
problem
where
we
can't
preemptively
figure
out
your
search
path.
So
it's
it's
a
less
than
perfect
solution
to
that
problem.
The
node
local
dns
cache
probably
will
help
you,
because
I
believe
it
will
cache
those
negative
responses,
so
they
will
never
leave
the
machine
that
they'll
leave
the
node
but
yeah.
It's
unfortunately
not
a
fully
solved
problem,
and
you
know
that's
probably
your
best
option.