►
From YouTube: Elevating Trust with cert-manager in Hyperledger Fabric - Alexander Meijer, Corsha, Inc.
Description
Elevating Trust with cert-manager in Hyperledger Fabric - Alexander Meijer, Corsha, Inc.
Trust is a key issue in any network, but is perhaps of paramount importance in blockchain networks. It is so crucial, in fact, that "Trust" and its derivatives are mentioned no less than 14 times in Satoshi's seminal Bitcoin whitepaper. Hyperledger Fabric networks are no exception; in Fabric, trust flows from the Certificate Authorities (CAs) to every participant on the network who utilize that trust to transact according to their role. Operating these CAs properly, then, is central to the trust and security model of Fabric. In this discussion, Alex will reinforce why using centralized PKI generators (such as cryptogen) presents a trust risk for production networks. Corsha's experiences with the Fabric CA will also be discussed, and why it was ultimately decided that the operational overhead of that system made ongoing maintenance challenging. Alex will introduce the cert-manager project and go over how Corsha integrated that with their Kubernetes-native Hyperledger Fabric networks. Finally, Alex will propose a Fabric network architecture that does not rely on a trusted setup and does not require any CA servers, with fully automated certificate management.
Sessions Details & Slides (if available): https://sched.co/14H3z
Access All Sessions: https://events.linuxfoundation.org/hyperledger-global-forum/program/schedule/
----
Hyperledger – Open Source Blockchain Technologies
Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use
Discord: https://discord.gg/rVCn4j7UFd
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory
Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter
Follow-us on Twitter @Hyperledger
Learn about Hyperledger Membership: https://www.hyperledger.org/about/join
#Hyperledger #Blockchain
License
Creative Commons Attribution license (reuse allowed)
A
My
name
is
Alex
I'm,
corsa's,
head
of
infrastructure,
so
I
handle
things
like
our
infrastructure's
code
and
building
containers
and
everything
sort
of
of
that
nature,
and
today
I'd
love
to
discuss
sort
of
some
recent
work.
We've
done
as
part
of
sort
of
venify
development,
partnership
with
server
manager.
A
Essentially,
I
look
forward
to
demoing
at
just
sort
of
a
quick
poke
around
our
kubernetes
cluster.
We
can
view
a
one
of
our
staging
clusters
and
view
cert
manager
in
that
context
and
then
finally,
end
with
sort
of
some
extending
of
these
Concepts.
So
we
talk
about
how
we
push
search
manager
to
the
fabric,
Network
Edge
and
then
conclude
with
some
lessons
learned.
A
A
A
So
in
the
official
fabric
documentation,
there
are
two
primary
ways
of
provisioning:
these
certificates,
when
you
bootstrap
your
x509
pki
and
start
up
your
fabric,
Network
right
via
the
cryptogen
tool,
which
is
a
binary
command
line
and
then
via
the
fabric
CA
and
throughout
Fabrics
documentation
right,
they
refer
to
Concepts,
you
can
extend
when
using
other
Cas,
but
they
sort
of
say.
Well,
you
know
we
support
this.
You
can
use
them,
but
you
know,
there's
no,
no
documentation
on
them
and
so
we'll
explore
another
one
of
those
today.
A
So
to
sort
of
go
into
each
of
these
sort
of
off-the-shelf
tools,
it
does
document
quickly.
The
first
one
is
cryptogen
right.
This
is
sort
of
what
people
are
are
pushed
to
when
they
first
start
getting
involved
with
fabric
right.
It's
a
binary
program
runs
on
your
local
machine
and
it
generates
a
full
pki
right.
It
just
sits
on
a
on
a
directory
structure
on
your
machine
and
then
it's
basically
up
to
the
operator
to
manually
distribute
this
pki
to
all
the
different
components.
Right,
sometimes
it's
Docker
shared
volume.
A
Sometimes
people
are
manually
copying
things
over.
It's
very
simple
right:
it's
it's!
This
nice
manifest
you
run
and
you
specify
out
what
Cas
you
want,
whose
certificates
you
want
them
to
sign.
It
comes
at
a
pretty
steep
cost
right
and
weak
term.
This
trusted
setup
so
specifically
right.
It's
all
in
one
place.
So
if
I'm,
an
attacker
right,
all
I
need
to
do
is
basically
somehow
intercept
this.
This
bundle
of
certificates
and
private
key
material
and
I
own
your
network
right.
We
have
our
little
attacker
here.
A
Maybe
you
could
spin
out
phony
components
right
using
legitimate
pki
or
you
could
swap
out.
You
know
your
legitimate
certificates,
with
their
corrupt
or
otherwise
bad
bki,
and
just
one
of
many
examples
right
Beyond,
like
the
sort
of
Byzantine
implications
of
this
it
in
general,
makes
maintenance
difficult
right.
How
do
we
handle
a
certificate
rotation?
How
do
we
swap
that
out
right?
It's
a
lot
of
manual
steps
and
transferring
these
certificates
from
where
they're
generated
out
to
where
they're
actually
used,
and
it
raises
secondary
concerns
like
how
do
we
securely
store
these
certificates?
A
So
the
way
that
this
works
at
a
high
level
again
is
clients,
submit
a
certificate
signing
request
that
is
signed
by
their
private
key
right.
So
there's
a
huge
benefit
here
in
that
the
client
generates
this
key
pair
on
their
own
device
right
and
the
secret
key
never
leaves
the
client
for
the
entire
duration
that
the
client
interacts
with
the
network.
So
we
get
a
green
check
mark
here
that
addresses.
A
Probably
the
biggest
drawback
of
cryptogen
right
is
that
the
certificates
are
generated
in
a
distributed
fashion
and
the
way
that
a
client
sort
of
authenticates
itself
to
the
ca
says.
Please
sign
my
certificate.
You
know
it
says
it
provides
a
username
and
password
right
and
it
counts
on
someone
else
to
say.
Okay,
there
will
be
a
time
in
the
future
where
a
client
will
present
this
username
password
combination.
A
So
this
is
a
graphic
that
I
took
right
from
the
fabric.
Ce
documentation
itself
right
fabric
CA
is
designed
to
be
run
as
a
separate
service,
and
it's
pretty
much
a
standard
two-tier
web
app
right.
You
have
your
application
tier,
which
runs
the
business
logic
of
the
ca,
performs
the
actual
cert
signing
and
then
that's
backed
by
a
persistence
layer.
So
we
can
see
that
at
the
extreme
right
of
this
diagram
right,
so
you
can
back
it
with
either
an
ldap
database
or
various
rdbms's.
A
You
know,
fabric
sea
has
its
own
configuration
format
right.
It
has
the
concept
of
a
bootstrapping.
So
if
you
have
a
root
Cas
in
this
diagram,
the
intermediency
it
can
contact
the
root
CA
get
its
own
CA
search,
signed
and
then
participate
in
the
network.
And
again
this
is
all
handled
sort
of
asynchronously
through
a
config.
So
what
does
this
mean
that
you
know
you
if
you're
deploying
this,
and
you
want
to
use
this
in
prod?
Well,
you
have
to
follow
traditional.
A
You
know
service
deployment,
best
practices
right,
do
things
like
load
balancing
as
they
sort
of
reference
in
this
diagram
again
and
then
in
the
top
right
and
you
you
have
to
address
concerns
like
rdbms
scaling
right,
it
could
become
a
scaling,
bottleneck
and
again.
These
are
all
well
understood,
Technologies,
but
again
it
is
just
sort
of
things
that
you
need
to
keep
in
in
mind
and
keep
in
view.
A
So
what
does
this
all
mean?
I
mean
in
we've
discovered
that
this
is
hard
right
running
a
CA
is
hard.
There
are
many
large
companies
that
make
an
entire
business
out
of
doing
this,
like
verisign
that
you
serve
right
challenges.
There
are
always
challenges
around
rdbms's,
right,
right,
replicas
and
that
sort
of
thing
and
just
scale
in
general
right.
Do
you
provision
for
Peak?
Do
you
Auto
scale?
How
do
you
implement
that?
A
A
What
we
also
found
is
a
challenge
with
the
cas.
It
has
its
own
our
back
system
right.
It
uses
attributes
encoded
in
Json
and
x509
V3,
extended,
right
attributes
and
certificates
to
allow
you
know:
okay,
you're
allowed
to
sign
certs
for
this
type
of
a
client
you're
allowed
to
sign
search
for
this.
Who
can
create
usernames
passwords
right
and
it's
basically,
you
have
this
whole
other
layer
of
authentication
right
and
that
who's
allowed
to
sign
certificates.
A
On
top
of
your
existing
fabric,
enrollment
search,
we
found
that
you
know
the
bootstrapping
process
can't
really
be
extended
well
to
third-party
Cas
right.
The
concept
of
a
bootstrapping
sort
of
necessitates
the
ICA
be
talking
to
another
fabric
CA
at
the
root
right
at
the
bootstrapping
is
sort
of
a
unique
thing
here,
so
you
can't
talk
to
venify
or
you
can't
go
out
and
talk
to
like
digicert
or
anything
like
that.
To
get
your
certificate
using
the
bootstrapping
process,
specifically
in
terms
of
longer
term
maintenance
concerns
right.
A
So
that
begs
the
question:
well,
what
happens
upon
cert
expiration
right
you'd,
like
we
never
went
through
with
implementing
this,
but
just
sort
of
a
quick
thought.
Experiment
right:
you
need
to
detect
your
certificates
expiring
soon
perform
the
ciao.
The
CLI
client
call
right
and
then
somehow
swap
that
into
your
node
and
and
that's
tough
right.
We
did
it
manually.
So
it's
it's
truly
a
hard
challenge
and
we
had
to
tackle
it
because
we
care
a
lot
about
trust
right.
So
what
we
fundamentally
do
is
we
offer
a
security
product
right.
A
We
built
a
cyber
security
Solution
on
top
of
hyperledger
fabric
and
essentially,
what
we
offer
is
an
MFA
solution
for
apis.
So
we
have
a
little
diagram
here
of
the
the
dln
is
again
top
right
and
what
we
do
is
generate
unique
identities
per
API
client
that
we
then
validate
down
low
at
the
at
the
service
right
and
then
go
from
there.
A
What
this
means
is
that
our
deployments
are
typically
Mission
critical
right
if
you're,
a
customer
who
is
looking
for
this
level
of
security
for
your
API
clients,
a
lot
of
government
users
and
that
sort
of
thing,
so
we
need
to
drive
the
attack
cost
through
the
roof
right.
We
can't
have
one
rdbms,
because
you
know
there
are
some
well-resourced
adversaries
here
that
will
attack
any
single
point
of
failure.
A
So
just
a
quick
sort
of
aside
here
to
give
a
little
context.
We
deploy
fabric
on
on
a
kubernetes
cluster
using
a
tool
called
Helm
right.
So
fundamentally
we
use
one
kubernetes
namespace
to
deploy
the
core
fabric
components
along
with
some
license
management.
Stuff
and
again
we
have
our
fabric
maintenance,
job
scripted.
Some
some
are
kubernetes.
Cron
drops,
some
are
just
one-off
jobs.
A
We
have
rebased
all
of
the
fabric
containers
on
Department
of
Defense
iron
band
containers
to
minimize
our
attack
surface
there
and
all
of
this
to
deploy
multi-tenant
dlns.
So
we
have
on
the
left
a
distributed,
Ledger
namespace
that
will
be
shared
between
and
many
customers
again
all
separated
out
and-
and
you
know,
these
are
fronted
by
certain
small
microservices
that
again,
we
Implement
as
a
company
to
sort
of
facilitate
this
interaction
with
fabric.
A
So
on
top
of
this,
you
know
by
using
kubernetes
right,
we
gain
Cloud
agnosticism
right.
We
run
equally
well
on-prem.
There
are
customers
that
run
us
in
a
totally
air
gapped
environment
on
Virtual
machines
and
that
sort
of
thing
gcp,
AWS
Azure.
We
run
on
all
three
within
the
last
year
and
honestly
with
any
even
the
maybe
the
last
four
months
so
yeah.
A
This
is
sort
of
you
know
what
we,
what
we're
trying
to
protect
right
and
we
have
so
many
challenges
with
scaling
out
the
CA
in
particular,
as
we
scale
out
like
we've
done
some
research
work
recently,
where
we've
deployed
like
49,
49
peers
out
right
right,
and
it's
like
how
do
you?
How
do
you
handle
that
scale
and
manage
that
trust
right?
A
So
just
for
those
of
us
who
aren't
familiar
with
it,
cert
manager
is
a
open
source
project
and
what
it
essentially
does
is
it
signs
x509
certificates
using
pre-configured
backends,
which
in
certain
manager
parlance
is
called
an
issuer
here
we
in
this
graphic
like
we
have.
We
have
multiple
different
types
of
back
ends.
We
support
right.
The
ones
I've
listed
here
are
the
ones
that
it
supports.
Out
of
the
box.
There
are
tons
of
third-party
plugins
for
cloudflare
or
most
of
your
other
issuers.
A
Honestly,
you
know
if,
if
your
DNS
system
supports
the
Acme
protocol
right,
you
can
issue
search
that
way.
It
works
with
hash
record
Vault
right
and
it
works
with
all
of
these
simultaneously.
You
could
be
running
50
of
these
and
again
it's
a
form
of
indirection
right
and
on
the
bottom.
It's
issuing
certs
using
this
to
it,
can
issue
it
to
peers
fabric
clients
themselves,
we'll
get
into
specifically
how
that's
done
later
or
just
your
typical
web
service.
That
needs
an
fqdn
the
search
sign
for
to
serve
its
its
web
UI
or
something
like
that.
A
The
way
they
accomplish
this
is
cert
manager
uses
the
kubernetes
API
exclusively
right.
So
a
in
kubernetes.
Again,
it's
declarative
infrastructure,
right
resources,
declare
I,
need
a
certificate
and
they
specify
what
what
they
need
and
again
we'll
get
into
the
specifics
here.
Assert
manager
is
always
watching
the
cluster.
It
effectively
subscribes
to
the
kubernetes
API
server
and
as
soon
as
that
request
is
put
on
the
cluster,
it
does
the
back
end
work
right.
A
It
says:
okay,
you
know
this
peer,
wants
a
certificate
signed
by
this
CA
and
it
goes
it
handles
the
interaction
and
it
just
gives
it
back
to
the
to
the
peer
all
of
the
authentication.
Authorization
is
done
by
by
the
kubernetes
rbac
API
right,
so
none
of
that
is
implemented
in
cert
manager.
Kubernetes
gives
us
so
many
features
here
that
they
just
sort
of
delegate
that
out
there.
A
So
if
we
can
add
a
little
depth
here,
this
is
an
actual
certificate
object
that
we
that
I
pulled
off
of
one
of
our
clusters
right.
So
this
is
a
it's
a
custom
resource
definition
right
and
just
sort
of
going
top
down.
We,
we
specified
a
cert
manager.
How
long
before
the
cert
is
typically
expired.
Should
cert
manager
handle
the
renewal
right
beneath
that
I'm
sure
we'll
all
recognize
the
standard
x509
Fields
like
operate
organizational
units,
common
names,
expiration,
dates
right.
A
A
Serp
manager,
of
course,
supports
x509v3,
extended
attributes
so
for
an
enrollment
cert
right,
we
need
the
digital
signature
usage
and
then,
finally,
at
the
bottom.
This
is
the
linkage
to
tell
it
which
CA
we
actually
wanted
to
get
signed
right.
We
tell
it:
okay,
I
want
the
issuer
ca
issue
Oregon.
This
is
a
local
key
pair,
but
this
could
be
venify
right,
an
Acme
provider,
whatever
you'd
want
and
that's
it.
You
just
keep
cuddle,
create
this
object
and
and
it
works.
A
So
this
is
sort
of
a
architecture
diagram
of
how
trust
flows
now
through
one
of
our
production
configurations
using
this
sort
of
Paradigm
right.
So
we
root
all
of
our
trust
in
air
gapped
root
Cas.
So
this
is
just
offline
key
pair
that
never
sort
of
leave
that
never
hits
the
internet
and
is
never
live
in
the
cluster.
We
manually
sign
intermediate
CA
certs
from
those
we
use
a
little
go
binary.
A
We
wrote
internally,
but
you
can
use
open,
SSL
or
anything
and
those
are
what
are
given
to
cert
manager
and
a
secret
right
as
a
key
pair,
and
it
is
then
registered
as
a
CA
and
then
from
each
one
of
those,
and
we
have
three
orgs
in
our
production
networks
and
those
all
flow
down
into
the
peers.
We
run
our
orders
in
order
one
so
the
truss
flows
from
there
and
then
towards
the
bottom.
A
We
have
our
course
off
server,
which
is
how
we
again
run
rbac
and
authorization
on
clients
and
if
a
client
is
earns
its,
it
earns
its
Trust.
Basically,
we
issue
a
cert
there
and
the
trust
flows
through
there
and
just
a
quick
aside.
Our
consensus
policy
is
two
of
three
right,
so
we
can
tolerate
the
loss
of
any
one
or
cool,
so
yeah
now
I
think
we
have
time
for
a
quick
demo
we're
just
going
to
explore
a
cluster
or
one
of
our
staging
clusters.
So
hopefully
this
will
be
large
enough.
A
So,
first
we'll
look
at
one
of
our
production
clusters.
Actually,
this
is
in
staging
right,
so
this
matches
a
production,
cluster
topology,
but
we're
not
in
prod
right
now.
What
I
would
like
to
point
out,
too,
is
that
you
know
at
the
bottom.
We
have
our
nine
peers
right
all
running.
We
use
external
chain
code,
so
this
is
all
the
chain
code
that
backs
up
each
Pier.
A
A
An
issuer,
so
an
issuer
is
a
custom
resource
introduced
by
cert
manager,
and
here
are
our
three
Cas
backing
this
cluster.
Now
they
don't
exist
in
the
traditional
sense
right.
It's
just
sort
of
a
group
of
keys
that
tell
cert
manager.
Okay,
when
someone
wants
assert
for
orc0
use,
this
key
pair
sign
it
and
get
it
back
to
them.
Basically,
so
if
we
want
to
go,
look
at
cert
manager
itself,.
A
We
store
those
in
our
Ingress.
You
know
all
right
that
wrap
here
we
store
those
in
our
Ingress,
namespace
and
basically
sort
manager
consists
of
three
main
components.
There's
the
logic
itself
here.
This
is
the
ca
injector.
So
this
configures,
the
kubernetes
API
server
to
basically
trust,
serve
manager
and
to
send
all
of
its
objects
that
it's
creating
to
cert
manager
to
see
if
it
needs
to
be
signed.
A
These
are
waiting
there
and
they
are
listening
and
then
you
know
they
make
changes
to
the
object,
invoke
cert
manager
itself
or
whatever
else
is
sort
of
needed
to
get
to
get
taken.
Care
of
this
is
an
NHA
configuration.
So
we
we
are
scaled
out
and
you
know
we
can
sign
pretty
several
hundred.
You
know
certs
per
second
I,
think
honestly
in
this
configuration.
A
A
Cert
manager
is
what's
actually
modifying
these
statuses
right,
so
we
have
a
a
real-time
update
of
whether
the
certificate
has
been
signed
and
it's
available
and
ready
to
be
used,
and
then
we
finally
point
to
the
actual
kubernetes
secret
object
where
the
cert
data
itself
is
stored.
This
is
just
more
of
an
abstraction.
So
if
we
dig
in.
A
We'll
grab
this
or
we'll
grab
an
enrollment
shirt.
One
of
these
enrollment
sits
at
the
bottom,
actually
we'll
describe
it.
So
we
get
the
full
sort
of
range
of
kubernetes
API
commands
again.
This
is
all
implemented
in
a
custom
resource
definition,
so
we
get
basically
real-time
status
on
all
of
our
certificates
in
the
cluster
manager
has,
let
us
know
the
the
last
message
was
okay.
You
know,
I've
signed
the
certificate.
It's
not
expired.
It's
continuing
to
check
right.
We
have
the
time
it
expects
to
renew
the
certificate.
We
have
the
maximum
age
right.
A
We
see
all
these
sort
of
fields
that
we
declared
a
few
slides
earlier
in
our
definition
and
yeah
again
our
usages,
so
this
is
sort
of
the
abstraction
that
kubernetes
holds
and
that
the
cert
manager
binary
itself
is
watching
to
know
when
it's
time
to
renew
if
anything's
up
yeah,
so
the
benefits
of
cert
manager
aren't
just
in
this
sort
of
automation.
Here
right.
There
are
other
add-ons
that
again
work
with
this
binary.
It's
an
indirection
point.
So
one
of
the
things
that
we
utilize
at
caution
is
what's
called
jet
stack
secure.
A
So
these
are
the
makers
basically
of
cert
manager
itself,
and
what
this
is
is.
This
is
an
additional
sort
of
add-on
that
we've
put
into
our
shirt
manager
object
and
what
this
does
is
it
synchronizes
our
cluster
certificates?
Now
just
the
certificate
portions
right,
no
secret
information
leaves
the
cluster,
and
this
is
a
SAS.
So
now
we
and
our
security
team
can
get
a
full
sort
of
insight
here
into
every
certificate.
That's
in
our
cluster
right.
A
So
if
we
dig
in
here
to
one
of
our
shirts
right,
we
could
see
it
make
some
recommendations
right
in
terms
of
certificate
Health,
how
we
can
better
utilize
these
and
better
write
these.
Some
of
them,
you
know,
might
not
apply
right,
like
we
sort
of
have
to
use
ecdsa
for
fabric,
but
again
we
can
view
the
cert
metadata
we
can.
We
can
sort
of
Trace
through
who
issued
it
right.
So
here
are
cluster
issuers
right.
We
we
looked
at
these
a
little
earlier,
so
it
provides
us
with
this
sort
of
all-in-one
monitoring
tool.
A
A
Right,
yeah,
so
standard
production
deployment
yeah
this
dln
just
for
your
information.
We
sustain
about
100
transactions
per
second
right
on
this
2000
transaction
per
second
read
give
or
take
so
to
sort
of
carry
this
forward
right.
How?
How
do
we
sustain
this
over
long
periods
right
so
cert
manager
gives
us
a
lot
of
benefits
like
automatic
rotation
right.
We
support
a
number
of
CAS
it
completely
abstracts
CA
management
from
the
application
code
and
in
general,
just
improve
security
hygiene.
A
Can
you
know
reprovision
them
quickly
enough,
but
you
might
have
lost
consensus
and
and
you're
in
big
trouble.
So
I
just
wanted
to
include
this
little
bit
of
pseudocode
that
we
have
implemented
where
we
essentially
hash
the
name
of
every
individual
component,
Right
strip
out
anything
that
isn't
a
isn't
a
digit
to
get
a
pseudo-random
number
and
then
normalize
that
so
that
every
component
has
a
different
lifespan:
that's
somewhere
in
the
last
50
percent
of
the
certificate's
life,
so
that
different
parts
are,
you
know,
maybe
they're
getting
rotated
more
often
than
they
need
to.
A
A
So
as
we
sort
of
wind
down
here,
I
just
wanted
to
go
over
some
Concepts
around
extending
cert
manager
outside
of
the
kubernetes
cluster
right.
So
you
know
cert,
managers,
power
and
capability.
A
lot
of
that
lies
in
the
fact
that
it
runs
inside
of
Kate's
right,
but
it
it
is
totally
dependent
on
it
and
that's
nice
in
a
way,
but
the
question
is
and
that
we
faced
was
well.
You
know
a
fabric
network,
isn't
just
peers
and
orders
right.
People
are
using
it
right,
that's
how
it
delivers
business
value.
A
So
how
do
we
basically
push
this
capability
to
sign
certificates
out
of
the
cluster?
And
our
answer
to
this
was
well.
We
handle
it
in
our
app
right,
so
a
client
when
it's
first
booting
up
will
generate
its
own
CSR
and
we'll
submit
it
to
our
services
right.
So
that's
this
inner
dash
line
right
so
when
it's
first
booting
up
IT,
Supplies
CSR,
and
then
it
runs
through
our
control,
plane,
microservices
logic
fronting
this,
where
it
goes
through
sort
of
our
business
logic,
our
back
checking
right.
A
We
check
that
it's
allowed
to
get
a
certificate
signed,
and
then
we
we
pass
it
right
into
fabric
right.
Our
chain
code
has
a
cert
manager,
really
it's
just
a
kubernetes
client
in
it.
The
chain
code
is
responsible
for
submitting
this
Certificate
request
onto
the
cluster.
The
chain
code
then
Waits
until
the
signed
certificate
is
ready
and
then
it
follows
this
Green
Path
back
out
the
sign
certificate,
and
then,
at
that
point
we
have
a
signed
enrollment
cert.
We
can
interact
directly
with
fabric
services
and
we
did
not
have
to
expose
a
public
CA.
A
So
I'd
just
like
to
conclude
here
with
some
lessons
we
learned
through
the
through
this
effort.
We
have
been
running
this
cert
manager,
integration
in
prod
for
during
close
to
a
year
now,
maybe
10
months.
So
you
know
we
found
that
you
know
fabric
components,
read
their
certificates
once
at
start
right.
So
it's
not
like
an
Ingress
controller
or
something
where
you
can
swap
the
contents
of
the
file
system
and
it
will
immediately
get
used
in
the
logic.
This
makes
automated
cert
rotation
challenging
right.
A
You
need
to
figure
out
how
to
basically
get
this,
get
this
application
to
read
its
updated
certs,
which
are
just
silently
updated
in
the
background
by
cert
manager.
So
to
get
around
this,
we
just
use
an
off-the-shelf
tool
called
the
reloader
Helm
chart
and
what
it
does
is
it
watches
all
the
secret
information
that's
attached
to
a
pod
and
when
anything
changes
it
it
bounces
the
Pod
and
restarts
it,
and
then
that
that
solved,
that
problem,
cert
manager
is
a
best
effort
service
right.
A
They
don't
really
do
a
blocking
Opera
duration
or
anything
like
that
when
you
want
a
certificate
signed.
So
what
we
have
to
do
is
sort
of
a
two-phase.
Here
we
submit
the
certificate
and
then
we
pull
or
we
submit
the
Certificate
request,
and
then
we
pull
until
the
certificate
is
signed
and
ready
to
come
back.
A
Sir
manager
supports
raw
pem
right.
You
can.
If,
if
the
API
doesn't
support
exactly
the
field,
you
need,
you
can
just
give
it
a
raw
Certificate
request,
pen
and
that'll
get
signed.
And
lastly,
if
you
manually
create
certificate
requests,
so
it's
not
matching
a
certificate
object,
which
is
what
we
do
in
the
slide
before
for
outside
clients
that
are
outside
of
kubernetes,
make
sure
to
clean
them
up.
Cert
manager
will
dutifully
monitor
them
and
keep
them
up.
A
To
date,
even
if
the
client
has
long
since
come
up,
you've
done
its
business
and
and
stopped
existing,
so
we
had
we
eventually
wind
up
with
thousands
of
these
certificate
requests
knocking
around
so
just
make
sure
to
clean
those
up.
We
we
implemented
just
a
tiny
little
Cron
job
to
take
care
of
that,
and
that's
it.
For
my
end,
you
know
happy
to
handle
any
questions
and
yeah.
Thank
you
again.
A
A
I,
don't
see
any
reason
why
cert
manager,
you
couldn't
use
it
to
sign
an
intermediate
CA
cert
from
a
root
CA,
we
just
sort
of
do
it
manually.
That
is
a
deliberate
business
decision
to
keep
that
a
manual
process,
because
the
root
is
sort
of
the
root
of
everything
right
and
that
way
we
don't
have
a
root
CA
that
ever
goes
live
on
the
cert,
but
I
mean
it's
an
incredibly
flexible
tool
right
and
at
the
end
of
the
day,
it
is
signing
for
x509
certificate,
so
I'm
sure
you
could
chain
it
yeah.