►
From YouTube: 9-1-1: Cloud Native - Kubernetes For Public Safety Systems - Marc Boorshtein, Tremolo Security, Inc.
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
9-1-1: Cloud Native - Kubernetes For Public Safety Systems - Marc Boorshtein, Tremolo Security, Inc.
Speakers: Marc Boorshtein
This session is a case study on using Kubernetes for the US National Capital Region’s Identity and Access Management System (IAMS), managed by Fairfax County VA. IAMS provides secure access to applications for first responders and public safety personnel in the DC region. This session will walk through IAMS’ ten year journey from “virtual appliances'', to trying docker containers, and ultimately implementing Kubernetes. The case study will cover the program’s journey and how it managed issues around automation, scalability, and cost. The case study will work through how the program approached automating builds, updating dependencies, moving from stateful log management to stateless log management and the program’s move to GitOps covering how it manages multiple environments and secrets. Finally, the case study will also address the challenges in particular to building an automated system in a public safety environment where there is a diverse mix of cloud native and legacy systems.
A
Hey
everybody
last
session
of
the
day,
so
I
appreciate
you
coming
out.
My
name
is
Mark
Borstein
I'm,
the
CTO
of
tremel
security
and
we're
going
to
be
talking
about
kubernetes
in
Public
Safety
Systems.
What
to
what
were
some
of
the
things
that
we
had
to
do
that
were
different
from
maybe
your
typical
Enterprise
and
our
journey
over
the
course
of
about
a
decade
from
running
on
Virtual
appliances,
all
the
way
to
containers.
A
So,
like
I
said,
my
name
is
Mark
borsteen
I'm,
the
CTO
at
tremo
security
I've
got
20
plus
years
of
identity
management.
Experience,
I've
been
working
in
kubernetes
world
since
1.3
with
openid
connect
and
are
back
coming
out.
So
if
you've
ever
like
read
the
documentation
for
oidc
on
the
kubernetes
website,
I
helped
redo
that
a
few
years
ago,
I'm
also
the
co-author
of
kubernetes
and
Enterprise
guide
second
edition
and
contributor
to
multiple
projects,
kubernetes
the
dashboard
kiali.
A
But
you
don't
really
want
to
hear
about
me.
Let's
talk
about
the
case
study
here.
So
what
is
the
national
capital
region
IMS
program?
Im
stands
for
identity
and
access
management.
It
is
a
identity
service
for
the
Washington
DC
area
that
provides
identity
and
access
for
regional
systems
and
applications
across
22
jurisdictions.
So
in
DC
we
have
local
governments.
We
have
state
governments
in
Maryland
and
Virginia.
We
have
the
D.C
government
and
we
also
have
the
U.S
federal
government
and
the
U.S
federal
government.
A
What's
kind
of
interesting
about
our
platform
is
that
we
do
both
authentication
and
authorization,
so
users
can
log
in
authenticate
using
their
jurisdiction
credentials,
but
they
can
also
request
access
for
applications
in
a
way
that's
independent
of
the
jurisdiction.
So
it
takes
a
little
bit
of
the
load
off
the
jurisdiction,
I.T
systems,
the
applications
that
are
integrated
with
IMS
kind
of
run,
a
Gambit
from
content
management
and
collaboration
systems,
SharePoint
sites,
WordPress
sites,
Drupal
sites,
Etc
all
the
way
to
emergency
response.
A
Agencies
will
use
our
system
for
authentication
to
get
Regional
folks,
like
folks
from
universities
Etc
into
their
systems
during
a
crisis.
So
we
run
that
entire
Gambit
from
the
whole
content
management
all
the
way
to
Emergency
Management
and
we've
been
in
production
since
2013..
So
we've
got
quite
a
nice
long
journey
and
the
entire
platform
is
built
on
top
of
tremo
Securities
open
Unison.
So
it's
all
open
source
trouble
security.
A
There
were
a
lot
of
lessons
learned
things
like
you
know:
the
regions
need
to
have
interoperable
systems,
radios
need
to
work
together
and
one
of
the
things
that
came
out
of
it
was
this
attempt
to
build
a
network
that
would
survive
an
outage
on
the
commercial
internet.
So
if
the
commercial
internet
went
down,
Public
Safety
officers
were
still
able
to
communicate
with
each
other.
This
network
was
built
and
folks
didn't
want
to
put
any
applications
on
because
they
didn't
want
to
have
yet
another
username
and
password.
A
So
that's
where
we
came
in
so
when
we're
talking
about
Public
Safety,
the
priorities
are
a
little
different
than
you
might
be
used
to
in
your
typical
Enterprise.
The
first
is
availability.
You
might
think
security
is
the
first
one,
but
it
isn't
because
a
lot
of
the
information,
if
you
just
have
a
radio
you
can
hear
so
what's
far
more
important,
is
making
sure
the
system
is
up
and
available.
A
As
an
example,
you
get
arrested,
your
mug
shot
goes
into
a
database.
There
are
a
lot
of
security
controls
around
that
to
maintain
your
privacy.
These
controls
are
important.
The
ACLU
comes
up
in
almost
every
conversation
when
we
start
talking
about
how
we're
going
to
manage
access
to
these
things
and
making
sure
that
everything
is
kept
above
board.
So
we
have
to
be
very
mindful
of
that.
Now,
of
course,
security
does
play
into
this.
You
can't
really
have
a
good
privacy
system.
A
If
you
don't
have
a
security
system
built,
that's
built
on
top
of,
and
so
we
use
MFA.
We
use
open,
Unison
itself
to
control
access
to
the
Clusters
and
the
finally
performance.
It's
got
to
perform,
it's
got
to
be
usable,
but
we're
not
talking
hyperscale.
There
are
about
10
000
users
across
the
region.
Maybe
a
few
hundred
at
any
given
time
are
logged
in
so
it's
got
to
be
available,
but
you
know
we're
not
exactly
worried
about.
You
know
serving
millions
of
people.
A
So
what
are
some
of
the
challenges
this
bring
about?
The
first
is
we
have
to
deal
with
a
lot
of
Legacy
systems.
If
it
ain't
broke,
don't
fix.
It
has
a
really
special
meaning
in
the
public
safety
world
because
of
the
need
to
maintain
availability,
been
a
slow
space
to
get
into
the
constant
movement
of
constantly
upgrading,
constantly
keeping
things
patched.
A
There
are
always
exceptions
to
the
rule,
so
as
an
example,
the
police
applications
we
work
with
the
first
time
thing
they
say
is
we
want
to
make
sure
only
sworn
officers
have
access
and
the
second
thing
they
say
is
well
actually
crime
analysts
need
to
be
able
to
access
it,
they're,
not
sworn
officers,
so
we
have
to
be
able
to
manage
to
that.
There
are
silos
of
silos.
You
have
every
jurisdiction,
maintains
a
silo
for
the
police
department
for
Fire
and
then
even
inside
police
departments.
A
So
we've
been
talking
a
lot
about
what
Public
Safety
is
like:
let's
get
into
the
fun
part,
which
is
technology.
So
when
we
originally
launched
in
2013,
we
were
running
in
Virtual
appliances.
We
thought
it
would
be
great
to
just
have
a
ISO
with
the
system
on
it
drop
it
into
a
VM,
get
up
and
running
and
have
it
work,
and
nobody
was
going
to
log
in
to
put
anything
on
it
and
that
lasted
about
15
minutes.
Until
we
were
told
no,
we
need
SNMP
on
there.
We
need
this
agent
on
there.
A
We
need
that
agent
on
there.
So
what
was
a
virtual
Appliance
really
just
became
another
VM.
It
was
a
mix
of
Windows
and
Linux,
so
our
product
was
running
on
Linux.
The
jurisdictions
wanted
to
be
able
to
take
ownership
of
it
once
it
was
up
and
running
so
the
most
jurisdictions
local
jurisdictions
run
on
windows,
so
they
want
things
to
be
running
on
IIs
SQL,
Server
active
directory.
A
After
a
few
years
we
decided
to
migrate
off
of
that
platform.
We
were
able
to
migrate
to
a
different
data
center
that
data
center
made
a
little
bit
easier
to
run
VMS,
and
so
we
moved
still
in
Virtual
appliances,
but
everything
was
Linux.
We
got
rid
of
IIs,
we
got
rid
of
SQL
server
and
moved
over
to
mySQL
or
actually
was
Maria
DB
with
percona
for
high
availability.
A
We
had
set
up
a
free
IPA
service
to
manage
authentication
authorization
into
our
servers
and,
for
the
most
part,
worked
fine,
and
then,
at
that
point
we
was
when
Docker
was
starting
to
get
a
little
popular
and
we
said
all
right
we're
going
to
go
ahead
and
try
using
Docker
containers
for
all
this,
and
we
made
every
mistake
you
could
possibly
make
with
containers.
We
treated
it
like
a
VM.
We
didn't
really
know
how
we
wanted
to
handle
persistent
storage.
We
didn't
have
a
good
way
to
handle
networking
or
scheduling.
A
A
Well,
it
took
a
little
longer
than
we
had
hoped
and
we
ended
up
moving
to
the
public
cloud
in
2019..
We
didn't
have
as
long
as
an
on-ramp
as
we
hoped
we
would
so.
Instead
of
doing
kubernetes,
we
first
went
to
ansible
managed
VMS,
which
we'll
talk
about
and
then
finally
we
did
make
it
into
kubernetes.
A
So
the
first
major
move
was
virtual
appliances
to
ansible
our
original
product.
Unison
was
not
open
source,
it
had
a
GUI
for
configuring
things,
and
so,
when
we
created
open
Unison,
we
got
rid
of
the
GUI
went
to
configurationist
code.
People
like
that.
A
lot
more
was
a
lot
easier
to
manage
and
we
externalized
the
environment
configuration
so
now
we're
able
to
generate
something
that
kind
of
looked
like
a
container.
It
wasn't
a
container,
but
it
was
very
easy
to
deploy.
We
were
able
to
use
ansible
playbooks
and
it
worked.
A
You
know
we
were
able
to
take
a
VM
and
get
a
a
system
up
and
running
in.
You
know
five
six
minutes,
which
is
fine
for
us,
the
other
major
change
here
was
we
changed
from
a
legacy
monitoring
solution
that
the
region
had
been
using.
We
we,
as
a
company
tremo,
had
taken
over
management,
and
so
we
wanted
to
do
our
own
monitoring,
we'll
talk
about
that
and
we
did
all
of
our
patching
via
ansible.
A
So
when
it
came
to
monitoring,
we
didn't
have
the
capacity
to
just
start,
deploying
Prometheus
and
getting
it
up
and
running
we're
a
little
bit
budget
constrained.
So
what
we
decided
to
do
was
we
were
going
to
run
Prometheus
and
grafana
inside
of
tremolo's
cloud,
but
do
all
the
scraping
against
are
in
data
center
or
in
the
the
other
Cloud
VMS.
A
So
what
we
decided
to
do
was
we
had
on
the
left
there.
That's
the
tremolo
Cloud.
We
would
have
a
Prometheus
instance
scrape
open,
Unison,
open
Unison,
we'll
put
a
message
on
an
Amazon
sqs
bus.
They
get
picked
up
inside
the
cloud
open.
Usm
would
then
scrape
all
the
open
unisins
inside
of
the
network,
compile
all
the
responses
put
it
back
on
the
bus,
send
it
back
up
to
the
cloud
and
then
the
open
Us
in
the
cloud
we
give
it
back
to
grafana.
A
A
All
of
the
security
rules
across
the
regions
are
all
still
built
on
firewalls
and
IPS
and
adding
so
that
was
a
bit
of
an
interesting
conversation
with
the
security
folks
and
they
they
saw
what
we
were
doing
and
they
were
able
to
sign
off
on
it,
and
then
we
had
again
the
CI
CD.
This
is
where
we
really
started
to
have
some
fun,
because
what
went
from
generating
what
was
essentially
a
executable
War
file
now
became.
A
A
So
let's
talk
about
our
pipeline,
so
tremble
security
manages
multiple
base
images
in
our
Docker
repo
and
we
keep
them
up
to
date.
We
use
gripe
from
anchor
to
mon
to
scan
those
containers
every
single
night
and
if
canonical,
all
of
our
images
are
built
on
Ubuntu.
So
if
canonical
releases
Ace
patch
for
a
cve,
we
get
triggered
to
go
ahead
and
rebuild
the
container.
That
rebuild
then
triggers
another
rebuild
to
go
ahead
and
build
our
NCR
containers.
A
So
we
execute
a
web
hook
that
triggers
a
patch
into
a
GitHub
repo
that
stores
the
application
code.
That
then
triggers
the
cloud
CI
CD
pipeline
to
do
a
rebuild
that
rebuild
then
patches,
the
API
server
and
the
test
environment
to
make
sure
to
go
ahead
and
load
the
new
container,
and
because
we
have
continuous
monitoring
that
tells
us
that
everything
is
still
working
once
we're
happy
that
everything
is
still
working.
We
go
ahead
and
update
our
production,
API
server
with
the
new
tags
to
run
the
new
version.
A
Yeah,
we
all
know
public
cloud
data
centers
do
go
down,
doesn't
happen
often,
but
it
does
happen,
and
so
we
wanted
to
be
able
to
fail
over
to
a
different
data
center.
A
different
region
entirely,
and
we
didn't
want
to
have
to
maintain
two
different
clusters
from
nothing
so
get
Ops
seems
like
the
right
approach
well
before
we
could
get
to
get
Ops
working
backwards
here.
A
A
So,
let's
talk
about
Secrets
management,
you
need
to
be
able
to
externalize
your
secrets
for
get
Ops.
Secrets
should
never
go
and
get
I
know
there
are
projects
out
there,
like
sealed
secret
for
encrypting
secrets.
The
problem
with
that
approach
is
a
couple
of
fold.
The
first
is,
if
you
git
is
specifically
designed
to
be
easily
distributed
and
to
lose
track
of
it,
and
so,
if
I
download
my
git
repo
I
pull
my
git
repo
accidentally
I
push
it
up
to
gitlab
or
GitHub.
I
have
now
lost
control
of
it.
There's
nothing.
A
I
can
do
at
that
point
other
than
try
and
go
delete
it,
but
once
it's
out
there
there's
no
centralized
control
you've
out.
You've
now
lost
control
of
it.
So
that's
issue
one
issue:
two,
the
big
point:
the
biggest
value
of
Secrets
Vault.
Isn't
the
fact
that
you're
walking
down
the
secret
itself
I
mean
it's
good
to
lock
that
down.
A
Don't
get
me
wrong,
but
it's
that
you
now
have
control
over
who's
accessing
it
and
you
have
control
over
or
visibility
and
who's
accessing
it
and
the
ability
to
quickly
rotate
it,
which
you
can't
do
if
you're
using
encrypted
secrets
in
git,
if
you're
using
encrypted
secrets
in
git.
Now
you
have
a
key
that
you
have
to
maintain
and
if
that
key
gets
lost
and
you
need
to
rotate
it,
you
have
to
go
to
find
each
secret
that
is
encrypted
by
that
key.
A
So
I
understand
why
people
do
it,
but
for
operations
it
we
were
not
going
to
do
it.
Another
point
about
using
vaults.
We
wanted
to
use
a
vault
as
a
centralized
database
for
Secrets,
but
we
were
not
worried
about
using
kubernetes
secrets.
It's
a
common
API.
All
the
systems
we
use
know
how
to
use
it,
and
so
kubernetes
secrets
were
fine
and
the
blog
post
I
linked
here
came
out.
A
Early
in
2022
does
a
really
great
job
of
looking
at
the
different
potential
threats
to
secrets
and
kubernetes
and
explaining
why
it's
fine
to
put
your
secrets
for
the
most
part,
into
etcd,
most
of
the
value
that
you're
going
to
get
out
of
Vault
beyond
what
putting
the
secrets
synchronizing
the
secrets
into
etcd
would
get.
You
would
be
defeated
by
automation
anyway,
especially
now
that
kubernetes
has
a
way
to
provide
an
identity,
to
workloads,
to
talk
to
your
vaults
without
having
to
have
a
specific
password
or
unlock
things.
A
So
we
decided
to
go
with
the
CSI
Secret
store
driver.
There
are
other
projects
that
let
you
synchronize
from
a
vault
into
kubernetes
secrets.
We
like
this
one,
because
it
was
part
of
cncf
the
kind
of
trick
with
it
was
that
you
still
had
to
have
the
secrets
mounted
to
a
pod,
even
though
you
were
generating
secrets,
so
that
was
a
little
bit
fun
to
figure
out.
A
So
first
thing
we
did
was
we
wrote
a
script
to
convert
our
existing
secrets
to
a
vault?
We
automated
it.
So
we
wrote
a
script
that
went
through
each
secret
generated
the
correct,
CSI
object.
You
know
the
the
CSI
object
is
kind
of
standard,
but
each
implementation
of
the
CSI
secret
driver
every
vendor
has
its
own
implementation,
so
the
Secret
store
driver
for
Amazon
has
different
things
in
the
object
than
the
Secret
store
driver
for
Vault
for
AKs
Etc.
A
So
it
it's
a
little
bit
of
hard-coded
from
that
perspective,
but
once
we
did
that
we
were
able
to
connect
our
cluster
to
Vault,
generate
the
correct
objects,
and
then
we
had
to
synchronize
the
objects,
the
secrets
into
our
cluster,
so
the
way
that
we
handled
that
was,
we
used
the
pause
container
or
the
pause
image.
It
hasn't
changed
in
like
six
years
I
think
so
we
created
an
appointment
with
pause.
A
It
takes
almost
no
resources,
don't
have
to
worry
about
patching
it
that
would
Mount
the
secrets
internally
and
that
would
then
get
the
CSI
Secret
store
driver
to
synchronize
the
secrets
into
our
the
Secrets
From,
the
Vault
into
secret
objects
into
etcd.
So
that
way
we're
getting
our
cake
and
eat
it
too.
A
So
once
we
had
our
secrets
figured
out
and
working,
then
we
could
start
having
fun
with
Git
Ops,
and
the
first
thing
we
learned
very
quickly
is
that
eventual
consistency
is
a
lie.
The
reason
I
say
that
is,
if
you've
ever
run
a
Helm
chart
and
a
failed
kind
of
halfway
through
Helm
doesn't
try
again.
You
know
you've
got
to
make
the
decision
to
do
that.
A
My
original
thought
was:
we
were
going
to
use
Helm
for
everything
and
just
have
a
different
values:
file
for
each
environment.
We
ended
up
deciding
to
go
with
environment.
Specific
repos,
though
our
Dev
environment
is
much
more
complex
in
our
production
environment.
So
I
talked
about
22
jurisdictions
across
the
region
in
Dev.
I
have
to
replicate
that
so
that
we
can
test
and
make
sure
that
things
work
before
we
push
it
into
prod.
A
Those
are
the
that
infrastructure
that's
needed
to
provide
those
testing
identities
doesn't
need
to
exist
in
production,
we're
not
going
to
sink
it
into
production,
so
we
decided
to
go
with
the
the
separate
repos
and
that
made
for
a
really
easy
migration
from
Dev
to
prod
and
for
get
Ops
controller.
We
decided
to
go
with
Argo
CD
using
the
app
of
app
pattern.
A
Some
of
the
things
that
we
had
to
that
were
interesting
to
deal
with
was
learning
how
to
use
waves
properly
in
Argo
CD,
so
making
sure
that
I
talked
about
sequences.
Yeah
config
Maps
were
deploy.
Certain
config
Maps
were
deployed
before
deployments
were
deployed
or
making
sure
that
certain
CRS
were
deployed
before
the
relying
deployments
were
there.
So
that
was
an
interesting
Journey,
seeing
how
all
that
fit
together.
A
So
once
we
had
excuse
me
so
once
we
had
our
environment
ready
to
go,
the
next
thing
we
had
to
do
is
move
all
of
our
configuration
out
of
our
old
cluster
into
our
new
cluster,
and
there
are
surprisingly
few
tools.
That'll.
Let
you
do
that
if
you
have
a
cluster
and
you
use
like
the
coupe
control,
get
all
plug-in.
That
literally
just
gets
everything
from
a
namespace.
A
A
Then
we
also
wanted
to
strip
out
all
of
the
metadata
information
that
was
very
cluster
specific
uuids
resource
versions,
the
The
annotation
that
stores
all
of
the
State
Management.
So
you
can
do
Coupe
control
apply
it's
a
blanking
on
it
right
now,
so
I
could
not
find
a
tool
that
would
do
it.
So
I
wrote
One,
there's
the
URL
for
it
you're
welcome
to
use.
It
is
not
a
polished,
CLI
tool,
but
you're
perfectly
welcome
to
to
use
it
and
what
it
does
is.
A
So
some
of
the
challenges
we
ran
into
with
Git
Ops
were
cicd.
So
when
we
talked
about
our
CI
CD
pipeline,
previously
we
were
using
the
the
cloud
vendors
CI
CD
system
to
build,
generate
the
container
and
then
patch
the
API
server
deployment
to
go
ahead
and
roll
out
the
new
container.
Well
in
the
git
Ops
World.
We
don't
want
to
do
that.
A
We
want
to
patch
a
git
Ops
repo,
so
most
of
the
CI
CD
platforms
out
there
make
it
really
easy
to
pull
from
a
single
repo,
but
if
you're
going
to
pull
and
write
to
a
different
repo,
you
basically
have
to
do
it
manually
as
if
you're
sitting
at
your
laptop.
So
that
was
a
little
interesting
trying
to
figure
out
how
to
get
that
to
work.
A
One
thing,
that's
really
good
is
you
can
use
the
coupe
control
patch
against
the
local
manifest
to
do
the
patching,
so
you
don't
have
to
worry
about
using
set
or
graph
or
awk
or
any
of
the
other
text-based
tools
to
update
it.
You
can
do
Cube
control
patch
the
same
way.
You
just
do
it
locally
against
the
yamla.
The
yaml
file
in
your
manifest
monitoring
was
a
little
interesting
when
we
moved
to
kubernetes.
We
stopped
using
the
tremolo,
hosted
monitoring
solution
and
we
moved
instead
to
prom
stack
Prometheus
stack
running
inside
the
cluster.
A
The
issue
we
ran
into,
though,
was
that
Prometheus
Tech
isn't
really
designed
to
be
customized,
and
so
we
had
specific
dashboards
and
specific
metrics
that
we
wanted
to
track,
and
so
we
ended.
Actually,
we
ended
up
writing
a
script
that
would
download
The
Source
from
git,
make
our
changes
and
then
one
on
a
lot
of
them
and
then
bring
that
in
to
get,
and
that
worked
pretty
well
that
actually
wasn't
as
bad
as
I
thought.
A
It
was
going
to
be
already
talked
about
leveraging
waves
in
Argo
and
then
the
last
is
you
know
once
once
the
repo
is
updated,
making
sure
that
Argo
is
synchronizing
in
the
proper
order.
So
we
ended
up
writing
some
scripts
to
do
that,
I'm.
Looking
at
doing
something,
maybe
with
Argo
workflows,
to
make
that
a
little
bit
more
automated.
A
Yeah
and
that's
pretty
much
it
I'm
going
to
do
a
little
Shameless
self-promotion
here
you
can
find
me
on
Twitter,
either
myself
or
terminal
security,
and
we've
got
a
20
discount
on
kubernetes
and
Enterprise
guide.
Second
edition:
you
get
it
off
of
Amazon,
that's
what
that
QR
code
is
for
and
you've
got
the
the
code
there.
Any
questions.