►
From YouTube: OCB: Microsoft SQL Server Big Data Clusters on OpenShift with Tushar (Red Hat) & Mihaela (Microsoft)
Description
Livestream version
Big Data Clusters (BDC) is a new set of capabilities introduced in SQL Server 2019 that allows customers to achieve data driven business insights from the high-value relational data, and high-volume big data. With BDC, Microsoft allows customers to run containerized Apache Spark and Hadoop Data File System (HDFS) natively as part of SQL Server 2019, in addition to running relational databases, Microsoft Machine Learning capabilities, and Polybase data virtualization. BDC requires Linux containers & Kubernetes. Microsoft now supports OpenShift Container Platform, Azure Red Hat OpenShift (ARO) and Microsoft’s AKS service in Azure. In this session, we'll discuss BDC use cases and solution architecture.
Full OpenShift Commons Briefings playlist:
https://www.youtube.com/watch?v=OcIZ3SfHyFY&list=PLaR6Rq6Z4IqdIM7LtosKqi3LlYXyxjwnj
https://commons.openshift.org#join
https://commons.openshift.org/events.html
A
All
right
welcome
to
another
openshift
commons
briefing.
My
name
is
beverly.
Now
this
week
we
have
red
hats,
tushar,
kartaki
and
microsoft,
mihala
joining
us
to
discuss
big
data
clusters,
which
is
a
new
set
of
capabilities
introduced
in
sql
server
2019
that
allows
customers
to
achieve
big,
data-driven
business
insights
from
high-volume,
big
data.
It's
important
to
note
that
bdc
requires
linux
containers
and
kubernetes,
and
microsoft
now
supports
openshift
container
platform,
azure
red
red
hat
openshift
and
microsoft.
A
Aks
service
in
azure,
tushar
and
michaela
will
discuss
the
solution
architecture
as
well
as
delve
into
a
relevant
use
case,
so
you
want
to
stick
around
for
the
entire
session.
Of
course,
if
you
have
questions
feel
free
to
add
them
in
the
chat
wherever
you're
watching
this
live
stream
from
and
we'll
relay
them
back
here.
With
that
I'll
hand
it
over
to
our
presenters
to
share.
B
B
For
you
know
everything
from
analytics
to
you
know
ai
and
machine
learning.
So
you
know
I'm
happy
to
have
with
me
mlandia,
who
is
a
principal
program
manager
at
microsoft,
and
she
is
going
to
do
most
of
the
talking
since
she's
the
expert
on
this
topic.
So
we
are
excited
to
have
her
and
I'll.
I
and
chris
and
others
will
monitor
the
chats.
If
you
have
any
questions,
please
do
put
them
in
the
chat
and
then
we
can
perhaps
have
some
time
in
the
end.
A
C
C
I'm
excited
to
join
the
session.
Today.
We
will
start
by
studying
some
context
with
today's
landscape
when
it
comes
to
collecting
and
using
big
data
that
will
later
lead
to
the
solutions
that
we're
actually
gonna
gonna
present
involving
six
simple
server,
big
data
clusters
and
openshift,
then
we
will
show
you
how
this
technology
solves
some
of
the
challenges
organizations
face
today
when
it
comes
to
analyzing
their
data
and
how
they
can
do
this
in
a
scalable
and
secure
way.
C
Also,
with
ever
increasing
sources
of
data
organizations
need
to
find
solutions
to
store
and
analyze
multiple
and
diverse
sources
of
data,
and
that
is
stored
again
in
various
formats,
and
there
are
other
too
many
enterprises
out
there
that
do
not
have
implemented
systems.
Historical
tracking
systems
with
bi
reporting
on
top
of
that
or
systems
that
are
collecting
signals
from
all
kinds
of
iot
devices.
C
Like
sensor
data
from
medical
trackers,
for
example,
departments
within
the
organizations
have
their
own
data
to
store
and
access,
they
have
data
mods,
they
depend
on
and
and
so
on.
So
it
is
more
obvious
each
day
that
not
keeping
up
with
the
taking
advantage
of
all
this
technology.
Technological
innovation
in
the
big
data
space
and
transform
that
data
into
meaningful
insights
will
be
a
competitive
disadvantage.
C
Resourcing
and
application
data,
warehousing
and
machine
learning
for
revenue,
projection
and
so
on,
and
then
there
is
the
length
of
stain
analysis
that
has
obviously
a
deep
advanced
analytics
aspect
to
it
and
we're
actually
going
to
touch
on
this
specific
use
case.
A
bit
more
in
details
later
today
and
same
pattern
applies
for
all
the
scenarios
you
see
here.
There
are
definitely
others
and
we
put
out
there.
C
But
you
know,
building
and
managing
and
using
an
iot
solution
is
not
at
all
easy.
One
of
the
main
solution
challenges
are
related
as
expected
to
to
data
itself,
as
I've
already
mentioned
in
time
and
more
since
hadoop
came
into
market
as
well
as
in
you
know,
most
recent
past
the
requirements
around
analyzing,
a
higher
volume
with
an
increased
velocity
of
data,
had
a
significant
boost
as
well.
C
They
give
you
the
choice
of
the
software.
You
want
to
run
without
the
infrastructure
dependencies
and
they
also
enable
easy
sharing
of
the
software,
enable
automation,
versioning,
easy
portability
across
various
types
of
infrastructure
like
physical,
virtual,
public
and
private
and
so
on.
Then
you
need
obviously
a
solid
data
platform
that
you
can
deploy
on
top
of
this
infrastructure
and
that
leverages
the
platform
capabilities
in
terms
of
scale
not
only
scale
but
also.
C
C
Orchestrated
by
kubernetes
and
can
be
deployed
on
premises
or
in
the
cloud
and
on
the
infrastructure
side,
as
mentioned
before,
red
hat
open
shift
is
supported
by
bdr
clusters
and
it
provides
this
end-to-end
commercially
distributed
infrastructure
based
on
we
recently
early
summer.
Actually,
we
announced
support
for
sql
server
clusters
deployments
on
openshift
on
premises
as
well
as
in
in
the
public
cloud
environments
with
azure
red
hat
openshift.
C
This
announcement
that
we
made
in
this
effort
is
the
outcome
of
a
close,
very
close
partnership
between
the
sql
server
team
and
red
hat
open
shift
teams.
We
collaborated
very
well
on
anything
from
the
design
and,
more
importantly,
on
on
the
security
capabilities
that
we
are
going
to
discuss
today
in
this
session.
A
B
C
One
one
example
that
I
like
to
highlight
here
is
the
built-in
automated
high
availability
capabilities
that
don't
involve
any
additional
manageability
overhead
from
user
side,
because
everything
is
is
built
in
very
important.
This
sql
server
has
the
same
data
virtualization
capabilities
that
we
released
with
sql
server
2019
as
when
deployed
on
on
other
platforms.
C
As
I
said
on
windows
on
linux,
outside
of
bdc,
and
to
to
go
into
more
details
about
this
use
case.
It
is
very
likely,
as
I
said,
that
in
addition
to
data
stored
within
bdc
is
relational
or
in
hadoop.
Some
of
you
know.
Enterprises
have
data
that
might
reside
in
other
systems
like
oracle,
mongodb
or
other
hdfs
systems,
in
azure
and
and
so
on.
C
So
no
data
is
moved
only
the
results
said
very
time
is
returned
to
to
the
user
in
in
a
more
efficient
way,
as
I
said,
but
when
it
comes
to
data
scientists
right
and
data
engineers,
their
use
cases
might
be
a
bit
different
and
they
want
to
run
their
workloads
in
in
a
slightly
different
manner.
That's
why,
when
you
deploy
big
data
clusters,
as
I
said
earlier,
you're
not
only
deploying
sql
server,
but
also
spark
and
hdfs
storage
again
as
a
unit
in
the
same
single
deployment
pattern.
C
So
what
we
call
the
storage
pool
includes,
as
you
see
here,
hdfs
we've
also
contributed
to
the
hdfs
hadoop
project,
with
a
concept
called
hdfs
steering
which
allows
you
to
mount
other
object
stores
like
s3
endpoints,
azure
data
linked
and
to
another
hdfs.
Like
stores
into
the
internal
bdc
hdfs
as
mount
points,
so
then
you
can
essentially
cache
all
this
remote
data
within
the
storage
pool
with
in
pdc.
C
C
Is
that
the
compute
power
of
sql
engine
is
scaled
out
when
accessing
the
the
data
in
the
lake
internally
again
it's
stored
in
hdfs
or
outside
through
this,
what
we
call
compute
pool
this
is
being
done
transparently
at
query
time
and
does
not
necessarily
you
know,
involve
any
user
interaction.
It's
it's
all
built
in
with
the
query.
C
So
to
to
summarize,
you
can
query
your
data
lake
in
real
time
and
sql,
but
also
with
spark
you
can
ingest
strings
of
data
with
spark
streaming.
You
can
do
training
of
machine
learning
models
using
spark
ml,
but
you
can
also
build
and
consume
models
into
the
database
with
sql
machine
learning.
Services
against
sql
server
master
instance
have
the
same
capabilities
as
any
other
sql
server
deployed
on
other
platforms.
C
C
As
you
know,
in
order
for
hospitals
to
optimize
resource
allocation,
it
is
very
important
to
for
them
to
predict.
You
know
to
be
able
to
predict
how
long
a
newly
admitted
patient
will
stay
in
in
the
hospital
and
so
advanced
length
of
stay
prediction
at
a
time
of
admission
can
help
hospitals
to
enhance
the
quality
of
care,
to
improve
efficiency,
help
with
accurate
planning
for
discharges,
as
well
as
reduce
costs
right.
C
C
C
They
can
do
this
either
by
using
sql
server
master
instance.
Again
they
can
leverage
the
invalid
machine
learning
services
that
are
in
sql
server
with
r
or
python,
or
they
can
use
spark
this
time
if
they
are
more
familiar
with
the
technology
or
for
a
targeted
use
case
like
data
preparation,
for
example,
data
scientists,
their
preparation,
the
model,
training
and
evaluation
from
azure
data
studio,
which
is
the
facto
tool
for
managing
and
interacting
with
sql
server.
C
C
B
C
C
C
C
C
We
release
new
versions
of
the
of
the
software
of
those
images
every
pretty
much
every
month,
and
in
order
to
upgrade,
you
can
simply
trigger
an
upgrade
command
using
the
easy
data.
Big
data
cluster
tools
and
this
controller
component
that
is
running
inside
the
beta
cluster
will
orchestrate
a
rolling
upgrade
of
the
software
throughout
the
entire
cluster.
A
C
C
So
when
it
comes
to,
we
talked
about
management
when
it
comes
to
administration.
We
know
that
for
years,
sql
server
commitment
was
to
support
this
mission.
Critical
applications
from
a
security
perspective,
high
availability,
automated
upgrade
operations,
as
well
as
monitoring
and
troubleshooting
active
directory
authentication.
C
One
one
thing
that
I
would
like
to
highlight
here
when
it
comes
to
security,
is
that
we've
enhanced
the
security
design
of
bdc,
specifically
to
take
full
advantage
of
the
openshift
container
platform,
and
one
of
the
most
important
enhancements
in
this
regard
is
that
now,
containers
that
are
running
within
the
bdc
are
running
as
non-root
users
as
a
default
configuration
when
you
deploy
the
cluster
for
an
in-depth
detail.
Analysis
of
this,
this
security
design,
we
com
jointly
published
a
white
paper.
C
It
was
co-authored
by
sql
server
and
red
hat
security
teams
and
again
I
put
it
in
the
resource
section
as
well
as
you
can
find
it
in
our
documentation
and
it's
very
important
to
to
go
through
that.
As
it
describes
the
the
designing
details,
it's
highlighting
what
and
why
we
require
certain
security
policies
when
deploying
bdc
on
openshift
bbc
also
deploys
additional
components
to
manage
high
availability,
as
I
said
earlier,
as
well
as
management
and
monitoring
components.
C
So
each
and
every
container
inside
dc
has
this
sidecar
monitoring
container
that
takes
the
logs
in
elasticsearch
and
the
monitoring
telemetry
into
a
influx
db
database.
So
you
can
later
visualize
this
metrics
and
logs
through
grafana,
as
well
as
kibana
dashboards
that
are
provided
and
deployed
by
default
within
the
cluster
as
well.
B
C
First,
I'm
gonna
take
care
of
the
setting
up
the
security
context,
so
we'll
create
a
custom
one
for
the
namespace
that
we're
going
to
deploy
bdc
in.
As
you
know,
ss.
These
sscs
are
groups
of
security,
related
configurations
or
deployments
on
openshift
that
support
secure
enterprise
deployments
for
various
use
cases.
C
You
see
the
dc
sec
and,
as
I
said,
we
we
create
the
namespace
and
to
complete
the
security
prerequisites.
We
must
bind
this
service,
this
name
space
with
the
sec
and
this
account
used
by
various
bdc
application
for
the
above,
create
namespace.
Only
so
this
way
the
namespace
becomes
the
security
boundary
for
for
bdc
users.
C
The
next
step
is
to
choose
what
configuration
templates
to
use
as
baseline
for
your
deployment
and,
as
I
mentioned
earlier,
for
a
tighter
integration
with
openshift,
we
included
deployment
templates
with
the
appropriate
bolts
for
the
openshift
platform.
So
you
see
here
the
openshift
one,
as
well
as
the
azure
red
hat
openstreet
in
in
my
case
here,
I'm
gonna
use
the.
C
One
and
the
neat
thing
about
this
built-in
profile
is
that
all
the
defaults
make
sense
for
a
test
environment
and
everything
is
pretty
much
ready
to
go.
I
don't
have
to
change
much,
but
just
the
name
of
the
cluster,
since
I
have
to
match
the
namespace
I
created
earlier,
but
if
other
edits
are
necessary
to
update
like
if
you
want
to
customize
storage
or
the
scaling
of
various
components,
or
if
you
want
to
customize
service
specific
configurations,
you
can
use
either
use
the
az
data
command
to
to
as
well.
C
C
So
we
will
we
issue
the
bdc
create
command
and,
from
this
point
on,
we
are
hands
off
all
the
bdc
components
are
laid
out,
starting
with
the
controller
and
once
the
central
management
service
of
the
cluster
is
up,
it
will
take
the
responsibility
of
coordinating
with
the
kubernetes
api
service.
The
deployment
of
all
the
other
bdc
applications
check
it
check
their
status
and
and
so
on,
for
a
deployment
of
this
size,
the
default
size.
B
C
Class,
like
sql
server,
spark
the
monitoring
services
and
so
on
this.
This
endpoint
list
command
is,
is
giving
me
the
information
about
all
the
endpoints
that
are
available
in
another
quick
example.
As
I
said,
I
want
to
show
you
how
to
leverage
the
the
power
of
containers
and
the
orchestration
with
kubernetes
and
openshift.
In
this
case,
during
an
upgrade,
you
can
simply
trigger
the
upgrade
using
the
easy
data
tool.
It's
pretty
straightforward.
C
We
just
specify
the
target
repo,
as
well
as
the
image
tag
that
you
want
to
use
to
upgrade
to,
and
then
this
controller
running
inside
a
big
data
cluster
will
orchestrate
a
rolling
upgrade
of
the
software
throughout
the
entire
cluster.
Again
it
takes
care
of
the
service
dependencies.
It
does
it
in
a
specific
order.
It
checks
the
service
health.
C
C
I
am
putting
here
some
other
resources
that
we
made
available
for
you
at
the
announcement
term,
and
then
we
constantly
enhance
our
documentation
and
articles
on
more
with
more
use
cases
and
examples
and
details
on
on
the
architecture
and
so
on
so
check
out
this
latest
sticker
several
blogs,
as
well
as
the
bbc
documentation,
to
learn
about
the
improvements
that
are
being
released
each
month
and
for
for
what's
next
right.
We
are
continuing
our
collaboration
with
the.
B
C
C
C
B
C
C
B
Yeah,
thank
you.
This
was
very
good,
I
mean
just
to
reiterate.
You
know
we
are
very
excited
about
that
too,
about
this
solution.
Working
with
microsoft,
we
have,
as
mahela
mentioned,
we
have
several
joint
customers.
One
of
them
is
a
that.
I
recall
is
a
bank
and
I
mean,
and
their
big
challenge
really
was
their
goal
really
was
to
build.
B
Customer
acquisition
and
retention
do
more
comprehensive
central
marketing
reporting
and
this
overall
improve
customer
experience,
kind
of
no
different
than
what
other
companies
are
trying
to
use,
ai
machine
learning
and
analytics
for
in
general
and
the
biggest
issue
that
they
have,
and
you
all
might
find
yourself
in-
is
the
idea
that
you
know
these
bespoke
solutions.
I
mean
doing
it
yourself,
you
know,
are
difficult.
It
takes
several
weeks
and
then
they're
very
hard
to
debug
so
when
they
they
when.
B
Finally,
when
we
introduced
them
to
this
bdc
solution
on
openshift,
they
were
able
to
complete
that
in
a
from
design
to
implementation
in
a
matter
of
a
few
weeks,
and
now
it's
a
stable
solution
within
that
bank,
so
they
were
able
to
you
know
so
they
were
so
the
the
solution
itself
is
comprised
of
bdc
with
apache
spark
and
hdfs,
and
you
know
at
all
running
on
top
of
openshift.
So
so
this
is
very
exciting
for
us
to
continue
to,
as
marilyn
said,
will
we
have
several
other
prospects?
B
B
You
know
the
idea
that
if
you
have
data
in
nfs
like,
for
example,
some
direct
questions
saying
that
what
if
I
have
unstructured
data
such
as
mri
and
x-ray
data
in
a
external
nfs,
how
would
you
you
know,
bring
that?
How
do
you
access
that
using
the
solution?
Mahala?
What
I
said
really
was
that
you
know
you
can
basically
use
things
like
kafka.
C
B
B
C
A
Yeah,
thank
you
so
much
mihaila.
Thank
you
to
char
big
date.
I'm
sure
it's
a
hot
topic
right
now
and,
like
you
said
in
the
roadmap,
there's
gonna
be
a
lot
of
use
cases
coming
up,
so
this
was
very
great
and
timely.
It
was
a
very
great
and
timely
presentation,
so
thank
you
once
again,
also
a
huge
shout
out
to
chris
as
well
for
producing
this
session
and
making
the
live
stream
very
seamless.