Home
Contribute Contact Us Browse all meetings
Home Contribute Contact Us Browse all meetings
Ceph / General Ceph Videos / 23 Jan 2017
Ceph / General Ceph Videos / 23 Jan 2017
Previous Meeting
Next Meeting
⏯
Add meeting Rate page Subscribe
youtube image
►
From YouTube: Install Ceph Server on Proxmox VE

Description

The video demonstrates the installation of a distributed Ceph storage server on an existing three node Proxmox VE cluster. At the end of this tutorial you will know how to build a free and open source hyper-converged virtualization and storage cluster using Proxmox VE and Ceph.

More help:
Proxmox VE Wiki - https://pve.proxmox.com/wiki/Ceph_Server
Ceph PGs per Pool Calculator - http://ceph.com/pgcalc/

Tutorial content:
- Hardware
- Network
- Installing Ceph Jewel
- Initializing Ceph
- Installing Ceph monitors
- Creating pools
- Ceph Dashboard
- Simple benchmark
- Configure Proxmox Storage
- Copy keyring

A

Hello and welcome to this proxmox tutorial on ceph storage.

A

I am william, and in this video I will show you the installation of a distributed ceph storage on an existing three node proxmox ve cluster.

A

At the end of this tutorial, you will know how to build a free and open source, hyper-converged, virtualization and storage cluster.

A

In this example, I use three identical xeon d servers for my three-node proxmox ve cluster.

A

Every node has got a xeon d with 6 cores and hyperthreading and 64 gb ram.

A

The os is installed on a fast samsung 240 gb nvm express ssd.

A

As we will run the ceph monitors on these disks,, it is very important that we only use fast and reliable ssds.

A

For the ceph osds I have two enterprise class samsung sm863 ssds in each server.

A

For this demo I only use 240 gb ssds, but you can get them also with 2 terabyte.

A

This xeon d servers are in a 1u chassis with 4 hot swap cases in a very space and energy efficient, package.

A

All three nodes have the same hardware: configuration.

A

The mainboard offers dual 1 gbit network and dual 10 gbit.. This is perfectly suited for our needs.

A

The first gigabit nic “eth0” is used for the vm bridge, zero (vmbr0).

A

The second nic “eth1” is used for the proxmox ve cluster communication.

A

And both 10 gbit nics – “eth2” and “eth3” are used for the ceph cluster in bonding mode.

A

In this special case – just three nodes –.

A

I can connect these servers without using a 10, gbit switch,. Therefore, for bond mode, I choose “broadcast”.

A

All three nodes have the same network configuration only with different ip addresses.

A

The first step is to install the ceph packages.

A

Just open a shell and install the ceph packages with the command: pveceph install -- version jewel.

A

Do the same on the second node: pveceph install -- version jewel.

A

And also on the third node: pveceph install -- version jewel.

A

As soon as we’ve got all packages installed, we need to initialize ceph on the first node.

A

Open a shell and type: pveceph, init --network 10.10.10.0/24.

A

Make sure you use the 10 gbit network. in this example, this is the network from bond zero.

A

Now create the first ceph monitor by typing pveceph createmon then continue on the gui.

A

To obtain redundancy create a second ceph monitor on the second node.

A

And a third ceph monitor on the third node.

A

With three monitors you’ll still have quorum, even if one node is down.

A

Now we are ready to create all the ceph osds.

A

The osd is the place where your data will be stored.. Each osd needs a complete disk.

A

Go to ceph, osd, and click “create: osd”.

A

As I already use fast ssds here, I can store the journal on the same device.

A

Let’S do the same for the second osd.

A

After reloading the page, you will see osd.0 and osd.1 on the first node.

A

Now go to the second node. click again on “create: osd” for the first ssd.

A

And also for the second one.

A

After reloading the page, you will see osd.2 and osd.3 on the second node.

A

Repeat the same for the third node., you will get osd.4 and osd.5.

A

Finally,, you will see six osds in total on all three nodes.

A

In order to use ceph for your virtual machines and your containers, you need to create two pools.

A

One for virtual machines and one for containers.

A

Go to pools, and click “create”. enter a suitable name.. I choose: “ceph-vm”.

A

The size parameter specifies the replication count.

A

To get a reliable pool, you need to set the count to three., and the minimum replication is two.

A

Be careful when choosing the number of placement groups.

A

You can easily increase the number later, but you never can reduce the number!.

A

Before starting calculate the right number for your expected use case with the calculator on the ceph website at http://ceph.com/pgcalc/.,.

A

My second pool will be used for lxc containers,, so I name it “ceph-lxc”.

A

To do benchmarks, I also create a “testpool” named "test3".

A

And because I don’t need the rbd pool,, I remove it.

A

Now let’s take a look on the ceph dashboard.. It shows the status of your monitor, hosts.

A

The health_warn shows a monitor clock, skew detected.. This will go away automatically as soon as the time is fine again.

A

Also, the dashboard shows your osds,, the placement groups, and some performance statistics.

A

Let’S do a simple rados, benchmark.

A

Just open the shell and run a 10 seconds, rados write benchmark.

A

Type: rados -p test3 bench 10 write, --no-cleanup.

A

We have a write performance of 420 mb per second.

A

To do a read test: type: rados, -p, test, 3 bench, 10, seq,.

A

Here we have a read performance of 2600 mb per second.

A

If you want to get rid of this test data, just remove the test3 pool.

A

In order to use a pool as proxmox storage, you need to add the rbd storage via gui.

A

First, click on datacenter and on storage and add rbd storage.

A

First, I create the storage for virtual machines using the pool "ceph-vm".

A

I also use "ceph-vm" for the storage name.

A

In “monitors” add all three ip addresses from your ceph monitors.

A

Do the same for the ceph-lxc pool.? But this time you need to enable container as well as krbd, because this is needed for lxc containers.

A

The last step is copying the keyring.

A

Open a shell and change directory into /etc/pve/priv/.

A

Create the ceph directory with: mkdir ceph.

A

And place a copy of the keyring for each storage, name.

A

Type: cp /etc/ceph/ceph.client.admin.keyring ceph/ceph-vm.keyring.

A

And for the second lxc pool, type: cp /etc/ceph/ceph.client.admin.keyring ceph/ceph-lxc.keyring.

A

Now you can see the storage in the gui summary. and we are ready to use ceph for our virtual machines and containers.

A

Let’S add a ceph hard disk to the debian stretch virtual machine in this example to do a simple benchmark.

A

Go to hardware – “add” – choose hardware – and choose the storage ceph-vm.

A

Check “discard”. and click “add”.

A

Open a console with double-clicking.

A

You can see the new harddisk and can start the benchmark.

A

I have about 380 mb write and 2.5 gigabyte read speed, slightly slower than the rados benchmark., so this looks great.

A

Check again, your ceph dashboard., the dashboard provides an overview about your ceph status,, the monitors,, the osds, and, of course, some performance. Statistics.

A

To learn more check out the article on ceph server in our wiki documentation at https://pve.proxmox.com/wiki/ceph_server.

A

I hope the video could help you to install ceph server on your proxmox ve to build a complete.

A

Free and open source, hyper-converged, virtualization and storage cluster.

A

If you need further help,, ask a question in the proxmox community forum.

A

Or purchase a subscription for direct help from our support team..

A

Thank you for watching!.