Proxmox VE Cluster Ceph-Storage Configure¶

1. Ceph disk prepare¶

1.1 Setup disk to non-raid¶

Reboot server and press F2 to Entering System Setup

Click Device Settings

Click RAID card

Click Configure

Click Convert to Non-RAID Disk

Click Check All -> OK to covert data disk to Non-RAID Disk

Click OK to confirm

Click Back to return to the RAID main menu

1.2 Check disk status¶

Click Main Menu to check disk status

Click Physical Disk Management

Check disk type

Click Back -> Finish to finish disk setup, and reboot server

2. Install ceph¶

Please confirm this node can access PVE enterprise repository (https://enterprise.proxmox.com)

Install PVE Ceph and check ceph version (@all nodes)

Login PVE webUI, select node and click >_Shell

pveceph install
apt policy ceph

If there no purchase subscription, you can use local APT repository or 3-third repository, please open temporary access internet

Replace APT repository to local (Optional: If use nosubscription)

sed -i 's|https://enterprise.proxmox.com|https://wzs-yum.wistron.com/proxmox|g' /etc/apt/sources.list.d/ceph.sources
sed -i 's|enterprise|no-subscription|g' /etc/apt/sources.list.d/ceph.sources

3. Ceph config¶

3.1 Ceph initial¶

Initialize cluster configuration (@first node)

Login PVE cluster website

Select first node ZSXXXXX, and click Ceph -> Configure Ceph

Select Public Network IP/CIDR and Cluster Network IP/CIDR, then click Next

Click Finish

3.2 Create monitor¶

⚠️The production environment requires at least 3 MON nodes (distributed on different physical nodes) to ensure HA

Select node, and click Monitor, and click Create under Monitor

Select node Host, and click Create to add new Monitor

Check Monitor status

3.3 Create manager¶

⚠️Recommend to create at least 2 MGRs to ensure HA

Click Create under Manager

Select node Host, and click Create to add new Manager

Check Manager status

3.4 Create OSD¶

⚠️add all node's HDD as OSD disks (use command or WebGUI)

Command:

pveceph osd create /dev/sdb
pveceph osd create /dev/sdc
......

WebGUI:

Select node, and click OSD -> Create: OSD

Select unused Disk (example: /dev/sdb...), and click Create

Check all OSD status

3.5 Create pools¶

Select node, and click Pools -> Create

Input ceph pool Name / Size, and click Create

Name: assign PVE ceph storage name

Size: number of replicas (minimum set to 2)

Add as Storage: checked (Automatically add to the cluster after creation)

Check PVE cluster storage