/etc/pve/priv/
/etc/pve/nodes/${NAME}/priv/
The Proxmox Cluster file system (“pmxcfs”) is a database-driven file
system for storing configuration files, replicated in real time to all
cluster nodes using corosync
. We use this to store all PVE related
configuration files.
Although the file system stores all data inside a persistent database on disk, a copy of the data resides in RAM. That imposes restriction on the maximum size, which is currently 30MB. This is still enough to store the configuration of several thousand virtual machines.
This system provides the following advantages:
seamless replication of all configuration to all nodes in real time
provides strong consistency checks to avoid duplicate VM IDs
read-only when a node loses quorum
automatic updates of the corosync cluster configuration to all nodes
includes a distributed locking mechanism
The file system is based on FUSE, so the behavior is POSIX like. But some feature are simply not implemented, because we do not need them:
you can just generate normal files and directories, but no symbolic links, …
you can’t rename non-empty directories (because this makes it easier to guarantee that VMIDs are unique).
you can’t change file permissions (permissions are based on path)
O_EXCL
creates were not atomic (like old NFS)
O_TRUNC
creates are not atomic (FUSE restriction)
All files and directories are owned by user root
and have group
www-data
. Only root has write permissions, but group www-data
can
read most files. Files below the following paths:
/etc/pve/priv/
/etc/pve/nodes/${NAME}/priv/
are only accessible by root.
We use the Corosync Cluster Engine for cluster communication, and SQlite for the database file. The file system is implemented in user space using FUSE.
The file system is mounted at:
/etc/pve
|
Corosync cluster configuration file (previous to Proxmox VE 4.x this file was called cluster.conf) |
|
Proxmox VE storage configuration |
|
Proxmox VE datacenter wide configuration (keyboard layout, proxy, …) |
|
Proxmox VE access control configuration (users/groups/…) |
|
Proxmox VE authentication domains |
|
Public key used by ticket system |
|
Public certificate of cluster CA |
|
Shadow password file |
|
Private key used by ticket system |
|
Private key of cluster CA |
|
Public SSL certificate for web server (signed by cluster CA) |
|
Private SSL key for |
|
Public SSL certificate (chain) for web server (optional override for |
|
Private SSL key for |
|
VM configuration data for KVM VMs |
|
VM configuration data for LXC containers |
|
Firewall configuration applied to all nodes |
|
Firewall configuration for individual nodes |
|
Firewall configuration for VMs and Containers |
|
|
|
|
|
|
|
File versions (to detect file modifications) |
|
Info about cluster members |
|
List of all VMs |
|
Cluster log (last 50 entries) |
|
RRD data (most recent entries) |
You can enable verbose syslog messages with:
echo "1" >/etc/pve/.debug
And disable verbose syslog messages with:
echo "0" >/etc/pve/.debug
If you have major problems with your Proxmox VE host, e.g. hardware
issues, it could be helpful to just copy the pmxcfs database file
/var/lib/pve-cluster/config.db
and move it to a new Proxmox VE
host. On the new host (with nothing running), you need to stop the
pve-cluster
service and replace the config.db
file (needed permissions
0600
). Second, adapt /etc/hostname
and /etc/hosts
according to the
lost Proxmox VE host, then reboot and check. (And don’t forget your
VM/CT data)
The recommended way is to reinstall the node after you removed it from your cluster. This makes sure that all secret cluster/ssh keys and any shared configuration data is destroyed.
In some cases, you might prefer to put a node back to local mode without reinstall, which is described here:
stop the cluster file system in /etc/pve/
# systemctl stop pve-cluster
start it again but forcing local mode
# pmxcfs -l
remove the cluster configuration
# rm /etc/pve/cluster.conf
# rm /etc/cluster/cluster.conf
# rm /var/lib/pve-cluster/corosync.authkey
stop the cluster file system again
# systemctl stop pve-cluster
restart PVE services (or reboot)
# systemctl start pve-cluster
# systemctl restart pvedaemon
# systemctl restart pveproxy
# systemctl restart pvestatd