proxmox HTTPS certificates for load balancing

generate a PKI and trust the Root CA in your browser (using XCA ?)
I used the CN as proxmox.domain.net, and then SAN to every proxmox$i.domain.net to use the same certificate for every node and the Virtual Server IP of the Load balancer (here the Fortigate)

export the certificate chain without the CA(.crt) and the key (.pem)

then copy on every node :


for i in {1..n}; \
do \
scp proxmox.domain.net.crt proxmox5:/etc/pve/nodes/proxmox$i/pve-ssl.pem; \
scp proxmox.domain.net.pem proxmox5:/etc/pve/nodes/proxmox$i/pve-ssl.key; \
ssh proxmox$i systemctl restart pveproxy;\
done



Then I use the Fortigate to load balance :
https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/713497/virtual-server

Kubernetes

Dashboard :

the URL is
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

in order to login with the config file in .kube/ folder, add the token of a service account with privileges (here cluster admin) :

get the token for user <toto>

TOKEN=$(kubectl -n kube-system describe secret toto| awk '$1=="token:"{print $2}')

add it to the config file :

kubectl config set-credentials kubernetes-admin --token="${TOKEN}" 

Waiting for Quorum on Proxmox

When an addition of a node to the cluster fails, there is the message 'waiting for quorum...' and on the GUI it is 'Invalid PVE Ticket' and you cannot login anymore.

Here is my script to retry a failed cluster :

systemctl stop pve-cluster.service
systemctl stop corosync
pmxcfs -l
rm /etc/pve/corosync.conf
rm -Rf /etc/corosync/*
killall pmxcfs
systemctl start pve-cluster.service

rm -Rf /etc/pve/nodes/proxmox5

echo "#### NOW on a cluster member, remove previous reference in these files:

root@proxmox1:~# rm -Rf /etc/pve/nodes/proxmox5
root@proxmox1:~# nano /etc/pve/corosync.conf
root@proxmox1:~# rm -Rf /var/lib/corosync/*
"

remote CEC server for Home assistant

I wanted to connect my Home assistant to the TV, so the pycec module on home assistant can connect to a Raspberry Pi that is connected to the TV and can be connected to Home assistant via TCP/IP.

I used this repository :
https://github.com/konikvranik/pycec/

First, enable the pycec module on home assistant.
In the configuration.yaml file, add :

hdmi_cec: !include config/hdmi_cec.yaml

and in the config/hdmi_cec.yaml file :

host: !secret rpi3b_ip

Then on the Raspberry pi :
cd
git clone  https://github.com/konikvranik/pycec/

First, compile the libcec :
https://github.com/Pulse-Eight/libcec

For compiling it, I followed the Linux procedure :

apt-get update
apt-get install cmake libudev-dev libxrandr-dev python-dev swig
git clone https://github.com/Pulse-Eight/libcec.git
mkdir libcec/build
cd libcec/build
cmake ..
make -j4
sudo make install
sudo ldconfig

Then symlink the service in the pycec repository :
ln -s /path/of/pycec.service /etc/systemd/system/pycec.service
systemctl start pycec

then enable the service so it starts at boot :
systemctl enable pycec

restore ceph admin rights

I did overwrite the client.admin entry by importing a wrong keyfile. So I could not manage the cluster again.

The only solution was to deactivate cephx authentication on the cluster by changing the configuration file to :

auth cluster required = none
auth service required = none
auth client required = none
auth supported = none

and then restart the ceph daemons on each node. As I'm using proxmox, I had to use the systemctl:

systemctl restart ceph.target 

source :

https://docs.ceph.com/docs/emperor/rados/operations/authentication/

then import the admin profile with full rights 

Then put the authentication back and restart the daemons.

connect Debian to Proxmox CEPH cluster and mount Cephfs

on client create the destination directory

mkdir -p /etc/pve/priv/

copy the keyring on the debian client: (uses admin keyring !!)

scp <proxmox_ip>:/etc/pve/priv/ceph.client.admin.keyring /etc/pve/priv/.

copy the configuration file :

scp <proxmox_ip>:/etc/pve/ceph.conf /etc/ceph/.

test

ceph status

#####

mount Cephfs on debian

1. on ceph:


generate a keyfile for client foo on proxmox:

ceph auth get-or-create client.foo \

    mds 'allow rw path=/ceph/mount/point' \

    mon 'allow r' \

    osd 'allow rw pool=cephfs_data' \

    -o /etc/pve/priv/ceph.client.foo.keyring

check the client with :
ceph auth list

generate a minimal config :
ceph config generate-minimal-conf

and copy the output in /etc/ceph/ceph.conf on the client

2. on client:

aptitude install libcephfs2 ceph-common ceph-fuse 

copy the keyring file from the client

mkdir -p /local/mount/point

add an entry in /etc/fstab like :

id=foo,conf=/etc/ceph/ceph.conf,client_mountpoint=/ceph/mount/point /local/mount/point fuse.ceph _netdev,defaults 0 0

_netdev here is important, otherwhise it does not boot 

3. then mount it 

sudo mount -a 

migrating from XCP-ng (xen) to Proxmox (LVM)

create the VM, be the closest to the XCP-ng VM on Proxmox
detach the disk

export the VM with Xencenter in ova/ovf format (GUI)

extract the ova : 

tar -xvf <ova_file.ova>
this result in a vhd + ovf file

convert the filedisk format:

qemu-img convert -f vpc <disk_file.vhd> -O qcow2 <disk_filename.qcow2>

import the disk to the vm :  

qm importdisk <vm_id> <disk_filename.qcow2> <storage> -format qcow2

attach the disk to the vm, then boot

CEPH

mount a LVM logical volume as OSD:

vgresize / vgcreate ...
lvcreate -l 100%FREE -n ceph pve
ceph-volume lvm prepare --bluestore --data pve/ceph
ceph-volume lvm activate --all

clean the disk for OSD usage :

ceph-volume lvm zap /dev/sd[X] --destroy

edit the crushmap

ceph osd tree

ceph osd crush reweight {name} {weight}

ceph osd crush set {id-or-name} {weight} root={pool-name} [{bucket-type}={bucket-name} ...]

or by hand :

ceph osd getcrushmap -o map.bin

crushtool -d map.bin -o map.txt

edit map.txt

crushtool -c map.txt -o map.bin

ceph osd setcrushmap -i map.bin

check the difference with the previous map :

crushtool -i crushmap --compare crushmap.new

create a 'bestred' rule:

ceph osd crush rule create-replicated bestred default osd


assign the rule to the pool / change the rule of the pool
ceph osd pool set redpool crush_rule bestred


create a rbd :

rbd create --size {megabytes} {pool-name}/{image-name}

see where the object is mapped :

ceph osd map <pool_name> <rbd_name> 

increase the number of replicas
ceph osd pool set bestred size 8
ceph osd pool set bestred min_size 4

change attributes

ceph osd pool set {pool-name} {field} {value}

Valid fields are:

• size: Sets the number of copies of data in the pool.
• pg_num: The placement group number.
• pgp_num: Effective number when calculating pg placement.
• crush_rule: rule number for mapping placement.

autoscaling

since Nautilus, enable the module first :

ceph mgr module enable pg_autoscaler

ceph osd pool autoscale-status

change : ceph osd pool set pool2 pg_autoscale_mode {warn,off}

reduce osd cache memory usage

(for osd.7 in this example)

change the memory usage (previously 4G) :
ceph daemon osd.7 config set osd_memory_target 1610612736
or
ceph config set osd.7 osd_memory_target 1610612736 
to be permanent 

or as global (all osd in this case):
ceph config set global osd_memory_target <value>

we can also add it in the config ceph.conf :
[global]
... some config
     osd_memory_target = 939524096

enable XCP-ng nested virtualisation

Enable XCP-ng nested virtualisation:

used here to test proxmox under XCP-ng as host

# xe vm-param-set uuid=<uuid> platform:exp-nested-hvm=true
# xe vm-param-set uuid=<uuid> platform:nic_type="e1000"




found here :
https://github.com/xcp-ng/xcp/wiki/Testing-XCP-ng-in-Virtual-Machine-(Nested-Virtualization)

Mysql / Adminer / Apache in FreeBSD jail

Mysql / Adminer / Apache in FreeBSD jail

 

Prerequisites :
create the jail, and add a ZFS mount to store the databases in another dataset
mount the dataset to /mnt/db-data

install mysql

# install + start at boot with custom config file
pkg install mariadb104-server
sysrc mysql_enable="YES"
sysrc mysql_pidfile=/var/db/mysql/mysql.pid
sysrc mysql_optfile=/usr/local/etc/mysql/my.cnf

# set proper owner for the zfs mounted dataset
chown -R mysql:mysql /mnt/db-data

-> restart the jail

# setup the database
mysql_secure_installation --socket=/mnt/db-data/mysql.sock



logs location : /var/db/mysql/*.log

install adminer

pkg install adminer
the adminer php file is at
/usr/local/www/adminer/adminer/index.php

install apache

# in order to serve adminer
pkg install apache24
sysrc apache24_enable="yes"
service apache24 start
add to /usr/local/etc/apache24/Includes/adminer.conf the config :
<VirtualHost *:80>
        ServerName 127.0.0.1
        ServerAlias adminer
        DocumentRoot "/usr/local/www/adminer/adminer/"

        ErrorLog "/var/log/adminer-error.log"
        CustomLog "/var/log/adminer-access_log" combined

</VirtualHost>

setup php 

# give a php configuration file
cp /usr/local/etc/php.ini-production /usr/local/etc/php.ini

# needed to work with adminer
change line mysqli.default_socket=
mysqli.default_socket = /mnt/db-data/mysql.sock


# make apache interpret the php
add this to /usr/local/etc/apache24/Includes/php.conf
<IfModule dir_module>
    DirectoryIndex index.php index.html
    <FilesMatch "\.php$">
        SetHandler application/x-httpd-php
    </FilesMatch>
    <FilesMatch "\.phps$">
        SetHandler application/x-httpd-php-source
    </FilesMatch>
</IfModule>


service apache24 reload