Ceph for Kubernetes

follow :
https://github.com/kubernetes-incubator/external-storage/

Straight forward with some modifications, as the claim has to have a different name, and pay attention that for CephFS and RDB, the admin secret name is different, but can be set the same.

I also used the same cephfs namespace for both.

EDIT : I made a PR that uses now the same kubernetes user for both storageclass
here : https://github.com/kubernetes-incubator/external-storage/pull/1306

ZFS storage driver for docker

This is used to have a docker setup directly on Proxmox 6 with ZFS 


stop the docker service

service docker stop


delete all the docker stuff

rm -rf /var/lib/docker


create a zfs pool dedicated for docker
  
zfs create -o mountpoint=/var/lib/docker rpool/docker


create the systemd entry for the docker service

mkdir /etc/systemd/system/docker.service.d


setup the driver for docker

nano /etc/systemd/system/docker.service.d/storage-driver.conf


[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=zfs -H fd://


then restart daemons and docker

systemctl daemon-reload
service docker start





Samba over CephFS

I wanted to use NFS to share data on the network, but Windows 10 still isn't able to use NFS ... what a joke.

So first mount CephFS (see previous article)

Then configure the Samba server

1. install samba server 

apt-get install samba

2. edit the configuration file

set the workgroup

add a samba share, for example

[photos]
        comment = photos
        read only = no
        path = /mnt/cephfs/nas/photos
        guest ok = no
 

2. configure users

I use the same setup users as the user on the windows machine

add the user :

adduser toto

setup its password: 

passwd toto

then add this user to Samba :

smbpasswd -a toto

then restart the daemon to take into account the config :

systemctl restart smbd 

proxmox HTTPS certificates for load balancing

generate a PKI and trust the Root CA in your browser (using XCA ?)
I used the CN as proxmox.domain.net, and then SAN to every proxmox$i.domain.net to use the same certificate for every node and the Virtual Server IP of the Load balancer (here the Fortigate)

export the certificate chain without the CA(.crt) and the key (.pem)

then copy on every node :


for i in {1..n}; \
do \
scp proxmox.domain.net.crt proxmox5:/etc/pve/nodes/proxmox$i/pve-ssl.pem; \
scp proxmox.domain.net.pem proxmox5:/etc/pve/nodes/proxmox$i/pve-ssl.key; \
ssh proxmox$i systemctl restart pveproxy;\
done



Then I use the Fortigate to load balance :
https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/713497/virtual-server

Kubernetes

Dashboard :

the URL is
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

in order to login with the config file in .kube/ folder, add the token of a service account with privileges (here cluster admin) :

get the token for user <toto>

TOKEN=$(kubectl -n kube-system describe secret toto| awk '$1=="token:"{print $2}')

add it to the config file :

kubectl config set-credentials kubernetes-admin --token="${TOKEN}" 

Waiting for Quorum on Proxmox

When an addition of a node to the cluster fails, there is the message 'waiting for quorum...' and on the GUI it is 'Invalid PVE Ticket' and you cannot login anymore.

Here is my script to retry a failed cluster :

systemctl stop pve-cluster.service
systemctl stop corosync
pmxcfs -l
rm /etc/pve/corosync.conf
rm -Rf /etc/corosync/*
killall pmxcfs
systemctl start pve-cluster.service

rm -Rf /etc/pve/nodes/proxmox5

echo "#### NOW on a cluster member, remove previous reference in these files:

root@proxmox1:~# rm -Rf /etc/pve/nodes/proxmox5
root@proxmox1:~# nano /etc/pve/corosync.conf
root@proxmox1:~# rm -Rf /var/lib/corosync/*
"

remote CEC server for Home assistant

I wanted to connect my Home assistant to the TV, so the pycec module on home assistant can connect to a Raspberry Pi that is connected to the TV and can be connected to Home assistant via TCP/IP.

I used this repository :
https://github.com/konikvranik/pycec/

First, enable the pycec module on home assistant.
In the configuration.yaml file, add :

hdmi_cec: !include config/hdmi_cec.yaml

and in the config/hdmi_cec.yaml file :

host: !secret rpi3b_ip

Then on the Raspberry pi :
cd
git clone  https://github.com/konikvranik/pycec/

First, compile the libcec :
https://github.com/Pulse-Eight/libcec

For compiling it, I followed the Linux procedure :

apt-get update
apt-get install cmake libudev-dev libxrandr-dev python-dev swig
git clone https://github.com/Pulse-Eight/libcec.git
mkdir libcec/build
cd libcec/build
cmake ..
make -j4
sudo make install
sudo ldconfig

Then symlink the service in the pycec repository :
ln -s /path/of/pycec.service /etc/systemd/system/pycec.service
systemctl start pycec

then enable the service so it starts at boot :
systemctl enable pycec

restore ceph admin rights

I did overwrite the client.admin entry by importing a wrong keyfile. So I could not manage the cluster again.

The only solution was to deactivate cephx authentication on the cluster by changing the configuration file to :

auth cluster required = none
auth service required = none
auth client required = none
auth supported = none

and then restart the ceph daemons on each node. As I'm using proxmox, I had to use the systemctl:

systemctl restart ceph.target 

source :

https://docs.ceph.com/docs/emperor/rados/operations/authentication/

then import the admin profile with full rights 

Then put the authentication back and restart the daemons.

connect Debian to Proxmox CEPH cluster and mount Cephfs

on client create the destination directory

mkdir -p /etc/pve/priv/

copy the keyring on the debian client: (uses admin keyring !!)

scp <proxmox_ip>:/etc/pve/priv/ceph.client.admin.keyring /etc/pve/priv/.

copy the configuration file :

scp <proxmox_ip>:/etc/pve/ceph.conf /etc/ceph/.

test

ceph status

#####

mount Cephfs on debian

1. on ceph:


generate a keyfile for client foo on proxmox:

ceph auth get-or-create client.foo \

    mds 'allow rw path=/ceph/mount/point' \

    mon 'allow r' \

    osd 'allow rw pool=cephfs_data' \

    -o /etc/pve/priv/ceph.client.foo.keyring

check the client with :
ceph auth list

generate a minimal config :
ceph config generate-minimal-conf

and copy the output in /etc/ceph/ceph.conf on the client

2. on client:

aptitude install libcephfs2 ceph-common ceph-fuse 

copy the keyring file from the client

mkdir -p /local/mount/point

add an entry in /etc/fstab like :

id=foo,conf=/etc/ceph/ceph.conf,client_mountpoint=/ceph/mount/point /local/mount/point fuse.ceph _netdev,defaults 0 0

_netdev here is important, otherwhise it does not boot 

3. then mount it 

sudo mount -a 

migrating from XCP-ng (xen) to Proxmox (LVM)

create the VM, be the closest to the XCP-ng VM on Proxmox
detach the disk

export the VM with Xencenter in ova/ovf format (GUI)

extract the ova : 

tar -xvf <ova_file.ova>
this result in a vhd + ovf file

convert the filedisk format:

qemu-img convert -f vpc <disk_file.vhd> -O qcow2 <disk_filename.qcow2>

import the disk to the vm :  

qm importdisk <vm_id> <disk_filename.qcow2> <storage> -format qcow2

attach the disk to the vm, then boot