External email with OVH mail on proxmox

configure the /etc/postfix/main.cf like the below

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

#CHANGEME
myhostname=proxmox1.gatom.net

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
#mydestination = $myhostname, localhost.$mydomain, localhost
#relayhost =
mynetworks = 127.0.0.0/8
inet_interfaces = loopback-only
recipient_delimiter = +

compatibility_level = 2

# OVH SMTP/TLS
relayhost = [ssl0.ovh.net]:465
smtp_use_tls=yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_sasl_security_options = noanonymous

smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt

the two last are required for SMTPS or error message is :

(TCP port 465) requires setting "smtp_tls_wrappermode = yes", and "smtp_tls_security_level = encrypt" (or stronger)

add credentials to /etc/postfix/sasl_passwd

 [ssl0.ovh.net]:465     login:password

set permissions

chmod 600 /etc/postfix/sasl_passwd

create the password db

postmap /etc/postfix/sasl_passwd

of the db is not correct with the given credentials, error is

(SASL authentication failed; server ssl0.ovh.net[193.70.18.144] said: 535 5.7.1 Authentication failed)

install the package for sasl

apt-get install libsasl2-modules

otherwise error is

(SASL authentication failed; cannot authenticate to server ssl0.ovh.net[193.70.18.144]: no mechanism available)

restart postfix

systemctl restart postfix

test

echo "test ok" | mail -s "Test" dest-email@domain.com

 

change Ceph permissions

ceph auth list

...
client.cloud
    key: AQB6hPddq....HcQpUMPSNunT2w==
    caps: [mds] allow rw path=/nas/path/1, allow rw path=/nas/path/2
    caps: [mon] allow r
    caps: [osd] allow rw pool=cephfs_data
...

change with :

ceph auth caps client.cloud \
     mon 'allow r' \
     osd 'allow rw pool=cephfs_data' \
     mds 'allow rw path=/nas/path/1, allow rw path=/nas/path/3'

then check with ceph auth list

...
client.cloud
    key: AQB6hPddq....HcQpUMPSNunT2w==
    caps: [mds] allow rw path=/nas/path/1, allow rw path=/nas/path/3
    caps: [mon] allow r
    caps: [osd] allow rw pool=cephfs_data
...

LoadBalance Service for Bare metal / VM Kubernetes : Metallb

I use Metallb for the LoadBalance service of my kubernetes cluster.

helm install metallb --namespace=metallb stable/metallb

apply the configmap that is the default for the chart.

remember that for the Helm chart the configmap name has to be metallb-config, which is different from the metallb documentation.

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb
  name: metallb-config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
        - 10.0.14.100-10.0.14.200

I prefer the BGP implementation :

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb
  name: metallb-config
data:
  config: |
    peers:
    - peer-address: 10.0.14.1
      peer-asn: 64501
      my-asn: 64500
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 10.0.12.128/25

I peer the BGP peers with the Fortigate firewall which is 10.0.14.1 :

config router bgp
    set as 64501
    set router-id 0.0.0.1
    config neighbor
        edit "10.0.14.11"
            set remote-as 64500
        next
        edit "10.0.14.12"
            set remote-as 64500
        next

        ... (and the remaining nodes)

        next
    end
    config redistribute "connected"
    end
    config redistribute "rip"
    end
    config redistribute "ospf"
    end
    config redistribute "static"
    end
    config redistribute "isis"
    end
    config redistribute6 "connected"
    end
    config redistribute6 "rip"
    end
    config redistribute6 "ospf"
    end
    config redistribute6 "static"
    end
    config redistribute6 "isis"
    end
end

do not forget to open the firewall rules then.

reference for the Helm chart :

https://hub.helm.sh/charts/stable/metallb

Ceph for Kubernetes

follow :
https://github.com/kubernetes-incubator/external-storage/

Straight forward with some modifications, as the claim has to have a different name, and pay attention that for CephFS and RDB, the admin secret name is different, but can be set the same.

I also used the same cephfs namespace for both.

EDIT : I made a PR that uses now the same kubernetes user for both storageclass
here : https://github.com/kubernetes-incubator/external-storage/pull/1306

ZFS storage driver for docker

This is used to have a docker setup directly on Proxmox 6 with ZFS 


stop the docker service

service docker stop


delete all the docker stuff

rm -rf /var/lib/docker


create a zfs pool dedicated for docker
  
zfs create -o mountpoint=/var/lib/docker rpool/docker


create the systemd entry for the docker service

mkdir /etc/systemd/system/docker.service.d


setup the driver for docker

nano /etc/systemd/system/docker.service.d/storage-driver.conf


[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=zfs -H fd://


then restart daemons and docker

systemctl daemon-reload
service docker start





Samba over CephFS

I wanted to use NFS to share data on the network, but Windows 10 still isn't able to use NFS ... what a joke.

So first mount CephFS (see previous article)

Then configure the Samba server

1. install samba server 

apt-get install samba

2. edit the configuration file

set the workgroup

add a samba share, for example

[photos]
        comment = photos
        read only = no
        path = /mnt/cephfs/nas/photos
        guest ok = no
 

2. configure users

I use the same setup users as the user on the windows machine

add the user :

adduser toto

setup its password: 

passwd toto

then add this user to Samba :

smbpasswd -a toto

then restart the daemon to take into account the config :

systemctl restart smbd 

proxmox HTTPS certificates for load balancing

generate a PKI and trust the Root CA in your browser (using XCA ?)
I used the CN as proxmox.domain.net, and then SAN to every proxmox$i.domain.net to use the same certificate for every node and the Virtual Server IP of the Load balancer (here the Fortigate)

export the certificate chain without the CA(.crt) and the key (.pem)

then copy on every node :


for i in {1..n}; \
do \
scp proxmox.domain.net.crt proxmox5:/etc/pve/nodes/proxmox$i/pve-ssl.pem; \
scp proxmox.domain.net.pem proxmox5:/etc/pve/nodes/proxmox$i/pve-ssl.key; \
ssh proxmox$i systemctl restart pveproxy;\
done



Then I use the Fortigate to load balance :
https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/713497/virtual-server

Kubernetes

Dashboard :

the URL is
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

in order to login with the config file in .kube/ folder, add the token of a service account with privileges (here cluster admin) :

get the token for user <toto>

TOKEN=$(kubectl -n kube-system describe secret toto| awk '$1=="token:"{print $2}')

add it to the config file :

kubectl config set-credentials kubernetes-admin --token="${TOKEN}" 

Waiting for Quorum on Proxmox

When an addition of a node to the cluster fails, there is the message 'waiting for quorum...' and on the GUI it is 'Invalid PVE Ticket' and you cannot login anymore.

Here is my script to retry a failed cluster :

systemctl stop pve-cluster.service
systemctl stop corosync
pmxcfs -l
rm /etc/pve/corosync.conf
rm -Rf /etc/corosync/*
killall pmxcfs
systemctl start pve-cluster.service

rm -Rf /etc/pve/nodes/proxmox5

echo "#### NOW on a cluster member, remove previous reference in these files:

root@proxmox1:~# rm -Rf /etc/pve/nodes/proxmox5
root@proxmox1:~# nano /etc/pve/corosync.conf
root@proxmox1:~# rm -Rf /var/lib/corosync/*
"

remote CEC server for Home assistant

I wanted to connect my Home assistant to the TV, so the pycec module on home assistant can connect to a Raspberry Pi that is connected to the TV and can be connected to Home assistant via TCP/IP.

I used this repository :
https://github.com/konikvranik/pycec/

First, enable the pycec module on home assistant.
In the configuration.yaml file, add :

hdmi_cec: !include config/hdmi_cec.yaml

and in the config/hdmi_cec.yaml file :

host: !secret rpi3b_ip

Then on the Raspberry pi :
cd
git clone  https://github.com/konikvranik/pycec/

First, compile the libcec :
https://github.com/Pulse-Eight/libcec

For compiling it, I followed the Linux procedure :

apt-get update
apt-get install cmake libudev-dev libxrandr-dev python-dev swig
git clone https://github.com/Pulse-Eight/libcec.git
mkdir libcec/build
cd libcec/build
cmake ..
make -j4
sudo make install
sudo ldconfig

Then symlink the service in the pycec repository :
ln -s /path/of/pycec.service /etc/systemd/system/pycec.service
systemctl start pycec

then enable the service so it starts at boot :
systemctl enable pycec