change Ceph permissions

ceph auth list

...
client.cloud
    key: AQB6hPddq....HcQpUMPSNunT2w==
    caps: [mds] allow rw path=/nas/path/1, allow rw path=/nas/path/2
    caps: [mon] allow r
    caps: [osd] allow rw pool=cephfs_data
...

change with :

ceph auth caps client.cloud \
     mon 'allow r' \
     osd 'allow rw pool=cephfs_data' \
     mds 'allow rw path=/nas/path/1, allow rw path=/nas/path/3'

then check with ceph auth list

...
client.cloud
    key: AQB6hPddq....HcQpUMPSNunT2w==
    caps: [mds] allow rw path=/nas/path/1, allow rw path=/nas/path/3
    caps: [mon] allow r
    caps: [osd] allow rw pool=cephfs_data
...

LoadBalance Service for Bare metal / VM Kubernetes : Metallb

I use Metallb for the LoadBalance service of my kubernetes cluster.

helm install metallb --namespace=metallb stable/metallb

apply the configmap that is the default for the chart.

remember that for the Helm chart the configmap name has to be metallb-config, which is different from the metallb documentation.

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb
  name: metallb-config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
        - 10.0.14.100-10.0.14.200

I prefer the BGP implementation :

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb
  name: metallb-config
data:
  config: |
    peers:
    - peer-address: 10.0.14.1
      peer-asn: 64501
      my-asn: 64500
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 10.0.12.128/25

I peer the BGP peers with the Fortigate firewall which is 10.0.14.1 :

config router bgp
    set as 64501
    set router-id 0.0.0.1
    config neighbor
        edit "10.0.14.11"
            set remote-as 64500
        next
        edit "10.0.14.12"
            set remote-as 64500
        next

        ... (and the remaining nodes)

        next
    end
    config redistribute "connected"
    end
    config redistribute "rip"
    end
    config redistribute "ospf"
    end
    config redistribute "static"
    end
    config redistribute "isis"
    end
    config redistribute6 "connected"
    end
    config redistribute6 "rip"
    end
    config redistribute6 "ospf"
    end
    config redistribute6 "static"
    end
    config redistribute6 "isis"
    end
end

do not forget to open the firewall rules then.

reference for the Helm chart :

https://hub.helm.sh/charts/stable/metallb

Ceph for Kubernetes

follow :
https://github.com/kubernetes-incubator/external-storage/

Straight forward with some modifications, as the claim has to have a different name, and pay attention that for CephFS and RDB, the admin secret name is different, but can be set the same.

I also used the same cephfs namespace for both.

EDIT : I made a PR that uses now the same kubernetes user for both storageclass
here : https://github.com/kubernetes-incubator/external-storage/pull/1306

ZFS storage driver for docker

This is used to have a docker setup directly on Proxmox 6 with ZFS 


stop the docker service

service docker stop


delete all the docker stuff

rm -rf /var/lib/docker


create a zfs pool dedicated for docker
  
zfs create -o mountpoint=/var/lib/docker rpool/docker


create the systemd entry for the docker service

mkdir /etc/systemd/system/docker.service.d


setup the driver for docker

nano /etc/systemd/system/docker.service.d/storage-driver.conf


[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=zfs -H fd://


then restart daemons and docker

systemctl daemon-reload
service docker start





Samba over CephFS

I wanted to use NFS to share data on the network, but Windows 10 still isn't able to use NFS ... what a joke.

So first mount CephFS (see previous article)

Then configure the Samba server

1. install samba server 

apt-get install samba

2. edit the configuration file

set the workgroup

add a samba share, for example

[photos]
        comment = photos
        read only = no
        path = /mnt/cephfs/nas/photos
        guest ok = no
 

2. configure users

I use the same setup users as the user on the windows machine

add the user :

adduser toto

setup its password: 

passwd toto

then add this user to Samba :

smbpasswd -a toto

then restart the daemon to take into account the config :

systemctl restart smbd