GenAp CVMFS

GenAp CVMFS New Features Release Notes

2015 - Jan - 29th

2015 - Jan - 13th

2015 - Jan - 5th

2014 - July - 5th

GenAp CVMFS Objective

CVMFS Overview

CVMFS Advantages

Overview Elements of CVMFS File System

How does it work

GenAp CVMFS Overview

Software Modules Currently Installed in GenAp CVMFS as of 2015-Jan-30

Dependency Softwares Currently Installed in GenAp CVMFS as of 2014-Jan-30.

GenAp CVMFS Stratums and Squid Monitoring.

GenAp CVMFS Site Requirements

GenAp CVMFS Current/Future Sites

GenAp CVMFS License

GenAp CVMFS Support

Quick Demo Using Main Components of CVMFS Stratum-0, Stratum-1, Squid and CVMFS Client

Requirements for Running the CVMFS Quick Demo

Objective of the Quick Demo

Setup Instruction for Stratum 0:

Setup Instruction for Stratum 1:

Setup Instruction for Squid Cache Proxy:

Setup Instruction for CVMFS clients (a machine you have provided to install CVMFS client):

GenAp CVMFS New Features Release Notes

2015 - Jan - 29th

Dependency library added:

Module added:

Network infrastructure:

Security infrastructure:

2015 - Jan - 13th

Dependency library added:

Module added:

Network infrastructure:

Security infrastructure:

2015 - Jan - 5th

Dependency library added:

Module added:

Network infrastructure:

Security infrastructure:

2014 - July - 5th

Dependency library added:

Module added:

Network infrastructure:

Security infrastructure:

GenAp CVMFS Objective

Install softwares securely in one location and have them propagated to the remote GenAp participating sites without worrying any prerequisites, nor linux distribution and release level.

CVMFS Overview

CVMFS stands for CERN virtual Machine File System.  It is a file system hosted on standard web servers and accessed by clients using FUSE (file system in userspace).  Software can be installed in one location and cached on demand anywhere using CVMFS.  Originally developed for the LHC (Large Hadron Collider) experiments to optimally deliver software for VM images and as a replacement for different software installation areas at many distributed locations.  Designed to deliver software in a fast, scalable and reliable fashion.  Hundreds of compute sites using CVMFS over the last 2 to 3 years with success.

CVMFS Advantages

Maintain one copy of the files for multiple sites.  Softwares need not to be embedded in the original image or virtual machine.  Virtual machine images and softwares can be maintained separately.  Files are fetched using standard http protocol from the server.  No additional ports are required to be opened in most environment.  Files are downloaded locally and cached on demand on CVMFS client in order to reduce network traffic.  The CVMFS structure is highly scalable and redundant.  Remote squid can be used as a failover cache server for local CVMFS client.  CVMFS clients are only working on a read-only copy of the file on the server.  Changes on the files are not propagated until a commit has been issued.  Files are encrypted using RSA.  Software and prerequisites can be installed in CVMFS in order to reduce remote software administration.  Software can be installed in one location and propagated to multiple sites.  Software versioning is reinforced across multiple sites.

Overview Elements of CVMFS File System

How does it work

Files are injected by an application software maintainer / librarian using dedicated Compute Canada Guillimin login node (guillimin1.clumeq.ca) through sshfs using libuser account.  The committed change will propagate the injected files from stratum-0 to stratum-1 and then only cached to squid and to the client node (compute node) on demand.

Our GenAp CVMFS implementation focus on flexibility of code injecting without giving shell access to Stratum-0 and at the same time be able to execute CVMFS librarian commands.  Multiple developers can test programs on the dedicated login node.  Using a sudo privileged command to switch from developer to CVMFS librarian, the CVMFS librarian commands can be executed.  Only one single librarian session is allowed.  This will in turn guaranteed the right content being published.Librarian.png

GenAp CVMFS Overview

GenAp CVMFS participating sites do not have to use same linux distribution nor the same release level because softwares and other prerequisites are installed in CVMFS.  Hence, no softwares are local OS dependent.  No additional tasks shall be performed on the remote site except for installing the CVMFS client software.  

cvmfs_org_proxy_clients_info_fit.png

Software Modules Currently Installed in GenAp CVMFS as of 2015-Jan-30

Dependency Softwares Currently Installed in GenAp CVMFS as of 2014-Jan-30.

GenAp CVMFS Stratums and Squid Monitoring.

Replication between Stratums are being monitored just in case if they are out of sync and so is Squid cache size since having an optimal size can reduce unnecessary files fetching due to files being swapped out.

cvmfs-guillimin-librarian-monitoring.jpg

GenAp CVMFS Site Requirements

Cache size at the squid level should be about 500Gb.  Cache size at the CVMFS client level should be about 30Gb.

GenAp CVMFS Current/Future Sites

GenAp CVMFS License

The CVMFS license and most linux open source software uses GNU GPL.  The details of the license can be found at http://www.gnu.org/copyleft/gpl.html

GenAp CVMFS Support

Question regarding GenAp CVMFS can be send to guillimin@calculquebec.ca

Quick Demo Using Main Components of CVMFS Stratum-0, Stratum-1, Squid and CVMFS Client

Requirements for Running the CVMFS Quick Demo

The version of CVMFS used in the demo is 2.1.19.  KVM is required to load Stratum-0, Stratum-1 and Squid virtual machines.  A CVMFS client has to be provided on your end to complete the demo.

Objective of the Quick Demo

The quick demo is used to demonstrate file distribution from Stratum-0 to CVMFS client using the VMs provided for Stratum-0, Stratum-1, Squid and CVMFS client configuration.

Setup Instruction for Stratum 0:

  1. KVM host: download the files ns-vm32.hda.qcow2, ns-vm32.hdb.qcow2 and ns-vm32.xml
  2. KVM host: virsh define ns-vm32.xml
  3. KVM host: virsh start ns-vm32
  4. KVM guest ns-vm32: remount the / filesystem and change the password:

[root@ns-vm32 ~] # mount -rw -o remount /

[root@ns-vm32 ~] # passwd

  1. KVM guest ns-vm32: mount the /boot filesystem and remove init=/bin/bash in the file /boot/grub/grub.conf

        [root@ns-vm33 ~] # mount /boot

        [root@ns-vm33 ~] # sed -i ‘s@init=/bin/bash@@g’ /boot/grub/grub.conf

  1. KVM host: Perform a hard reset on the VM.  Your VM will now boot all the way with the new root password that you have set.

        virsh reboot ns-vm32 or virsh destroy ns-vm32

KVM host: virsh start ns-vm32

  1. KVM guest ns-vm32: login as root
  2. Add the following hostnames and IP addresses /etc/hosts:

10.243.0.32 ns-vm32

10.243.0.33 ns-vm33

10.243.0.34 ns-vm34

  1. KVM guest ns-vm32: create a testing CVMFS filesystem and use root as the main file maintainer

[root@ns-vm32 ~] # cvmfs_server mkfs test.domain

Owner of test.domain [root]: root

  1. KVM guest ns-vm32: check test.domain to see if the CVMFS filesystem has been created properly.

        [root@ns-vm32 ~] # cvmfs_server info

Repository name: test.domain

Created by CernVM-FS 2.1.19

Stratum1 Replication Allowed: yes

Whitelist is valid for another 30 days

Client configuration:

Add test.domain to CVMFS_REPOSITORIES in /etc/cvmfs/default.local

Create /etc/cvmfs/config.d/test.domain.conf and set

          CVMFS_SERVER_URL=http://localhost/cvmfs/test.domain

          CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/test.domain.pub

Copy /etc/cvmfs/keys/test.domain.pub to the client

  1. KVM guest ns-vm32: Inject a file for testing.

        [root@ns-vm32 ~] # cvmfs_server transaction test.domain

        [root@ns-vm32 ~] # touch /cvmfs/test.domain/hello

  1. KVM guest ns-vm32: Publish the file hello so that other CVMFS clients will see it

        [root@ns-vm32 ~] # cvmfs_server publish test.domain

Processing changes...

Waiting for upload of files before committing…

Committing file catalogs…

Exporting repository manifest

Tagging test.domain

Signing /var/spool/cvmfs/test.domain/tmp/manifest

Remounting newly created repository revision

  1. KVM guest ns-vm32: create a cron job /etc/cron.d/resign to resign the CVMFS repository for the test.domain

[root@ns-vm32 ~] # echo “*/5 * * * * root /usr/bin/cvmfs_server resign  test.domain” > /etc/cron.d/resign

 

Setup Instruction for Stratum 1:

  1. KVM host: download the files ns-vm33.hda.qcow2, ns-vm33.hdb.qcow2 and ns-vm33.xml
  2. KVM host: virsh define ns-vm33.xml
  3. KVM host: virsh start ns-vm33
  4. KVM guest ns-vm33: remount the / filesystem and change the password:

[root@ns-vm33 ~] # mount -rw -o remount /

[root@ns-vm33 ~] # passwd

  1. KVM guest ns-vm33: mount the /boot filesystem and remove init=/bin/bash in the file /boot/grub/grub.conf

        [root@ns-vm33 ~] # mount /boot

        [root@ns-vm33 ~] # sed -i ‘s@init=/bin/bash@@g’ /boot/grub/grub.conf

  1. KVM host: Perform a hard reset on the VM.  Your VM will now boot all the way with the new root password that you have set.

        virsh reboot ns-vm33 or virsh destroy ns-vm33

KVM host: virsh start ns-vm33

  1. KVM guest ns-vm33: login as root
  2. Add the following hostnames and IP addresses /etc/hosts:

10.243.0.32 ns-vm32

10.243.0.33 ns-vm33

10.243.0.34 ns-vm34

  1. KVM guest ns-vm33: copy the public key ns-vm32:/etc/cvmfs/keys/test.domain.pub to ns-vm33:/etc/cvmfs/keys

[root@ns-vm33 ~] # scp -p root@ns-vm32:/etc/cvmfs/keys/test.domain.pub /etc/cvmfs/keys/

  1. KVM guest ns-vm33: add a replication to ns-vm33 and take a snapshot of the content on ns-vm32.

[root@ns-vm33 ~] # cvmfs_server add-replica -o cvmfs http://ns-vm32/cvmfs/test.domain    /etc/cvmfs/keys/test.domain.pub

[root@ns-vm33 ~] # cvmfs_server snapshot test.domain

Initial snapshot

CernVM-FS: replicating from http://ns-vm32/cvmfs/test.domain

CernVM-FS: using public key(s) /etc/cvmfs/keys/test.domain.pub

Found 2 named snapshots

Uploading history database

Starting 16 workers

Replicating from trunk catalog at /

  Processing chunks: . fetched 2 new chunks out of 2 processed chunks

Replicating from trunk repository tag

  Catalog up to date

Replicating from trunk-previous repository tag

  Processing chunks:  fetched 0 new chunks out of 1 processed chunks

Stopping 16 workers

Uploading manifest ensemble

Fetched 2 new chunks out of 3 processed chunks

  1. KVM guest ns-vm33: verify that ns-vm33 is a stratum 1

[root@ns-vm33 ~] # cvmfs_server list

  1. KVM guest ns-vm33: create a cron job /etc/cron.d/cvmfs.cron to regularly take a snapshot using ns-vm32.  The content of /etc/cron.d/cvmfs.cron should be

*/5 * * * *    root    (for r in `cd /srv/cvmfs;echo *.*`; do echo; if [ -f /srv/cvmfs/$r/.cvmfspublished ]; then echo "Starting $r at `date`";if ! cvmfs_server snapshot $r;then echo "ERROR from cvmfs_server!"; fi;echo "Finished $r at `date`";fi;done) >>/var/log/cvmfs.cron.log 2>&1

Setup Instruction for Squid Cache Proxy:

  1. KVM host: download the files ns-vm34.hda.qcow2, ns-vm34.hdb.qcow2 and ns-vm34.xml
  2. KVM host: virsh define ns-vm34.xml
  3. KVM host: virsh start ns-vm34
  4. KVM guest ns-vm34: remount the / filesystem and change the password:

[root@ns-vm34 ~] # mount -rw -o remount /

[root@ns-vm34 ~] # passwd

  1. KVM guest ns-vm34: mount the /boot filesystem and remove init=/bin/bash in the file /boot/grub/grub.conf

        [root@ns-vm34 ~] # mount /boot

        [root@ns-vm34 ~] # sed -i ‘s@init=/bin/bash@@g’ /boot/grub/grub.conf

  1. KVM host: Perform a hard reset on the VM.  Your VM will now boot all the way with the new root password that you have set.

        virsh reboot ns-vm34 or virsh destroy ns-vm34

KVM host: virsh start ns-vm34

  1. KVM guest ns-vm34: login as root
  2. Add the following hostnames and IP addresses /etc/hosts:

10.243.0.32 ns-vm32

10.243.0.33 ns-vm33

10.243.0.34 ns-vm34

  1. make sure ns-vm32 and ns-vm33 can be reached.

[root@ns-vm34 ~] #  wget -e "http_proxy=http://localhost:3128" http://ns-vm32/cvmfs/test.domain/.cvmfspublished

[root@ns-vm34 ~] #  wget -e "http_proxy=http://localhost:3128" http://ns-vm33/cvmfs/test.domain/.cvmfspublished

Setup Instruction for CVMFS clients (a machine you have provided to install CVMFS client):

  1. Add the following hostnames and IP addresses /etc/hosts:

10.243.0.32 ns-vm32

10.243.0.33 ns-vm33

10.243.0.34 ns-vm34

  1. Save the following files in /tmp:

        [root@cvmfsclient ~] # cd /tmp

[root@cvmfsclient ~] # wget https://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM 

[root@cvmfsclient ~] # wget --no-check-certificate https://ecsft.cern.ch/dist/cvmfs/cvmfs-init-scripts/cvmfs-init-scripts-1.0.20-1.noarch.rpm [root@cvmfsclient ~] # wget --no-check-certificate https://ecsft.cern.ch/dist/cvmfs/cvmfs-keys/cvmfs-keys-1.4-1.noarch.rpm

[root@cvmfsclient ~] # wget --no-check-certificate https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.19/cvmfs-2.1.19-1.el6.x86_64.rpm

  1. Install the rpm files you have downloaded

[root@cvmfsclient ~] # yum localinstall cvmfs-2.1.19-1.el6.x86_64.rpm cvmfs-keys-1.4-1.noarch.rpm cvmfs-init-scripts-1.0.20-1.noarch.rpm

  1. Add the following lines to /etc/cvmfs/default.local

CVMFS_REPOSITORIES=test.domain

CVMFS_CACHE_BASE=/cvmfs-cache/cache/cvmfs2

CVMFS_HTTP_PROXY="http://ns-vm34:3128"

CVMFS_QUOTA_LIMIT=25500

  1. Copy the public key ns-vm32:/etc/cvmfs/keys/test.domain.pub to cvmfsclient:/etc/cvmfs/keys

[root@cvmfsclient ~] # scp -p root@ns-vm32:/etc/cvmfs/keys/test.domain.pub /etc/cvmfs/keys/

  1. Add the following to In /etc/cvmfs/config.d/test.domain.conf:

CVMFS_REPOSITORIES=test.domain

CVMFS_CACHE_BASE=/cvmfs-cache/cache/cvmfs2

CVMFS_QUOTA_LIMIT=24423

CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/test.domain.pub

CVMFS_SERVER_URL=http://ns-vm33/cvmfs/test.domain

CVMFS_HTTP_PROXY=http://ns-vm34:3128

  1. Add the following In /etc/autofs, add the following if it is not there

/cvmfs /etc/auto.cvmfs

  1. Make sure autofs will start automatically.

[root@cvmfsclient ~] # chkconfig autofs on

  1. Start autofs.

[root@cvmfsclient ~] # /etc/init.d/autofs start

  1. Make sure ns-vm32 and ns-vm33 can be reached.

        [root@cvmfsclient ~] # wget -e "http_proxy=http://ns-vm34:3128" http://ns-vm32/cvmfs/test.domain/.cvmfspublished

        [root@cvmfsclient ~] # wget -e "http_proxy=http://ns-vm34:3128" http://ns-vm33/cvmfs/test.domain/.cvmfspublished

  1. Testing CVMFS client:

[root@cvmfsclient ~] # /usr/bin/cvmfs_config chksetup

[root@cvmfsclient ~] # /usr/bin/cvmfs_config probe

Probing /cvmfs/test.domain... OK

cd /cvmfs/test.domain && ls