GenAp CVMFS New Features Release Notes
Overview Elements of CVMFS File System
Software Modules Currently Installed in GenAp CVMFS as of 2015-Jan-30
Dependency Softwares Currently Installed in GenAp CVMFS as of 2014-Jan-30.
GenAp CVMFS Stratums and Squid Monitoring.
GenAp CVMFS Current/Future Sites
Quick Demo Using Main Components of CVMFS Stratum-0, Stratum-1, Squid and CVMFS Client
Requirements for Running the CVMFS Quick Demo
Setup Instruction for Stratum 0:
Setup Instruction for Stratum 1:
Setup Instruction for Squid Cache Proxy:
Setup Instruction for CVMFS clients (a machine you have provided to install CVMFS client):
Dependency library added:
Module added:
Network infrastructure:
Security infrastructure:
Dependency library added:
Module added:
Network infrastructure:
Security infrastructure:
Dependency library added:
Module added:
Network infrastructure:
Security infrastructure:
Dependency library added:
Module added:
Network infrastructure:
Security infrastructure:
Install softwares securely in one location and have them propagated to the remote GenAp participating sites without worrying any prerequisites, nor linux distribution and release level.
CVMFS stands for CERN virtual Machine File System. It is a file system hosted on standard web servers and accessed by clients using FUSE (file system in userspace). Software can be installed in one location and cached on demand anywhere using CVMFS. Originally developed for the LHC (Large Hadron Collider) experiments to optimally deliver software for VM images and as a replacement for different software installation areas at many distributed locations. Designed to deliver software in a fast, scalable and reliable fashion. Hundreds of compute sites using CVMFS over the last 2 to 3 years with success.
Maintain one copy of the files for multiple sites. Softwares need not to be embedded in the original image or virtual machine. Virtual machine images and softwares can be maintained separately. Files are fetched using standard http protocol from the server. No additional ports are required to be opened in most environment. Files are downloaded locally and cached on demand on CVMFS client in order to reduce network traffic. The CVMFS structure is highly scalable and redundant. Remote squid can be used as a failover cache server for local CVMFS client. CVMFS clients are only working on a read-only copy of the file on the server. Changes on the files are not propagated until a commit has been issued. Files are encrypted using RSA. Software and prerequisites can be installed in CVMFS in order to reduce remote software administration. Software can be installed in one location and propagated to multiple sites. Software versioning is reinforced across multiple sites.
Files are injected by an application software maintainer / librarian using dedicated Compute Canada Guillimin login node (guillimin1.clumeq.ca) through sshfs using libuser account. The committed change will propagate the injected files from stratum-0 to stratum-1 and then only cached to squid and to the client node (compute node) on demand.
Our GenAp CVMFS implementation focus on flexibility of code injecting without giving shell access to Stratum-0 and at the same time be able to execute CVMFS librarian commands. Multiple developers can test programs on the dedicated login node. Using a sudo privileged command to switch from developer to CVMFS librarian, the CVMFS librarian commands can be executed. Only one single librarian session is allowed. This will in turn guaranteed the right content being published.
GenAp CVMFS participating sites do not have to use same linux distribution nor the same release level because softwares and other prerequisites are installed in CVMFS. Hence, no softwares are local OS dependent. No additional tasks shall be performed on the remote site except for installing the CVMFS client software.
Replication between Stratums are being monitored just in case if they are out of sync and so is Squid cache size since having an optimal size can reduce unnecessary files fetching due to files being swapped out.
Cache size at the squid level should be about 500Gb. Cache size at the CVMFS client level should be about 30Gb.
The CVMFS license and most linux open source software uses GNU GPL. The details of the license can be found at http://www.gnu.org/copyleft/gpl.html
Question regarding GenAp CVMFS can be send to guillimin@calculquebec.ca
The version of CVMFS used in the demo is 2.1.19. KVM is required to load Stratum-0, Stratum-1 and Squid virtual machines. A CVMFS client has to be provided on your end to complete the demo.
The quick demo is used to demonstrate file distribution from Stratum-0 to CVMFS client using the VMs provided for Stratum-0, Stratum-1, Squid and CVMFS client configuration.
[root@ns-vm32 ~] # mount -rw -o remount /
[root@ns-vm32 ~] # passwd
[root@ns-vm33 ~] # mount /boot
[root@ns-vm33 ~] # sed -i ‘s@init=/bin/bash@@g’ /boot/grub/grub.conf
virsh reboot ns-vm32 or virsh destroy ns-vm32
KVM host: virsh start ns-vm32
10.243.0.32 ns-vm32
10.243.0.33 ns-vm33
10.243.0.34 ns-vm34
[root@ns-vm32 ~] # cvmfs_server mkfs test.domain
Owner of test.domain [root]: root
[root@ns-vm32 ~] # cvmfs_server info
Repository name: test.domain
Created by CernVM-FS 2.1.19
Stratum1 Replication Allowed: yes
Whitelist is valid for another 30 days
Client configuration:
Add test.domain to CVMFS_REPOSITORIES in /etc/cvmfs/default.local
Create /etc/cvmfs/config.d/test.domain.conf and set
CVMFS_SERVER_URL=http://localhost/cvmfs/test.domain
CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/test.domain.pub
Copy /etc/cvmfs/keys/test.domain.pub to the client
[root@ns-vm32 ~] # cvmfs_server transaction test.domain
[root@ns-vm32 ~] # touch /cvmfs/test.domain/hello
[root@ns-vm32 ~] # cvmfs_server publish test.domain
Processing changes...
Waiting for upload of files before committing…
Committing file catalogs…
Exporting repository manifest
Tagging test.domain
Signing /var/spool/cvmfs/test.domain/tmp/manifest
Remounting newly created repository revision
[root@ns-vm32 ~] # echo “*/5 * * * * root /usr/bin/cvmfs_server resign test.domain” > /etc/cron.d/resign
[root@ns-vm33 ~] # mount -rw -o remount /
[root@ns-vm33 ~] # passwd
[root@ns-vm33 ~] # mount /boot
[root@ns-vm33 ~] # sed -i ‘s@init=/bin/bash@@g’ /boot/grub/grub.conf
virsh reboot ns-vm33 or virsh destroy ns-vm33
KVM host: virsh start ns-vm33
10.243.0.32 ns-vm32
10.243.0.33 ns-vm33
10.243.0.34 ns-vm34
[root@ns-vm33 ~] # scp -p root@ns-vm32:/etc/cvmfs/keys/test.domain.pub /etc/cvmfs/keys/
[root@ns-vm33 ~] # cvmfs_server add-replica -o cvmfs http://ns-vm32/cvmfs/test.domain /etc/cvmfs/keys/test.domain.pub
[root@ns-vm33 ~] # cvmfs_server snapshot test.domain
Initial snapshot
CernVM-FS: replicating from http://ns-vm32/cvmfs/test.domain
CernVM-FS: using public key(s) /etc/cvmfs/keys/test.domain.pub
Found 2 named snapshots
Uploading history database
Starting 16 workers
Replicating from trunk catalog at /
Processing chunks: . fetched 2 new chunks out of 2 processed chunks
Replicating from trunk repository tag
Catalog up to date
Replicating from trunk-previous repository tag
Processing chunks: fetched 0 new chunks out of 1 processed chunks
Stopping 16 workers
Uploading manifest ensemble
Fetched 2 new chunks out of 3 processed chunks
[root@ns-vm33 ~] # cvmfs_server list
*/5 * * * * root (for r in `cd /srv/cvmfs;echo *.*`; do echo; if [ -f /srv/cvmfs/$r/.cvmfspublished ]; then echo "Starting $r at `date`";if ! cvmfs_server snapshot $r;then echo "ERROR from cvmfs_server!"; fi;echo "Finished $r at `date`";fi;done) >>/var/log/cvmfs.cron.log 2>&1
[root@ns-vm34 ~] # mount -rw -o remount /
[root@ns-vm34 ~] # passwd
[root@ns-vm34 ~] # mount /boot
[root@ns-vm34 ~] # sed -i ‘s@init=/bin/bash@@g’ /boot/grub/grub.conf
virsh reboot ns-vm34 or virsh destroy ns-vm34
KVM host: virsh start ns-vm34
10.243.0.32 ns-vm32
10.243.0.33 ns-vm33
10.243.0.34 ns-vm34
[root@ns-vm34 ~] # wget -e "http_proxy=http://localhost:3128" http://ns-vm32/cvmfs/test.domain/.cvmfspublished
[root@ns-vm34 ~] # wget -e "http_proxy=http://localhost:3128" http://ns-vm33/cvmfs/test.domain/.cvmfspublished
10.243.0.32 ns-vm32
10.243.0.33 ns-vm33
10.243.0.34 ns-vm34
[root@cvmfsclient ~] # cd /tmp
[root@cvmfsclient ~] # wget https://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM
[root@cvmfsclient ~] # wget --no-check-certificate https://ecsft.cern.ch/dist/cvmfs/cvmfs-init-scripts/cvmfs-init-scripts-1.0.20-1.noarch.rpm [root@cvmfsclient ~] # wget --no-check-certificate https://ecsft.cern.ch/dist/cvmfs/cvmfs-keys/cvmfs-keys-1.4-1.noarch.rpm
[root@cvmfsclient ~] # wget --no-check-certificate https://ecsft.cern.ch/dist/cvmfs/cvmfs-2.1.19/cvmfs-2.1.19-1.el6.x86_64.rpm
[root@cvmfsclient ~] # yum localinstall cvmfs-2.1.19-1.el6.x86_64.rpm cvmfs-keys-1.4-1.noarch.rpm cvmfs-init-scripts-1.0.20-1.noarch.rpm
CVMFS_REPOSITORIES=test.domain
CVMFS_CACHE_BASE=/cvmfs-cache/cache/cvmfs2
CVMFS_HTTP_PROXY="http://ns-vm34:3128"
CVMFS_QUOTA_LIMIT=25500
[root@cvmfsclient ~] # scp -p root@ns-vm32:/etc/cvmfs/keys/test.domain.pub /etc/cvmfs/keys/
CVMFS_REPOSITORIES=test.domain
CVMFS_CACHE_BASE=/cvmfs-cache/cache/cvmfs2
CVMFS_QUOTA_LIMIT=24423
CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/test.domain.pub
CVMFS_SERVER_URL=http://ns-vm33/cvmfs/test.domain
CVMFS_HTTP_PROXY=http://ns-vm34:3128
/cvmfs /etc/auto.cvmfs
[root@cvmfsclient ~] # chkconfig autofs on
[root@cvmfsclient ~] # /etc/init.d/autofs start
[root@cvmfsclient ~] # wget -e "http_proxy=http://ns-vm34:3128" http://ns-vm32/cvmfs/test.domain/.cvmfspublished
[root@cvmfsclient ~] # wget -e "http_proxy=http://ns-vm34:3128" http://ns-vm33/cvmfs/test.domain/.cvmfspublished
[root@cvmfsclient ~] # /usr/bin/cvmfs_config chksetup
[root@cvmfsclient ~] # /usr/bin/cvmfs_config probe
Probing /cvmfs/test.domain... OK
cd /cvmfs/test.domain && ls