About

  • cmor-fixer is a tool developed by the EC-EARTH community, for ‘on-site fixing of cmorized data’.
  • cmor-fixer_env is a containerized deployment of the cmor-fixer tool.
  • Contains a wrapper script to allow for scripted/batch-job driven fixing.

The container

  • A predeployed singularity container cmorfixerenv.simg is now available on tetralith.
  • Available as /proj/s-cmip/singularity/cmorfixerenv.simg.
  • SHA256SUM: 8fff400a5acb62fefac8b62870fd8dbe9ec2baa61c2ae6bdd2c5ed4ad19d7f33
  • Wrapper script for non-interactive execution.
  • Gitlab repo for cmorfixer_env

General instructions

  • Singularity is only available on compute nodes on tetralith, so in order to use the container, ensure that you are already on a compute node.
  • A bug with cmorfixer causes the list of changed files to remain empty if the --npp value is greater than 1, so it is perhaps best to request for a single task, instead of a dedicated node.
  • You need to mount the directory containing the data (absolute path) to be scanned/fixed, so the container can see it.

Usage instructions (interactive)

  • On the compute node, do the following
singularity run --bind <path_to_datadirectory>:/mnt /proj/s-cmip/singularity/cmorfixerenv.simg
  • Example
singularity run --bind /home/pchengi/cmorfixer_test/CMIP6:/mnt /proj/s-cmip/singularity/cmorfixerenv.simg
Singularity> source /etc/bashrc
Singularity> activateminiconda3
Singularity> conda activate cmorfixer
(cmorfixer) Singularity>
(cmorfixer) Singularity> ls /mnt
CMIP6
(cmorfixer) Singularity> /opt/cmor_fixer/cmor-fixer/cmor-fixer.py /mnt/CMIP6 --verbose --olist --npp 1 --dry

Usage instructions (non-interactive)

[pchengi@n481 ~]$ singularity exec --bind /home/pchengi/cmorfix_test:/mnt /proj/s-cmip/singularity/cmorfixerenv.simg /opt/scripts/cmorfixerwrapper.sh /mnt/CMIP6 --verbose --olist --npp 1 --dry
2020-12-28 12:16:58 WARNING:cmor-fixer.py: Output file name list-of-modified-files.txt already exists, trying list-of-modified-files-2.txt...

Dry-run vs actual run

  • The flag --dry only checks if any issues are present.
  • If the flag --dry is used, the list-of-modified-files.txt will always be empty, even if there are files with issues.
  • If the flag --dry is used, the standard output would contain lines that would indicate issues.
  • To perform an actual cmorfix run, leave out the --dry flag.
  • If there would be unaffected files along with the affected files in the data directory, please use the --forceid files, to ensure that they too get new tracking ids.