BioMAX scripts at online and offline cluster

BioMAX offers two login opportunities using thinlinc

  • online cluster clu0-fe-1.maxiv.lu.se with 24 cores per compute node and hyperthreaded to handle 48 tasks per compute node
  • offline cluster offline-fe1.maxiv.lu.se with 20 cores per compute node and hyperthreaded to handle 40 tasks per compute node

Please find a description of the online cluster under 'Manual data reduction' and the offline cluster at BioMAX. The MAXIV online and offline clusters share the same PReSTO software installation.

BioMAX is equipped with an Eiger 16M detector and GlobalPhasing made a useful list of beamline specific settings around the world however here we discuss settings and scripts suitable for BioMAX online and offline clusters.

Evaluating diffraction on Eiger detectors

ALBULA and ADXV are software used to monitor crystal X-ray diffraction during screening and first characterization. ALBULA, developed by Eiger detector manufacturers, have no issues on reading the HFD5 format. For ADXV, we developed an adxv_eiger.sh script to read all metadata from the master file when using ADXV as:

module load adxv
adxv_eiger.sh protein_x001_1_master.h5

The adxv_eiger.sh script has to be runnable e.g. by chmod 755 adxv_eiger.sh.

XDSAPP, autoPROC and DIALS at MAX IV online cluster

The native.script and anomalous.script below is intended for native/anomalous data processing with XDSAPP, autoPROC and xia2/DIALS at the MAX IV kluster. XDSAPP runs at two compute nodes while xia2/DIALS and autoPROC use a single compute node each. In contrast to XDSAPP, that benefit from several nodes, xia2/DIALS and autoPROC contain many serial subroutines and use of more than a single core does not save much wall-clock time. To run native.script copy-paste it and make it runnable by chmod 755 native.script, create the output directory and execute ./native.script /path/file_master.h5 /path/output-directory 1 3600 and please note the output directory must exist or otherwise the scripts does not run.

A few parameters in native.script/anomalous.script are BioMAX specific such as goniometer.axes=0,1,0 and since every compute node at MAXIV online cluster have 24 cores that are hyperthreaded to handle 48 tasks (check with sinfo -N -o "%N %c"), JOBS x PROCESSORS should equal 48 in autoPROC using a single compute node and 96 in XDSAPP using two compute nodes.

The native.script and anomalous.script below is very similar in fact there are three changes only

  • exchange --fried=true for --fried=false in xdsapp
  • exchange -noANO for -ANO in autoproc
  • add a new row atom=X for xia2/dials

in the native script below, and viola you have a script for anomalous data processing.

native.script for online cluster handling 48 tasks per compute node

#!/bin/sh -eu

#
# Argument list:
# $1: indata
# $2: utdata
# $3: fromrange 
# $4: torange  
#

indir=`realpath -es "$1"`
outdir=`realpath -es "$2"`

xdsapp="\
module load XDSAPP
xdsapp --cmd --dir=$outdir/xdsapp -j 12 -c 8 -i $indir  --fried=true --range=$3\ $4 
"

autoproc="\
module load autoPROC
process -h5 $indir \
    -noANO \
    autoPROC_XdsKeyword_LIB=\$EBROOTDURIN/lib/durin-plugin.so \
    autoPROC_XdsKeyword_ROTATION_AXIS='0  -1 0' \
    autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_JOBS=6 \
    autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_PROCESSORS=8 \
    autoPROC_XdsKeyword_DATA_RANGE=$3\ $4 \
    autoPROC_XdsKeyword_SPOT_RANGE=$3\ $4 \
    -d $outdir/autoproc 
"

dials="\
module load DIALS
cd $outdir/dials
xia2 goniometer.axes=0,1,0 \
    pipeline=dials failover=true \
    image=$indir:$3:$4 \
    multiprocessing.mode=serial \
    multiprocessing.njob=1 \
    multiprocessing.nproc=auto
"

# autoproc bails out if its outdir basename exists; don't make it
mkdir "$outdir/xdsapp" "$outdir/dials"

#echo "$xdsapp"
#echo "$autoproc"
#echo "$dials"

sbatch -N2 --exclusive --ntasks-per-node=48 -J XDSAPP -o "$outdir/xdsapp.out" --wrap="$xdsapp"
sbatch -N1 --exclusive --ntasks-per-node=48 -J autoPROC -o "$outdir/autoproc.out" --wrap="$autoproc"
sbatch -N1 --exclusive --ntasks-per-node=48 -J DIALS -o "$outdir/dials.out" --wrap="$dials"

anomalous.script for online cluster handling 48 tasks per compute node

#!/bin/sh -eu

#
# Argument list:
# $1: indata
# $2: utdata
# $3: fromrange 
# $4: torange  
#

indir=`realpath -es "$1"`
outdir=`realpath -es "$2"`

xdsapp="\
module load XDSAPP
xdsapp --cmd --dir=$outdir/xdsapp -j 12 -c 8 -i $indir  --fried=false --range=$3\ $4 
"

autoproc="\
module load autoPROC
process -h5 $indir \
    -ANO \
    autoPROC_XdsKeyword_LIB=\$EBROOTDURIN/lib/durin-plugin.so \
    autoPROC_XdsKeyword_ROTATION_AXIS='0  -1 0' \
    autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_JOBS=8 \
    autoPROC_XdsKeyword_MAXIMUM_NUMBER_OF_PROCESSORS=6 \
    autoPROC_XdsKeyword_DATA_RANGE=$3\ $4 \
    autoPROC_XdsKeyword_SPOT_RANGE=$3\ $4 \
    -d $outdir/autoproc 
"

dials="\
module load DIALS
cd $outdir/dials
xia2 goniometer.axes=0,1,0 \
    atom=X \
    pipeline=dials failover=true \
    image=$indir:$3:$4 \
    multiprocessing.mode=serial \
    multiprocessing.njob=1 \
    multiprocessing.nproc=auto
"

# autoproc bails out if its outdir basename exists; don't make it
mkdir "$outdir/xdsapp" "$outdir/dials"

#echo "$xdsapp"
#echo "$autoproc"
#echo "$dials"

sbatch -N2 --exclusive --ntasks-per-node=48 -J XDSAPP   -o "$outdir/xdsapp.out"   --wrap="$xdsapp"
sbatch -N1 --exclusive --ntasks-per-node=48 -J autoPROC -o "$outdir/autoproc.out" --wrap="$autoproc"
sbatch -N1 --exclusive --ntasks-per-node=48 -J DIALS    -o "$outdir/dials.out"    --wrap="$dials"

XDS and its derivatives

ROTATION_AXIS parameter

The crystal rotation axis used by BioMAX is a key parameter that needs to be fed into XDS and its associated softwares XDSAPP, XDSGUI and autoPROC

Software Keyword Input file or script

XDS

ROTATION_AXIS= 0 -1 0

XDS.INP

XDSGUI

ROTATION_AXIS= 0 -1 0

XDS.INP

XDSAPP --cmd

--raxis="0 -1 0"

xdsapp.script

autoPROC

autoPROC_XdsKeyword_ROTATION_AXIS="0 -1 0"

process.script

Table 1. Various ways to set crystal rotation axis to "0 -1 0".

When using graphical user interfaces (GUIs) crystal rotation axis are set inside these GUIs

Figure 1. How to set crystal "Rotation axis" in XDSAPP GUI.

Figure 2. How to set crystal "Rotation axis" in XDSGUI

Lines for XDS.INP in XDSGUI

To run with HDF5 containers XDSGUI require the following steps

  1. Launch XDSGUI from the PReSTO Menu
  2. Use the slider to select 24 cores online cluster, and 20 cores at offline cluster
  3. Select "Folder with XDS configuration and output file" in XDSGUI
  4. Load the master container and press "generate_XDS.INP" button
  5. Add these three lines to XDS.INP

    **online cluster gn0-fe-1.maxiv.lu.se**
    MAXIMUM_NUMBER_OF_JOBS= 4
    MAXIMUM_NUMBER_OF_PROCESSORS= 6
    LIB=/sw/pkg/presto/software/Durin/2019v1-foss-2018b-3/lib/durin-plugin.so
    
    **offline cluster offline-fe1.maxiv.lu.se**
    MAXIMUM_NUMBER_OF_JOBS= 4
    MAXIMUM_NUMBER_OF_PROCESSORS= 5
    LIB=/sw/pkg/presto/software/Durin/2019v1-foss-2018b-3/lib/durin-plugin.so
  6. Now press "Run XDS" and off you go

XDSME script for BioMAX data

#!/bin/sh
#SBATCH -t 00:30:00
#SBATCH -N 1 --exclusive
#SBATCH -A snic2020-5-368
#SBATCH --mail-type=ALL
#SBATCH --mail-user=martin.moche@ki.se
module load XDSME/0.6.6-7-PReSTO
xdsme \
-i "ROTATION_AXIS= 0.0 -1.0 0.0" \
/proj/xray/users/x_marmo/tau/tau1-tau_1_??????.h5

More on DIALS at BioMAX

DIALS is software in rapid development for integration of MX diffraction data. Graeme Winter from the DIALS development team shared a patch that enable DIALS running with data from BioMAX. DIALS is most often used via xia2 -dials option and when processing data from BioMax the goniometer axis need to be explicitly stated in the input script. DIALS can also be run in a step-wise manner following this tutorial and when using dials.image_viewer or dials.reciprocal_latttice_viewer vglrun should be used for optimal experience at the BioMAX cluster like

  • vglrun dials.image_viewer datablock.json strong.pickle
  • vglrun dials.reciprocal_lattice_viewer experiments.json indexed.pickle

Simplified script for xia2 -dials only with BioMAX data

#!/bin/sh
#SBATCH -t 1:00:00
#SBATCH -N 1 --exclusive
module load DIALS
xia2 goniometer.axes=0,1,0 \
pipeline=dials failover=true \
image=/lunarc/nobackup/users/name/tau1/tau1-tau_1_master.h5 \
multiprocessing.mode=serial \
multiprocessing.njob=1 \
multiprocessing.nproc=auto \

Simplified script for xia2 -3dii (i.e. XDS) only with BioMAX data

#!/bin/sh
#SBATCH -t 1:00:00
#SBATCH -N 1 --exclusive
module load DIALS
xia2 goniometer.axes=0,1,0 \
pipeline=3dii failover=true \
image=/lunarc/nobackup/users/name/tau1/tau1-tau_1_master.h5 \
multiprocessing.mode=serial \
multiprocessing.njob=1 \
multiprocessing.nproc=auto \

LUNARC Aurora data processing

Data from Swedish academics will be automatically transfer to LUNARC Aurora and available in ´/projects/maxiv/visitors/biomax/proposalNr´

If you cannot access this folder at Aurora please contact

PReSTO access at LUNARC Aurora

Use the "id" command at LUNARC and check if you have access to presto group.

For instance:

$ id
uid=16804(mochma) gid=16800(liu-tora) groups=16800(liu-tora),24500(presto),30400(liu-mamo),350003(x20180077),350212(max4xp_350212),20193561(sto20193561),796200020(prestoadm)

If you do not have access to presto, review the MX-licenses according to instructions for presto access and contact mailto:support@lunarc.lu.se.


User Area

User support

Guides, documentation and FAQ.

Getting access

Applying for projects and login accounts.

System status

Everything OK!

No reported problems

Self-service

SUPR
NSC Express