This week, I learned CWL, which is a pipeline framework. Take this as an early write-up with limited experience.
Previously, I used make files, xargs and bash scripts to identify computation steps to be done and execute the relevant commands. I use dynamic rules with pattern replacement. In that (software-building) framework, the files live next to each other (*.o is built from *.c) in the same folder. If the output file does not exist, or dependency indicates files have been modified (by being newer), that build step is executed.
Both make and CWL can do parallelisation. You can use scatter and map-reduce patterns, and make it come together into a single product, then fan out again, and scheduling will be dynamic.
Make is somewhat cumbersome for dynamically building the final list of files. In CWL this is not possible at all: you are supposed to make such a list (called records) completely outside beforehand with a separate preprocessing script. I guess this is similar to ninja’s take on make.
The major conceptual difference in CWL I had to grapple with is that each CWL step receives a directory in which it works, which is empty by default and in cwltool a temporary folder somewhere inside /tmp. You can and should pass a directory from one step to the next, or individual files. This is implemented neatly with symlinks, so it is very fast (no copying). The implemention being handed a file name instead of relying on guessing the file name by convention makes the pipeline more robust.
Writing the tasks (one .cwl file per task, and one bash/python script next to it with the same name) allows me to clearly see what information is used in each step. A workflow.cwl wires the steps together, defines the input records list and the final output products.
Following Unix fashion, in Make there are only files as a concept, although some are phony (helper names that are then not physically realised, such as in “make all”). In CWL, there are more types, including records, strings, floats, files and directories, as well as arrays or trees of these.
At the end of a successful execution of a workflow, CWL copies the final results into a user-designated output folder. The benefit of CWL I see is that it is not keeping broken step execution results around – broken files never leave the temporary folders. That can be an issue with make when killed.
The language isn’t too bad, albeit a tad verbose: you have to wire each step’s list of input parameters to output parameters. You can dynamically manipulate variables with Javascript one-liners (executed internally with Node.js). The scripts for the computation steps still live in Bash or Python. The data language is YAML, which seems a bit slow to parse when having thousands of records with many fields. Overall, not too complicated coming from Github Actions or CircleCI workflows specified also in YAML.
A selling point of CWL for astronomy data pipelines is that it can be deterministic and reproducible by connecting with docker images and connectable to cluster execution (SLURM). I’m going to look at this soon.
Because my steps are very compute-intensive, I chose a somewhat custom way of memoization/caching for my first pipeline. Each step checks if the output is already available in the final output directory, and if so, symlinks to there. The final step removes all symlinks from the temporary output folder, to avoid overwriting anything when the workflow copies the result into the output directory. This way, when nothing needs to be done, the workflow runs through very quickly.
Currently, I am performing X-ray spectral fitting of some 30,000 objects in the eROSITA all-sky survey, inferring black hole mass accretion rates and obscuration properties of quasars from the last 10 billion years.
