Cosmic Coherence

Blog

  • “The first step to lose a game is to accept the rules.” ― Johannes Buchner

  • “If you’re thinking without writing, you only think you’re thinking.” ― Leslie Lamport

  • It is good to love many things, for therein lies the true strength, and whosoever loves much performs much, and can accomplish much, and what is done in love is well done. ― Vincent Van Gogh

  • Phones are consumer devices, laptops are creator devices

    On phones, you cannot create new concepts. Phones can be creative for photos, but they are not for creating new ideas. They are for consuming images, films, and other people’s ideas. It is difficult to create new things until you interrupt consuming other people’s ideas.

    Don’t spend your time on a device that cannot create. I rather reach for my laptop, where I can write down ideas and create new programs.

    I have noticed this and have practiced it for ~10 years now.

    In the meantime, the same thought has also been written down by Zak El Fassi at https://zakelfassi.com/command-interface-device-power. If you want more words on this, look there.

  • Common Workflow Language (CWL) for astronomy data pipelines: pros and cons

    This week, I learned CWL, which is a pipeline framework. Take this as an early write-up with limited experience.

    Previously, I used make files, xargs and bash scripts to identify computation steps to be done and execute the relevant commands. I use dynamic rules with pattern replacement. In that (software-building) framework, the files live next to each other (*.o is built from *.c) in the same folder. If the output file does not exist, or dependency indicates files have been modified (by being newer), that build step is executed.

    Both make and CWL can do parallelisation. You can use scatter and map-reduce patterns, and make it come together into a single product, then fan out again, and scheduling will be dynamic.

    Make is somewhat cumbersome for dynamically building the final list of files. In CWL this is not possible at all: you are supposed to make such a list (called records) completely outside beforehand with a separate preprocessing script. I guess this is similar to ninja’s take on make.

    The major conceptual difference in CWL I had to grapple with is that each CWL step receives a directory in which it works, which is empty by default and in cwltool a temporary folder somewhere inside /tmp. You can and should pass a directory from one step to the next, or individual files. This is implemented neatly with symlinks, so it is very fast (no copying). The implemention being handed a file name instead of relying on guessing the file name by convention makes the pipeline more robust.

    Writing the tasks (one .cwl file per task, and one bash/python script next to it with the same name) allows me to clearly see what information is used in each step. A workflow.cwl wires the steps together, defines the input records list and the final output products.

    Following Unix fashion, in Make there are only files as a concept, although some are phony (helper names that are then not physically realised, such as in “make all”). In CWL, there are more types, including records, strings, floats, files and directories, as well as arrays or trees of these.

    At the end of a successful execution of a workflow, CWL copies the final results into a user-designated output folder. The benefit of CWL I see is that it is not keeping broken step execution results around – broken files never leave the temporary folders. That can be an issue with make when killed.

    The language isn’t too bad, albeit a tad verbose: you have to wire each step’s list of input parameters to output parameters. You can dynamically manipulate variables with Javascript one-liners (executed internally with Node.js). The scripts for the computation steps still live in Bash or Python. The data language is YAML, which seems a bit slow to parse when having thousands of records with many fields. Overall, not too complicated coming from Github Actions or CircleCI workflows specified also in YAML.

    A selling point of CWL for astronomy data pipelines is that it can be deterministic and reproducible by connecting with docker images and connectable to cluster execution (SLURM). I’m going to look at this soon.

    Because my steps are very compute-intensive, I chose a somewhat custom way of memoization/caching for my first pipeline. Each step checks if the output is already available in the final output directory, and if so, symlinks to there. The final step removes all symlinks from the temporary output folder, to avoid overwriting anything when the workflow copies the result into the output directory. This way, when nothing needs to be done, the workflow runs through very quickly.

    Currently, I am performing X-ray spectral fitting of some 30,000 objects in the eROSITA all-sky survey, inferring black hole mass accretion rates and obscuration properties of quasars from the last 10 billion years.

  • MSCI World ETF value per consumer price index

    Let’s consider the value of a piece of land, or a basket of goods over a year. The dollar or Euro number will fluctuate with inflation. What is more interesting is the value of a piece of land relative to the basket of goods you can buy with it – the consumer price index.

    I have not yet seen a plot of stock price normalised by consumer price index. Wouldn’t it make sense that you would want all stock price charts over the years normalised to your consumer price index, to see if you are really gaining in value? Yet I have not seen this plotted before.

    Here is a comparison, all normalised at 2010 in USD.

    The consumer price indices are different for different countries. Here is the MSCI World value relative to what you could buy from it in that country:

    This makes sense to me as a way of looking at an investment. On the other hand, inflation does not seem to be extremely important.

    Data sources:

  • To bring humans to Moon or to Mars?

    To bring humans to Moon or to Mars?

    Either is a multi-decade task. This requires stable leadership and institutional knowledge. Components and people build on each other to achieve complex goals.

    The moon landing was a monumental achievement. The closest mental anologies are expeditions to the north and south pole, or high-altitute human tests in the atmosphere by the military. It was a quick peek into hostile territory with a high chance of death every time. It was not the establishment of a longer-term presence.

    For Antarctica, establishment of a base was the next step. In part, because of resources, military strategy. But there are also scientific experiments conducted there (South Pole Telescope for studying the Big Bang, Icecube for studying the origin of neutrinos).

    Whether a longer-term presence outside Earth is the next step and which solar system body it should be on has changed over the years. Here is a chart of what NASA was directed to focus on:

    Each change in direction incurs a high cost. Tailored solutions are shelved, and expertise is lost. It matters to some degree which target is chosen, but it also matters to stick to it for decades. This is not easy in a democracy with changing leaders, and when everyone has an opinion.

    Here is mine. I am not an expert in solar system exploration, or engineering rockets, space suits or bases.

    The priority to choose, Mars or Moon, is flexible if you know little about what Mars and Moon are like. From Earth looking up, they are both balls where you can point rockets at. Indeed, their volume, radius, landarea is not hugely different.

    Pros and cons of Mars:
    + Mars is a planet, and it sounds more romantic to have a multi-planetary society than to spread out to the moon.
    – Mars soil is toxic.
    – The atmosphere is very thin and does not have oxygen.
    – Gravity is there but low.
    – Dust storms.
    – Flight time only for a return rescue mission is 9 month to 2 years .
    – If we place humans there, we will contaminate the planet, and might never find out if there was ever life on Mars.

    Pros and cons of the Moon:
    + Flight time only for a return rescue mission is a few days.
    + solar power all day round
    + No dust storms, but
    – Asteroid impacts: every square meter there are a few per week.
    – Moon dust sticks to everything with its microscopic spikes. It teared up the Apollo space suits, making them trash after a few days. Also, it causes cancer.
    – No radiation protection, unless you go underground.
    – No gravity.

    Neither are great starting points. We can again compare to Antarctica. The temperature is similar on Mars, but Antarctica comes with breathable atmosphere, radiation protection and low rate of meteorites. I’m not saying we should stick to Antarctica, been there done that, but we should be aware on just how many fronts progress and improvements are needed.

    My suggestion would be:
    1) to stick to one plan
    2) to let go of romantic ideas and start with an underground moon base, for example, by sealing off a large cavity (lava tube) near the poles, with permanently irradiated solar panels on a hill.
    3) If anything romantic, develop self-replicating additive manufacturing.

    There is also the issue of modifying an environment to create, at least for some limited area, a breathable atmosphere, and maintaining this atmosphere against failures. For the last century, we have made headway in understanding Earth’s atmosphere and its processes in great detail, and have accelerated modifications. But it has been in the inhabitable direction, driven by societal processes that we do not fully and consciously control. Practicing good stewardship of this Earth is not an alternative to Moon and Mars bases, it is a prerequisite.

  • Applied Computer Science and “Real” Physics

    In Computer Science, there is a divide between general methods research (such as: algorithms, data structures, security research) and applications (find a solution for a company or societal problem). When studying at University, I was surrounded by the opinion that the more general method is of higher value, and the other is “just application” and messy. I held this opinion for a while, too.

    In physics, there is a divide between software method research (such as: new data analysis methods) and finding out something about a physical process (application of a method to a specific case and interpreting the result). The latter is much higher regarded. I’ll give some examples I have experienced:

    1. Physicists would say “But what’s the new physics here?” and consistently display a dismissive attitude to automating, robustifying or generalising analyses.
    2. Astronomy journals reject papers that present general methods research without application.

    There is a cognitive dissonance there too, because methods papers are very highly cited. Physicists find them extremely useful. For example, the 2014 paper of PyMultiNest/BXA and the 2021 paper of UltraNest have 1500 and 500 citations, respectively.

    Pure data (catalog) papers are somewhere in the middle. They receive a substantial number of citations and moderate regard. But physicists would recommend not to publish a pure catalog paper with no direct scientific content.

    Part of the above can be explained by people liking similar people. If you are a computer scientist at a University, likely you are doing general research and hold that in high regard. If you are a physicist famous for some subfield, you will value progress in that subfield, not general-purpose methods that could help all fields.

    The tide is changing a bit recently. There are a few professorships for astrostatistics and machine learning in astronomy. And of course, the above is a simplification: there have been well-regarded experts in methods before, for example specific simulation techniques, observation techniques (direct imaging), going at least back to Joseph Fraunhofer developing the best optics.

  • The fear of writing thoughts down

    I am not hesitant to give opinions in person. But I have been hesitant to publish them online. I feel the many eyes of astronomers, many more senior. Judging any misstep I might make. Inconsiderate thoughts and just dumb arguments.

    What you put online also holds permanency. Years later, I might have a completely different, hopefully wiser and more informed opinion. Old written arguments may make me look bad.

    I am writing now because I accept this. What I will write here will be imperfect, and a snapshot of my thoughts in time.

    I invite you to leave comments on my posts. Feel free to correct mistakes and add with your life experience. Please be generous.

  • About this blog

    I want to practice writing. Writing is structured thinking. Ideas sound good in your head, until written down. That’s when it is noticeable that there are gaps in the logic. Or that explaining something isn’t as easy as it seemed.

    “If you’re thinking without writing, you only think you’re thinking.” ― Leslie Lamport

    I suggest you start with this post: The fear of writing thoughts down.