Category: Uncategorized

  • How many people does it take for society to be against you?

    Let’s say society is divided into two factions. One faction is univocally for equality between men and women in all areas. Another faction thinks that due to various factors, women and men should in some areas have different societal roles.

    People treat others according to their values.

    Let’s say the ratio is 80:20 in favor of your value system. You are a woman and interact with 100 people per day from all factions.

    What would this feel like?

    First, in interactions you would regularly interactions get comments that are inappropriate for your value system. Proportionally, you would get “Why are you still working?” and “When are you going to start working again?”. So you will get confronted with the minority values that most of society disagrees with, every day.

    In a previous post, I looked at the small number of bad actors it takes to make a community of neutral people a toxic environment.

    Most people you are friends with are probably sharing your values, or are neutral. So you will feel the society at large is wrong and against you. Even if you are in the majority.

    Major news outlets, because the 80% are boring non-news, will report on the incidents highlighting the 20% faction, skewing perception further.

    If the confidence of the 20% in expressing their values changes up and down over the years, it will look like major changes in society.

    Academia

    Now let’s consider the impact in academia. Much has been said about the leaky pipeline, which has many causes, including poor support and pressure on mothers, harassment, sexual misconduct, and unequal share of care work, among others.

    Still, let’s say in your institute, out of 10 women who enter, 8 are for equality and 2 are in favor of traditional gender roles.

    Now let’s say one of the 8 writes an email to the department in favor of equality, the other 7 may not feel the need to say anything because of course they agree. But the other 2 may give opposing views. They may also be supported by their equivalent camp of men. And now you have a feeling of being in a minority, and questioning whether you are right, why is everyone against you, can you fit in, etc.

    This skewing effect is probably most drastic on social media, where responding takes effort, and that effort is most commonly taken up for a backlash reaction, rarely for a supportive action.

    The skew can be addressed by stating your values openly, and by expressing support more often to people when they are so obviously right that it feels like it does not need to be said.

  • Movies are unrealistic because of communication bandwidth

    Movies have scientific inaccuracies, for example:

    • Sounds: sounds in space, eagle sounds, loudness of guns
    • Zooming in infinitely, blood spatter, etc
    • On crime scenes, one person handles what in reality would be many individual experts

    We also often see unrealistic or unfair depictions:

    • Aliens have a humanoid shape, and talk English.
    • Robots have a soul and just want to be human, or are killers
    • Depression and mental illnesses leads to violence; Depictions of autists, nerds, cheerleaders, etc are stereotypical
    • How societal processes work; for example, the judicial system in the US, fraction of crimes solved by confession; Hunting a criminal down to the end of the world vs. blocking them from participating in society and beaurocracy.

    People complain about these, but I have some sympathy with movie makers.

    As a movie maker, you have to communicate relatively quickly to get the story along. Using tropes and concepts that the audience is already familiar with is not only handy, but essential.

    You cannot start a movie with a scientific lecture and debunking, and then build upon what the audience has just learned.

    So when you see an alien arriving and speaking English, don’t think “That’s a giant plot hole”, but “OK, the movie writers are dumbing it down for me and skipping some steps, because otherwise the audience would be bored with the details here.”

    In this view, the communication is “movie makers” -> “agreed-upon communication building blocks” plus ~1 new element this movie manages to introduce -> “audience”.

    The limiting factor is clearly the number of communication building blocks available.

    The situation gets more dramatic as big film studies aim for world-wide audiences. Then, the number of universally understood concepts that can be used for communication are further limited. This leads to less humor in movies.

    The antidote to this is to watch smaller productions aimed at more local audiences. Or, to accept that there is this additional communication layer, and a laser, a space ship, an alien or a detective should not be taken literally, but as a stand-in for a communicated concept.

    Still, I have some reservations:

    • 24 promoted the misconception that torture is an effective way to obtain truthful information or a long-term helpful strategy eased up an audience to false intelligence leading to the US’s Iraq War and the Abu Ghraib torture and prisoner abuse.
    • Police shows give false impressions of the science (or lack thereof) in analysing crime scenes (blood spatter, fingerprints, burning, video evidence, witness account reliability, false confessions).
    • Portraying war without showing the many wounded, the innocent people who get in the cross-fire, the destruction and disruption and ensuing poverty, the power vacuum and people abusing it, and the women and children who suffer from selfish men.
  • Astronomer’s logarithms

    A few generations ago, you might have learned how to take logarithms with the slide rule scale. Here I’ll teach you the fast algorithm astronomers use to take logarithms in their head.

    Algorithm:

    • If the number is already in scientific format (easy case):
      • 1 * 10^A, then the log10 is A.
      • 3 * 10^A, then the log10 is A.5.
      • There are no other numbers, pick the closest of the two cases above.
    • If the number is >=1:
      • 1xxxxxxxxx: count the number of x digits and call it A –> the log10 is A.
      • 3xxxxxxxxx: count the number of x digits and call it A –> the log10 is A.5
    • If the number is <1:
      • 0.xxxxxx1: the log10 is the number of digits after the period.
      • 0.xxxxxx3: the log10 is the number of digits after the period, plus 0.5.

    dex: the units of logarithm differences

    Finally, we can talk differences in logarithmic plots. A difference of 1 in a log plot is a factor of ten, a difference of 2.5 is a factor of 300.

    Astronomers say “dex” after the difference to indicate log-differences. So. “There is a 2.5 dex difference”.

    It’s quite nice shorthand.

    Astronomers don’t know that non-astronomers (even physicists) don’t know about dex.

    dex is analogous to decibel, or magnitudes.

    dex stands for decimal exponent.

  • The half-dead tree of standard models of particle physics

    Today, we speak about the standard models of particle physics, and often students of physics call it beautiful. But this version of the theory was not conceived at once. It is the product of many iterations until it was settled to its current form in ~1975. Many of the alternatives were more mathematically appealing, but experiments showed that they were not describing our Universe.

    Below is a brief historic review of more elegant theories that did not make it. The text below is AI generated, but I found it an interesting read. Open this to see the prompt.

    Claude Opus 4.6: Please review versions of the standard model of particle physics since 1900, with a focus on promising and elegant versions of the standard model that were discarded because they were ruled out by experiment. I do not care much about the people involved here.

    The Classical Era (1900–1930s): Building Blocks

    What We Had

    The early 20th century began with just electrons, protons, and the photon. The theoretical framework was classical electromagnetism plus the emerging quantum theory.

    Elegant Casualty: The Pure Electromagnetic World

    Around 1900–1910, there was a deeply appealing idea that all of physics could be reduced to electromagnetism — that the electron’s mass was entirely electromagnetic in origin (the electron was just a self-energy of its own field), and that ultimately all matter was electromagnetic. This was aesthetically beautiful: one force, one substance.

    Killed by: The discovery of the nuclear force. The existence of a compact, positively charged nucleus (1911) immediately demanded some non-electromagnetic force holding it together. No amount of electromagnetic cleverness could bind protons together against their mutual repulsion.

    Elegant Casualty: The Proton-Electron Nuclear Model

    Before the neutron’s discovery (1932), the nucleus was modeled as containing protons and electrons. A nitrogen-14 nucleus would have 14 protons and 7 electrons, giving it charge +7 and mass 14. This was economical — only two fundamental particles needed.

    Killed by: Multiple problems. The uncertainty principle made confining electrons in a nucleus energetically absurd. Worse, nitrogen-14 was observed to obey Bose-Einstein statistics, but 21 fermions (14p + 7e) should make it a fermion. Nuclear spin values were also wrong. The neutron resolved all of this, at the cost of adding a new particle.


    The Yukawa Era and Nuclear Democracy (1930s–1960s)

    The Framework

    After the neutron, positron, and neutrino entered the picture, the 1930s saw the first attempts at a real theory of nuclear forces.

    Elegant Casualty: Fermi’s Four-Fermion Interaction as Fundamental

    Fermi’s 1934 theory of beta decay — a direct, contact interaction between four fermions (neutron → proton + electron + antineutrino) — was remarkably simple and calculationally useful at low energies. There was a reasonable hope it might be fundamental.

    Killed by: Non-renormalizability. The theory predicted cross-sections that grew without bound at high energy, eventually violating unitarity (the probability of something happening can’t exceed 100%). This was a theoretical death sentence even before the W boson was found, though the precise energy scale where it failed (~300 GeV) was identified clearly by the 1960s. It turned out to be the low-energy limit of W boson exchange.

    Elegant Casualty: Yukawa’s Meson Theory as the Fundamental Strong Force

    Yukawa (1935) predicted a ~100 MeV boson mediating the nuclear force, directly paralleling how the photon mediates electromagnetism. When the pion was found in 1947, this seemed like a triumph — and it was, operationally. The vision was: the strong force is mediated by pion exchange, just as electromagnetism is mediated by photon exchange. Simple, symmetric, beautiful.

    Killed by: The “particle zoo.” Through the 1950s and 1960s, hundreds of hadronic resonances were discovered. The pion wasn’t special — it was just one of many hadrons. Pion-exchange theory also became non-renormalizable and incalculable at short distances. The pion turned out not to be a fundamental field but a composite object made of quarks. Yukawa’s theory survives as an effective theory of nuclear physics (and is still used!), but it isn’t fundamental.

    The Magnificent Casualty: Nuclear Democracy / The Bootstrap

    This is perhaps the most philosophically radical and elegant framework ever abandoned in particle physics.

    The idea, developed primarily in the late 1950s and 1960s, was that there are no fundamental hadrons at all. Every hadron is a composite of every other hadron. The proton is a bound state of pions and other particles, which are themselves bound states involving protons, in an entirely self-consistent “bootstrap.” The S-matrix (which encodes all scattering information) was supposed to be uniquely determined by internal consistency requirements: analyticity, unitarity, and crossing symmetry, with no need for an underlying field theory at all.

    This was coupled with Regge theory, which organized particles into families on trajectories in the angular momentum–mass² plane, and the dual resonance model, which eventually morphed into string theory.

    The philosophical appeal was enormous: no arbitrary inputs, no fundamental entities, complete self-determination from consistency alone. It was also explicitly democratic — no particle was more fundamental than any other.

    Killed by:

    • Deep inelastic scattering at SLAC (1968–69), which revealed point-like constituents inside the proton, exactly what the bootstrap denied.
    • The success of the quark model in classifying all known hadrons.
    • The discovery of asymptotic freedom in QCD (1973), which explained why quarks looked free at short distances — something the bootstrap program could never accommodate.
    • Ultimately, QCD was renormalizable, calculable, and predictive where the bootstrap was not.

    The bootstrap’s spiritual descendant is string theory, which inherited the dual resonance model’s mathematics. Some argue the bootstrap philosophy has partly returned in the modern “S-matrix bootstrap” and conformal bootstrap programs, but the original vision of replacing field theory entirely with S-matrix consistency was decisively abandoned for hadronic physics.


    The Electroweak Unification Era (1960s–1970s)

    Elegant Casualty: Vector–Axial Vector Ambiguity and Parity Conservation

    Before 1956, it was universally assumed that the laws of physics were symmetric under parity (mirror reflection). This was an extremely natural symmetry — why should nature care about left vs. right? The weak interaction was written in forms that preserved parity.

    Killed by: The observation of parity violation in cobalt-60 decay and pion decay (1957). The weak force violates parity maximally — it couples only to left-handed particles. This was profoundly shocking. Many physicists found it almost aesthetically offensive. To this day, the fact that the weak force distinguishes left from right is one of the most disturbing features of the Standard Model.

    Elegant Casualty: The V-A Theory Without Gauge Bosons

    After parity violation was established, the weak interaction was reformulated as a V-A (vector minus axial vector) current-current interaction. This was a clean, elegant structure that captured all low-energy weak phenomena.

    Like Fermi’s theory, it was killed by non-renormalizability — but the path to fixing it produced multiple candidates.

    Elegant Casualty: Ambidextrous / Left-Right Symmetric Weak Interactions

    Given the ugliness of maximal parity violation, an appealing idea emerged: perhaps parity is a fundamental symmetry, but it’s spontaneously broken. Left-right symmetric models (based on gauge group SU(2)_L × SU(2)_R × U(1)) posit that there are heavy right-handed W_R and Z_R bosons that are partners of the known left-handed ones. At high energies, parity is restored.

    Status: Not quite dead, but severely constrained. The LHC has pushed the mass of W_R bosons above ~5 TeV, and various precision measurements constrain these models tightly. The minimal versions are in serious tension with data. This remains one of the most aesthetically motivated extensions — it would explain why parity appears violated (it’s spontaneously broken, like electroweak symmetry itself) — but nature doesn’t seem to have taken this path, at least not at accessible energies.

    The Path Taken: SU(2)_L × U(1)_Y

    The Glashow-Salam-Weinberg model unified weak and electromagnetic interactions. But even here, there were alternative routes:

    Elegant Casualty: Unified Electroweak Models Without a Higgs Mechanism (Technicolor)

    The Higgs mechanism works, but it has features many theorists found unappealing — a fundamental scalar field with a seemingly arbitrary and fine-tuned mass. Technicolor proposed that electroweak symmetry breaking is driven by a new strong force (analogous to QCD) acting on new fermions. Just as the chiral condensate of QCD breaks chiral symmetry and gives most of the proton’s mass, the “technifermion” condensate would break electroweak symmetry and give the W and Z their masses.

    This was elegant because it explained electroweak symmetry breaking dynamically (no fundamental scalars, no fine-tuning) and it followed a pattern already established in nature by QCD.

    Killed by:

    • Precision electroweak measurements at LEP (1990s), which severely constrained the simplest technicolor models through their contributions to the S, T, U oblique parameters.
    • The discovery of the Higgs boson at 125 GeV (2012) behaving exactly like a fundamental scalar, with couplings proportional to particle masses as the Standard Model predicts. Technicolor predicted a very different phenomenology in the Higgs sector — broad resonances rather than a sharp, elementary-looking particle.
    • Flavor-changing neutral currents: “extended technicolor” models (needed to give fermions their masses) generically produced unacceptably large FCNCs.

    Some walking/conformal technicolor variants survive on life support, but the program is largely abandoned.


    The Quark Model and QCD (1960s–1970s)

    Elegant Casualty: The Sakata Model

    Before the quark model, the Sakata model (1956) proposed that all hadrons were composed of proton, neutron, and lambda (p, n, Λ) and their antiparticles. This had the appeal of using known particles as fundamental building blocks rather than introducing hypothetical new entities.

    Killed by: It couldn’t accommodate the full SU(3) flavor structure of hadrons. It predicted the wrong representations — in particular, it got the baryons wrong. The quark model, using fractionally charged particles that were not observed in isolation, was more mathematically natural and ultimately far more successful.

    Elegant Casualty: Han-Nambu Colored Quarks with Integer Charge

    One of the most aesthetically troubling features of quarks was their fractional electric charges (2/3 and -1/3). The Han-Nambu model (1965) proposed that quarks carry a “color” charge (predating QCD’s use of color) but have integer electric charges that depend on their color. The fractional charges of hadrons would emerge as averages. This was elegant because it eliminated the bizarre fractional charges and because it introduced what would become the color degree of freedom.

    Killed by: Deep inelastic scattering and electron-positron annihilation measurements in the 1970s. The cross-section ratio R = σ(e⁺e⁻ → hadrons)/σ(e⁺e⁻ → μ⁺μ⁻) depends on the sum of the squares of quark charges. Fractional charges (giving R = 2 for three flavors with three colors) matched beautifully; integer charges gave the wrong value. Neutrino scattering experiments confirmed this. Nature chose the stranger option.

    Elegant Casualty: Quarks as Pure Mathematics (No Real Constituents)

    Through the 1960s, many physicists treated the quark model as a mathematical bookkeeping trick — quarks encoded symmetry patterns but didn’t “really” exist as physical entities inside protons. This was motivated by the failure to observe isolated quarks and by the philosophical framework of the bootstrap.

    Killed by: Deep inelastic scattering at SLAC (1968-69), which showed hard, point-like scattering centers inside the proton, exactly as if it contained nearly-free constituents at short distances. Combined with asymptotic freedom (1973), which explained both why quarks appear free at high energies and why they can’t be isolated (confinement), this established quarks as physically real.

    Elegant Casualty: QCD Alternatives — Abelian Color

    Could the strong force be described by an abelian (QED-like) gauge theory rather than non-abelian SU(3)? This would have been mathematically much simpler.

    Killed by: Non-abelian gauge theories have unique properties — asymptotic freedom (abelian theories don’t have it) and confinement. Three-gluon and four-gluon self-interaction vertices, characteristic of non-abelian theories, have been experimentally verified in jet physics at LEP and the LHC. The observed pattern of gluon jets is inconsistent with an abelian theory.


    Grand Unification and Beyond (1970s–present)

    The Magnificent Casualty: SU(5) Grand Unification (Georgi-Glashow)

    This deserves special attention as perhaps the most beautiful theory clearly killed by experiment.

    SU(5) unified the Standard Model gauge group SU(3) × SU(2) × U(1) into a single simple group. Its elegances were staggering:

    • Charge quantization explained. The Standard Model gives no reason why the proton and electron have exactly equal and opposite charges. In SU(5), quarks and leptons sit in the same multiplets, and charge quantization is automatic.
    • Quantum number patterns explained. The bizarre-seeming pattern of Standard Model fermion quantum numbers (why does a left-handed down quark have charge -1/3 and hypercharge 1/3?) follows inevitably from the group theory of SU(5). One generation of fermions fits perfectly into a 5̄ + 10 representation.
    • Coupling constant unification. The three Standard Model coupling constants, extrapolated to high energies using the renormalization group, approximately converge to a single value at ~10¹⁵ GeV. This is what you’d expect if they’re all manifestations of a single force.
    • Explains anomaly cancellation. The seemingly miraculous cancellation of gauge anomalies in the Standard Model becomes automatic.

    The model’s most dramatic prediction: proton decay, with a lifetime of ~10³⁰⁻³¹ years, primarily through p → π⁰ + e⁺.

    Killed by: Super-Kamiokande (and predecessors like IMB and Kamiokande). The proton lifetime has been pushed to >10³⁴ years for the π⁰e⁺ channel, ruling out minimal SU(5) by several orders of magnitude. Additionally, precision measurements of the three coupling constants at LEP showed that they don’t quite unify in non-supersymmetric SU(5) — they miss at a statistically significant level.

    This is perhaps the most painful loss in theoretical physics. The theory explained so many otherwise arbitrary features of the Standard Model, and it was wrong. (Supersymmetric SU(5) fares better on coupling unification but has its own problems, and SUSY itself is now heavily constrained — see below.)

    Elegant Casualty: SO(10) Grand Unification (Minimal Versions)

    SO(10) is even more beautiful than SU(5): an entire generation of fermions, including a right-handed neutrino, fits into a single 16-dimensional spinor representation. It naturally incorporates left-right symmetry, explains the seesaw mechanism for neutrino masses, and contains SU(5) as a subgroup.

    Status: Minimal versions are heavily constrained by proton decay limits and detailed fitting of fermion masses. Not definitively dead, because SO(10) has many possible breaking chains, each with different predictions. But the simplest, most predictive versions are excluded or under severe pressure.

    Elegant Casualty: Minimal Supersymmetric Standard Model (MSSM)

    Supersymmetry — a symmetry relating fermions and bosons — was the dominant beyond-Standard-Model framework from the 1980s through 2012. Its elegances:

    • Hierarchy problem solved. The Higgs mass receives enormous quantum corrections from every heavy particle in nature. In the Standard Model, these must cancel to ~34 decimal places to give the observed Higgs mass. SUSY provides a natural cancellation: every boson loop is cancelled by its fermionic partner and vice versa.
    • Gauge coupling unification. With SUSY particles at the ~TeV scale, the three gauge couplings unify precisely at ~10¹⁶ GeV, much better than without SUSY. This was considered the single strongest piece of indirect evidence.
    • Dark matter candidate. The lightest supersymmetric particle (LSP), typically a neutralino, is stable and has roughly the right properties to be dark matter.
    • Radiative electroweak symmetry breaking. The Higgs mass-squared parameter is driven negative by quantum corrections from the top squark, naturally explaining why electroweak symmetry breaks. This was considered remarkable.
    • Required by string theory. Most realistic string compactifications required spacetime SUSY.

    Killed by (at least in its natural/minimal form): The LHC. The first runs (2010–2012) excluded gluinos and squarks below ~1–2 TeV. The 13 TeV run pushed these limits to ~2–2.5 TeV. The measured Higgs mass of 125 GeV is possible in the MSSM but requires heavy stops (~multi-TeV) and large mixing, meaning SUSY doesn’t solve the hierarchy problem without its own fine-tuning of ~1% or worse. Direct searches for charginos, neutralinos, and sleptons have found nothing. The original motivation — natural electroweak symmetry breaking without fine-tuning — requires SUSY partners near the electroweak scale (~100–500 GeV), and they’re simply not there.

    SUSY is not formally excluded (you can always push masses higher and add complexity), but the natural, minimal, predictive version that motivated a generation of theorists is dead. This has been one of the most consequential null results in the history of physics.

    Elegant Casualty: Large Extra Dimensions / TeV-Scale Gravity

    Models where gravity becomes strong at the TeV scale (because it propagates in large extra dimensions while Standard Model fields are confined to a 3+1 dimensional brane) offered a radical solution to the hierarchy problem: there is no hierarchy. The Planck scale is a mirage caused by dilution of gravity into extra dimensions; the fundamental scale of gravity is ~TeV.

    Predictions included microscopic black holes at the LHC, Kaluza-Klein graviton towers, and modifications of gravity at sub-millimeter distances.

    Killed by: The LHC has seen no microscopic black holes, no KK gravitons, and no deviations from the Standard Model. Tabletop gravity experiments have verified the inverse-square law down to ~50 micrometers. The simplest versions are excluded.

    Elegant Casualty: Composite Higgs / Minimal Little Higgs Models

    These models tried to make the Higgs a pseudo-Nambu-Goldstone boson of a broken global symmetry — naturally light, like pions in QCD. This was a beautiful hybrid between technicolor and the Standard Model: a light Higgs exists, but its lightness is explained by symmetry rather than fine-tuning.

    Status: Increasingly constrained by LHC precision measurements of Higgs couplings (which look very Standard-Model-like) and by direct searches for the predicted top-partner resonances. Not fully dead, but the natural parameter space is substantially reduced.


    Cross-Cutting Elegant Casualties

    CP Conservation in the Strong Force: The Axion Window Closing

    QCD naturally allows a CP-violating term (the θ-term) that would give the neutron an electric dipole moment. Experimentally, θ < 10⁻¹⁰. The Peccei-Quinn mechanism explains this by introducing a new symmetry whose breaking produces the axion — a light pseudoscalar particle.

    Status: The original “standard” axion (at the electroweak scale) was quickly killed by accelerator experiments and rare decay searches. The “invisible” axion (with much higher symmetry-breaking scale) survives and is actively searched for. This is more a story of the parameter space narrowing than a theory dying.

    Massless Neutrinos

    The original Standard Model had massless neutrinos — elegant because it meant you could choose neutrinos to be purely left-handed with no right-handed counterpart, simplifying the theory and making lepton number an exact symmetry.

    Killed by: Neutrino oscillations (Super-Kamiokande 1998, SNO 2001). Neutrinos have mass. This requires either right-handed neutrinos (which add new particles) or Majorana mass terms (which violate lepton number) or both. The seesaw mechanism is the most elegant solution but remains unconfirmed.


    What Survives: The Actual Standard Model

    The Standard Model as it stands in 2024 is SU(3)_C × SU(2)_L × U(1)_Y with three generations of fermions, one Higgs doublet, and neutrino masses added (minimally via dimension-5 operators or right-handed neutrinos). It is:

    • Spectacularly successful. Every prediction tested has been confirmed, most recently the Higgs boson properties.
    • Clearly incomplete. It doesn’t include gravity, dark matter, dark energy, the baryon asymmetry, neutrino mass mechanism, or inflation.
    • Aesthetically disturbing. It has ~19-26 free parameters (depending on how you count neutrino masses), unexplained patterns (why three generations? why the mass hierarchies? why the specific gauge group?), and the strong CP problem.

    The deepest lesson from this history may be that nature does not share physicists’ aesthetic preferences. Every attempt to make the Standard Model less arbitrary — to explain its patterns, reduce its parameters, or embed it in something more symmetric — has either been ruled out or remains unconfirmed. The theory that works is not the most beautiful one anyone could imagine; it’s a peculiar, lopsided structure that has defeated every prettier challenger.

    — End of AI generated text —

  • Why I sign emails with “Cheers”

    In New Zealand, I was lucky enough to have as a teacher the extraordinarily knowledgeable and patient senior scientist Tim Natusch. Tim brought me for observing at the Auckland observatory, where we many times looked from the dome at the long, white cloud lending the Maori name to the island (Aoteroa), and to the radio observatory in Warkworth.

    Tim was at the time heavily involved in building up the radio telescope infrastructure, and taught me many aspects of radio astronomy and antennas. Ultimately, I remotely oriented the radio dish and recorded data, performed a small radio scan of the sky, and did some manual Fourier transforms of the recorded data. All great fun!

    Tim always signed his emails with “Cheers,”, followed on the next line by a manually indented spaces and his name. I’ve adopted this myself and keep thinking of all the things I learned in my time at the Institute for Radio Astronomy and Space Research at AUT.

    Cheers,
    Johannes
  • Competition

    I don’t really understand sport competition. There is no denying that sport competitions are popular. I don’t participate in them much – to start with, I don’t like crowds or being yelled at.

    I suppose the core is that humans value extremes for their rarity. Competing is one way, but not the only way, to encourage reaching and exceeding extremes.

    It is interesting that the number of competitions we value is very limited – only a few sports take most of people’s interest, foremost running, skiing and ball sports. This leading to a somewhat one-dimensional thinking.

    As a statistics interlude, the fastest 0.1% of people would be in the 3 sigma tail of human distribution. But how much does it take to be good in two or three axes? If you are a two sigma outlier in two unrelated disciplines, that makes you rarer than a more than 3 sigma outlier in one discipline. If you are merely a one sigma outlier in three disciplines, that makes you a 2.65 sigma outlier in one discipline. That assumes independence of course, if the disciplines are highly related, than it would be common to be simultaneously good at them. Nevertheless, this illustrates that combining disciplines is gold.

    I’m curious whether more competitions (chess boxing, triathlons) and jobs (computational astrostatistics) are possible that encourage excellence in multiple disciplines, to avoid one-dimensional thinking.

    Whether are sport competitions are interesting to watch, depends crucially on their randomness. Soccer for example has very few points during a 90 minutes match, and it is extremely hard to predict which team will win if they are not extremely different. The satisfaction of a fan whose team rarely wins is that much higher, and random rewards are more addictive than predictable ones. Running perhaps the opposite: People know their marathon time quite precisely, there are no direct interactions, so the randomness is very limited, and so marathons are not such interesting watch.

  • Common Workflow Language (CWL) for astronomy data pipelines: pros and cons

    This week, I learned CWL, which is a pipeline framework. Take this as an early write-up with limited experience.

    Previously, I used make files, xargs and bash scripts to identify computation steps to be done and execute the relevant commands. I use dynamic rules with pattern replacement. In that (software-building) framework, the files live next to each other (*.o is built from *.c) in the same folder. If the output file does not exist, or dependency indicates files have been modified (by being newer), that build step is executed.

    Both make and CWL can do parallelisation. You can use scatter and map-reduce patterns, and make it come together into a single product, then fan out again, and scheduling will be dynamic.

    Make is somewhat cumbersome for dynamically building the final list of files. In CWL this is not possible at all: you are supposed to make such a list (called records) completely outside beforehand with a separate preprocessing script. I guess this is similar to ninja’s take on make.

    The major conceptual difference in CWL I had to grapple with is that each CWL step receives a directory in which it works, which is empty by default and in cwltool a temporary folder somewhere inside /tmp. You can and should pass a directory from one step to the next, or individual files. This is implemented neatly with symlinks, so it is very fast (no copying). The implemention being handed a file name instead of relying on guessing the file name by convention makes the pipeline more robust.

    Writing the tasks (one .cwl file per task, and one bash/python script next to it with the same name) allows me to clearly see what information is used in each step. A workflow.cwl wires the steps together, defines the input records list and the final output products.

    Following Unix fashion, in Make there are only files as a concept, although some are phony (helper names that are then not physically realised, such as in “make all”). In CWL, there are more types, including records, strings, floats, files and directories, as well as arrays or trees of these.

    At the end of a successful execution of a workflow, CWL copies the final results into a user-designated output folder. The benefit of CWL I see is that it is not keeping broken step execution results around – broken files never leave the temporary folders. That can be an issue with make when killed.

    The language isn’t too bad, albeit a tad verbose: you have to wire each step’s list of input parameters to output parameters. You can dynamically manipulate variables with Javascript one-liners (executed internally with Node.js). The scripts for the computation steps still live in Bash or Python. The data language is YAML, which seems a bit slow to parse when having thousands of records with many fields. Overall, not too complicated coming from Github Actions or CircleCI workflows specified also in YAML.

    A selling point of CWL for astronomy data pipelines is that it can be deterministic and reproducible by connecting with docker images and connectable to cluster execution (SLURM). I’m going to look at this soon.

    Because my steps are very compute-intensive, I chose a somewhat custom way of memoization/caching for my first pipeline. Each step checks if the output is already available in the final output directory, and if so, symlinks to there. The final step removes all symlinks from the temporary output folder, to avoid overwriting anything when the workflow copies the result into the output directory. This way, when nothing needs to be done, the workflow runs through very quickly.

    Currently, I am performing X-ray spectral fitting of some 30,000 objects in the eROSITA all-sky survey, inferring black hole mass accretion rates and obscuration properties of quasars from the last 10 billion years.

  • MSCI World ETF value per consumer price index

    Let’s consider the value of a piece of land, or a basket of goods over a year. The dollar or Euro number will fluctuate with inflation. What is more interesting is the value of a piece of land relative to the basket of goods you can buy with it – the consumer price index.

    I have not yet seen a plot of stock price normalised by consumer price index. Wouldn’t it make sense that you would want all stock price charts over the years normalised to your consumer price index, to see if you are really gaining in value? Yet I have not seen this plotted before.

    Here is a comparison, all normalised at 2010 in USD.

    The consumer price indices are different for different countries. Here is the MSCI World value relative to what you could buy from it in that country:

    This makes sense to me as a way of looking at an investment. On the other hand, inflation does not seem to be extremely important.

    Data sources:

  • About this blog

    I want to practice writing. Writing is structured thinking. Ideas sound good in your head, until written down. That’s when it is noticeable that there are gaps in the logic. Or that explaining something isn’t as easy as it seemed.

    “If you’re thinking without writing, you only think you’re thinking.” ― Leslie Lamport

    I suggest you start with this post: The fear of writing thoughts down.