Results
Every method per stage, every full-pipeline combination, and the patterns behind the numbers.
Combination matrix
Dipole inversion ↓ × background removal →, chained end-to-end · best methods top-left · click a cell to inspect.
Combined single-method algorithms (one step straight to a χ map)
No pipeline combinations for this field mapping yet.
Scored from the , so no upstream stage contributes error. For these methods after a real field-mapping step, open and pick a field mapping.
Patterns the leaderboard tables don't surface, computed live from the same results. The recurring lesson: a method's standing depends heavily on what you score, and on the pipeline it sits in, not just on a single number.
The isolated ranking misleads about real pipelines
Dipole inversion ranked on the clean ground-truth field (left) versus chained after real background removal (right), by NRMSE, best at top. Green rises when composed (robust to upstream error); red falls. TV-family methods top the clean ranking, then collapse; several networks climb.
Accurate but fragile, or reliable but modest
Each dipole method by its mean NRMSE across background-removal partners (x) versus how much that error varies between partners (y). Bottom-left is the sweet spot: accurate and consistent. The networks cluster there, accurate and dependable across upstream partners; the iterative methods scatter, and while the best few rival the networks, many degrade sharply with a poor upstream stage.
Greedy pipeline-building is close, but not optimal
Picking the strongest method at each stage independently lands within NRMSE of the best full pipeline; composition is fairly forgiving. But the two share no methods: stages interact, so the actual best combo isn't the one you'd assemble greedily.
Top pipelines (of )
Which stage decides your result?
How much the choice at each stage reshuffles the full-pipeline ranking (rank-based, so one broken method can't skew it), across every pipeline. Background removal and dipole inversion matter about equally: each reshuffles the result far more than field mapping. The notable part: the background-removal choice is as influential as the dipole-inversion step the field usually fixates on.
methods compared at this stage
Rank-based on purpose: under a raw-NRMSE variance measure a single catastrophic method (AMP-PE fails on in-silico, ~4000% NRMSE) would make its stage look all-important. Ranking each pipeline first keeps the comparison robust.
Error compounds through the pipeline
The best achievable xSIM (blue) and the median (green box = interquartile range) at three pipeline depths: dipole inversion alone from the ground-truth local field, then with real background removal added (from the ground-truth total field), then a full pipeline from raw phase. Each real upstream stage roughly halves the median and pulls the ceiling down with it. Upstream error is never recovered downstream, so a method's isolated score badly overstates its end-to-end result.
Pick your dipole independently of background removal
For each dipole method, its rank among dipoles (dot = mean, bar = min–max) across the strong background-removal methods. Most bars are tight: the dipole ranking barely shifts with the upstream stage, so the stages are near-separable — the methods at the top lead wherever they sit, and those at the bottom trail wherever they sit. The few with wide bars swing; those are the genuinely upstream-sensitive methods. Coloured by family.
The best background-removal method depends on what you measure
Each background-removal method's rank (1 = best, green good → red poor) under five metrics, rows sorted by xSIM. The similarity metrics (xSIM, NRMSE, HFEN) are nearly interchangeable (their columns match). But calcification and iron linearity reshuffle the ranking: iron linearity is essentially uncorrelated with similarity (near-zero rank correlation), and the deep-learning bfrnet is worst-tier on similarity yet best on iron quantification. A single headline number hides which sources a method actually recovers.
The best method depends on what you measure
Isolated dipole methods across five metrics, each normalised so higher is better. Grey lines are all methods; a coloured line is the winner of one axis, showing it gives up ground elsewhere: the global-error (NRMSE / xSIM) leader slips down the calcification axis, and the fast direct methods under-quantify iron in deep grey matter. No single method wins every metric.
Where the error lives
Isolated dipole error by region and artefact (rows sorted by overall NRMSE). Green is good, red is poor, coloured within each column. Venous blood is universally hardest; the calcification columns reshuffle the ranking entirely.
Which metrics are redundant
Rank agreement (Spearman ρ) between metrics over the isolated dipole methods, sign-aligned so +1 means they reward the same methods. NRMSE, detrended NRMSE and correlation are near-interchangeable; DGM linearity, calcification and runtime are their own axes, so a single headline number can't stand in for them.
One region, every method
Reconstructed susceptibility inside a single segmented region, per isolated dipole method: dot = mean, bar = ±1 std, tick = median (ppm). The green band is ground truth (mean ± std; dashed line = median). Methods are sorted by how far their regional mean lands from the truth, a per-region view of quantification bias that the whole-brain metrics average away.
Every method, across every region
Every isolated dipole method's regional mean (one coloured dot per method) against the grey ground-truth band (mean ± std; tick = median), for all regions at once. Hover a dot or a legend entry to trace that method across every region; drag the plot sideways to pan the χ axis (calcification sits far to the left). Click a dot to open its page.
Statistics are computed over the full brain mask (a voxel an eroding method dropped counts as zero), so from the next full re-score the paired ground-truth values are identical for every run on a phantom; runs scored earlier used each run's own valid support and can differ slightly. Currently populated for isolated dipole runs; composed pipelines fill in at the next full re-score.
Accuracy vs compute
Isolated dipole inversion: structural similarity (xSIM, higher is better) against wall-clock runtime on a log scale. Green points are Pareto-optimal (nothing is both faster and more accurate); grey points are dominated. Some methods cost minutes for no accuracy gain.
What tuning buys
Isolated dipole NRMSE at default parameters (grey) versus tuned on the scoring phantom (green). Only a handful of methods are tunable (pretrained networks aren't), which is why defaults is the fair cross-method ranking. Tuning on the scored data is a mild form of overfitting.
Build your own comparison
Plot any two metrics against each other for any isolated stage. Points are methods, coloured by family; hover to identify, click to open a method. Runtime uses a log scale.
In silico · QSM Challenge 2.0 (2019)
The main dataset is the QSM Reconstruction Challenge 2.0 in-silico head phantom, forward-simulated to a BIDS multi-echo acquisition with qsm-forward. Its per-voxel susceptibility is known exactly, so QSM-CI scores each pipeline stage on its own (field mapping, background removal, dipole inversion) rather than only the final map. It backs the Full pipelines, Stages and Findings views.
Why a synthetic ground truth
QSM is an ill-posed dipole deconvolution, so different algorithms produce visibly different maps from the same phase. Judging them needs a true susceptibility map, which no in vivo acquisition provides, so every in vivo reference is a surrogate that biases the comparison: a COSMOS reconstruction (Liu et al., 2009) averages anisotropic susceptibility across orientations, and simple piecewise-constant phantoms (de Rochefort, 2008; Wharton, 2010) flatter the TV/TGV methods that reconstruct piecewise-constant maps by construction (Knoll, 2011). A ground truth is never neutral; its texture decides which assumptions get rewarded.
Challenge 2.0 (Marques et al., 2021) answered this with an in-silico head phantom: a realistic digital brain from a 0.64 mm 7 T acquisition, with an interhemispheric calcification, whose susceptibility is known exactly and from which a data-consistent gradient-echo acquisition is forward-simulated. It deliberately carries physiological texture so it does not reward piecewise-smooth solutions the way earlier phantoms did.
What QSM-CI scores, and what it leaves out
Because the total field, local field and tissue susceptibility are all known, QSM-CI scores each stage independently and reruns methods as they appear. The simulation carries realistic background fields from air/tissue interfaces and wrapped multi-echo phase, so field mapping and background removal genuinely have work to do, not just dipole inversion.
It stays idealised in known ways: susceptibility is an isotropic scalar (no white-matter anisotropy), there is no flow or motion, microstructure enters only through the χ-separation compartments, noise is complex Gaussian, and the phantom comes from a single subject. A QSM-CI ranking is a strong controlled test, not a guarantee of in vivo performance.
Reproduce the phantom
qsm-forward simulates the acquisition from the Challenge 2.0 tissue-property maps. Request those from the Donders repository (data.ru.nl/…/DSC_3015069.02_542, DOI 10.34973/m20r-jt17), then generate:
pip install qsm-forward
# 7 T, 4 echoes, 1 mm, fixed seed: matches the scored phantom
qsm-forward head ./qsm-challenge-2.0 bids_qsm --save-field
--save-field emits the total field (fieldmap) and local field (fieldmap-local) that the isolated background-removal and dipole stages are scored against. Acquisition parameters (--B0, --TEs, --voxel-size, --peak-snr, --random-seed) default to the challenge phantom.
Citations
- Head phantom: Marques J.P., Meineke J., Milovic C., et al. “QSM reconstruction challenge 2.0: A realistic in silico head phantom for MRI data simulation and evaluation of susceptibility mapping procedures.” Magn Reson Med 2021;86(1):526–542. doi:10.1002/mrm.28716. Data: doi:10.34973/m20r-jt17.
- Challenge 2.0 design & results: QSM Challenge 2.0 Organization Committee. “QSM reconstruction challenge 2.0: Design and report of results.” Magn Reson Med 2021;86(3):1241–1255. doi:10.1002/mrm.28754.
- COSMOS reference: Liu T., Spincemaille P., de Rochefort L., et al. “Calculation of susceptibility through multiple orientation sampling (COSMOS).” Magn Reson Med 2009;61(1):196–204. doi:10.1002/mrm.21828.
- Piecewise-constant inversion: de Rochefort L., et al. Magn Reson Med 2008;60(4):1003–1009. doi:10.1002/mrm.21710. Wharton S., et al. Magn Reson Med 2010;63(5):1292–1304. doi:10.1002/mrm.22334.
- TGV regularisation: Knoll F., Bredies K., Pock T., Stollberger R. “Second order total generalized variation (TGV) for MRI.” Magn Reson Med 2011;65(2):480–491. doi:10.1002/mrm.22595.
- Forward simulation: qsm-forward (Stewart A., et al.).
| # | Algorithm | |
|---|---|---|
| GRE-native paper computing… DNF |
R2′ generators
Candidate R2′ substitutes, each scored directly against the phantom's true (spin-echo-derived) R2′. The best one (lowest detrended NRMSE) supplies R2′ to every pipeline above; the Findings tab compares how far each substitute carries every method. A generator reads only the GRE magnitude, which is the same at both input levels, so it is scored once and the same substitute supplies R2′ under both ground-truth and realistic inputs.
| Generator | NRMSE | Detrended | Corr. | Runtime |
|---|---|---|---|---|
| supplies R2′ above DNF |
Myelin (χ−) is harder to recover than iron (χ+)
Structural similarity (xSIM) of the recovered paramagnetic χ+ (iron) source versus the diamagnetic χ− (myelin / calcium) source. Every method drops from left to right, so the diamagnetic map is systematically the harder of the two.
Cross-contamination between the two sources
How much each source map bleeds into the other (regression slope, 0 = clean): iron leaking into the myelin map (x) versus myelin leaking into iron (y). The origin is perfect separation; distance from it is contamination that xSIM can miss.
What does dropping the spin-echo cost?
χ-separation wants R2′ = R2* − R2, and measuring R2 takes a separate (multi-echo) spin-echo acquisition. This figure shows each method's separation error (mean of the χ+ and χ− detrended NRMSE) when fed the true R2′ (● full MEGRE+MESE protocol) versus a GRE-derived substitute: ● R2PRIMEnet (learned R2*→R2′) or ● the fixed 0.52·R2* scaling. The gap is the price of GRE-only; methods that never read R2′ (◆) pay nothing by construction.
χ-separation (2026)
Susceptibility source separation splits a voxel’s net susceptibility into a paramagnetic part, χ⁺ (mostly iron), and a diamagnetic part, χ⁻ (mostly myelin, also calcium). QSM measures only their signed sum, χtotal = χ⁺ + χ⁻, in which the two partly cancel. Separation needs a second, independent piece of information: either the reversible relaxation rate R2′, which grows with the combined magnitude |χ⁺| + |χ⁻|, or the shape of the multi-echo signal itself. Each method produces χ⁺ and χ⁻, and the two maps are scored separately.
What methods are given
Each dataset gives methods the multi-echo gradient-echo (GRE) magnitude and phase, a brain mask and the acquisition parameters. Where a method needs them, it also gets the local field, χtotal, R2′, the per-echo χ, and a matched multi-echo spin echo. Methods read only the inputs they declare:
- R2′-based methods (χ-separation iLSQR and MEDI, χ-sepnet, SUSEP-Net, WaveSep) combine χtotal and the local field with R2′.
- R2*-QSM uses the GRE magnitude instead of R2′.
- DECOMPOSE fits the multi-echo complex signal and takes a QSM per echo.
GRE-only condition. Without a spin echo there is no measured R2, and so no measured R2′. The leaderboard’s GRE-only view feeds every R2′-based method an R2′ estimated from the GRE magnitude alone. Three estimators are scored against the true R2′ (R2PRIMEnet, R2PRIMEnet-7T and the fixed 0.52·R2* scaling), and the one closest to it on each phantom (lowest detrended NRMSE) supplies R2′ to every method. R2*-QSM and DECOMPOSE never read R2′, so they appear with their usual results.
Ground-truth vs realistic inputs (L3)
- Ground-truth inputs (the default) give each method the phantom’s true local field, χtotal and brain mask, with the R2′ the dataset ships. Nothing upstream is in error.
- Realistic inputs (L3) give each method what a typical real QSM pipeline would hand it: the same maps with noise from the dataset’s SNR, the losses of background-field removal and dipole inversion (low spatial frequencies, magic-cone attenuation, smoothing), residual smooth backgrounds, an error near the brain boundary, a mean reference, a 2 mm-eroded mask, and R2′ fitted from the noisy gradient-echo and spin-echo signal. The Ridani phantom is noiseless, so its realistic inputs carry the pipeline errors but no noise.
- Calibration. Each error term is the median over six real QSMxT pipelines (iLSQR, TKD, NLTV, MEDI, TFI and iR2QSM) run on the Mesoscopic 3T phantom. The spread across the six is recorded with every dataset.
- Validation. Methods were also run on the outputs of ten real pipelines, four of which were not used for calibration. Fed the L3 inputs, they rank almost as they do on those real outputs: Spearman ρ = 0.92 on Mesoscopic 3T and 0.89 on WM microstructure 7T, and 0.94 on both with DECOMPOSE included. Their scores come out about 0.02 xSIM higher, because the model leaves out the non-linear streaking of real inversions.
- How to read them. Ground-truth inputs give an upper bound on what each method can do. They favour methods that trust their inputs voxel by voxel: with exact inputs, a closed-form solve has nothing to correct, while regularised methods pay for their smoothing. L3 is the realistic comparison. Both levels are scored against the same ground truth, over the same full brain mask, with every method at its default settings.
- Toggles. The MEGRE+MESE / GRE-only toggle is available under both input levels. Under L3, the GRE-only estimators read the same noisy GRE magnitude with the eroded mask, and the same estimator supplies R2′ as under ground-truth inputs. Tuned parameters are shown only under ground-truth inputs, where they were optimised.
- More detail: the degradation model is described in qsm-forward’s Degraded derived inputs section, and implemented in
qsm_forward/degrade.py.
Per-echo χ for DECOMPOSE
DECOMPOSE takes a QSM reconstructed at each echo, and separates the sources from how χ changes with echo time. The datasets ship one at both input levels:
- Ground-truth inputs: the true χ, plus the echo dependence that a per-echo STAR-QSM reconstruction finds in the noiseless signal. The same reconstruction of a perfectly linear phase is subtracted, so STAR-QSM’s own echo dependence cancels and only the tissue’s remains.
- Realistic inputs (L3): each echo is degraded like χtotal, with its own noise, plus a smooth per-echo drift calibrated so that the echo dependence matches that of real per-echo pipelines.
- Ridani 3T anisotropic: its phase is linear in echo time, so the per-echo χ is the same map at every echo.
On Mesoscopic 3T, DECOMPOSE scores slightly higher on the L3 per-echo χ than on the best real per-echo pipeline (mean χ⁺/χ⁻ xSIM 0.263 against 0.254). This is part of L3’s slight optimism, and it is in DECOMPOSE’s favour. The DECOMPOSE (QSM.rs) entry reads the per-echo χ once QSM-CI moves to the next QSMxT release; until then it is given χtotal at every echo.
The datasets
All three are built with qsm-forward on the QSM Challenge 2.0 head model (Marques et al., 2021), with the same χ⁺ and χ⁻ anatomy as the Ridani et al. (2026) separation phantom. They differ in how the signal is produced.
| Dataset | Field / echoes | How the signal is produced | Role |
|---|---|---|---|
| Mesoscopic 3T (default) | 3 T, 16 echoes 2.7–46.2 ms + spin echo, SNR 100, 1 mm | from explicit sub-voxel sources | realistic benchmark |
| WM microstructure 7T | 7 T, 8 echoes 3–45 ms + spin echo, SNR 100, 1 mm | from explicit sub-voxel sources | realistic benchmark at high field |
| Ridani 3T anisotropic | 3 T, 6 echoes, noiseless | the published phantom’s formula | published reference |
The mesoscopic datasets. These don’t write R2′ as a formula. Each voxel’s signal is computed from sub-voxel sources placed explicitly:
- Myelin: myelinated axons modelled as hollow cylinders, with three water compartments (Wharton & Bowtell, 2012).
- Iron: iron-rich spheres, mostly in 3 µm clusters with a share at ferritin scale.
- Calcification: the head model’s calcification is a purely diamagnetic source, χ⁻ = −3.3 ppm (Marques et al., 2021, SIM2), simulated as solid diamagnetic spheres with no water inside.
- Water diffusion: simulated by Monte Carlo through that geometry.
Relaxivity, fibre-orientation effects and the shape of the decay all come out of the simulation rather than being put in. The paramagnetic relaxivity that results is about 140 Hz/ppm at 3 T, close to the in-vivo calibration of Shin et al. (2021). In white matter, both the field and R2′ depend on fibre orientation, as they do in real tissue. The χ⁻ that is scored is the orientation-independent myelin susceptibility. The orientation-dependent apparent χ⁻ is published alongside it. The look-up tables behind the simulation are published on Hugging Face.
WM microstructure 7T stresses the same physics at high field. Here white-matter orientation effects are stronger, and several methods’ 3 T assumptions no longer hold.
Ridani 3T anisotropic reproduces the published phantom of Ridani et al. (2026), so results are comparable with that paper. R2′ is fitted from the signal, as for the other datasets. Its relaxation model is simpler than real tissue: iron relaxes only outside white matter, and myelin only inside it, with an orientation-dependent relaxivity. The published phantom has no calcification, so this dataset has none either.
Retired datasets. Ridani 3T isotropic, Ridani 7T anisotropic and the 7 T multicompartment phantom are retired: they no longer appear on the leaderboard, and their past results are kept in QSM-CI’s results file. The 7 T Ridani set duplicated the 3 T one for most methods, and the multicompartment phantom’s white-matter model was the same as one submitted method’s.
Scoring
- Headline, split error (ppb, lower is better). Each method is given χtotal, and most keep it, so an error in χ⁺ comes with the opposite error in χ⁻. Scoring the two maps separately would count that one error twice. The split error, ½[(χ⁺ error) + (|χ⁻| error)] at each voxel, is susceptibility put in the wrong source. The total error, shown beside it, is the error in χ⁺ − |χ⁻|, which mostly comes from the inputs. Both are RMS values with each tissue class (iron nuclei, thalamus, white matter, cortical grey matter, CSF, veins) weighted equally, so the iron nuclei, under 2 % of the brain, count as much as white matter. The leaderboard is sorted by split error, lowest first.
- Region calibration: the slope and offset of estimated against true region means: χ⁺ over the iron nuclei, and χ⁻ over the white-matter fibre bundles and the iron nuclei. A slope of 1 and an offset of 0 are ideal.
- Leakage: a whole-brain regression of each map on the other source’s truth (0 is ideal), and the excess of each source over its truth where the other source dominates (χ⁻ in the iron nuclei, where true |χ⁻| is 10–30 ppb, and χ⁺ in white matter).
- Secondary metrics: χ⁺ and χ⁻ are also scored separately with xSIM, NRMSE (also detrended) and correlation, and with the per-ROI mean squared percentage error of Ridani et al. (2026), which compares region means and so is comparable with that paper. xSIM is averaged over the whole brain, where 86 % of voxels have χ⁺ below 0.01 ppm, so it is lenient to scale errors. The Ridani MSPE scores χ⁻ in white matter only, so it does not see false χ⁻ in the iron nuclei.
- Sign convention: χ⁻ is stored and shown as a positive magnitude.
- Baselines: two closed-form baselines are listed and ranked with their numbers, like every method. The null baseline solves χtotal and R2′ per voxel with one fixed relaxivity; the Ridani baseline inverts that phantom’s region-split model. A method shows separation skill by beating them.
Caveats
- Learning-based methods were trained on in-vivo labels (χ-sepnet) or on maps derived from another method (SUSEP-Net), at particular field strengths and relaxivities. Results on a 7 T or simulated dataset say little about their in-vivo performance.
- DECOMPOSE depends on how phase evolves within a voxel over the echoes. That echo dependence appears only if water inside an iron or myelin source precesses as the method’s model assumes, and only for sources weak enough not to wrap within the echo train. In the mesoscopic datasets, built with standard physics, it is small. Its scores should be read with that in mind.
- Relaxivity matters. Methods assume different relaxivities: about 114, 142 or 324 Hz/ppm at 3 T. Part of the spread between methods reflects how close each assumption is to the tissue’s effective value, rather than separation quality.
Known simplifications
- The realistic inputs leave out the non-linear streaking of real dipole inversions, including the streaks around calcifications.
- The realistic inputs are calibrated at 3 T on one head anatomy.
- The mesoscopic phantoms have no vessels.
- In the iron-rich nuclei, the iron in the simulation adds more R2 than the head model leaves room for. The water’s own R2 is held at the CSF value there rather than lowered further, so R2 in those nuclei is slightly higher than in the head model.
Reproduce the datasets
The datasets are generated from qsm-forward presets (meso-3t, wm-micro-7t, ridani-3t-aniso). The mesoscopic look-up tables download automatically on first use. You also need the Challenge 2.0 tissue maps and the diffusion V1 eigenvector (Donders repository, doi:10.34973/m20r-jt17), and the white-matter tract atlas from Ridani et al. (OSF 9xwhz). QSM-CI’s scripts/gen_chisep_preset.py builds each dataset, including its realistic (L3) inputs, from the pinned preset in scripts/datasets.json.
Citations
- Separation phantom: Ridani D., De Leener B., Alonso-Ortiz E. “A realistic in-silico brain phantom for quantifying susceptibility anisotropy-induced error in susceptibility separation.” Magn Reson Med 2026. doi:10.1002/mrm.70468.
- Head model and calcification susceptibility (SIM2): Marques J.P., Meineke J., Milovic C., et al. “QSM reconstruction challenge 2.0: a realistic in silico head phantom for MRI data simulation and evaluation of susceptibility mapping procedures.” Magn Reson Med 2021;86(1):526–542. doi:10.1002/mrm.28716.
- χ-separation model & Dr: Shin H.G., Lee J., Yun Y.H., et al. “χ-separation: Magnetic susceptibility source separation toward iron and myelin mapping in the brain.” NeuroImage 2021;240:118371. doi:10.1016/j.neuroimage.2021.118371.
- Static-dephasing regime: Yablonskiy D.A., Haacke E.M. “Theory of NMR signal behavior in magnetically inhomogeneous tissues: the static dephasing regime.” Magn Reson Med 1994;32(6):749–763. doi:10.1002/mrm.1910320610.
- DECOMPOSE: Chen J., et al. “Decompose QSM to sub-voxel diamagnetic and paramagnetic components based on gradient-echo MRI data.” NeuroImage 2021;242:118477. doi:10.1016/j.neuroimage.2021.118477.
- SUSEP-Net: Li, Gao, Sun, et al. “SUSEP-Net: simulation-supervised contrastive learning for susceptibility source separation via a sub-voxel QSM network.” arXiv:2506.13293, 2025. doi:10.48550/arXiv.2506.13293.
- WaveSep: Fang Z., Shin H.G., van Zijl P., Li X., Sulam J. “WaveSep: A Flexible Wavelet-Based Approach for Source Separation in Susceptibility Imaging.” MLCN, MICCAI 2023. doi:10.1007/978-3-031-44858-4_6.
- R2*-QSM: Dimov A.V., et al. “Magnetic Susceptibility Source Separation Solely from Gradient Echo Data: Histological Validation.” Tomography / J Neuroimaging 2022. doi:10.1111/jon.13014.
- Multicompartment white-matter signal: Wharton S., Bowtell R. “Fiber orientation-dependent white matter contrast in gradient echo MRI.” PNAS 2012;109(45):18559–18564. doi:10.1073/pnas.1211075109.
| # | Algorithm | vs COSMOS | vs STI χ33 | Runtime |
|---|---|---|---|---|
| DNF |
No in-vivo results yet.
COSMOS vs STI χ33: the harder reference
Each method's structural similarity against the two references. Every point sits below the diagonal: STI χ33 is a systematically harder target than COSMOS (roughly +20 NRMSE across the board), yet the two references rank methods almost identically. Coloured by family; hover for values.
Do NRMSE and HFEN agree?
The headline error metric (NRMSE) against the fine-detail metric (HFEN, the 2016 challenge's signature), both vs COSMOS. A high Spearman ρ means they reward the same methods; where a point strays off the trend, a method wins on bulk error but loses fine structure, or vice-versa.
Runtime vs accuracy
In-vivo structural similarity (xSIM vs COSMOS) against wall-clock runtime on a log scale. Green points are Pareto-optimal (nothing is both faster and more accurate); grey points are dominated.
Tuning rescues classical methods in vivo
Isolated dipole xSIM vs COSMOS at default parameters (grey) versus tuned on the in-vivo data (green), ordered by gain. At their defaults the classical iterative methods look far weaker than the pretrained networks; tuned, several leap most of the way to the best learned method (dashed). The payoff dwarfs the phantom's (medi and l1-qsm each gain roughly +0.3 xSIM) because their defaults are mistuned for real-data noise. Tuning on the scored data is a mild form of overfitting, and only some methods are tunable, so defaults stays the fair cross-method ranking.
In vivo · 2016 Reconstruction Challenge
This dataset is real acquired brain data from the 2016 QSM Reconstruction Challenge: a single-orientation 3 T scan of one volunteer. Each method inverts the ground-truth local tissue field to a susceptibility map, scored against two reference reconstructions rather than a true susceptibility (none exists in vivo). Only the dipole-inversion stage is scored, so the ranking reads as agreement with a reference, not accuracy against ground truth.
The challenge and its references
The 2016 Reconstruction Challenge (Langkammer et al., 2018) took the χ33 component of a susceptibility-tensor (STI) reconstruction as its reference, hoping to sidestep the orientation bias of COSMOS. Twenty-seven submissions were scored on RMSE, HFEN, SSIM and regional error. The lesson was as much about metrics as algorithms: tuned to minimise those errors, the winning maps were over-smoothed and lost fine structure, and near-identical scores hid visibly different maps. A follow-up (Milovic et al., 2020) then showed the χ33 reference itself did not match the measured phase once anisotropy, microstructure and noise were accounted for.
QSM-CI scores each method against both available references: the multi-orientation COSMOS reconstruction and the STI χ33 map. Source: the 2016 challenge (qsm.neuroimaging.at); the local tissue field and both reference χ maps are in ppm.
Citations
- 2016 Challenge report: Langkammer C., Schweser F., Shmueli K., et al. “Quantitative susceptibility mapping: Report from the 2016 reconstruction challenge.” Magn Reson Med 2018;79(3):1661–1673. doi:10.1002/mrm.26830.
- 2016 Challenge lessons: Milovic C., Tejos C., Acosta-Cabronero J., et al. “The 2016 QSM Challenge: Lessons learned and considerations for a future challenge design.” Magn Reson Med 2020;84(3):1624–1637. doi:10.1002/mrm.28185.
- COSMOS reference: Liu T., Spincemaille P., de Rochefort L., Kressler B., Wang Y. “Calculation of susceptibility through multiple orientation sampling (COSMOS).” Magn Reson Med 2009;61(1):196–204. doi:10.1002/mrm.21828.
How reproducible is each acquisition protocol, marginalised over reconstruction pipelines? Slope deviation |a−1| across all pipelines, as median (mean) [5th–95th percentile].
How wide is the spread, protocol by protocol?
Where the mean sits well right of the median, a minority of pipelines is dragging the average out: the protocol works for most pipelines and fails for a few.
Hover a dot for its pipeline, or click to open it.
| Protocol | Test–retest |a−1| | Inter-scanner |a−1| | BA bias | Pipelines |
|---|---|---|---|---|
Reproducibility results are not computed yet; the reconstruction matrix over all acquisitions is pending.
pipeline(s) with in-silico xSIM ≥ , ranked by reproducibility.
How reproducible is each reconstruction pipeline across acquisitions of the same head? |a−1| over the pairwise per-ROI orthogonal fits in each comparison class. A pipeline that harmonizes well has a small inter-scanner slope deviation without giving up test–retest precision. Both statistics are shown in every cell — where they disagree sharply the pipeline is bimodal, reproducible on some protocols and not others.
Combination matrix
Dipole inversion ↓ × background removal → · best combinations top-left · bar length = in-silico accuracy · click a cell to inspect the pipeline.
Combined & end-to-end methods
These span background removal and dipole inversion in one step (or the whole reconstruction), so they have no cell on the axes above. Same colour scale.
No pipelines match the current field mapping and accuracy floor.
| # | Pipeline | Test–retest |a−1| | Inter-scanner |a−1| | vs bridge |a−1| | In-silico xSIM | Pairs |
|---|---|---|---|---|---|---|
Reproducibility results are not computed yet; the reconstruction matrix over all acquisitions is pending.
Which stage decides reproducibility?
How much each stage's method choice reshuffles the full-pipeline |a−1| ranking, across every pipeline in the factorial (rank-based η², so one degenerate fit can't dominate). A larger share means that stage's choice matters more for reproducibility.
methods compared at this stage
Rank-based η² on each pipeline's median |a−1|: the share of the outcome ordering each stage's choice explains. Because the design is the complete factorial, these shares aren't confounded by the other stages.
Which methods are most reproducible, stage by stage?
For each method, its median |a−1| marginalised over every pipeline it appears in (dot), with the interquartile spread (bar) showing how much its partner stages swing it. Lower is more reproducible. Coloured by family; hover for the numbers.
Reproducible and accurate? Every method, both axes
Each point is a method, placed by its median |a−1| across every pipeline it appears in (y, lower is more reproducible) against its median in-silico (2019) xSIM over those same pipelines (x, higher is more accurate). Coloured by stage: field mapping, background removal, and the final χ-producing step (dipole inversion plus combined bf-removal+dipole and end-to-end methods). The bottom-right corner is the sweet spot: reproducible and accurate. Hover any point for its numbers. method(s) with a reproducibility fit but no in-silico run are omitted (no accuracy value to place them).
The reproducibility frontier
Each point is a pipeline: test–retest |a−1| (run-to-run precision) on x, inter-scanner |a−1| (the same head on two scanners: harmonizability) on y. Bottom-left is both precise and harmonizable. Pick a method to light up every pipeline that uses it; click any point to open it.
: median inter-scanner across pipelines that use it, vs for those that don't.
Where does harmonization fail? Region ×
Median |Δχ| for each nucleus, per , marginalised over the other pipeline stages and protocols. Green = the region's mean χ agrees across the selected acquisition pair; red = it doesn't. Columns sorted best→worst.
Does one scanner read a nucleus systematically higher?
Forest plot of − mean-χ bias per region (same subject and protocol, every pair of runs), median ± IQR across all pipelines. Deep-grey nuclei are bold. A dot off the dashed zero line is a systematic cross-scanner offset.
χ–iron calibration: do the scanners agree?
Median χ per deep-grey nucleus vs literature non-heme iron, one regression per scanner. The slope is χ-sensitivity to iron; matching slopes and overlapping points mean the scanners are harmonized on the iron axis. A tight linear χ–iron relation is expected (it’s one of QSM’s foundational validations), so this is a physical-plausibility & agreement check, not a precise calibration (n = 5 nuclei, and the iron values are fixed population constants, not these subjects’). Iron (µg Fe/g wet wt) from Hallgren & Sourander 1958: pallidum 213, putamen 133, caudate 93, ventral DC (SN/RN) ~190, thalamus ~55.
Which protocol shifts which region?
Each protocol's per-region mean-χ deviation from the cross-protocol consensus (same scanner/run/pipeline), averaged over everything else. This is a signed map: pale/near-white = the protocol agrees with the consensus (good); saturated blue = it reads that region lower, saturated red = higher. So both strong colours mean the protocol shifts that region, just in opposite directions (e.g. whether Pulseq pulls a nucleus up or down versus bridge/local).
Which stage drives deep-grey variability?
For 3-stage pipelines, swap the method at one stage while holding the other two plus the acquisition fixed, and measure the spread of deep-grey mean χ. A taller bar means that stage moves the clinically-relevant χ more, attributing reproducibility to unwrapping vs background removal vs dipole inversion.
Is inter-nucleus contrast preserved across scanners?
The χ difference between neighbouring nuclei on the (y) vs the (x), one dot per pipeline × subject × protocol × pair of runs. On the dashed identity line the contrast (e.g. pallidum−putamen) reproduces across scanners; scatter off it means the contrast a clinician reads changes with the scanner even when global χ looks fine.
Does the headline metric reflect the clinical regions?
Each pipeline's whole-brain inter-scanner slope-deviation (today's harmonization headline) vs its deep-grey regional χ agreement. A tight correlation means the global number is a good proxy; scatter means pipelines can look harmonized globally yet disagree in the nuclei that matter (or vice versa).
Harmonization · across scanners, protocols and runs
The same brains, scanned again and again: subjects on scanners, multi-echo GRE acquisitions in all. There is no ground truth and none is possible; the question is reproducibility: how consistent are a pipeline's susceptibility values across runs, protocols and scanners of the same brain? Every comparison stays within one subject.
EMTP traveling heads. Each subject was scanned at their home site (KKI: Philips Achieva or Ingenia Elition X) and, for four of them, at one or two further sites (Weill Cornell CBIC: GE MR750 and Siemens Prisma Fit; UMass: Siemens Skyra; Hamburg: Philips Ingenia), with the shared bridge protocol and each site's own local protocol, three runs each. The MGH pilot acquisitions are a separate dataset, Harmonization pilot (MGH).
MGH pilot. One volunteer on two Siemens 3 T systems at MGH (bays 4–5: a MAGNETOM Prisma Fit and a MAGNETOM Cima.X), with four protocols, three runs each (one Pulseq offline run could not be reconstructed). Data acquired by Berkin Bilgic. The EMTP traveling heads are a separate dataset, Harmonization (2026).
Software. These results come from QSMxT v11.0.0. The re-runs of fixed methods used a pre-release build of QSMxT (main at commit 625b9f3, from before QSMxT took the scanner's centre frequency for the phase-to-ppm conversion; image qsm-ci/qsmxt@sha256:d9e80a9b…, tag interim-625b9f3): iSMV, HARPERELLA's default (now the whole rim; the STI Suite rim is the .sti variant), iHARPERELLA's whole-rim variant, and the NDI and WH-QSM weighting variants. V-SHARP's default no longer deconvolves; the deconvolved runs are the .deconv variant. NLTV and FANSI nlTV/nlTGV on the single-echo site 2 local acquisitions (and three further NLTV runs) were re-run with a guarded Newton step (the same pre-release build with QSM.rs 2828152, image qsm-ci/qsmxt@sha256:001d0768…, tag interim-625b9f3-qsmrs2828152): the original solvers' unguarded step diverges on long-echo single-echo data. Still to come in later updates: the mask-erosion variants and local-field previews for the re-run methods.
Software and brain masks. The MGH pilot results shown here come from QSMxT v11.0.0 on the original brain masks. A re-run on the tighter masks that match the traveling-head data, with the re-runs of fixed methods (pre-release build of QSMxT main at commit 625b9f3), replaces them in a later update. V-SHARP's default no longer deconvolves (the deconvolved runs are the .deconv variant), and HARPERELLA's v11.0.0 rim is the .sti variant.
The datasets come from the QSM harmonization initiative of the ISMRM Electro-Magnetic Tissue Properties Study Group (Costagli, Lancione, Bilgic, Robinson, Schweser, Li, and colleagues). The analysis method (per-ROI orthogonal fits χB = a·χA + b between acquisitions, with |a−1| as the headline) follows Marta Lancione's two-pipeline analysis, which QSM-CI extends across its full pipeline matrix.
Who was scanned where
| Subject | Scanners (acquisitions) | Protocols | Acquisitions |
|---|---|---|---|
The first scanner listed is the subject's home scanner, the reference side (x) of every inter-scanner fit. Local means each site's own protocol, so it is a different sequence at every site; pooled "local" numbers mix them.
The four MGH pilot protocols
| Protocol | Sequence | TEs (ms) | TR / FA |
|---|---|---|---|
| Bridge | Siemens product GRE implementing the consensus protocol (GRAPPA R=2, 1 mm iso) | 5 / 11 / 17 / 23 / 29 | 35 ms / 15° |
| Local | Each site’s optimised protocol: Prisma is GRE 2×2, sum-of-squares; Cima is 3D-EPI, CAIPIRINHA 3×2 | Prisma: 7.04 / 13.4 / 19.76 / 26.12 Cima: 6.5 / 12 / 17.5 / 23 | 31 ms / 15° |
| Pulseq (online) | Vendor-neutral Pulseq consensus sequence (HarmonizedMRI/megre_label), Siemens ICE reconstruction | 5 / 11 / 17 / 23 / 29 | 35 ms / 15° |
| Pulseq (offline) | Same acquisition, offline GRAPPA reconstruction (removes the vendor recon from the chain) | 5 / 11 / 17 / 23 / 29 | 35 ms / 15° |
Echo times were recovered from the scanners’ exam-card PDFs and the Pulseq sequence source (the exported DICOM sidecars carry none, and the Pulseq exam cards show placeholder timings). The full recovered acquisition manifest ships with the dataset as protocols.json.
How it is evaluated
Every QSM-CI pipeline combination (field mapping × background removal × dipole inversion, plus single-step methods) reconstructs all acquisitions from raw phase. Each χ map is rigid-registered to its subject's target acquisition, referenced to its whole-brain mean, and reduced to per-ROI means over one SynthSeg segmentation per subject (FreeSurfer mri_synthseg, which is license-free and emits standard FreeSurfer aseg labels (the deep-grey nuclei used in the regional Findings figures below), so ROI definitions cannot vary between a subject's acquisitions). Every acquisition pair then gets an orthogonal (total-least-squares) fit χB = a·χA + b: |a−1| measures scale disagreement, b the bias, and Bland–Altman limits the ROI-level scatter. Test–retest pairs bound what each protocol/pipeline could possibly achieve across scanners.
A fit is rejected outright when its slope is non-physical (|a−1| > , i.e. a outside [, ]), which would otherwise let one blown-up number dominate an average. This is rare: of pairs (%), and not a borderline call — rejected slopes run to a median |a−1| of and a maximum of . They are not spread evenly: and % are comparisons, , where a handful of solvers lose χ contrast entirely. A further pairs are missing because pipelines failed to reconstruct an acquisition. In total of possible pairs (%) survive.
Data availability
The MGH pilot is on OSF at osf.io/gkemr: raw/ holds the original exports per scanner/protocol (with corrected sidecars and exam-card PDFs), and qsmci-inputs/ one prepacked zip per acquisition in the QSM-CI canonical layout (4D magnitude/phase in radians, SynthStrip mask, params.json). The EMTP traveling-head inputs will be released with the paper.
Dipole inversion across datasets
The same isolated dipole-inversion methods, scored on the in-silico phantom (2019, true χ) and the in-vivo challenge (2016, COSMOS reference). Does a ranking on the idealised phantom carry over to real acquired data?
Phantom rank does not predict in-vivo rank
Each method's xSIM on the in-silico phantom (x) against its xSIM in vivo (y), coloured by family. The two are essentially uncorrelated: classical iterative and direct methods that win on the clean phantom collapse in vivo, while deep-learning methods hold up. Hover for values; click to open a method.
How methods move between datasets
Each method's rank on the in-silico phantom (left) linked to its rank in vivo (right), by xSIM. Lines coloured by family: deep-learning methods tend to climb left-to-right; classical methods fall.
Domain robustness by family
Mean xSIM per method family on each dataset. Deep-learning methods barely drop from phantom to in-vivo; classical iterative and direct methods lose far more: tuned to the idealised phantom, they generalise worse to real 3 T data.
The tuned optimum moves with the data
For every dipole method tuned on both datasets, its optimal regularisation strength on the in-silico phantom (left) versus in vivo (right), on a log scale. The in-vivo optimum sits roughly 10–30× higher almost everywhere: real 3 T data (with acquisition noise and forward-model mismatch) needs far heavier regularisation than the idealised phantom, so a single “recommended” value can't serve both. Coloured by family; hover for the shift.