NF01 · Displacement null model: is motion more directed than chance?

An earlier exploratory draft (DV-motility_metrics_sanity_check.ipynb) made "flower plots" of real vs. step-shuffled trajectories, but shuffling a trajectory's own steps by order can't change its net displacement or path length — vector summation is order-invariant, so the "shuffled" and "real" endpoints are always identical. No formal statistic was ever computed.

A first pass here tried keeping each step's speed but replacing its direction with a synthetic random angle — that can differ from the observed trajectory, but the angles aren't real data (too far from what a cell actually does).

This version instead pools every real (dx, dy) step from every trajectory into one big pot, and builds each null trajectory by resampling that many real steps from the pot. Every step in the null is something a real cell actually did somewhere in the movie -- no fabricated angles -- while a null trajectory's steps come from many different cells at many different moments, so any one cell's own directional persistence ("memory") is destroyed. Runs on all live trajectories (not just ones reaching the final frame).

4,375 live trajectories, frames <= 59

	trajectory_id	frame_pos	frame_index	cell_id	label	centroid_x	centroid_y
0	0	0	0	Time00000_ChannelBrightfield LEB_Seq0000_cellp...	2	1929.510272	5.657058
1	0	1	1	Time00001_ChannelBrightfield LEB_Seq0001_cellp...	1	1932.016109	3.705700
2	0	2	2	Time00002_ChannelBrightfield LEB_Seq0002_cellp...	5	1928.963498	5.802046
3	0	3	3	Time00003_ChannelBrightfield LEB_Seq0003_cellp...	5	1929.228883	6.069633
4	0	4	4	Time00004_ChannelBrightfield LEB_Seq0004_cellp...	3	1930.711566	5.201693

Permutation test: observed net displacement vs. pool-resampled null (real steps only)

3,721 trajectories with >=10 frames tested
12.8% individually significant at p<0.05 (observed > null at least 95% of shuffles)
Population-level Wilcoxon signed-rank test (observed vs. each trajectory's own null mean):
  n=3721, statistic=3570086.0, p=0.0502, median(observed-null)=-0.42 px

	observed_displacement	null_mean	p_value
count	3721.000000	3721.000000	3721.000000
mean	13.349865	12.238796	0.484259
std	9.666990	2.480865	0.336299
min	0.099254	4.964155	0.000000
25%	6.033110	11.871413	0.162000
50%	11.351658	13.468303	0.474000
75%	18.356116	13.696235	0.803000
max	68.137434	14.427789	1.000000

Flower plots: real vs. one pool-resampled realization

Both panels re-base each trajectory to a common origin (its own start = (0,0)). The null panel's path is built from real steps drawn at random from the whole-population pool (not fabricated angles), so any coherent look in the real panel reflects genuine directional persistence, not an artifact of the null construction.

Bonus: turning-angle persistence

Autocorrelation of step heading vs. lag — real motion should decay slower than the null (which is flat near 0 by construction).

	lag	real_mean	null_mean	null_std
0	1	-0.171016	0.000335	0.010544
1	2	0.076488	0.000153	0.011034
2	3	0.068609	0.000467	0.010890
3	4	0.056782	0.000407	0.011371
4	5	0.057649	0.000794	0.011338
5	6	0.043809	0.000799	0.011706
6	7	0.046111	0.000367	0.011891
7	8	0.035329	0.000227	0.011634

Is the turning-angle persistence actually significant?

The plot above shows real vs. null autocorrelation curves, but no formal test yet. Since each trajectory has both a real value and its own null mean (averaged over resamples) at every lag, this is a paired comparison -- same Wilcoxon signed-rank approach as the displacement test above, run separately per lag.

	lag	n	statistic	p_value	median_diff
0	1	3521	603506.0	1.000000e+00	-0.196560
1	2	3521	4820008.0	4.643464e-179	0.073080
2	3	3521	4703330.0	6.782679e-156	0.066483
3	4	3521	4470368.0	1.695341e-114	0.052242
4	5	3521	4519866.0	9.425528e-123	0.054306
5	6	3521	4172210.0	6.060609e-71	0.043154
6	7	3521	4241207.0	4.414574e-80	0.043941
7	8	3521	4025028.0	2.411394e-53	0.034457

Why doesn't this show up as excess net displacement?

Good question to ask, since the persistence is real (p < 1e-50 at every lag 2-8 above). Two reasons:

1. The effect nets out to nearly zero once summed over lags. A weak persistent-random-walk boosts expected displacement roughly in proportion to the sum of the autocorrelation across lags (each lag contributing to how much a step "remembers" the previous heading). Here that sum is small: the significant lag-1 negative value (-0.20 median diff, likely measurement jitter -- see caveats below) largely cancels the smaller positive contributions from lags 2-8 (+0.03 to +0.07 each). Net effect on displacement: close to zero, consistent with the population displacement test's borderline p=0.05.
2. Statistical power differs hugely between the two tests. The turning-angle test uses every individual step-pair as a (weak) data point -- effectively ~180,000 of them across the population -- so even a tiny, consistent effect is easy to detect. The displacement test has only one number per trajectory (net displacement, n=3,721) -- a single trajectory's net displacement has already accumulated ~46 steps' worth of noise on top of whatever small persistent bias exists, diluting the signal into a much smaller, much noisier sample. The same underlying weak memory is real in both, but far easier to see in the step-level data than in the once-aggregated trajectory-level statistic.

Caveats: could technical error explain some of this?

The effect sizes throughout this notebook are small (a fraction of a pixel, a few hundredths of a correlation coefficient) -- exactly the regime where segmentation noise, tracking mistakes, or mislinking can masquerade as biology. Worth checking directly rather than assuming either way.

post-fixation 'should be still' steps: n=7,611, median=0.999 px
live steps: n=184,214, median=1.049 px
noise floor as a fraction of the live median step: 95%

This came out higher than expected: the post-fixation "should be motionless" median step is ~95% of the live median step size. That doesn't mean 95% of displacement is noise (a persistent random walk vs. pure jitter have very different statistical signatures even at similar step magnitudes), but it does mean the raw per-step signal-to-noise ratio here is low, and is a very plausible explanation for the negative lag-1 turning-angle autocorrelation above -- independent per-frame localization noise is a textbook cause of exactly that artifact in single-particle tracking. Two more specific technical-error concerns:

• Segmentation jitter. Cellpose mask boundaries can shift slightly frame-to-frame for a perfectly stationary cell (sub-pixel to ~1px), which is comparable to the median live step (1.05px) -- see above.
• Tracking/mislinking errors, especially under crowding. The tracker's distance-fallback and gap-bridging stages (pre/trajectories.py) are more likely to mislink in dense regions. A wrong link would attribute a "step" to the wrong physical cell. This is a particular concern for NF03's crowding-slows-speed finding: if mislinking is more common in crowded regions and systematically produces smaller apparent displacement (e.g. a broken/restarted track looks slower than one continuous real track), that alone could produce "crowded cells look slower" as an artifact, without any real biology behind it.

Next steps to rule this out, roughly in order of effort:

1. pre/trajectories.py's track() already records how each link was made (method: "overlap" vs. "distance" fallback). Re-run the key correlations (crowding vs. speed, shape-variability vs. speed) restricted to trajectories built entirely from confident overlap links, and check whether the effects survive.
2. Compare the noise-floor step-size distribution against the live step-size distribution directly (not just medians) -- if they're statistically indistinguishable in shape, that's a stronger signal the live data is noise-dominated at the single-step level.
3. Re-run NF01's displacement/turning-angle tests after a mild temporal smoothing (or a minimum-step threshold) on centroids, and see how much of the signal survives -- if it vanishes under smoothing, it was likely jitter.
4. Once the scrubber's search-by-id feature exists (queued separately), manually spot-check a sample of high-collision-count or high-shape-variability cells to visually confirm they're real tracked cells, not segmentation artifacts or mislinked identities.

---

⭐ Key finding

No population-level excess net displacement against the pool-resampled null (real steps only): Wilcoxon p = 0.050 (borderline), median(observed - null) = -0.42 px (n = 3,721 trajectories, >=10 live frames), 12.8% individually significant.

Turning-angle persistence IS formally significant, lags 2-8 (paired Wilcoxon per lag, n=3,521 trajectories): p ranges from 4.6e-179 (lag 2) to 2.4e-53 (lag 8), median diff +0.03 to +0.07. Lag 1 is significant in the opposite direction (p=1.0 for "greater", i.e. real is below null there) -- consistent with measurement jitter, not real anti-persistence.

Why doesn't real persistence show up as excess displacement? Two reasons, worked out with the actual numbers: (1) summed across lags the effect nearly cancels -- lag 1's negative median diff offsets most of lags 2-8's small positive ones; (2) the turning-angle test uses ~180,000 step-pairs vs. the displacement test's 3,721 once-aggregated trajectory values, so the same weak real signal is far easier to detect at the step level than after ~46 steps of accumulated noise per trajectory.

Caveat, checked directly: the post-fixation "should be motionless" step size (median 0.999px, frames 61-64) is 95% of the live median step size (1.049px) -- the raw per-step signal-to-noise ratio is low, a very plausible driver of the negative lag-1 autocorrelation (a textbook artifact of localization noise in particle tracking). This doesn't invalidate the lag 2-8 result (a persistent random walk and pure jitter have different signatures even at similar step magnitudes), but it's a real reason for caution, and a specific concern for NF03's crowding-slows-speed finding if mislinking is more common in crowded regions. See the caveats section above for concrete next steps (overlap-only vs. distance-fallback comparison, smoothing sensitivity, manual spot-checks).