New pixels, old trajectories

Change the colours, move the camera, replace the background, compress the video again. The pixels are new. The event may not be. If a model saw the same collision during training, how much should a correct prediction on the new rendering count as generalisation?

We built a collision simulator that can render the same recorded motion in different ways, and an audit that measures how reusing an event changes a predictor's score. A trajectory is the sequence of physical states through time. We tested the audit with a predictor that stores trajectories, compared simple predictors on public collision data, and trained a first pixel-conditioned ConvLSTM.

Exact file identity can be tested by hashing. Semantic similarity alone does not establish reuse of the same physical event. Video retrieval work already distinguishes transformed duplicates, complementary views and related incidents, while memorisation work on video diffusion separates repeated content from repeated motion. Known simulator lineage gives us a more specific relationship: we can hold the physical event fixed, change its pixels, and measure the effect of that exposure on evaluation.

Four relationships, not one duplicate flag

We record each video's relationship to a source event. An exact duplicate repeats its pixels. A rerender changes appearance but preserves every stored position and velocity. A nearby trajectory is generated separately and meets a similarity rule fixed in advance. An independent sample is drawn afresh without selecting for similarity. We call the renderings of one trajectory a source family.

The planned experiment compares copies of one starting model. One copy trains on source trajectories; the reference copy trains on independently generated replacements. Both begin with identical model and optimiser states and receive the same number of updates and examples. We will test them on new renderings of source events and on independently sampled events.

For each test set, the gain is the reference model's average error minus the source-trained model's average error:

A positive gap means that the source-trained branch has a larger relative advantage over the replacement-trained branch on source events than on independent events. This can come from source improvement, independent-event degradation or both, so we report both models' absolute errors and their changes from the starting checkpoint.

First, test a predictor that remembers

We first tested a predictor designed to remember: it stores the opening state frames of each trajectory and returns the recorded future of the closest match. It sees eight frames of object positions and predicts the next sixteen. Across sixty-four independent runs, we compared two memory banks of thirty-two trajectories each and tested them on source events and 128 freshly sampled events per run. Error is the average distance between predicted and true object centres, measured in arena widths, with each distance capped at one.

Compared with the replacement bank, source memory reduced error by 0.03337 arena widths on source events and 0.00012 on independent events. Reweighting those measured gains to a hypothetical test set with fifty-percent source-event reuse increases the estimated advantage over independent-only evaluation by 0.5 × (0.03337 − 0.00012) = 0.01662 arena widths. The mean exceeded the saved threshold of 1/64 arena width, roughly one rendered pixel. The lower endpoint of its normal-approximation 95% interval was 0.01613, above zero. Together with exact retrieval of source events, that satisfied the saved pass rule.

Because it receives exact state positions, this predictor can retrieve a stored event perfectly. That makes it a useful check for a reversed sign, a mismatched memory bank or an incorrect aggregation. Rendering, visual matching and learned-model behaviour are separate gates.

A harder baseline, and a metric that changed the answer

We next asked how well trajectory lookup works on collisions from another dataset. We used the public collision evaluation file from PhyWorld, a study of physical-law learning in video generators. It contains 1,635 examples in three numbered groups, each with thirty-two frames of recorded object positions, initial radii and velocities, and a video.

We gave both predictors four frames of recorded positions and asked for the next twenty-eight. Constant velocity continues the displacement between the last two observed frames. Retrieval uses the exact radii and first four ground-truth position frames, standardises those features using the retrieval bank, and returns the closest example's stored future. Velocity metadata is excluded, although motion is visible across the position frames.

On the original all-state metric, retrieval won clearly. With 745 group-00000 examples in memory, it reduced mean error on 187 held-out examples from 0.2577 to 0.1653. Using all 932 examples from that group, it reduced error from 0.4729 to 0.2329 on group 00001 and from 0.5090 to 0.2604 on group 00002. For each fixed partition and retrieval bank, a 2,000-resample paired example bootstrap put the retrieval-minus-constant difference below zero.

Then we checked what the metric was counting. The file's state trajectories continue after objects leave the nominal image. Object centres are outside the scene in 15.2%, 33.2% and 43.6% of future object-frames across the three splits. Restrict the calculation to frames where both disks remain fully visible and the ranking reverses: constant velocity scores 0.0643, 0.0597 and 0.0634; retrieval scores 0.1054, 0.1083 and 0.1240.

The first result was arithmetically correct and scientifically incomplete. Retrieval was much better at extrapolating stored states after objects left the image. Constant velocity was better while both objects remained visible. The choice of evaluation window changed which baseline won.

This is exactly why the measurement comes before the model. “Beat constant velocity” and “beat retrieval” are not one undifferentiated gate.

Pixels in, future states out

We then trained a compact ConvLSTM visual-state predictor. It consumes eight RGB frames and predicts the next sixteen object-centre states. Each of three seeded runs used 512 generated training videos, 128 independently generated test videos and 300 updates. Every sampled video contains an equal-mass collision after the observation prefix and before the final four frames, with trajectories that remain inside the arena. The model has 1,085,552 parameters; all three runs together took 35.5 seconds on Apple MPS.

The ConvLSTM reduced mean centre error by 39.3% relative to constant velocity and 70.0% relative to persistence. Those reference baselines use ground-truth object positions from the observed prefix; nearest-trajectory retrieval also searches ground-truth position histories. Retrieval remained slightly better, 0.02903 against 0.02965. The ConvLSTM passes the feasibility gate on this conditioned collision distribution and gives us a starting model for the exposure experiment.

Next, expose the learned model

Copies of one starting ConvLSTM will receive independent replacement trajectories, one rendering of each source trajectory, or nearby independently generated trajectories. An additional condition will receive exact copies of source videos. We will match successful update counts, schedules and batch positions.

We will score independently trained blocks on exact copies, new renderings of source events, nearby events and unrestricted independent events. Each error stays visible. A larger source-specific gap counts as source learning only when the source-trained branch also improves on source events relative to both the starting checkpoint and the matched replacement branch.

We will also test methods for finding overlap: exact file matching, visual frame matching, temporal alignment and motion comparison. The simulator's lineage records provide the answer key. Detector thresholds will be fixed on separate development families, at matched false-removal rates, before they touch the evaluation set.

Several outcomes would weaken the case for a special advantage from reusing events: equal gains on source and independent events, an advantage fully explained by nearby trajectories, or matching methods that catch every overlap large enough to affect the score. Other outcomes would leave the question unresolved: tracking may be unreliable on generated frames, or the number of independent training runs required for a useful interval may exceed the compute budget.

The controlled study begins with one-dimensional collision dynamics rendered into 2D video, where lineage and event state are exact. The same lineage design can scale to two- and three-dimensional simulations, multiple camera views, RGB-D observations or point-cloud world models by grouping every observation derived from one latent trajectory.

A new file is not necessarily a new example, and a new rendering is not necessarily a new event. Before treating a held-out score as evidence of a world model, we should be able to say what, exactly, was held out.

Code, experiment configurations, machine-readable results and the findings record: physicalreasoning/trajectory-lineage.

References. B. Kang et al., How Far Is Video Generation from World Model: A Physical Law Perspective, ICML 2025. C. Chen et al., Investigating Memorization in Video Diffusion Models, 2024. G. Kordopatis-Zilos et al., FIVR: Fine-grained Incident Video Retrieval, 2019. S. He et al., A Large-Scale Comprehensive Dataset and Copy-Overlap Aware Evaluation Protocol for Segment-Level Video Copy Detection, CVPR 2022.