Airliner video shows complex treatment of depth

I used some basic computer vision techniques to analyze the airliner satellite video (see this thread if this video is new to you). tl;dr: I found that the video shows complex treatment of depth that would come from 3D VFX possibly combined with custom software, or from a real video, but not from 2D VFX.

The oldest version of the video from RegicideAnon has two unique perspectives forming a stereo pair. The apparent distance between the same object in both images of a pair is called “disparity” (given in pixel units). Using disparity, we may be able to make an estimate of the orientation of the cameras. This would help identify candidate satellites, or rule out the possibility of any satellite ever taking this video.

To start, I tried using StereoSGBM to get a dense disparity map. It showed generally what I expected: the depth increasing towards the top of the frame, with the plane popping out. But all the compression noise gives a very messy result and details are not resolved well.

StereoSGBM disparity map for a single stereo pair (left RGB image shown for reference).

I tried to get a clean background image by taking the median over time. I ran this for each section of video where the video was not being manually panned. That turned noisy image pairs like this:

Noisy image pair from frame 1428.

Into clean image pairs like this:

Denoised image pair from sixth section of video (frames 1135-1428).

I tried recomputing the disparity map using StereoSGBM, but I found that it was still messy. StereoSGBM uses block matching, and it only really works up to 11 pixel blocks. Because this video has very sparse features, I decided to take another approach that would allow for much larger blocks: a technique called phase cross correlation (PCC). Given two images of any size, PCC will use frequency-domain analysis to estimate the x/y offset.

I divided both the left and right image into large rectangular blocks. Then I used PCC to estimate the offset between each block pair.

PCC results on sixth section of video (frames 1135-1428).

In this case, red means that there is a larger x offset, and gray means there is no x offset (this failure case happens inside clouds and empty ocean). This visualization shows that the top of the image is farther away and the bottom is closer. If you are able to view the video in 3D by crossing your eyes, or some other way, you may have already noticed this. But with exact numbers, we can get a more precise characterization of this pattern.

So I ran PCC across all the median filtered image pairs. I collected all the shifts relative to their y position.

Showing a line fit with slope of -0.0069.

In short, what this line says is that the disparity has a range of 6 pixels, and that at any given y position the disparity has a range of around 2 pixels. If the camera was directly above this location, we would expect the line fit to be fairly flat. If the camera was at an extreme angle, we would expect the line fit to drastically increase towards the top of the image. Instead we see something in-between.

Declination of the cameras: In theory we should be able to use disparity plot above to figure this out, but I think to do it properly you might have to solve the angle between the cameras and the declination at the same time—for which I am unprepared. So all I will say is that it looks high without being directly above! Angle between the cameras: When the airplane is traveling from left to right, it’s around 46 pixels wide for its 64m length. That’s 1.4 m/pixel. If the cameras were directly above the scene, that would give us a triangle with a 2px=2.8m wide base and 12,000m height. That’s around 0.015 degrees. Since the camera is not directly above, then the distance from the plane to the ocean will be larger, and the angle will be more narrow than 0.015 degrees. Distance to the cameras: If we are working with Keyhole-style optics (2.4m lens for 6cm resolution at 250 km) then we could be 23x farther away than usual and still have 1.4m resolution (up to 5,750km, nearly half the diameter of earth).

Next, instead of analyzing the whole image, we can analyze the plane alone by subtracting the background.

Frame 816 before and after background subtraction.

Using PCC on the airplane shows a similar pattern of having a smaller disparity towards the bottom of the image, and larger towards the top of the image. The colors in the following diagram correspond to different sections of video, in-between panning.

https://preview.redd.it/p3h0gkulm1ib1.png?width=554&format=png&auto=webp&s=5db5cddca099de578093c91b132394e77e1b2b77

(Some of the random outlier points are errors from moments when the plane is not in the scene.)

Here’s the main thing I discovered. Notice that as the plane flies towards the bottom of the screen (from left to right on the x axis in this plot), we would expect the disparity to keep decreasing until it becomes negative. But instead, when the user pans the image downward, the disparity increases again in the next section, keeping it positive. If this video a hoax, this disparity compensation feature would have to be carefully designed—possibly with custom software. It would be counterintuitive to render a large scene in 3D and then comp the mouse cursor and panning in 2D afterwards. Instead you would want to move the orthographic camera itself when rendering, and also render the 2D mouse cursor overlay at the same time. Or build custom software that knows about the disparity and compensates for it. Analyzing the disparity during the panning might yield more insight here.

My main conclusion is that if this is fake, there are an immense number of details taken into consideration.

Details shared by both videos: Full volumetric cloud simulation with slow movement/evolution, plane contrails with dissipation, the entire “portal flash” sequence, camera characteristics like resolution, framerate, motion blur (see frame 371 or 620 on the satellite video for example), knowledge of airplane performance (speed, max bank angle, etc).

Details in the satellite video: The disparity compensation I just mentioned, and the telemetry that goes with it. Rendering a stereo pair in the first place. My previous post about cloud illumination. And small details like self-shadowing on the plane and bloom from the clouds. Might the camera positions prove to match known satellites?

Details in the thermal video: the drone shape and FLIR mounting position. Keeping the crosshairs, but picking some unusual choices like rainbow color scheme and no HUD. But especially the orb rendering is careful: the orbs reflect/refract the plane heat, they leave cold trails, and project a Lazar-style “gravity well”.

If this is all interesting to you, I’ve posted the most useful parts of my code as a notebook on GitHub.