Worker Communication

AMQP Communication

This page documents the AMQP (Advanced Message Queuing Protocol) communication used in the nerf-or-nothing project. It covers the message formats and queue structures used for communication between the web server and workers.

Queue Structure

The following messaging queues are used in the system:

• sfm-in: Input queue for SFM (Structure from Motion) jobs
• sfm-out: Output queue for completed SFM jobs
• nerf-in: Input queue for NeRF (Neural Radiance Fields) jobs
• nerf-out: Output queue for completed NeRF jobs

Additional scene queue lists managed by the QueueListManager:

• sfm_list: List of SFM jobs in progress
• nerf_list: List of NeRF jobs in progress
• queue_list: Overall list of jobs in the pipeline

Message Formats

SFM Job (sfm-in)

When a new video is uploaded, a SFM job is published to the sfm-in queue:

Example

{
  "id": "5f9b2c1b3e7a8d1234567890",
  "file_path": "http://web-server:5000/worker-data/data/raw/videos/5f9b2c1b3e7a8d1234567890.mp4"
}

SFM Job Result (sfm-out)

After processing, the SFM worker publishes the result to the sfm-out queue, which is read by a dedicated goroutine.

Example

{
  "id": "5f9b2c1b3e7a8d1234567890",
  "vid_width": 1920,
  "vid_height": 1080,
  "sfm": {
    "intrinsic_matrix": [
       [focal_x, 0, center_x],
       [0, focal_y, center_y],
       [0, 0, 1]
    ],
    "frames": [
      {
        "file_path": "http://sfm-worker:5100/data/outputs/5f9b2c1b3e7a8d1234567890/frame_0001.png",
        "extrinsic_matrix": [
          [r11, r12, r13, t1],
          [r21, r22, r23, t2],
          [r31, r32, r33, t3],
          [0, 0, 0, 1]
        ]
      },
      // ... more frames ...
    ],
    "white_background": true
  },
  "flag": 0
}

NeRF Job (nerf-in)

After processing the SFM result, a NeRF job is published to the nerf-in queue by the sfm-out goroutine. Due to previously supporting tensoRF and now Gaussian Splatting, the matrix transforms must be modified into an alternate form by nerf-worker for use:

Example TensoRF format

{
  "id": "5f9b2c1b3e7a8d1234567890",
  "vid_width": 1920,
  "vid_height": 1080,
  "frames": [
    {
      "file_path": "http://web-server:5000/worker-data/data/sfm/5f9b2c1b3e7a8d1234567890/frame_0001.png",
      "extrinsic_matrix": [
        [r11, r12, r13, t1],
        [r21, r22, r23, t2],
        [r31, r32, r33, t3],
        [0, 0, 0, 1]
      ]
    }
    // ... more frames ...
  ],
  "intrinsic_matrix": [
    [focal_x, 0, center_x],
       [0, focal_y, center_y],
       [0, 0, 1]
  ],
  "white_background": true,
  "output_types": ["splat_cloud", "video"],
  "training_mode": "gaussian",
  "save_iterations": [1000, 7000, 30000],
  "total_iterations": 30000
}

The gaussian splatting (blender3D) transform standard is:

{
 "camera_angle_x": fov_x,
 "camera_angle_y": fov_y,
 "frames": [
     {
       "file_path": "path/to/image",
       "transform_matrix": [
         [r11, r12, r13, t1],
         [r21, r22, r23, t2],
         [r31, r32, r33, t3],
         [0, 0, 0, 1]
       ]
     },
     // ... more frames ..
  ],
  // ... rest of nerf-in message ...
}

NeRF Job Result (nerf-out)

After processing, the NeRF worker publishes the result to the nerf-out queue:

Example

{
  "id": "5f9b2c1b3e7a8d1234567890",
  "file_paths": {
    "splat_cloud": {
      "1000": "http://nerf-worker:5200/data/nerf/5f9b2c1b3e7a8d1234567890/splat_cloud/iteration_1000/output.splat",
      "7000": "http://nerf-worker:5200/data/nerf/5f9b2c1b3e7a8d1234567890/splat_cloud/iteration_7000/output.splat",
      "30000": "http://nerf-worker:5200/data/nerf/5f9b2c1b3e7a8d1234567890/splat_cloud/iteration_30000/output.splat"
    },
    "video": {
      "1000": "http://nerf-worker:5200/data/nerf/5f9b2c1b3e7a8d1234567890/video/iteration_1000/output.mp4",
      "7000": "http://nerf-worker:5200/data/nerf/5f9b2c1b3e7a8d1234567890/video/iteration_7000/output.mp4",
      "30000": "http://nerf-worker:5200/data/nerf/5f9b2c1b3e7a8d1234567890/video/iteration_30000/output.mp4"
    }
  }
}

Processing Flow

1. User uploads a video, which is processed by HandleIncomingVideo.
2. A SFM job is published to sfm-in and the scene ID is added to sfm_list and queue_list.
3. The SFM worker processes the job and publishes the result to sfm-out.
4. The web server processes the SFM result, updates the database, and publishes a NeRF job to nerf-in.
5. The scene ID is removed from sfm_list and added to nerf_list.
6. The NeRF worker processes the job and publishes the result to nerf-out.
7. The web server processes the NeRF result, saves the output files, updates the database, and removes the scene ID from nerf_list and queue_list.

This communication flow allows for asynchronous processing of videos through the SFM and NeRF stages, with the web server coordinating the overall pipeline.