-
Notifications
You must be signed in to change notification settings - Fork 75
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
omninets train result is very low #47
Comments
[03/18 09:12:03 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3028 s/iter. Eval: 1.2359 s/iter. Total: 1.5396 s/iter. ETA=0:00:56
100%|██████████| 20/20 [00:07<00:00, 2.72it/s][03/18 09:14:17 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 09:45:01 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3173 s/iter. Eval: 1.0703 s/iter. Total: 1.3884 s/iter. ETA=0:00:51
100%|██████████| 20/20 [00:08<00:00, 2.46it/s][03/18 09:47:13 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 10:18:45 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3187 s/iter. Eval: 0.7140 s/iter. Total: 1.0338 s/iter. ETA=0:00:38
100%|██████████| 20/20 [00:08<00:00, 2.45it/s][03/18 10:20:32 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 10:52:04 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3022 s/iter. Eval: 0.6537 s/iter. Total: 0.9569 s/iter. ETA=0:00:35
100%|██████████| 20/20 [00:08<00:00, 2.39it/s][03/18 10:53:37 tridet]: Evaluation results for kitti_3d_val in csv format:
|
[03/18 10:53:55 d2.utils.events]: eta: 1 day, 3:08:42 iter: 10020 total_loss: 2.608 loss_box2d_reg: 0.1859 loss_box3d_depth: 0.6881 loss_box3d_proj_ctr: 0.1245 loss_box3d_quat: 0.188 loss_box3d_size: 0.09417 loss_centerness: 0.6208 loss_cls: 0.1405 loss_conf3d: 0.5876 lr: 0.002 max_mem: 5634M [03/18 11:25:24 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0011 s/iter. Inference: 0.3317 s/iter. Eval: 0.4896 s/iter. Total: 0.8225 s/iter. ETA=0:00:30
100%|██████████| 20/20 [00:08<00:00, 2.43it/s][03/18 11:26:46 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 11:57:43 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3177 s/iter. Eval: 0.3748 s/iter. Total: 0.6935 s/iter. ETA=0:00:25
100%|██████████| 20/20 [00:08<00:00, 2.35it/s][03/18 11:58:56 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 12:30:09 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3160 s/iter. Eval: 0.3866 s/iter. Total: 0.7035 s/iter. ETA=0:00:26
100%|██████████| 20/20 [00:08<00:00, 2.33it/s][03/18 12:31:22 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 13:02:48 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3115 s/iter. Eval: 0.3154 s/iter. Total: 0.6278 s/iter. ETA=0:00:23
100%|██████████| 20/20 [00:08<00:00, 2.32it/s][03/18 13:03:55 tridet]: Evaluation results for kitti_3d_val in csv format:
|
[03/18 13:04:13 d2.utils.events]: eta: 9:58:05 iter: 18020 total_loss: 2.398 loss_box2d_reg: 0.1528 loss_box3d_depth: 0.6664 loss_box3d_proj_ctr: 0.08491 loss_box3d_quat: 0.1119 loss_box3d_size: 0.07235 loss_centerness: 0.6144 loss_cls: 0.09708 loss_conf3d: 0.581 lr: 0.002 max_mem: 5634M [03/18 13:35:32 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3236 s/iter. Eval: 0.2405 s/iter. Total: 0.5652 s/iter. ETA=0:00:20
100%|██████████| 20/20 [00:08<00:00, 2.31it/s][03/18 13:36:35 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 14:09:09 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3296 s/iter. Eval: 0.2070 s/iter. Total: 0.5375 s/iter. ETA=0:00:19
100%|██████████| 20/20 [00:09<00:00, 2.21it/s][03/18 14:10:11 tridet]: Evaluation results for kitti_3d_val in csv format:
|
[03/18 14:10:30 d2.utils.events]: eta: 3:56:45 iter: 22020 total_loss: 1.756 loss_box2d_reg: 0.1245 loss_box3d_depth: 0.2839 loss_box3d_proj_ctr: 0.06876 loss_box3d_quat: 0.07323 loss_box3d_size: 0.05928 loss_centerness: 0.6114 loss_cls: 0.07886 loss_conf3d: 0.4414 lr: 0.0002 max_mem: 5634M [03/18 14:42:23 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0010 s/iter. Inference: 0.3297 s/iter. Eval: 0.2114 s/iter. Total: 0.5421 s/iter. ETA=0:00:20
100%|██████████| 20/20 [00:09<00:00, 2.20it/s][03/18 14:43:24 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 14:59:41 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0008 s/iter. Inference: 0.3292 s/iter. Eval: 0.1822 s/iter. Total: 0.5123 s/iter. ETA=0:00:18
100%|██████████| 20/20 [00:09<00:00, 2.20it/s][03/18 15:00:41 tridet]: Evaluation results for kitti_3d_val in csv format:
|
[03/18 15:00:41 tridet]: Starting inference with test-time augmentation. [03/18 15:00:50 d2.evaluation.evaluator]: Inference done 1/48. Dataloading: 1.1408 s/iter. Inference: 6.7434 s/iter. Eval: 0.5676 s/iter. Total: 8.4560 s/iter. ETA=0:06:37
100%|██████████| 20/20 [00:10<00:00, 1.86it/s][03/18 15:07:25 tridet]: Evaluation results for kitti_3d_val in csv format:
|
[03/18 08:06:01 tridet.utils.hydra.callbacks]: Rank of current process: 0. World size: 8
[03/18 08:06:01 tridet.utils.setup]: Working Directory: /home/azuryl/dd3d_test/omni/dd3d/outputs/2023-03-18/08-05-55
[03/18 08:06:01 tridet.utils.setup]: Full config:
{
"WANDB": {
"ENABLED": false,
"DRYRUN": false,
"PROJECT": "dd3d",
"GROUP": null,
"TAGS": [
"kitti-val",
"dla34",
"bn"
]
},
"EVAL_ONLY": false,
"EVAL_ON_START": false,
"ONLY_REGISTER_DATASETS": false,
"OUTPUT_ROOT": "./outputs",
"SYNC_OUTPUT_DIR_S3": {
"ENABLED": false,
"ROOT_IN_S3": "???",
"PERIOD": 1000
},
"DATASET_ROOT": "/hdd1/datasets_left/",
"TMP_DIR": "/tmp/",
"DATASETS": {
"TRAIN": {
"NAME": "kitti_3d_train",
"CANONICAL_BOX3D_SIZES": [
[
1.61876949,
3.89154523,
1.52969237
],
[
0.62806586,
0.82038497,
1.76784787
],
[
0.56898187,
1.77149234,
1.7237099
],
[
1.9134491,
5.15499603,
2.18998422
],
[
2.61168401,
9.22692319,
3.36492722
],
[
0.5390196,
1.08098042,
1.28392158
],
[
2.36044838,
15.56991038,
3.5289238
],
[
1.24489164,
2.51495357,
1.61402478
]
],
"DATASET_MAPPER": "default",
"NUM_CLASSES": 5,
"MEAN_DEPTH_PER_LEVEL": [
32.594,
15.178,
8.424,
5.004,
4.662
],
"STD_DEPTH_PER_LEVEL": [
14.682,
7.139,
4.345,
2.399,
2.587
]
},
"TEST": {
"NAME": "kitti_3d_val",
"NUSC_SAMPLE_AGGREGATE_IN_INFERENCE": false,
"DATASET_MAPPER": "default"
}
},
"FE": {
"FPN": {
"IN_FEATURES": [
"level3",
"level4",
"level5"
],
"OUT_FEATURES": null,
"OUT_CHANNELS": 256,
"NORM": "FrozenBN",
"FUSE_TYPE": "sum"
},
"BUILDER": "build_fcos_dla_fpn_backbone_p67",
"BACKBONE": {
"NAME": "DLA-34",
"OUT_FEATURES": [
"level3",
"level4",
"level5"
],
"NORM": "FrozenBN"
},
"OUT_FEATURES": null
},
"DD3D": {
"IN_FEATURES": null,
"NUM_CLASSES": 5,
"FEATURE_LOCATIONS_OFFSET": "none",
"SIZES_OF_INTEREST": [
64,
128,
256,
512
],
"INFERENCE": {
"DO_NMS": true,
"DO_POSTPROCESS": true,
"DO_BEV_NMS": false,
"BEV_NMS_IOU_THRESH": 0.3,
"NUSC_SAMPLE_AGGREGATE": false
},
"FCOS2D": {
"_VERSION": "v2",
"NORM": "BN",
"NUM_CLS_CONVS": 4,
"NUM_BOX_CONVS": 4,
"USE_DEFORMABLE": false,
"USE_SCALE": true,
"BOX2D_SCALE_INIT_FACTOR": 1.0,
"LOSS": {
"ALPHA": 0.25,
"GAMMA": 2.0,
"LOC_LOSS_TYPE": "giou"
},
"INFERENCE": {
"THRESH_WITH_CTR": true,
"PRE_NMS_THRESH": 0.05,
"PRE_NMS_TOPK": 1000,
"POST_NMS_TOPK": 100,
"NMS_THRESH": 0.75
}
},
"FCOS3D": {
"NORM": "FrozenBN",
"NUM_CONVS": 4,
"USE_DEFORMABLE": false,
"USE_SCALE": true,
"DEPTH_SCALE_INIT_FACTOR": 0.3,
"PROJ_CTR_SCALE_INIT_FACTOR": 1.0,
"PER_LEVEL_PREDICTORS": false,
"SCALE_DEPTH_BY_FOCAL_LENGTHS": true,
"SCALE_DEPTH_BY_FOCAL_LENGTHS_FACTOR": 500.0,
"MEAN_DEPTH_PER_LEVEL": [
32.594,
15.178,
8.424,
5.004,
4.662
],
"STD_DEPTH_PER_LEVEL": [
14.682,
7.139,
4.345,
2.399,
2.587
],
"MIN_DEPTH": 0.1,
"MAX_DEPTH": 80.0,
"CANONICAL_BOX3D_SIZES": [
[
1.61876949,
3.89154523,
1.52969237
],
[
0.62806586,
0.82038497,
1.76784787
],
[
0.56898187,
1.77149234,
1.7237099
],
[
1.9134491,
5.15499603,
2.18998422
],
[
2.61168401,
9.22692319,
3.36492722
],
[
0.5390196,
1.08098042,
1.28392158
],
[
2.36044838,
15.56991038,
3.5289238
],
[
1.24489164,
2.51495357,
1.61402478
]
],
"CLASS_AGNOSTIC_BOX3D": false,
"PREDICT_ALLOCENTRIC_ROT": true,
"PREDICT_DISTANCE": false,
"LOSS": {
"SMOOTH_L1_BETA": 0.05,
"MAX_LOSS_PER_GROUP_DISENT": 20.0,
"CONF_3D_TEMPERATURE": 1.0,
"WEIGHT_BOX3D": 2.0,
"WEIGHT_CONF3D": 1.0
},
"PREPARE_TARGET": {
"CENTER_SAMPLE": true,
"POS_RADIUS": 1.5
}
}
},
"VIS": {
"DATALOADER_ENABLED": true,
"DATALOADER_PERIOD": 1000,
"DATALOADER_MAX_NUM_SAMPLES": 10,
"PREDICTIONS_ENABLED": true,
"PREDICTIONS_MAX_NUM_SAMPLES": 20,
"D2": {
"DATALOADER": {
"ENABLED": true,
"SCALE": 1.0,
"COLOR_MODE": "image"
},
"PREDICTIONS": {
"ENABLED": true,
"SCALE": 1.0,
"COLOR_MODE": "image",
"THRESHOLD": 0.4
}
},
"BOX3D": {
"DATALOADER": {
"ENABLED": true,
"SCALE": 1.0,
"RENDER_LABELS": true
},
"PREDICTIONS": {
"ENABLED": true,
"SCALE": 1.0,
"RENDER_LABELS": true,
"THRESHOLD": 0.5,
"MIN_DEPTH_CENTER": 0.0
}
}
},
"INPUT": {
"FORMAT": "BGR",
"AUG_ENABLED": true,
"RESIZE": {
"ENABLED": true,
"MIN_SIZE_TRAIN": [
288,
304,
320,
336,
352,
368,
384,
400,
416,
448,
480,
512,
544,
576
],
"MIN_SIZE_TRAIN_SAMPLING": "choice",
"MAX_SIZE_TRAIN": 10000,
"MIN_SIZE_TEST": 384,
"MAX_SIZE_TEST": 100000
},
"CROP": {
"ENABLED": false,
"TYPE": "relative_range",
"SIZE": [
0.9,
0.9
]
},
"RANDOM_FLIP": {
"ENABLED": true,
"HORIZONTAL": true,
"VERTICAL": false
},
"COLOR_JITTER": {
"ENABLED": true,
"BRIGHTNESS": [
0.2,
0.2
],
"SATURATION": [
0.2,
0.2
],
"CONTRAST": [
0.2,
0.2
]
}
},
"MODEL": {
"DEVICE": "cuda",
"META_ARCHITECTURE": "DD3D",
"PIXEL_MEAN": [
103.53,
116.28,
123.675
],
"PIXEL_STD": [
57.375,
57.12,
58.395
],
"CKPT": "",
"BOX2D_ON": true,
"BOX3D_ON": true,
"DEPTH_ON": false,
"backbone_with_fpn": null,
"width_mult": 1.0,
"depth_mult": 1.0
},
"DATALOADER": {
"TRAIN": {
"NUM_WORKERS": 12,
"FILTER_EMPTY_ANNOTATIONS": true,
"SAMPLER": "RepeatFactorTrainingSampler",
"REPEAT_THRESHOLD": 0.4,
"ASPECT_RATIO_GROUPING": false
},
"TEST": {
"NUM_WORKERS": 4,
"SAMPLER": "InferenceSampler"
}
},
"SOLVER": {
"IMS_PER_BATCH": 64,
"BASE_LR": 0.002,
"MOMENTUM": 0.9,
"NESTEROV": false,
"WEIGHT_DECAY": 0.0001,
"WEIGHT_DECAY_NORM": 0.0,
"BIAS_LR_FACTOR": 1.0,
"WEIGHT_DECAY_BIAS": 0.0001,
"GAMMA": 0.1,
"LR_SCHEDULER_NAME": "WarmupMultiStepLR",
"STEPS": [
21500,
24000
],
"WARMUP_FACTOR": 0.0001,
"WARMUP_ITERS": 2000,
"WARMUP_METHOD": "linear",
"CLIP_GRADIENTS": {
"ENABLED": false,
"CLIP_TYPE": "value",
"CLIP_VALUE": 1.0,
"NORM_TYPE": 2.0
},
"CHECKPOINT_PERIOD": 2000,
"MIXED_PRECISION_ENABLED": true,
"DDP_FIND_UNUSED_PARAMETERS": false,
"ACCUMULATE_GRAD_BATCHES": 1,
"SYNCBN_USE_LOCAL_WORKERS": false,
"MAX_ITER": 25000
},
"TEST": {
"ENABLED": true,
"EVAL_PERIOD": 2000,
"EVAL_ON_START": false,
"ADDITIONAL_EVAL_STEPS": [],
"IMS_PER_BATCH": 80,
"AUG": {
"ENABLED": true,
"MIN_SIZES": [
320,
384,
448,
512,
576
],
"MAX_SIZE": 100000,
"FLIP": true
}
},
"EVALUATORS": {
"KITTI3D": {
"IOU_THRESHOLDS": [
0.5,
0.7
],
"ONLY_PREPARE_SUBMISSION": false
}
},
"CKPT": "https://tri-ml-public.s3.amazonaws.com/github/dd3d/pretrained/depth_pretrained_omninet-small-3nxjur71.pth"
}
[03/18 08:06:01 tridet.data.datasets.kitti_3d]: KITTI-3D dataset(s): kitti_3d_train, kitti_3d_val
[03/18 08:06:08 tridet.data.build]: Creating D2 dataset: 15 batches on rank 0.
7%|▋ | 1/15 [00:02<00:28, 2.04s/it][2023-03-18 08:06:11,063][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:11,068][root][INFO] - Reducer buckets have been rebuilt in this iteration.
13%|█▎ | 2/15 [00:02<00:11, 1.10it/s][2023-03-18 08:06:11,115][root][INFO] - Reducer buckets have been rebuilt in this iteration.
100%|██████████| 15/15 [00:03<00:00, 4.94it/s][03/18 08:06:11 tridet.data.build]: Gathering D2 dataset dicts from all GPU workers...
100%|██████████| 15/15 [00:02<00:00, 5.65it/s][03/18 08:06:13 tridet.data.build]: Done (length=3712, took=1.7s).
WARNING [03/18 08:06:13 tridet.utils.coco]: Using previously cached COCO format annotations at '/tmp/kitti_3d_train_coco_format.json'. You need to clear the cache file if your dataset has been modified.
[03/18 08:06:13 tridet.data.datasets.kitti_3d.build]: COCO json file: /tmp/kitti_3d_train_coco_format.json
[03/18 08:06:20 tridet.data.build]: Creating D2 dataset: 15 batches on rank 0.
7%|▋ | 1/15 [00:01<00:27, 1.97s/it][2023-03-18 08:06:23,076][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,091][root][INFO] - Reducer buckets have been rebuilt in this iteration.
13%|█▎ | 2/15 [00:02<00:11, 1.13it/s][2023-03-18 08:06:23,136][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:23,136][root][INFO] - Reducer buckets have been rebuilt in this iteration.
100%|██████████| 15/15 [00:02<00:00, 5.69it/s][03/18 08:06:23 tridet.data.build]: Gathering D2 dataset dicts from all GPU workers...
100%|██████████| 15/15 [00:02<00:00, 5.67it/s][03/18 08:06:25 tridet.data.build]: Done (length=3769, took=1.5s).
WARNING [03/18 08:06:25 tridet.utils.coco]: Using previously cached COCO format annotations at '/tmp/kitti_3d_val_coco_format.json'. You need to clear the cache file if your dataset has been modified.
[03/18 08:06:25 tridet.data.datasets.kitti_3d.build]: COCO json file: /tmp/kitti_3d_val_coco_format.json
[03/18 08:06:25 tridet]: Registered 2 datasets:
kitti_3d_train
kitti_3d_val
[03/18 08:06:25 tridet.data.dataset_mappers.dataset_mapper]: Augmentations used in training: [ResizeShortestEdge(short_edge_length=[288, 304, 320, 336, 352, 368, 384, 400, 416, 448, 480, 512, 544, 576], max_size=10000, sample_style='choice'), RandomFlip(), RandomBrightness(intensity_min=0.8, intensity_max=1.2), RandomSaturation(intensity_min=0.8, intensity_max=1.2), RandomContrast(intensity_min=0.8, intensity_max=1.2)]
[03/18 08:06:25 d2.data.build]: Removed 0 images with no usable annotations. 3712 images left.
[03/18 08:06:26 d2.data.build]: Distribution of instances among all 5 categories:
2023-03-18 08:06:26,882][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 0
[2023-03-18 08:06:26,884][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 2
[2023-03-18 08:06:26,885][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 1
[2023-03-18 08:06:26,895][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 3
[2023-03-18 08:06:26,900][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 5
[2023-03-18 08:06:26,901][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 4
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 6
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:3 to store for rank: 7
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,911][torch.distributed.distributed_c10d][INFO] - Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,912][torch.distributed.distributed_c10d][INFO] - Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,912][torch.distributed.distributed_c10d][INFO] - Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,914][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,916][torch.distributed.distributed_c10d][INFO] - Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,916][torch.distributed.distributed_c10d][INFO] - Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[2023-03-18 08:06:26,916][torch.distributed.distributed_c10d][INFO] - Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
[03/18 08:06:26 tridet]: Length of train dataset: 3712
[03/18 08:06:26 tridet]: Starting training
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,107][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,214][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,214][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,214][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[2023-03-18 08:06:38,297][root][INFO] - Reducer buckets have been rebuilt in this iteration.
[03/18 08:06:53 d2.utils.events]: iter: 20 total_loss: 313.7 loss_box2d_reg: 0.9839 loss_box3d_depth: 8.246 loss_box3d_proj_ctr: 0.5431 loss_box3d_quat: 2.739 loss_box3d_size: 1.337 loss_centerness: 0.829 loss_cls: 297.5 loss_conf3d: 1.106 lr: 1.9198e-05 max_mem: 5630M
[03/18 08:07:11 d2.utils.events]: eta: 6:05:01 iter: 40 total_loss: 16.35 loss_box2d_reg: 0.9829 loss_box3d_depth: 7.286 loss_box3d_proj_ctr: 0.4852 loss_box3d_quat: 2.643 loss_box3d_size: 1.003 loss_centerness: 0.7814 loss_cls: 2.374 loss_conf3d: 0.6532 lr: 3.9196e-05 max_mem: 5630M
[03/18 08:07:29 d2.utils.events]: eta: 6:15:22 iter: 60 total_loss: 15.62 loss_box2d_reg: 0.9841 loss_box3d_depth: 7.074 loss_box3d_proj_ctr: 0.4394 loss_box3d_quat: 2.56 loss_box3d_size: 0.8469 loss_centerness: 0.7516 loss_cls: 2.424 loss_conf3d: 0.4763 lr: 5.9194e-05 max_mem: 5630M
[03/18 08:07:45 d2.utils.events]: eta: 5:43:40 iter: 80 total_loss: 14.58 loss_box2d_reg: 0.9868 loss_box3d_depth: 6.905 loss_box3d_proj_ctr: 0.4186 loss_box3d_quat: 2.341 loss_box3d_size: 0.6894 loss_centerness: 0.7354 loss_cls: 2.056 loss_conf3d: 0.434 lr: 7.9192e-05 max_mem: 5630M
[03/18 08:08:02 d2.utils.events]: eta: 5:47:47 iter: 100 total_loss: 13.93 loss_box2d_reg: 0.9874 loss_box3d_depth: 6.831 loss_box3d_proj_ctr: 0.4029 loss_box3d_quat: 2.229 loss_box3d_size: 0.5956 loss_centerness: 0.7245 loss_cls: 1.662 loss_conf3d: 0.412 lr: 9.919e-05 max_mem: 5630M
[03/18 08:08:18 d2.utils.events]: eta: 5:30:20 iter: 120 total_loss: 13.23 loss_box2d_reg: 0.9885 loss_box3d_depth: 6.772 loss_box3d_proj_ctr: 0.4011 loss_box3d_quat: 2.174 loss_box3d_size: 0.5176 loss_centerness: 0.7142 loss_cls: 1.31 loss_conf3d: 0.4019 lr: 0.00011919 max_mem: 5630M
[03/18 08:08:35 d2.utils.events]: eta: 5:53:07 iter: 140 total_loss: 12.69 loss_box2d_reg: 0.9886 loss_box3d_depth: 6.417 loss_box3d_proj_ctr: 0.3833 loss_box3d_quat: 2.155 loss_box3d_size: 0.4419 loss_centerness: 0.7073 loss_cls: 1.047 loss_conf3d: 0.4043 lr: 0.00013919 max_mem: 5630M
[03/18 08:08:52 d2.utils.events]: eta: 5:50:52 iter: 160 total_loss: 12.11 loss_box2d_reg: 0.9847 loss_box3d_depth: 6.238 loss_box3d_proj_ctr: 0.3848 loss_box3d_quat: 2.105 loss_box3d_size: 0.4187 loss_centerness: 0.7041 loss_cls: 0.8634 loss_conf3d: 0.4038 lr: 0.00015918 max_mem: 5630M
[03/18 08:09:08 d2.utils.events]: eta: 5:36:01 iter: 180 total_loss: 11.85 loss_box2d_reg: 0.9781 loss_box3d_depth: 6.133 loss_box3d_proj_ctr: 0.3975 loss_box3d_quat: 2.138 loss_box3d_size: 0.4007 loss_centerness: 0.7013 loss_cls: 0.8141 loss_conf3d: 0.4097 lr: 0.00017918 max_mem: 5630M
[03/18 08:09:25 d2.utils.events]: eta: 5:49:53 iter: 200 total_loss: 11.32 loss_box2d_reg: 0.9691 loss_box3d_depth: 5.581 loss_box3d_proj_ctr: 0.3858 loss_box3d_quat: 2.057 loss_box3d_size: 0.3819 loss_centerness: 0.6958 loss_cls: 0.7868 loss_conf3d: 0.4348 lr: 0.00019918 max_mem: 5630M
[03/18 08:09:42 d2.utils.events]: eta: 5:50:24 iter: 220 total_loss: 10.9 loss_box2d_reg: 0.9443 loss_box3d_depth: 5.266 loss_box3d_proj_ctr: 0.3807 loss_box3d_quat: 2.119 loss_box3d_size: 0.358 loss_centerness: 0.6976 loss_cls: 0.7606 loss_conf3d: 0.436 lr: 0.00021918 max_mem: 5631M
[03/18 08:09:59 d2.utils.events]: eta: 5:36:50 iter: 240 total_loss: 10.65 loss_box2d_reg: 0.8994 loss_box3d_depth: 5.054 loss_box3d_proj_ctr: 0.3808 loss_box3d_quat: 2.041 loss_box3d_size: 0.3253 loss_centerness: 0.6955 loss_cls: 0.7061 loss_conf3d: 0.4434 lr: 0.00023918 max_mem: 5631M
[03/18 08:10:16 d2.utils.events]: eta: 6:03:12 iter: 260 total_loss: 10.58 loss_box2d_reg: 0.8071 loss_box3d_depth: 5.113 loss_box3d_proj_ctr: 0.3784 loss_box3d_quat: 2.126 loss_box3d_size: 0.3223 loss_centerness: 0.6966 loss_cls: 0.6927 loss_conf3d: 0.4359 lr: 0.00025917 max_mem: 5631M
[03/18 08:10:34 d2.utils.events]: eta: 6:12:29 iter: 280 total_loss: 9.967 loss_box2d_reg: 0.7079 loss_box3d_depth: 4.762 loss_box3d_proj_ctr: 0.3807 loss_box3d_quat: 2.071 loss_box3d_size: 0.3053 loss_centerness: 0.697 loss_cls: 0.6697 loss_conf3d: 0.4572 lr: 0.00027917 max_mem: 5631M
[03/18 08:10:52 d2.utils.events]: eta: 6:00:41 iter: 300 total_loss: 9.587 loss_box2d_reg: 0.6188 loss_box3d_depth: 4.493 loss_box3d_proj_ctr: 0.3809 loss_box3d_quat: 2.019 loss_box3d_size: 0.2915 loss_centerness: 0.6956 loss_cls: 0.6371 loss_conf3d: 0.4666 lr: 0.00029917 max_mem: 5631M
[03/18 08:11:09 d2.utils.events]: eta: 6:02:27 iter: 320 total_loss: 9.218 loss_box2d_reg: 0.5775 loss_box3d_depth: 4.211 loss_box3d_proj_ctr: 0.3782 loss_box3d_quat: 2.049 loss_box3d_size: 0.2695 loss_centerness: 0.6879 loss_cls: 0.6051 loss_conf3d: 0.4795 lr: 0.00031917 max_mem: 5631M
[03/18 08:11:26 d2.utils.events]: eta: 5:46:55 iter: 340 total_loss: 9.233 loss_box2d_reg: 0.5667 loss_box3d_depth: 4.277 loss_box3d_proj_ctr: 0.3803 loss_box3d_quat: 1.974 loss_box3d_size: 0.2613 loss_centerness: 0.6883 loss_cls: 0.5957 loss_conf3d: 0.4734 lr: 0.00033917 max_mem: 5631M
[03/18 08:11:42 d2.utils.events]: eta: 5:31:08 iter: 360 total_loss: 8.959 loss_box2d_reg: 0.5531 loss_box3d_depth: 3.969 loss_box3d_proj_ctr: 0.377 loss_box3d_quat: 1.966 loss_box3d_size: 0.2622 loss_centerness: 0.683 loss_cls: 0.5763 loss_conf3d: 0.4865 lr: 0.00035916 max_mem: 5631M
[03/18 08:11:59 d2.utils.events]: eta: 5:48:45 iter: 380 total_loss: 8.931 loss_box2d_reg: 0.5438 loss_box3d_depth: 3.986 loss_box3d_proj_ctr: 0.3742 loss_box3d_quat: 2.071 loss_box3d_size: 0.2466 loss_centerness: 0.6847 loss_cls: 0.5473 loss_conf3d: 0.4756 lr: 0.00037916 max_mem: 5631M
[03/18 08:12:16 d2.utils.events]: eta: 5:44:59 iter: 400 total_loss: 8.547 loss_box2d_reg: 0.53 loss_box3d_depth: 3.688 loss_box3d_proj_ctr: 0.3782 loss_box3d_quat: 2.032 loss_box3d_size: 0.2267 loss_centerness: 0.6795 loss_cls: 0.5292 loss_conf3d: 0.4964 lr: 0.00039916 max_mem: 5631M
[03/18 08:12:33 d2.utils.events]: eta: 5:41:55 iter: 420 total_loss: 8.449 loss_box2d_reg: 0.5203 loss_box3d_depth: 3.59 loss_box3d_proj_ctr: 0.3694 loss_box3d_quat: 2.06 loss_box3d_size: 0.2328 loss_centerness: 0.6779 loss_cls: 0.5298 loss_conf3d: 0.5062 lr: 0.00041916 max_mem: 5631M
[03/18 08:12:50 d2.utils.events]: eta: 5:58:25 iter: 440 total_loss: 8.701 loss_box2d_reg: 0.5144 loss_box3d_depth: 3.845 loss_box3d_proj_ctr: 0.3692 loss_box3d_quat: 2.052 loss_box3d_size: 0.2305 loss_centerness: 0.6765 loss_cls: 0.5357 loss_conf3d: 0.496 lr: 0.00043916 max_mem: 5631M
[03/18 08:13:08 d2.utils.events]: eta: 6:03:39 iter: 460 total_loss: 8.406 loss_box2d_reg: 0.5114 loss_box3d_depth: 3.646 loss_box3d_proj_ctr: 0.3705 loss_box3d_quat: 2.024 loss_box3d_size: 0.2223 loss_centerness: 0.6749 loss_cls: 0.5159 loss_conf3d: 0.5056 lr: 0.00045915 max_mem: 5631M
[03/18 08:13:25 d2.utils.events]: eta: 5:35:23 iter: 480 total_loss: 8.059 loss_box2d_reg: 0.495 loss_box3d_depth: 3.274 loss_box3d_proj_ctr: 0.3645 loss_box3d_quat: 1.989 loss_box3d_size: 0.2091 loss_centerness: 0.673 loss_cls: 0.4985 loss_conf3d: 0.5236 lr: 0.00047915 max_mem: 5631M
[03/18 08:13:43 d2.utils.events]: eta: 6:09:42 iter: 500 total_loss: 8.048 loss_box2d_reg: 0.4968 loss_box3d_depth: 3.301 loss_box3d_proj_ctr: 0.3631 loss_box3d_quat: 2.007 loss_box3d_size: 0.2059 loss_centerness: 0.6718 loss_cls: 0.4829 loss_conf3d: 0.5226 lr: 0.00049915 max_mem: 5631M
[03/18 08:14:02 d2.utils.events]: eta: 6:22:45 iter: 520 total_loss: 7.996 loss_box2d_reg: 0.4874 loss_box3d_depth: 3.196 loss_box3d_proj_ctr: 0.3659 loss_box3d_quat: 2.018 loss_box3d_size: 0.2089 loss_centerness: 0.6704 loss_cls: 0.482 loss_conf3d: 0.5192 lr: 0.00051915 max_mem: 5631M
[03/18 08:14:19 d2.utils.events]: eta: 5:52:08 iter: 540 total_loss: 8.394 loss_box2d_reg: 0.4856 loss_box3d_depth: 3.628 loss_box3d_proj_ctr: 0.3581 loss_box3d_quat: 2.013 loss_box3d_size: 0.2012 loss_centerness: 0.6691 loss_cls: 0.4796 loss_conf3d: 0.5054 lr: 0.00053915 max_mem: 5631M
[03/18 08:14:36 d2.utils.events]: eta: 5:56:55 iter: 560 total_loss: 7.878 loss_box2d_reg: 0.4753 loss_box3d_depth: 3.151 loss_box3d_proj_ctr: 0.3652 loss_box3d_quat: 2.065 loss_box3d_size: 0.1987 loss_centerness: 0.6684 loss_cls: 0.4594 loss_conf3d: 0.5235 lr: 0.00055914 max_mem: 5631M
[03/18 08:14:54 d2.utils.events]: eta: 5:59:40 iter: 580 total_loss: 7.752 loss_box2d_reg: 0.4706 loss_box3d_depth: 3.02 loss_box3d_proj_ctr: 0.3588 loss_box3d_quat: 2.003 loss_box3d_size: 0.1921 loss_centerness: 0.666 loss_cls: 0.4479 loss_conf3d: 0.5276 lr: 0.00057914 max_mem: 5631M
[03/18 08:15:11 d2.utils.events]: eta: 5:50:34 iter: 600 total_loss: 7.661 loss_box2d_reg: 0.4639 loss_box3d_depth: 2.968 loss_box3d_proj_ctr: 0.3508 loss_box3d_quat: 1.98 loss_box3d_size: 0.198 loss_centerness: 0.6653 loss_cls: 0.4515 loss_conf3d: 0.5431 lr: 0.00059914 max_mem: 5631M
[03/18 08:15:29 d2.utils.events]: eta: 6:02:12 iter: 620 total_loss: 7.837 loss_box2d_reg: 0.4569 loss_box3d_depth: 3.175 loss_box3d_proj_ctr: 0.3568 loss_box3d_quat: 1.978 loss_box3d_size: 0.1946 loss_centerness: 0.6636 loss_cls: 0.4433 loss_conf3d: 0.521 lr: 0.00061914 max_mem: 5631M
[03/18 08:15:47 d2.utils.events]: eta: 6:07:58 iter: 640 total_loss: 7.4 loss_box2d_reg: 0.4518 loss_box3d_depth: 2.776 loss_box3d_proj_ctr: 0.3484 loss_box3d_quat: 2.027 loss_box3d_size: 0.1891 loss_centerness: 0.6632 loss_cls: 0.4388 loss_conf3d: 0.5476 lr: 0.00063914 max_mem: 5631M
[03/18 08:16:05 d2.utils.events]: eta: 6:00:15 iter: 660 total_loss: 7.444 loss_box2d_reg: 0.4427 loss_box3d_depth: 2.746 loss_box3d_proj_ctr: 0.3471 loss_box3d_quat: 2.013 loss_box3d_size: 0.1857 loss_centerness: 0.6616 loss_cls: 0.4334 loss_conf3d: 0.5383 lr: 0.00065913 max_mem: 5631M
[03/18 08:16:23 d2.utils.events]: eta: 6:05:13 iter: 680 total_loss: 7.34 loss_box2d_reg: 0.4486 loss_box3d_depth: 2.729 loss_box3d_proj_ctr: 0.3551 loss_box3d_quat: 2.051 loss_box3d_size: 0.1861 loss_centerness: 0.6625 loss_cls: 0.4228 loss_conf3d: 0.5446 lr: 0.00067913 max_mem: 5631M
[03/18 08:16:41 d2.utils.events]: eta: 6:12:37 iter: 700 total_loss: 7.324 loss_box2d_reg: 0.4437 loss_box3d_depth: 2.778 loss_box3d_proj_ctr: 0.3369 loss_box3d_quat: 1.895 loss_box3d_size: 0.1822 loss_centerness: 0.6604 loss_cls: 0.421 loss_conf3d: 0.5549 lr: 0.00069913 max_mem: 5631M
[03/18 08:16:59 d2.utils.events]: eta: 5:53:25 iter: 720 total_loss: 7.247 loss_box2d_reg: 0.4349 loss_box3d_depth: 2.665 loss_box3d_proj_ctr: 0.3395 loss_box3d_quat: 2.071 loss_box3d_size: 0.1832 loss_centerness: 0.66 loss_cls: 0.4182 loss_conf3d: 0.5483 lr: 0.00071913 max_mem: 5631M
[03/18 08:17:18 d2.utils.events]: eta: 6:25:09 iter: 740 total_loss: 7.225 loss_box2d_reg: 0.427 loss_box3d_depth: 2.573 loss_box3d_proj_ctr: 0.3331 loss_box3d_quat: 2.058 loss_box3d_size: 0.1827 loss_centerness: 0.6577 loss_cls: 0.4156 loss_conf3d: 0.5472 lr: 0.00073913 max_mem: 5631M
[03/18 08:17:37 d2.utils.events]: eta: 6:18:08 iter: 760 total_loss: 7.479 loss_box2d_reg: 0.4307 loss_box3d_depth: 2.97 loss_box3d_proj_ctr: 0.3494 loss_box3d_quat: 2.002 loss_box3d_size: 0.1835 loss_centerness: 0.6579 loss_cls: 0.4124 loss_conf3d: 0.539 lr: 0.00075912 max_mem: 5631M
[03/18 08:17:55 d2.utils.events]: eta: 6:13:52 iter: 780 total_loss: 7.41 loss_box2d_reg: 0.4249 loss_box3d_depth: 2.805 loss_box3d_proj_ctr: 0.3338 loss_box3d_quat: 2.003 loss_box3d_size: 0.1762 loss_centerness: 0.6576 loss_cls: 0.4098 loss_conf3d: 0.5387 lr: 0.00077912 max_mem: 5631M
[03/18 08:18:14 d2.utils.events]: eta: 6:14:02 iter: 800 total_loss: 7.089 loss_box2d_reg: 0.4275 loss_box3d_depth: 2.618 loss_box3d_proj_ctr: 0.3316 loss_box3d_quat: 2.004 loss_box3d_size: 0.1814 loss_centerness: 0.6576 loss_cls: 0.4038 loss_conf3d: 0.5579 lr: 0.00079912 max_mem: 5632M
[03/18 08:18:32 d2.utils.events]: eta: 6:06:57 iter: 820 total_loss: 7.155 loss_box2d_reg: 0.4229 loss_box3d_depth: 2.65 loss_box3d_proj_ctr: 0.3232 loss_box3d_quat: 2.034 loss_box3d_size: 0.1791 loss_centerness: 0.6558 loss_cls: 0.4 loss_conf3d: 0.5496 lr: 0.00081912 max_mem: 5632M
[03/18 08:18:49 d2.utils.events]: eta: 5:45:03 iter: 840 total_loss: 6.939 loss_box2d_reg: 0.4146 loss_box3d_depth: 2.444 loss_box3d_proj_ctr: 0.3215 loss_box3d_quat: 2.012 loss_box3d_size: 0.1754 loss_centerness: 0.6562 loss_cls: 0.3967 loss_conf3d: 0.5615 lr: 0.00083912 max_mem: 5632M
[03/18 08:19:07 d2.utils.events]: eta: 6:01:24 iter: 860 total_loss: 7.042 loss_box2d_reg: 0.4196 loss_box3d_depth: 2.52 loss_box3d_proj_ctr: 0.3196 loss_box3d_quat: 1.966 loss_box3d_size: 0.1746 loss_centerness: 0.6566 loss_cls: 0.3916 loss_conf3d: 0.5585 lr: 0.00085911 max_mem: 5632M
[03/18 08:19:25 d2.utils.events]: eta: 6:08:57 iter: 880 total_loss: 7.178 loss_box2d_reg: 0.4112 loss_box3d_depth: 2.596 loss_box3d_proj_ctr: 0.3241 loss_box3d_quat: 1.964 loss_box3d_size: 0.1747 loss_centerness: 0.6557 loss_cls: 0.389 loss_conf3d: 0.5442 lr: 0.00087911 max_mem: 5632M
[03/18 08:19:43 d2.utils.events]: eta: 5:47:11 iter: 900 total_loss: 6.987 loss_box2d_reg: 0.4133 loss_box3d_depth: 2.505 loss_box3d_proj_ctr: 0.3177 loss_box3d_quat: 1.967 loss_box3d_size: 0.165 loss_centerness: 0.6551 loss_cls: 0.3895 loss_conf3d: 0.5635 lr: 0.00089911 max_mem: 5632M
[03/18 08:20:01 d2.utils.events]: eta: 6:12:12 iter: 920 total_loss: 6.741 loss_box2d_reg: 0.399 loss_box3d_depth: 2.223 loss_box3d_proj_ctr: 0.3157 loss_box3d_quat: 2.062 loss_box3d_size: 0.1692 loss_centerness: 0.6537 loss_cls: 0.3846 loss_conf3d: 0.572 lr: 0.00091911 max_mem: 5632M
[03/18 08:20:19 d2.utils.events]: eta: 6:05:16 iter: 940 total_loss: 6.783 loss_box2d_reg: 0.4011 loss_box3d_depth: 2.258 loss_box3d_proj_ctr: 0.315 loss_box3d_quat: 1.948 loss_box3d_size: 0.1725 loss_centerness: 0.6535 loss_cls: 0.3873 loss_conf3d: 0.5749 lr: 0.00093911 max_mem: 5632M
[03/18 08:20:37 d2.utils.events]: eta: 5:46:51 iter: 960 total_loss: 6.665 loss_box2d_reg: 0.3935 loss_box3d_depth: 2.254 loss_box3d_proj_ctr: 0.3167 loss_box3d_quat: 1.942 loss_box3d_size: 0.1703 loss_centerness: 0.6518 loss_cls: 0.3809 loss_conf3d: 0.5796 lr: 0.0009591 max_mem: 5632M
[03/18 08:20:56 d2.utils.events]: eta: 6:30:48 iter: 980 total_loss: 6.87 loss_box2d_reg: 0.3943 loss_box3d_depth: 2.313 loss_box3d_proj_ctr: 0.3179 loss_box3d_quat: 1.956 loss_box3d_size: 0.1642 loss_centerness: 0.6527 loss_cls: 0.38 loss_conf3d: 0.5673 lr: 0.0009791 max_mem: 5632M
[03/18 08:21:15 d2.utils.events]: eta: 6:18:15 iter: 1000 total_loss: 6.81 loss_box2d_reg: 0.3928 loss_box3d_depth: 2.418 loss_box3d_proj_ctr: 0.3128 loss_box3d_quat: 1.942 loss_box3d_size: 0.1678 loss_centerness: 0.6514 loss_cls: 0.3743 loss_conf3d: 0.5641 lr: 0.0009991 max_mem: 5632M
[03/18 08:21:38 d2.utils.events]: eta: 7:39:37 iter: 1020 total_loss: 6.905 loss_box2d_reg: 0.3801 loss_box3d_depth: 2.482 loss_box3d_proj_ctr: 0.3116 loss_box3d_quat: 1.942 loss_box3d_size: 0.1705 loss_centerness: 0.6507 loss_cls: 0.3672 loss_conf3d: 0.5583 lr: 0.0010191 max_mem: 5632M
[03/18 08:21:57 d2.utils.events]: eta: 6:12:08 iter: 1040 total_loss: 6.639 loss_box2d_reg: 0.3866 loss_box3d_depth: 2.269 loss_box3d_proj_ctr: 0.3061 loss_box3d_quat: 1.967 loss_box3d_size: 0.164 loss_centerness: 0.6509 loss_cls: 0.3658 loss_conf3d: 0.5779 lr: 0.0010391 max_mem: 5632M
[03/18 08:22:15 d2.utils.events]: eta: 6:11:48 iter: 1060 total_loss: 6.67 loss_box2d_reg: 0.3877 loss_box3d_depth: 2.281 loss_box3d_proj_ctr: 0.305 loss_box3d_quat: 1.972 loss_box3d_size: 0.1642 loss_centerness: 0.651 loss_cls: 0.3669 loss_conf3d: 0.5748 lr: 0.0010591 max_mem: 5632M
[03/18 08:22:33 d2.utils.events]: eta: 5:52:40 iter: 1080 total_loss: 6.735 loss_box2d_reg: 0.3809 loss_box3d_depth: 2.261 loss_box3d_proj_ctr: 0.3159 loss_box3d_quat: 1.955 loss_box3d_size: 0.1634 loss_centerness: 0.6518 loss_cls: 0.3699 loss_conf3d: 0.5751 lr: 0.0010791 max_mem: 5632M
[03/18 08:22:51 d2.utils.events]: eta: 5:57:27 iter: 1100 total_loss: 6.578 loss_box2d_reg: 0.3794 loss_box3d_depth: 2.198 loss_box3d_proj_ctr: 0.306 loss_box3d_quat: 1.953 loss_box3d_size: 0.1629 loss_centerness: 0.6504 loss_cls: 0.3662 loss_conf3d: 0.5751 lr: 0.0010991 max_mem: 5632M
[03/18 08:23:09 d2.utils.events]: eta: 5:59:44 iter: 1120 total_loss: 6.641 loss_box2d_reg: 0.3826 loss_box3d_depth: 2.281 loss_box3d_proj_ctr: 0.3073 loss_box3d_quat: 1.879 loss_box3d_size: 0.156 loss_centerness: 0.6519 loss_cls: 0.3631 loss_conf3d: 0.5691 lr: 0.0011191 max_mem: 5632M
[03/18 08:23:27 d2.utils.events]: eta: 6:01:38 iter: 1140 total_loss: 6.661 loss_box2d_reg: 0.3768 loss_box3d_depth: 2.177 loss_box3d_proj_ctr: 0.3096 loss_box3d_quat: 1.997 loss_box3d_size: 0.1643 loss_centerness: 0.6498 loss_cls: 0.3563 loss_conf3d: 0.5694 lr: 0.0011391 max_mem: 5632M
[03/18 08:23:46 d2.utils.events]: eta: 6:10:59 iter: 1160 total_loss: 6.691 loss_box2d_reg: 0.3709 loss_box3d_depth: 2.344 loss_box3d_proj_ctr: 0.3047 loss_box3d_quat: 1.934 loss_box3d_size: 0.1581 loss_centerness: 0.6487 loss_cls: 0.3619 loss_conf3d: 0.5679 lr: 0.0011591 max_mem: 5632M
[03/18 08:24:04 d2.utils.events]: eta: 6:04:55 iter: 1180 total_loss: 7.015 loss_box2d_reg: 0.3737 loss_box3d_depth: 2.619 loss_box3d_proj_ctr: 0.3101 loss_box3d_quat: 1.944 loss_box3d_size: 0.1604 loss_centerness: 0.6485 loss_cls: 0.3585 loss_conf3d: 0.5463 lr: 0.0011791 max_mem: 5632M
[03/18 08:24:23 d2.utils.events]: eta: 6:05:43 iter: 1200 total_loss: 6.973 loss_box2d_reg: 0.3769 loss_box3d_depth: 2.603 loss_box3d_proj_ctr: 0.3047 loss_box3d_quat: 1.943 loss_box3d_size: 0.1655 loss_centerness: 0.6503 loss_cls: 0.3687 loss_conf3d: 0.5576 lr: 0.0011991 max_mem: 5632M
[03/18 08:24:42 d2.utils.events]: eta: 6:18:14 iter: 1220 total_loss: 6.539 loss_box2d_reg: 0.375 loss_box3d_depth: 2.191 loss_box3d_proj_ctr: 0.3045 loss_box3d_quat: 1.939 loss_box3d_size: 0.1591 loss_centerness: 0.651 loss_cls: 0.3683 loss_conf3d: 0.5825 lr: 0.0012191 max_mem: 5632M
[03/18 08:25:02 d2.utils.events]: eta: 6:28:30 iter: 1240 total_loss: 6.376 loss_box2d_reg: 0.3713 loss_box3d_depth: 2.121 loss_box3d_proj_ctr: 0.2918 loss_box3d_quat: 1.792 loss_box3d_size: 0.1579 loss_centerness: 0.6496 loss_cls: 0.3558 loss_conf3d: 0.6053 lr: 0.0012391 max_mem: 5632M
[03/18 08:25:20 d2.utils.events]: eta: 6:05:09 iter: 1260 total_loss: 6.324 loss_box2d_reg: 0.3733 loss_box3d_depth: 2.279 loss_box3d_proj_ctr: 0.3035 loss_box3d_quat: 1.608 loss_box3d_size: 0.1526 loss_centerness: 0.6486 loss_cls: 0.3618 loss_conf3d: 0.6009 lr: 0.0012591 max_mem: 5632M
[03/18 08:25:39 d2.utils.events]: eta: 6:15:00 iter: 1280 total_loss: 6.361 loss_box2d_reg: 0.3683 loss_box3d_depth: 2.268 loss_box3d_proj_ctr: 0.31 loss_box3d_quat: 1.615 loss_box3d_size: 0.1639 loss_centerness: 0.6491 loss_cls: 0.3582 loss_conf3d: 0.6014 lr: 0.0012791 max_mem: 5632M
[03/18 08:25:58 d2.utils.events]: eta: 6:07:48 iter: 1300 total_loss: 6.099 loss_box2d_reg: 0.3677 loss_box3d_depth: 2.089 loss_box3d_proj_ctr: 0.3075 loss_box3d_quat: 1.525 loss_box3d_size: 0.165 loss_centerness: 0.6499 loss_cls: 0.3592 loss_conf3d: 0.6165 lr: 0.0012991 max_mem: 5632M
[03/18 08:26:16 d2.utils.events]: eta: 5:54:22 iter: 1320 total_loss: 6.157 loss_box2d_reg: 0.3632 loss_box3d_depth: 2.209 loss_box3d_proj_ctr: 0.3017 loss_box3d_quat: 1.495 loss_box3d_size: 0.1633 loss_centerness: 0.6475 loss_cls: 0.3518 loss_conf3d: 0.614 lr: 0.0013191 max_mem: 5632M
[03/18 08:26:35 d2.utils.events]: eta: 6:20:50 iter: 1340 total_loss: 6.085 loss_box2d_reg: 0.3638 loss_box3d_depth: 2.133 loss_box3d_proj_ctr: 0.3024 loss_box3d_quat: 1.456 loss_box3d_size: 0.1624 loss_centerness: 0.6482 loss_cls: 0.3557 loss_conf3d: 0.6216 lr: 0.0013391 max_mem: 5632M
[03/18 08:26:54 d2.utils.events]: eta: 6:10:26 iter: 1360 total_loss: 6.072 loss_box2d_reg: 0.368 loss_box3d_depth: 2.166 loss_box3d_proj_ctr: 0.3054 loss_box3d_quat: 1.49 loss_box3d_size: 0.1599 loss_centerness: 0.6474 loss_cls: 0.3537 loss_conf3d: 0.6147 lr: 0.0013591 max_mem: 5632M
[03/18 08:27:12 d2.utils.events]: eta: 6:05:03 iter: 1380 total_loss: 5.727 loss_box2d_reg: 0.3611 loss_box3d_depth: 1.988 loss_box3d_proj_ctr: 0.3012 loss_box3d_quat: 1.322 loss_box3d_size: 0.156 loss_centerness: 0.6484 loss_cls: 0.3463 loss_conf3d: 0.6342 lr: 0.0013791 max_mem: 5632M
[03/18 08:27:31 d2.utils.events]: eta: 6:09:04 iter: 1400 total_loss: 5.808 loss_box2d_reg: 0.3535 loss_box3d_depth: 2.093 loss_box3d_proj_ctr: 0.3004 loss_box3d_quat: 1.284 loss_box3d_size: 0.1502 loss_centerness: 0.6469 loss_cls: 0.3494 loss_conf3d: 0.6318 lr: 0.0013991 max_mem: 5632M
[03/18 08:27:50 d2.utils.events]: eta: 6:12:05 iter: 1420 total_loss: 5.784 loss_box2d_reg: 0.3496 loss_box3d_depth: 2.005 loss_box3d_proj_ctr: 0.2989 loss_box3d_quat: 1.343 loss_box3d_size: 0.1568 loss_centerness: 0.6455 loss_cls: 0.3433 loss_conf3d: 0.6302 lr: 0.0014191 max_mem: 5632M
[03/18 08:28:09 d2.utils.events]: eta: 6:04:13 iter: 1440 total_loss: 5.8 loss_box2d_reg: 0.3531 loss_box3d_depth: 2.089 loss_box3d_proj_ctr: 0.2929 loss_box3d_quat: 1.318 loss_box3d_size: 0.1598 loss_centerness: 0.6463 loss_cls: 0.3447 loss_conf3d: 0.6293 lr: 0.0014391 max_mem: 5632M
[03/18 08:28:28 d2.utils.events]: eta: 6:29:27 iter: 1460 total_loss: 5.891 loss_box2d_reg: 0.3608 loss_box3d_depth: 2.124 loss_box3d_proj_ctr: 0.292 loss_box3d_quat: 1.325 loss_box3d_size: 0.1578 loss_centerness: 0.6478 loss_cls: 0.3528 loss_conf3d: 0.6313 lr: 0.0014591 max_mem: 5632M
[03/18 08:28:48 d2.utils.events]: eta: 6:21:46 iter: 1480 total_loss: 5.804 loss_box2d_reg: 0.3537 loss_box3d_depth: 2.145 loss_box3d_proj_ctr: 0.3021 loss_box3d_quat: 1.24 loss_box3d_size: 0.1523 loss_centerness: 0.6464 loss_cls: 0.3407 loss_conf3d: 0.6331 lr: 0.0014791 max_mem: 5632M
[03/18 08:29:07 d2.utils.events]: eta: 6:08:08 iter: 1500 total_loss: 6.109 loss_box2d_reg: 0.3569 loss_box3d_depth: 2.405 loss_box3d_proj_ctr: 0.301 loss_box3d_quat: 1.284 loss_box3d_size: 0.1531 loss_centerness: 0.6459 loss_cls: 0.3489 loss_conf3d: 0.6149 lr: 0.0014991 max_mem: 5632M
[03/18 08:29:25 d2.utils.events]: eta: 6:05:07 iter: 1520 total_loss: 5.78 loss_box2d_reg: 0.3539 loss_box3d_depth: 2.094 loss_box3d_proj_ctr: 0.2956 loss_box3d_quat: 1.225 loss_box3d_size: 0.1588 loss_centerness: 0.6471 loss_cls: 0.3409 loss_conf3d: 0.6292 lr: 0.001519 max_mem: 5632M
[03/18 08:29:44 d2.utils.events]: eta: 5:57:01 iter: 1540 total_loss: 5.629 loss_box2d_reg: 0.3464 loss_box3d_depth: 2.034 loss_box3d_proj_ctr: 0.2877 loss_box3d_quat: 1.189 loss_box3d_size: 0.1499 loss_centerness: 0.6435 loss_cls: 0.3288 loss_conf3d: 0.6345 lr: 0.001539 max_mem: 5632M
[03/18 08:30:01 d2.utils.events]: eta: 5:48:09 iter: 1560 total_loss: 5.533 loss_box2d_reg: 0.3415 loss_box3d_depth: 1.951 loss_box3d_proj_ctr: 0.2918 loss_box3d_quat: 1.182 loss_box3d_size: 0.1534 loss_centerness: 0.6448 loss_cls: 0.3316 loss_conf3d: 0.637 lr: 0.001559 max_mem: 5632M
[03/18 08:30:21 d2.utils.events]: eta: 6:20:16 iter: 1580 total_loss: 6.106 loss_box2d_reg: 0.351 loss_box3d_depth: 2.564 loss_box3d_proj_ctr: 0.2969 loss_box3d_quat: 1.215 loss_box3d_size: 0.1525 loss_centerness: 0.6448 loss_cls: 0.3347 loss_conf3d: 0.6149 lr: 0.001579 max_mem: 5632M
[03/18 08:30:39 d2.utils.events]: eta: 6:00:58 iter: 1600 total_loss: 5.755 loss_box2d_reg: 0.3522 loss_box3d_depth: 2.135 loss_box3d_proj_ctr: 0.2831 loss_box3d_quat: 1.191 loss_box3d_size: 0.1524 loss_centerness: 0.646 loss_cls: 0.3369 loss_conf3d: 0.6341 lr: 0.001599 max_mem: 5632M
[03/18 08:30:57 d2.utils.events]: eta: 5:52:18 iter: 1620 total_loss: 5.558 loss_box2d_reg: 0.3479 loss_box3d_depth: 2.041 loss_box3d_proj_ctr: 0.2937 loss_box3d_quat: 1.187 loss_box3d_size: 0.1515 loss_centerness: 0.6454 loss_cls: 0.3347 loss_conf3d: 0.6389 lr: 0.001619 max_mem: 5632M
[03/18 08:31:16 d2.utils.events]: eta: 5:58:20 iter: 1640 total_loss: 5.535 loss_box2d_reg: 0.3432 loss_box3d_depth: 1.971 loss_box3d_proj_ctr: 0.2949 loss_box3d_quat: 1.18 loss_box3d_size: 0.1521 loss_centerness: 0.6447 loss_cls: 0.3311 loss_conf3d: 0.6382 lr: 0.001639 max_mem: 5632M
[03/18 08:31:35 d2.utils.events]: eta: 6:13:50 iter: 1660 total_loss: 5.641 loss_box2d_reg: 0.3409 loss_box3d_depth: 2.094 loss_box3d_proj_ctr: 0.2829 loss_box3d_quat: 1.119 loss_box3d_size: 0.1496 loss_centerness: 0.6433 loss_cls: 0.3285 loss_conf3d: 0.6339 lr: 0.001659 max_mem: 5632M
[03/18 08:31:53 d2.utils.events]: eta: 5:48:47 iter: 1680 total_loss: 5.577 loss_box2d_reg: 0.3472 loss_box3d_depth: 2.003 loss_box3d_proj_ctr: 0.2884 loss_box3d_quat: 1.109 loss_box3d_size: 0.1493 loss_centerness: 0.6456 loss_cls: 0.3313 loss_conf3d: 0.6348 lr: 0.001679 max_mem: 5632M
[03/18 08:32:13 d2.utils.events]: eta: 6:26:15 iter: 1700 total_loss: 5.497 loss_box2d_reg: 0.3394 loss_box3d_depth: 1.937 loss_box3d_proj_ctr: 0.2919 loss_box3d_quat: 1.165 loss_box3d_size: 0.1506 loss_centerness: 0.6444 loss_cls: 0.3255 loss_conf3d: 0.6378 lr: 0.001699 max_mem: 5632M
[03/18 08:32:33 d2.utils.events]: eta: 6:22:42 iter: 1720 total_loss: 5.457 loss_box2d_reg: 0.3388 loss_box3d_depth: 1.955 loss_box3d_proj_ctr: 0.2799 loss_box3d_quat: 1.086 loss_box3d_size: 0.1483 loss_centerness: 0.644 loss_cls: 0.3155 loss_conf3d: 0.6394 lr: 0.001719 max_mem: 5632M
[03/18 08:32:51 d2.utils.events]: eta: 5:59:36 iter: 1740 total_loss: 5.458 loss_box2d_reg: 0.3355 loss_box3d_depth: 1.962 loss_box3d_proj_ctr: 0.285 loss_box3d_quat: 1.095 loss_box3d_size: 0.1466 loss_centerness: 0.6434 loss_cls: 0.3263 loss_conf3d: 0.6384 lr: 0.001739 max_mem: 5632M
[03/18 08:33:10 d2.utils.events]: eta: 6:11:12 iter: 1760 total_loss: 5.344 loss_box2d_reg: 0.3334 loss_box3d_depth: 1.909 loss_box3d_proj_ctr: 0.2814 loss_box3d_quat: 1.049 loss_box3d_size: 0.1418 loss_centerness: 0.6441 loss_cls: 0.3185 loss_conf3d: 0.6423 lr: 0.001759 max_mem: 5632M
[03/18 08:33:29 d2.utils.events]: eta: 6:07:56 iter: 1780 total_loss: 5.221 loss_box2d_reg: 0.3335 loss_box3d_depth: 1.867 loss_box3d_proj_ctr: 0.2846 loss_box3d_quat: 1.031 loss_box3d_size: 0.1468 loss_centerness: 0.6424 loss_cls: 0.3144 loss_conf3d: 0.6463 lr: 0.001779 max_mem: 5632M
[03/18 08:33:47 d2.utils.events]: eta: 5:46:48 iter: 1800 total_loss: 5.386 loss_box2d_reg: 0.3327 loss_box3d_depth: 1.915 loss_box3d_proj_ctr: 0.2877 loss_box3d_quat: 1.108 loss_box3d_size: 0.1585 loss_centerness: 0.6427 loss_cls: 0.3144 loss_conf3d: 0.6437 lr: 0.001799 max_mem: 5632M
[03/18 08:34:06 d2.utils.events]: eta: 5:50:54 iter: 1820 total_loss: 5.412 loss_box2d_reg: 0.3317 loss_box3d_depth: 2.006 loss_box3d_proj_ctr: 0.2895 loss_box3d_quat: 1.022 loss_box3d_size: 0.1489 loss_centerness: 0.6423 loss_cls: 0.3128 loss_conf3d: 0.6448 lr: 0.001819 max_mem: 5632M
[03/18 08:34:24 d2.utils.events]: eta: 6:02:58 iter: 1840 total_loss: 5.868 loss_box2d_reg: 0.3389 loss_box3d_depth: 2.383 loss_box3d_proj_ctr: 0.2912 loss_box3d_quat: 1.055 loss_box3d_size: 0.1456 loss_centerness: 0.6443 loss_cls: 0.3225 loss_conf3d: 0.6344 lr: 0.001839 max_mem: 5632M
[03/18 08:34:42 d2.utils.events]: eta: 5:39:11 iter: 1860 total_loss: 5.477 loss_box2d_reg: 0.3404 loss_box3d_depth: 2.04 loss_box3d_proj_ctr: 0.2862 loss_box3d_quat: 1.025 loss_box3d_size: 0.148 loss_centerness: 0.6442 loss_cls: 0.3221 loss_conf3d: 0.6452 lr: 0.001859 max_mem: 5632M
[03/18 08:35:00 d2.utils.events]: eta: 5:49:06 iter: 1880 total_loss: 5.558 loss_box2d_reg: 0.3291 loss_box3d_depth: 2.17 loss_box3d_proj_ctr: 0.2895 loss_box3d_quat: 0.989 loss_box3d_size: 0.1437 loss_centerness: 0.6424 loss_cls: 0.3169 loss_conf3d: 0.638 lr: 0.001879 max_mem: 5632M
[03/18 08:35:19 d2.utils.events]: eta: 6:08:45 iter: 1900 total_loss: 5.253 loss_box2d_reg: 0.3289 loss_box3d_depth: 1.898 loss_box3d_proj_ctr: 0.2903 loss_box3d_quat: 0.9442 loss_box3d_size: 0.1506 loss_centerness: 0.6417 loss_cls: 0.3157 loss_conf3d: 0.6479 lr: 0.001899 max_mem: 5632M
[03/18 08:35:37 d2.utils.events]: eta: 5:41:41 iter: 1920 total_loss: 5.343 loss_box2d_reg: 0.3279 loss_box3d_depth: 1.974 loss_box3d_proj_ctr: 0.2896 loss_box3d_quat: 0.9662 loss_box3d_size: 0.1469 loss_centerness: 0.6409 loss_cls: 0.313 loss_conf3d: 0.6459 lr: 0.001919 max_mem: 5632M
[03/18 08:35:56 d2.utils.events]: eta: 6:06:53 iter: 1940 total_loss: 5.245 loss_box2d_reg: 0.3287 loss_box3d_depth: 1.981 loss_box3d_proj_ctr: 0.2698 loss_box3d_quat: 0.9367 loss_box3d_size: 0.1429 loss_centerness: 0.6427 loss_cls: 0.3073 loss_conf3d: 0.6487 lr: 0.001939 max_mem: 5632M
[03/18 08:36:16 d2.utils.events]: eta: 6:22:55 iter: 1960 total_loss: 5.244 loss_box2d_reg: 0.3229 loss_box3d_depth: 1.9 loss_box3d_proj_ctr: 0.2822 loss_box3d_quat: 1.009 loss_box3d_size: 0.1435 loss_centerness: 0.6422 loss_cls: 0.3102 loss_conf3d: 0.6472 lr: 0.001959 max_mem: 5632M
[03/18 08:36:34 d2.utils.events]: eta: 5:43:01 iter: 1980 total_loss: 5.142 loss_box2d_reg: 0.3235 loss_box3d_depth: 1.81 loss_box3d_proj_ctr: 0.2837 loss_box3d_quat: 0.945 loss_box3d_size: 0.1464 loss_centerness: 0.6411 loss_cls: 0.3118 loss_conf3d: 0.6522 lr: 0.001979 max_mem: 5632M
[03/18 08:36:52 d2.utils.events]: eta: 5:50:43 iter: 2000 total_loss: 5.204 loss_box2d_reg: 0.3229 loss_box3d_depth: 1.922 loss_box3d_proj_ctr: 0.2814 loss_box3d_quat: 0.9451 loss_box3d_size: 0.1452 loss_centerness: 0.6421 loss_cls: 0.3056 loss_conf3d: 0.6506 lr: 0.001999 max_mem: 5632M
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
[03/18 08:37:00 fvcore.common.checkpoint]: Saving checkpoint to ./model_0001999.pth
[03/18 08:37:01 tridet.data.dataset_mappers.dataset_mapper]: Augmentations used in training: [ResizeShortestEdge(short_edge_length=(384, 384), max_size=100000, sample_style='choice')]
[03/18 08:37:01 d2.data.common]: Serializing 3769 elements to byte tensors and concatenating them all ...
[03/18 08:37:01 d2.data.common]: Serialized dataset takes 13.38 MiB
[03/18 08:37:01 tridet.data.build]: Using test sampler InferenceSampler
[03/18 08:37:01 d2.evaluation.evaluator]: Start inference on 48 batches
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/azuryl/anaconda3/envs/dd3d/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of
lr_scheduler.step()
beforeoptimizer.step()
. In PyTorch 1.1.0 and later, you should call them in the opposite order:optimizer.step()
beforelr_scheduler.step()
. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:87: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
[03/18 08:37:21 d2.evaluation.evaluator]: Inference done 11/48. Dataloading: 0.0009 s/iter. Inference: 0.3040 s/iter. Eval: 1.2617 s/iter. Total: 1.5665 s/iter. ETA=0:00:57
[03/18 08:37:27 d2.evaluation.evaluator]: Inference done 15/48. Dataloading: 0.0009 s/iter. Inference: 0.3116 s/iter. Eval: 1.2645 s/iter. Total: 1.5771 s/iter. ETA=0:00:52
[03/18 08:37:32 d2.evaluation.evaluator]: Inference done 18/48. Dataloading: 0.0010 s/iter. Inference: 0.3153 s/iter. Eval: 1.3108 s/iter. Total: 1.6273 s/iter. ETA=0:00:48
[03/18 08:37:39 d2.evaluation.evaluator]: Inference done 22/48. Dataloading: 0.0010 s/iter. Inference: 0.3119 s/iter. Eval: 1.3120 s/iter. Total: 1.6249 s/iter. ETA=0:00:42
[03/18 08:37:45 d2.evaluation.evaluator]: Inference done 26/48. Dataloading: 0.0010 s/iter. Inference: 0.3091 s/iter. Eval: 1.3096 s/iter. Total: 1.6198 s/iter. ETA=0:00:35
[03/18 08:37:52 d2.evaluation.evaluator]: Inference done 30/48. Dataloading: 0.0010 s/iter. Inference: 0.3094 s/iter. Eval: 1.3118 s/iter. Total: 1.6223 s/iter. ETA=0:00:29
[03/18 08:37:58 d2.evaluation.evaluator]: Inference done 34/48. Dataloading: 0.0010 s/iter. Inference: 0.3091 s/iter. Eval: 1.3145 s/iter. Total: 1.6248 s/iter. ETA=0:00:22
[03/18 08:38:04 d2.evaluation.evaluator]: Inference done 38/48. Dataloading: 0.0010 s/iter. Inference: 0.3088 s/iter. Eval: 1.3051 s/iter. Total: 1.6150 s/iter. ETA=0:00:16
[03/18 08:38:10 d2.evaluation.evaluator]: Inference done 42/48. Dataloading: 0.0010 s/iter. Inference: 0.3082 s/iter. Eval: 1.2912 s/iter. Total: 1.6005 s/iter. ETA=0:00:09
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
/home/azuryl/dd3d_test/omni/dd3d/scripts/tridet/structures/boxes3d.py:210: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /opt/conda/conda-bld/pytorch_1640811805959/work/torch/csrc/utils/tensor_new.cpp:201.)
quats = torch.as_tensor(quats, dtype=torch.float32, device=device)
[03/18 08:38:16 d2.evaluation.evaluator]: Inference done 46/48. Dataloading: 0.0010 s/iter. Inference: 0.3067 s/iter. Eval: 1.2753 s/iter. Total: 1.5832 s/iter. ETA=0:00:03
[03/18 08:38:18 d2.evaluation.evaluator]: Total inference time: 0:01:06.936973 (1.556674 s / iter per device, on 8 devices)
[03/18 08:38:18 d2.evaluation.evaluator]: Total inference pure compute time: 0:00:13 (0.304431 s / iter per device, on 8 devices)
################coco_evaluation.py evaluate
[03/18 08:38:22 d2.evaluation.coco_evaluation]: @@@@@@@@@@@@@@@@@###############Preparing results for COCO format ...
[03/18 08:38:22 d2.evaluation.coco_evaluation]: Saving results to /home/azuryl/dd3d_test/omni/dd3d/outputs/2023-03-18/08-05-55/inference/step0002000/kitti_3d_val/coco_instances_results.json
[03/18 08:38:24 d2.evaluation.coco_evaluation]: Evaluating predictions with unofficial COCO API...
Loading and preparing results...
DONE (t=1.02s)
creating index...
index created!
[03/18 08:38:25 d2.evaluation.fast_eval_api]: Evaluate annotation type bbox
[03/18 08:38:29 d2.evaluation.fast_eval_api]: COCOeval_opt.evaluate() finished in 3.81 seconds.
[03/18 08:38:29 d2.evaluation.fast_eval_api]: Accumulating evaluation results...
[03/18 08:38:30 d2.evaluation.fast_eval_api]: COCOeval_opt.accumulate() finished in 1.12 seconds.
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.062
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.148
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.043
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.067
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.065
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.081
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.083
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.206
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.254
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.135
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.273
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.300
[03/18 08:38:30 d2.evaluation.coco_evaluation]: Evaluation results for bbox:
100%|██████████| 20/20 [00:07<00:00, 2.74it/s][03/18 08:40:01 tridet]: Evaluation results for kitti_3d_val in csv format:
[03/18 08:40:01 tridet.utils.train]: Test results:
The text was updated successfully, but these errors were encountered: