How can I obtain these ground truths?

[lyccnb]

When I generate the videos and directly run:
python evaluate.py --task block_slide --generated_videos ../data_creation/eval/v_reasonbench/wan2_2_origin/videos/,

the evaluation results indicate:
"./evaluations/Block_slide/predictions/wan2-2-origin_00_seed0.png": {
"pred_frame": "./evaluations/Block_slide/predictions/wan2-2-origin_00_seed0.png",
"error": "masks_not_found",
"gt_target": null,
"gt_mask": null,
"gt_bg_mask": null
},

Where should I obtain these ground truths?

[yangluo7]

Hi, the video name format should be <input_name>_<model>_seed<N>.mp4, for example, shape_fit_00_model1_seed0.mp4. For your case, it should be block_slide_00_wan_seed0.mp4.

[lyccnb]

Hi, the video name format should be <input_name>_<model>_seed<N>.mp4, for example, shape_fit_00_model1_seed0.mp4. For your case, it should be block_slide_00_wan_seed0.mp4.

Thank you for your reply.

My video format is set the same as yours, for example, “block_slide_00_wan2-2-origin_seed0.mp4”. The file ./evaluations/Block_slide/predictions/wan2-2-origin_00_seed0.png mentioned above is the result produced by vreason_bench.__init__.VReasonBench.distribute_videos, and there is no issue up to this step. However, the problem seems to be that the corresponding gt file cannot be found.

[yangluo7]

The required inputs/GT are updated. Sorry for the inconvenience.

[lyccnb]

The required inputs/GT are updated. Sorry for the inconvenience.

Alright, thank you for open-sourcing this project. I am now able to run the evaluation successfully. In addition, I noticed a small bug in lines 34–49 of vreason_bench.__init__, where the variable names used to extract model and gt_index seem to be incorrect. I replaced:

return {
"task": task,
"model": model,
"gt_index": gt_index,
"seed": seed
}

with

return {
"task": task,
"model": gt_index,
"gt_index": model,
"seed": seed
}

and after this change, it runs properly. Otherwise, it reports an "error": "invalid_filename" during evaluation. As mentioned earlier, I have confirmed that my video filenames follow the format <input_name>_<model_name>_seedx.mp4, so this might be a minor issue in the code.

[yangluo7]

The reason is that u used block_slide_00_wan2-2-origin_seed0.mp4, the correct format should be block_slide_wan2-2-origin_00_seed0.mp4, which is <input_name>_<model>_<gt_index>_seed<N>.mp4. The README is updated accordingly. Thx for your reply.