Ns reconstruction on friction dataset.

[YuechenXie]

Thank you for your amazing work.

The default configurations in your code work properly when I use them to reconstruct the abo500 dataset.

Since there are no parameter settings for the friction dataset in the code, I attempted to use your code and reconstruct your friction dataset in accordance with the parameter configurations specified in your paper.
However, the reconstruction results are incorrect and exhibit obvious artifacts. Could you please provide the specific configuration code, parameter settings, and running commands ... you used for this dataset?

Below are my parameter configurations and reconstruction results:

nerf reconstruction:

pointcloud export:
result = subprocess.run([
'ns-export', 'pointcloud',
'--load-config', os.path.join(ns_dir, 'config.yml'),
'--output-dir', base_dir,
'--num-points', str(args.num_points),
'--remove-outliers', 'True',
'--normal-method', 'open3d',
'--use-bounding-box', 'True',
'--bounding-box-min', str(cx - half_bbox_size), str(cy - half_bbox_size), str(cz - half_bbox_size),
'--bounding-box-max', str(cx + half_bbox_size), str(cy + half_bbox_size), str(cz + half_bbox_size),
], env=proc_env, check=True)
and:
--bbox_size 1.5 --bbox_center 0 0 -0.75

here's visualization:

[ajzhai]

Hi, thanks for your question. Going through some old wandb logs, it seems that the difference might be in pipeline.datamanager.camera_optimizer.mode and pipeline.model.proposal_initial_sampler (these settings seemed to be different in some of the runs). Try disabling lines 33 and 34 here:

NeRF2Physics/ns_reconstruction.py

Line 33 in 7311210

'--pipeline.datamanager.camera-optimizer.mode', 'off',

[YuechenXie]

Thank you for your reply. After commenting out these two lines, the reconstruction results have been significantly improved. Below are my specific configurations.

for scene in scenes: 
    base_dir = os.path.join(scenes_dir, scene, 'ns')
    result = subprocess.run([
        'ns-train', 'nerfacto',
        '--output-dir', base_dir,
        '--vis', args.vis_mode,
        '--project-name', args.project_name,
        '--experiment-name', scene,
        '--max-num-iterations', str(args.training_iters),
        '--pipeline.model.background-color', 'random',
        # '--pipeline.model.camera-optimizer.mode', 'off',
        # '--pipeline.model.proposal-initial-sampler', 'uniform',
        '--pipeline.model.near-plane', str(args.near_plane),
        '--pipeline.model.far-plane', str(args.far_plane),
        '--steps-per-eval-image', '1000',
        'nerfstudio-data',
        '--data', os.path.join(scenes_dir, scene),
        '--scene-scale', str(args.scene_scale),
    ], env=proc_env, check=True)

However, through visualization,

I have found that the reconstruction results of many scenes, especially in terms of depth geometry, exhibit very significant artifacts. I would like to ask whether this is due to problems with my experimental setup itself, or whether it is normal and caused by the inherent limitations of NeRF.

[ajzhai]

Good to hear that the results improved! Those artifacts come from fundamental limitations of NeRF in low-texture regions. I think I encountered them as well. Incorporating monocular depth priors might be able to fix that, but we did not do so in our work.

[YuechenXie]

I have encountered new problems in the evaluation on the friction dataset. The original evaluation.py appears to be designed for overall item-level metric evaluation, rather than point-level evaluation as required by this dataset. I have tried to implement the evaluation code based on my own understanding, but the results differ significantly from the metrics reported in the paper, and I have difficulty locating the issue.

Could you please provide the specific evaluation pipeline code or details for this dataset, as well as the GPT annotation files labeled with BLIP and GPT used for this dataset?

[ajzhai]

Sadly I cannot find the intermediate results from BLIP/GPT previously obtained. However, the inference code should be the same. The model should be gpt-4 and temperature should be 0.01.

I did find some old code for loading the CSV query points that might be helpful:

def get_query_pts(csv_file, scenes_dir):
    
    csv_data = np.loadtxt(csv_file, delimiter=',', skiprows=1 , dtype='str')
    
    query_pts = []
    scene_names = []
    vals = []
    for row in csv_data[:]:
        scene_name, img_name, x, y = row[:4]
        scene_name = scene_name + ('_nodepth' if scene_name in ['drawer', 'gray_mat'] else '_masked')  
        scene_dir = os.path.join(scenes_dir, scene_name)
        depth_dir = os.path.join(scene_dir, 'ns', 'renders', 'depth')
        sorted_depth_fnames = list(sorted(os.listdir(depth_dir)))
        c2ws, Ks = parse_transforms_json(os.path.join(scene_dir, 'transforms.json'), different_Ks=True)
        
        depth_file = img_name + '.npy.gz'
        with gzip.open(os.path.join(depth_dir, depth_file), 'rb') as f:
            dist = np.load(f)[:, :, 0]
        idx = sorted_depth_fnames.index(img_name + '.npy.gz')
        depth = distance_to_depth(dist, Ks[idx])
        
        pt_3d = unproject_point([int(x), int(y)], depth, c2ws[idx], Ks[idx])
        query_pts.append(pt_3d)
        scene_names.append(scene_name)
        vals.append(row[4:])

        scene_pts = load_ns_point_cloud(os.path.join(scene_dir, 'ns', 'point_cloud.ply'),
                                        os.path.join(scene_dir, 'ns', scene_name, 'nerfacto', 
                                                     os.path.basename(get_last_file_in_folder(os.path.join(scene_dir, 'ns', scene_name, 'nerfacto'))), 'dataparser_transforms.json'))

    return query_pts, scene_names, vals