pixel-perfect-sfm

For example, dense features are stored in memory by default. If we reconstruct a large scene or have limited RAM, we should instead write them to a cache file that is loaded on-demand. With the Python API, we can pass a configuration update:

refiner = PixSfM(conf={"dense_features": {"use_cache": True}})

or equivalently with the command line using a dotlist:

python -m pixsfm.refine_hloc reconstructor [...] dense_features.use_cache=true

We also provide ready-to-use configuration templates in pixsfm/configs/ covering the main use cases. For example, pixsfm/configs/low_memory.yaml reduces the memory consumption to scale to large scene and can be used as follow:

refiner = PixSfM(conf="low_memory")
# or
python -m pixsfm.refine_hloc reconstructor [...] --config low_memory

If camera poses are available, we can simply triangulate a 3D point cloud from an existing reference COLMAP model with:

model, _ = refiner.triangulation(..., path_to_reference_model, ...)

or

python -m pixsfm.refine_hloc triangulator [...] \
    --reference_sfm_model path_to_reference_model

By default, camera poses and intrinsics are optimized by the bundle adjustment. To keep them fixed, we can simply overwrite the corresponding options as:

conf = {"BA": {"optimizer": {
    "refine_focal_length": False,
    "refine_extra_params": False,  # distortion parameters
    "refine_extrinsics": False,    # camera poses
}}}
refiner = PixSfM(conf=conf)
refiner.triangulation(...)

or equivalently

python -m pixsfm.refine_hloc triangulator [...] \
  'BA.optimizer={refine_focal_length: false, refine_extra_params: false, refine_extrinsics: false}'

To refine keypoints stored in an hloc HDF5 feature file:

from pixsfm.refine_hloc import PixSfM
refiner = PixSfM()
keypoints, _, _ = refiner.refine_keypoints(
    path_to_output_keypoints.h5,
    path_to_input_keypoints.h5,
    path_to_list_of_image_pairs,
    path_to_matches.h5,
    path_to_image_dir,
)

To refine keypoints stored in a COLMAP database:

from pixsfm.refine_colmap import PixSfM
refiner = PixSfM()
keypoints, _, _ = refiner.refine_keypoints_from_db(
    path_to_output_database,  # pass path_to_input_database for in-place refinement
    path_to_input_database,
    path_to_image_dir,
)

In either case, there is an equivalent command line interface.

To refine a 3D model stored on file:

from pixsfm.refine_colmap import PixSfM
refiner = PixSfM()
model, _, _, = refiner.refine_reconstruction(
    path_to_input_model,
    path_to_output_model,
    path_to_image_dir,
)

Using the command line interface:

python -m pixsfm.refine_colmap bundle_adjuster \
    --input_path path_to_input_model \
    --output_path path_to_output_model \
    --image_dir path_to_image_dir

1. To recompute the references from scratch, pass the path to the reference images:

from pixsfm.localization import QueryLocalizer
localizer = QueryLocalizer(
    reference_model,  # pycolmap.Reconstruction 3D model
    image_dir=path_to_reference_image_dir,
    dense_features=cache_path,  # optional: cache to file for later reuse
)
pose_dict = localizer.localize(
    pnp_points2D      # keypoints with valid 3D correspondence (N, 2)
    pnp_point3D_ids,  # IDs of corresponding 3D points in the reconstruction
    query_camera,     # pycolmap.Camera
    image_path=path_to_query_image,
)
if pose_dict["success"]:
    # quaternion and translation of the query, from world to camera
    qvec, tvec = pose_dict["qvec"], pose_dict["tvec"]

The default localization configuration can be accessed with QueryLocalizer.default_conf.

2. Alternatively, if dense reference features have already been computed during the pixel-perfect SfM, it is more efficient to reuse them:

refiner = PixSfM()
model, outputs = refiner.reconstruction(...)
features = outputs["feature_manager"]
# or load the features manually
features = pixsfm.extract.load_features_from_cache(
    refiner.resolve_cache_path(output_dir=path_to_output_sfm)
)
localizer = QueryLocalizer(
    reference_model,  # pycolmap.Reconstruction 3D model
    dense_features=features,
)

We first need to install the ETH3D multi-view evaluation tool:

sudo apt install libpcl-dev  # linux only
git clone [email protected]:ETH3D/multi-view-evaluation.git
cd multi-view-evaluation && mkdir build && cd build
cmake .. && make -j

We can then evaluate the accuracy of the sparse 3D point cloud triangulated with Pixel-Perfect SfM, for example on the courtyard scene with SuperPoint keypoints:

python -m pixsfm.eval.eth3d.triangulation \
    --scenes courtyard \
    --methods superpoint \
    --tag pixsfm

• omit --scenes and --methods to run all scenes with all feature detectors.
• the results are written to ./outputs/ETH3D/ by default
• use --tag some_run_name to distinguish different runs
• add --config norefine to turn off any refinement or use the dotlist KA.apply=false BA.apply=false
• add --config photometric to run the photometric BA (no KA)

To aggregate the results and compare different runs, for example with and without refinement, we run:

python -m pixsfm.eval.eth3d.plot_triangulation \
    --scenes courtyard \
    --methods superpoint \
    --tags pixsfm raw

Running on all scenes and all detectors should yield the following results (±1%):

----scene---- -keypoints- -tag-- -accuracy @ X cm- completeness @ X cm
                                  1.0   2.0   5.0   1.0   2.0   5.0 
----------------------------------------------------------------------
indoor        sift        raw    75.95 85.50 92.88  0.21  0.88  3.65
                          pixsfm 83.16 89.94 94.94  0.25  0.96  3.77
              superpoint  raw    78.96 87.77 94.55  0.64  2.36  9.39
                          pixsfm 89.93 94.09 97.04  0.76  2.62  9.85
              r2d2        raw    67.91 80.25 90.45  0.55  2.12  8.85
                          pixsfm 81.09 87.78 93.41  0.67  2.32  9.04
----------------------------------------------------------------------
outdoor       sift        raw    57.70 72.90 86.41  0.06  0.34  2.46
                          pixsfm 68.10 80.57 91.59  0.08  0.42  2.75
              superpoint  raw    53.63 68.93 83.27  0.11  0.64  4.43
                          pixsfm 71.83 82.65 92.06  0.18  0.89  5.40
              r2d2        raw    49.33 66.21 83.37  0.11  0.55  3.62
                          pixsfm 67.94 81.02 91.68  0.16  0.71  3.99

The results of this evaluation can be different from the numbers reported in the paper. The trends are however similar and the conclusions of the paper still hold. This difference is due to improvements of the pixsfm code and to changes in the SuperPoint implementation: we initially used the setup of PatchFlow and later switched to hloc, which is strictly better and easier to install.

Similarly, we evaluate the accuracy of camera pose estimation given sparse 3D models triangulated from other views:

python -m pixsfm.eval.eth3d.localization --tag pixsfm

Again, we can also run on a subset of scenes or keypoint detectors. To aggregate the results and compare different runs, for example with and without KA and BA, we run:

python -m pixsfm.eval.eth3d.plot_localization --tags pixsfm raw

We should then obtain the following table and plot (±2%):

-keypoints- -tag-- -AUC @ X cm (%)--
                    0.1    1    10  
sift        raw    16.92 55.39 81.15
            pixsfm 23.08 60.47 84.01
superpoint  raw    15.38 63.41 87.24
            pixsfm 41.54 73.86 89.66
r2d2        raw     6.15 51.70 83.46
            pixsfm 23.85 62.41 86.89

Results for the 0.1cm threshold can vary across setups and therefore differ from the numbers reported in the paper. This might be due to changes in the PyTorch and COLMAP dependencies. We are investigating this but any help is welcome!

dense_features:  # refinement features
  model:  # the CNN that extracts the features
    name: s2dnet  # the name of one of the models defined in pixsfm/features/models/
    num_layers: 1  # the number of output layers (model-specific parameters)
  device: auto  # cpu, cuda, or auto-determined based on CUDA availability
  max_edge: 1600  # downscale the image such the largest dimension has this value
  resize: LANCZOS  # interpolation algorithm for the image resizing
  pyr_scales: [1.0]   # concat features extracted at multiple scales
  fast_image_load: false  # approximate resizing for large images
  l2_normalize: true  # whether to normalize the features so they have unit norm
  sparse: true  # whether to store sparse patches of features instead of the full feature maps
  patch_size: 8  # the size of the feature patches if sparse
  dtype: half  # the data type of features when stored, half float or double
  use_cache: false  # whether to cache the features on file or keep them in memory
  overwrite_cache: false  # whether to overwrite the cache file if it already exists
  cache_format: chunked
interpolation:
  nodes: [[0.0, 0.0]]  # grid over which to compute the cost, by default a single point
  mode: BICUBIC  # the interpolation algorithm
  l2_normalize: true
  ncc_normalize: false  # only works if len(nodes)>1, mostly for photometric
mapping:  # pixsfm.refine_colmap.PixSfM
  dense_features: ${..dense_features}
  KA:  # keypoint adjustment
    apply: true  # whether to apply or instead skip
    strategy: featuremetric  # regular, or alternatively topological_reference (much faster)
    interpolation: ${...interpolation}  # we can use a different interpolation for KA
    level_indices: null  # we can optimize a subset of levels, by default all
    split_in_subproblems: true  # parallelize the optimization
    max_kps_per_problem: 50  # parallelization, a lower value saves memory, conservative if -1
    optimizer:  # optimization problem and solving
      loss:
        name: cauchy  # name of the loss function, among {huber, soft_l1, ...}
        params: [0.25]  # loss-specific parameters
      solver:
        function_tolerance: 0.0
        gradient_tolerance: 0.0
        parameter_tolerance: 1.0e-05
        minimizer_progress_to_stdout: false  # print a progress bar
        max_num_iterations: 100  # maximum number of optimization iterations
        max_linear_solver_iterations: 200
        max_num_consecutive_invalid_steps: 10
        max_consecutive_nonmonotonic_steps: 10
        use_inner_iterations: false
        use_nonmonotonic_steps: false
        num_threads: 1
      root_regularize_weight: -1  # prevent drift by adding edges to the root node, disabled if -1
      print_summary: false  # whether to print a detailed summary after completion
      bound: 4.0  # constraint on the distance (in pixels) w.r.t. the initial values
      num_threads: -1  # number of threads if parallelize in subproblems
  BA:  # bundle adjustment
    apply: true  # whether to apply or instead skip
    strategy: feature_reference  # regular, or alternatively {costmaps, patch_warp}
    interpolation: ${...interpolation}  # we can use a different interpolation for BA
    level_indices: null  # we can optimize a subset of levels, by default all
    max_tracks_per_problem: 10  # parallelization of references/costmaps, a lower value saves memory
    num_threads: -1
    optimizer:
      loss:  # same config as KA.optimizer.loss
      solver:  # same config as KA.optimizer.solver
      print_summary: false
      refine_focal_length: true  # whether to optimize the focal length
      refine_principal_point: false  # whether to optimize the principal points
      refine_extra_params: true  # whether to optimize distortion parameters
      refine_extrinsics: true  # whether to optimize the camera poses
    references:  # if strategy==feature_reference
      loss:  # what to minimize to compute the robust mean
        name: cauchy
        params: [0.25]
      iters: 100  # number of iterations to compute the robust mean
      num_threads: -1
    repeats: 1
localization:  # pixsfm.localization.main.QueryLocalizer
  dense_features: ${..dense_features}
  target_reference: nearest  # how to select references, in {nearest, robust_mean, all_observations}
  overwrite_features_sparse: null  # overwrite dense_features.sparse in query localization only
  references:  # how to compute references
    loss:  # what to minimize to compute the robust mean, same as BA.references.loss
    iters: 100
    keep_observations: true  # required for target_reference in {nearest, all_observations}
    num_threads: -1
  max_tracks_per_problem: 50  # parallelization of references, a lower value saves memory
  unique_inliers: min_error  # how we select unique matches for each 3D point
  QKA:  # query keypoint adjustment
    apply: true  # whether to apply or instead skip
    interpolation: ${...interpolation}
    level_indices: null
    feature_inlier_thresh:

提示：代码过长已自动截断

Note that the config supports variable interpolation through omegaconf.

The main improvements are:

• dense_features
  • store as sparse patches: sparse=true
  • reduce the size of the patches: patch_size=8 (or smaller)
  • store in a cache file: use_cache=true
• KA
  • chunk the optimization, loading only a subset of features at once: split_in_subproblems=true
  • optimize at most around 50 keypoints per chunk: max_kps_per_problem=50
• BA
  • use the costmap approximation: strategy=costmaps (described in Section C of the paper)

Some keypoint detectors with low output resolution, like D2-Net, predict keypoints that are localized inaccurately. In this case, the refinement is highly beneficial but the default parameters are not optimal. It is necessary to increase the patch size and use multiple feature layers. An example configuration is given in pixsfm_eth3d_d2net to evaluate D2-Net on ETH3D.