SlicerVirtualReality
[!NOTE]
It is recommended to use a recent Slicer Preview Release, OpenXR interface, and Meta Quest 3 headset.
Other configurations may work, but not regularly tested.
Extension for 3D Slicer enabling interaction with the 3D scene through virtual reality (VR) or augmented reality (AR).
The extension works with all OpenXR-compatible virtual reality and augmented reality headsets. The recommended headset is the Meta Quest 3, which works wirelessly via Air Link or with a USB cable. Other supported headsets include HTC Vive, and HoloLens 2. The legacy OpenVR backend is also available as a fallback for headsets that do not support OpenXR.
See the YouTube video below or the Kitware blog post for some more background and application examples.
Features
Key features:
- View all content of any 3D viewer in Slicer with a single click.
- Display volumes as 2D image slices or volume renderings, render surfaces, points, etc.
- Visualize any 4D datasets using various rendering techniques (including volume rendering) provided by the Sequences extension.
- Align the headset’s view to match the viewpoint of the selected 3D view in Slicer.
- Fly around using the touchpad of the right controller: direction is specified by orientation of the controller; speed is determined by the position of the finger on the touchpad (touch at the top to fly forward, touch at the bottom to fly backward).
- Grab and reposition objects using either controller’s grab button.
- Translate, rotate, and scale the entire scene using a two-handed gesture (see How to use controllers for the exact buttons, which vary by headset).
- Advanced volume rendering performance tuning available in the Virtual Reality module to balance image quality and refresh rate.
- Make controller positions available as transforms in the Slicer scene. These transforms can be used in custom modules to reslice volumes (using Volume Reslice Driver module in SlicerIGT extension) or transform any nodes in the scene.
Continuous improvements are made to the feature set. Feedback and ideas for improvement can be submitted via the issue tracker.
Setup
The extension uses OpenXR as its primary backend, supporting both virtual reality (VR) and augmented reality (AR, via OpenXR Remoting). The legacy OpenVR backend is retained as a fallback.
Platform support: Currently, the extension only functions on Windows computers. Linux support is experimental and untested due to the lack of official OpenXR runtimes and limited SteamVR support on Linux. The extension is available on macOS but lacks backend support due to Apple’s lack of OpenXR support for the Vision Pro.
|
OpenXR (recommended) |
OpenVR (legacy) |
| Windows |
:white_check_mark: |
:white_check_mark: |
| Linux |
:x: |
:x: |
| macOS |
:x: |
:x: |
If you wish to use Virtual Reality extension on Linux or macOS and you have virtual or augmented reality capable hardware working well on your computer through an OpenXR runtime then add a comment in the issue tracker (macOS / Linux).
Configuring graphics: On systems with both integrated display card and high-performance GPU (typically this is the case on laptops with NVidia GPUs), configure the graphics card application settings to use high-performance GPU for SlicerApp-real.exe (it is not necessary to use high-performance GPU for the launcher, Slicer.exe).
- Install Meta Horizon Link app and set up your headset.
- In Meta Horizon Link, click on
Settings, then in General section at OpenXR Runtime click Set Meta Horizon Link as... button. If this step is skipped then Slicer will not be able to connect to the headset (it will log the error: “Error [GENERAL | | OpenXR-Loader] : RuntimeManifestFile::FindManifestFiles - failed to find active runtime file in registry”)
- Install Slicer and the SlicerVirtualReality extension.
- To view content of the selected 3D view in your headset, click on the “Show scene in virtual reality” button
on the Slicer toolbar.
OpenXR is used by default. If OpenXR does not work, the legacy OpenVR backend can be used instead:
- Install [Steam][Steam-link], [SteamVR][SteamVR-link] and start the SteamVR home application in your headset.
- In Slicer, click on the wrench icon :wrench: in the toolbar, then select the OpenVR backend and enable rendering.
Supported headsets: Quest 3 (recommended), Quest Pro, Quest 2, Quest
Supported XR modality: Virtual Reality (Augmented Reality via camera pass-through is coming soon)
HTC Vive headset
- Install VIVESetup and set up your headset.
- Install Slicer and the SlicerVirtualReality extension.
- To view content of the selected 3D view in your headset, click on the “Show scene in virtual reality” button on the Slicer toolbar.
OpenXR is used by default. If OpenXR does not work, the legacy OpenVR backend can be used instead:
- Start the SteamVR home application in your headset.
- In Slicer, click on the wrench icon :wrench: in the toolbar, then select the OpenVR backend and enable rendering.
Supported headsets: Pro 2, Pro, Vive
Supported XR modality: Virtual Reality
HoloLens 2 headset
HoloLens 2 is supported via OpenXR Holographic Remoting, which streams the rendered scene wirelessly from the PC to the headset.
- On the HoloLens 2, install and start the Holographic Remoting Player from the Microsoft Store. Note the IP address shown on the HoloLens display.
- Install Slicer and the SlicerVirtualReality extension on the PC.
- In Slicer, click on the wrench icon :wrench: in the toolbar, then:
- Select the OpenXR backend.
- Enable Remoting and enter the IP address shown on the HoloLens.
- Enable rendering.
- The PC and HoloLens 2 must be on the same network.
- Rendering is performed on the PC and streamed to the headset, so a high-performance GPU on the PC is important.
Supported XR modality: Augmented Reality
How to use controllers
| Action |
Meta Quest |
HTC Vive |
Valve Index |
HP Reverb G2 |
| Fly forward |
Right thumbstick forward |
Right touchpad forward |
Right thumbstick forward |
Right thumbstick forward |
| Fly backward |
Right thumbstick backward |
Right touchpad backward |
Right thumbstick backward |
Right thumbstick backward |
| Grab/move object |
Grip button (either controller) |
Grip button (either controller) |
Grip (either controller) |
Grip button (either controller) |
| Transform/scale entire scene |
X button (left) + A button (right), pressed together |
Trigger (both controllers), pressed together |
A button (both controllers), pressed together |
X button (left) + A button (right), pressed together |
Grabbing/moving objects is always available via either grip button, so there is no separate “interaction mode” to select. The remaining buttons (B, menu, …) are not mapped to anything by default, but every control is available for customization, see the developer guide. Headsets with other controllers work through the generic Khronos simple controller profile (select = grab, menu buttons pressed together = transform/scale the scene).
The legacy OpenVR backend keeps its own bindings: on HTC Vive with OpenVR, objects are grabbed with the trigger, the scene is transformed with both grip buttons, and the menu button selects the interaction mode.
Fly
Move around in space. Equivalent to physically walking around.
Orient the controller in the intended direction of motion and use Fly forward or Fly backward action.
- Flying direction is specified by the orientation of the controller.
- Speed is proportional to distance of the fingertip from the touchpad center.
- Maximum speed is configurable in Virtual Reality module.
Translate/rotate a selected object.
- Hold down either controller’s
Grip button when the controller is inside a selectable object, move the controller to translate/rotate the object, then release the Grip button. No interaction mode needs to be selected first.
- With the legacy OpenVR backend on HTC Vive: make sure interaction mode is set to
Grab (this is the default), then hold down the Trigger when the controller is inside a selectable object, move the controller to translate/rotate the object, then release the Trigger.
- When you grab and move object, a parent transform is automatically created for it (if it has not been under a transform already) and that transform is modified.
- To move a group of objects together, assign the same parent transform to them. You can do that in Data module’s Transform hierarchy tab by drag-and-dropping objects under the same transform (or by double-clicking in the Applied Transform column in Data module’s Transform hierarchy tab and selecting a transform; or by selecting a transform in Transforms module and applying it to all the nodes that must move together).
- By default all objects are selectable. An object can be made non-selectable (thus non-movable) in Data module / Subject hierarchy tab, right-clicking on the node and unchecking “Toggle Selectable”.
How to use hand interaction
See Action poses for hand interactions
Translate/rotate a selected object.
Controls: do the “grip” or “pinch” pose when the hand is inside a selectable object
Other features
Go to Virtual Reality module and check the desired checkbox to update linear transform nodes with the various devices’ positions.
Frequently asked questions
How to clip models
3D Slicer can clip models using slice planes. This feature can be used in virtual reality by moving the slice plane using a transform, with the help of Volume reslice driver module.
The models, transforms, etc. only need to be set up once, because the scene can then be saved to file and next time it can be readily used.
Simple clipping
Move clipping planes continuously, as the controllers are moved.
- Go to Models module, select the model to clip, and in Clipping section enable Clip selected model. By default the Red slice will be chosen for clipping.
- Go to Virtual Reality module and enable making controller transform appear as transforms in the scene.
- Go to Volume reslice driver module (in SlicerIGT extension) to make the controller transform move the Red slice.
- Moved the controller inside the model to clip it
Clipping with handles
Grab and move clipping planes using controllers (clipping plane remains in place).
- Go to Models module, select the model to clip, and in Clipping section enable Clip selected model. By default the Red slice will be chosen for clipping.
- Add a small model, for example a box to the scene. It can be loaded from STL or created using Create models module (in SlicerIGT extension).
- Move the model using a controller. This creates a parent transform for this model.
- Go to Volume reslice driver module (in SlicerIGT extension) to make the model’s parent transform move the Red slice.
- Grab and move the model using the controller to move the clipping plane.
Rendering is slow
There are several settings that help in increasing the performance of virtual reality rendering:
- If you are using a computer that has two graphics cards (for example, laptops often have an integrated Intel and a high-performance NVidia graphics card), make sure that Slicer is forced to use the high-performance card that the headset is connected to. Most laptops assign applications to the integrated card by default. When you need to select the application executable, choose
SlicerApp-real.exe (and not the Slicer launcher application Slicer.exe).
- Optimize scene for virtual reality button (magic wand icon on toolbar): this switches volume rendering to use GPU, turns off backface culling for all existing models (to see surfaces even when going inside an object), turns off slice intersection visibility for all existing models and segmentations (to make slice view updates faster)
- Settings in virtual reality module panel related to performance (click on wrench icon :wrench: in toolbar):
- Update rate: Volume rendering quality is set to produce the highest possible quality while keeping the desired frame per second
- Motion sensitivity: It is very important to keep rendering smooth when moving. This setting detects head movement and significantly lowers volume rendering quality while it is happening. At value of 0 motion is never detected, at high values a little motion triggers the quality change
- Settings in Volume rendering module: open “Advanced” section / “Techniques” tab, try “Adaptive” setting with different “Interactive speed” values. Also try and “Normal” setting: it disables the automatic mechanism that tries to dynamically adjust rendering quality based on predicted rendering time (in some cases the prediction does not work well and results in sub-optimal image quality).
- Switch layout of desktop Slicer: Any additional 3D view to render decreases virtual reality rendering performance. By switching to a layout that contains no 3D view (e.g. Red slice only), this can be prevented
Some scenes are too complex to render fluently by mid-range graphics cards. If the scene includes volume rendering of high-resolution CT for example, then it may be necessary to upgrade to a high-end GPU.
How to record virtual reality videos?
Use OBS Studio (free) to capture the VR headset view, application window, webcam, etc.:
- OpenXR (recommended): Most headset companion apps (Meta Horizon Link for Quest 3, SteamVR) provide a screen-mirror or casting window that OBS can capture directly. On Meta Quest 3, the mirror application can be found at
C:\Program Files\Oculus\Support\oculus-diagnostics\OculusMirror.exe or similar location. The Quest app on your phone can also cast directly to a TV or browser.
- OpenVR (SteamVR): Enable screen mirroring in SteamVR and capture the mirror window with OBS.
Note that OBS Studio may crash if NVidia hardware-based compression is used. If this happens, choose the CPU-based video compression option for recording.
How to ask questions, report problems, or suggest new features?
Visit Slicer forum and search for similar discussions. If you do not find related topics then create a new one. Add virtual-reality tag to make sure people who monitor virtual reality related questions get a notification about your question.
If you are certain that you have found a software bug and no similar issue has been reported in the issue tracker then please submit a new issue.
Please do not use “VR” acronym (you can spell out “virtual reality” instead), because “VR” may mean “volume rendering” just as well as “virtual reality” - you can even do volume rendering in virtual reality in Slicer - and so it becomes confusing very quickly.
For developers
Information for developers is available in the Developer Guide.
Contributors
Contributors include:
- Kitware: Jean-Christophe Fillion-Robin, Jean-Baptiste Vimort, Lucas Gandel, Sankhesh Jhaveri
- PerkLab (Queen’s University): Csaba Pinter, Andras Lasso
- VASST Lab (Robarts Research Institute): Adam Rankin
How to cite
Pinter, C., Lasso, A., Choueib, S., Asselin, M., Fillion-Robin, J. C., Vimort, J. B., Martin, K., Jolley, M. A. & Fichtinger, G. (2020). SlicerVR for Medical Intervention Training and Planning in Immersive Virtual Reality. IEEE Transactions on Medical Robotics and Bionics, vol. 2, no. 2, pp. 108-117, May 2020, doi: 10.1109/TMRB.2020.2983199
License
It is covered by the Apache License, Version 2.0:
http://www.apache.org/licenses/LICENSE-2.0
评论