
# Babylon.js WebXR features

This contains the documentation of all features that are not AR-Exclusive. For AR-specific Features, see the [BabylonJS WebXR AR Features](/features/featuresDeepDive/webXR/webXRARFeatures) page.



# WebXR

## Current state

The [WebXR W3C Proposal](https://immersive-web.github.io/webxr/) is currently in its draft phase. It is, however, already implemented in Chrome (check [caniuse.com](https://caniuse.com/#feat=webxr) to know about others browsers). Starting with version 79, WebVR has been deprecated and WebXR is enabled by default. Earlier browser versions had WebXR behind a configuration flag. One of its goal is deprecating WebVR and other AR implementation and provide a single VR and AR API.

As the API continuously changes, it is difficult to keep up with feature changes. The latest Chrome Canary is notably the most XR-feature-complete browser and Google continuously updates the browser with new features. This is the main reason we introduced the [Features Manager](/features/featuresDeepDive/webXR/webXRFeaturesManager), which allows us to implement the newest version of official features with internal versioning without breaking backwards compatibility.

Note that most of the time when we say WebXR, we actually mean WebXR **in VR immersive mode**. This is currently the most used mode of WebXR.

## Device and browser support

### PC

Chrome 79 on Windows officially supports WebXR with all [Microsoft Mixed Reality](https://en.wikipedia.org/wiki/Windows_Mixed_Reality) Devices. Unofficially, WebXR is working well with the Oculus SDK (Rift, Rift S, and Quest with Link). As of this writing, Oculus support is still behind a flag.

### Mobile and Quest

WebXR AR features on Android's Chrome Browser (Stable and Canary) can be enabled behind a flag at [chrome://flags](chrome://flags), including AR features such as plane detection, hit-tests and anchors. Note that the AR features' architecture is constantly changing, so expect different results from version to version.

Oculus Quest supports WebXR (in VR mode) in the latest Oculus browser. Babylon's specs implementation works well with the quest.

No official iOS/iPhone support is planed at the moment. Mozilla has built the [WebXR iOS Viewer](https://apps.apple.com/us/app/webxr-viewer/id1295998056) which is a (very) limited AR-oriented browser.

### Polyfill

For older browsers that support WebVR but not WebXR you can use the [WebXR Polyfill](https://github.com/immersive-web/webxr-polyfill) which is the WebXR API implementation using WebVR features. Some functions will not work (or will simply return without changes) but the basic functionality works well.

Babylon does not intend on integrating the polyfill in the framework itself or in the playground. We encourage the developer to offer the polyfill to users not using a WebXR-Supported browser.

To use the polyfill in the playground, please add the following code to your playground (before 'createScene'):

```javascript
const xrPolyfillPromise = new Promise((resolve) => {
    if (navigator.xr) {
        return resolve();
    }
    if (window.WebXRPolyfill) {
        new WebXRPolyfill();
        return resolve();
    } else {
        const url = "https://cdn.jsdelivr.net/npm/webxr-polyfill@latest/build/webxr-polyfill.js";
        const s = document.createElement("script");
        s.src = url;
        document.head.appendChild(s);
        s.onload = () => {
            new WebXRPolyfill();
            resolve();
        };
    }
});
```

afterwards, make sure to `await` it before initializing WebXR:

```javascript
const xrPolyfillPromise = new Promise((resolve) => {
    if (navigator.xr) {
        return resolve();
    }
    if (window.WebXRPolyfill) {
        new WebXRPolyfill();
        return resolve();
    } else {
        const url = "https://cdn.jsdelivr.net/npm/webxr-polyfill@latest/build/webxr-polyfill.js";
        const s = document.createElement("script");
        s.src = url;
        document.head.appendChild(s);
        s.onload = () => {
            new WebXRPolyfill();
            resolve();
        };
    }
});

var createScene = async function () {
  // wait for the polyfill to kick in
  await xrPolyfillPromise;
  console.log(navigator.xr); // should be there!
  console.log(await BABYLON.WebXRSessionManager.IsSessionSupportedAsync("immersive-vr")); // should be true
  // create your scene
  var scene = new BABYLON.Scene(engine);
  var camera = new BABYLON.DeviceOrientationCamera("DevOr_camera", new BABYLON.Vector3(-30, -30, -30), scene);
  camera.setTarget(BABYLON.Vector3.Zero());
  camera.attachControl(canvas, true);
  var light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 0, 0), scene);
  scale = 100;
  // initialize XR
  var xr = await scene.createDefaultXRExperienceAsync();

  return scene;
};
```

If you experience low-resolution when using the polyfill, make sure to resize the canvas to a higher resolution. This is a limitation of WebVR (that required resizing the canvas) which we didn't integrate for WebXR.

### The WebXR Emulator

If you want to debug your XR scene on your desktop you will need to emulate WebXR. The first method is using a browser extension.

Mozilla's [WebXR Emulator extension](https://blog.mozvr.com/webxr-emulator-extension/) has not been updated for quite some time. Instead, we recommend using Meta's [Immersive WebXR emulator](https://github.com/meta-quest/immersive-web-emulator/) which is available for both [Chrome](https://chrome.google.com/webstore/detail/immersive-web-emulator/cgffilbpcibhmcfbgggfhfolhkfbhmik) and [Edge](https://microsoftedge.microsoft.com/addons/detail/immersive-web-emulator/hhlkbhldhffpeibcfggfndbkfohndamj).

Another option to debug your scene on Window is emulating a headset using [Microsoft's Mixed Reality portal](https://www.microsoft.com/store/apps/9NG1H8B3ZC7M). Instructions can be found here: [https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/using-the-windows-mixed-reality-simulator](https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/using-the-windows-mixed-reality-simulator).

The main difference between the two approaches is that the first one works in your browser and emulates WebXR, and the second is directly using the WebXR implementation of the browser and emulates a headset in an external native application.

## Getting started

The simplest way to get started is using a WebXR-enabled browser and add a single line of code to your scene:

```javascript
const xr = scene.createDefaultXRExperienceAsync();
```

This will enable WebXR **in VR immersive mode**, including session init, input sources, the camera, teleportation and scene interactions. All using our [WebXR Default Experience Helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers#the-basic-experience-helper).

Note that the `xr` variable is a Promise. Using the async/await pattern will be simpler and more intuitive. It will also make sense to define floor meshes, so we can define our ground and move on it. Here is a sphere in XR:

```javascript
var createScene = async function () {
  var scene = new BABYLON.Scene(engine);
  var camera = new BABYLON.FreeCamera("camera1", new BABYLON.Vector3(0, 5, -10), scene);
  camera.setTarget(BABYLON.Vector3.Zero());
  camera.attachControl(canvas, true);
  var light = new BABYLON.HemisphericLight("light1", new BABYLON.Vector3(0, 1, 0), scene);
  light.intensity = 0.7;
  var sphere = BABYLON.MeshBuilder.CreateSphere("sphere1", { segments: 16, diameter: 2 }, scene);
  sphere.position.y = 1;

  const env = scene.createDefaultEnvironment();

  // here we add XR support
  const xr = await scene.createDefaultXRExperienceAsync({
    floorMeshes: [env.ground],
  });

  return scene;
};
```

<Playground id="#F41V6N" title="Sphere In WebXR Using Babylon.js" description="Simple example of a sphere in WebXR using Babylon.js" isMain={true} category="WebXR"/>

And that's it!

Make sure to read more on the [WebXR Experience Helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers) for further tips and tricks, and take a look at our [Demos and examples](/features/featuresDeepDive/webXR/webXRDemos) page.

### ES6 support with Tree Shaking

When using [Babylon.js ES6 support with tree shaking](/setup/frameworkPackages/es6Support), import WebXR modules from:

* `@babylonjs/core/XR/*`

And import loaders and side-effects for loading default controller models from the [WebXR Input Profiles](https://github.com/immersive-web/webxr-input-profiles) GitHub repository.

For example:

```javascript
import { WebXRDefaultExperience } from '@babylonjs/core/XR/webXRDefaultExperience.js'

// Enable GLTF/GLB loader for loading controller models from WebXR Input registry
import '@babylonjs/loaders/glTF'

// Without this next import, an error message like this occurs loading controller models:
//  Build of NodeMaterial failed" error when loading controller model
//  Uncaught (in promise) Build of NodeMaterial failed: input rgba from block
//  FragmentOutput[FragmentOutputBlock] is not connected and is not optional.
import '@babylonjs/core/Materials/Node/Blocks'

```

See also:

* [WebXR Controllers Support](/features/featuresDeepDive/webXR/webXRInputControllerSupport)
* [WebXR with Vite](/features/featuresDeepDive/webXR/webXRDemos#webxr-with-vite)

## Migrating from WebVR

WebVR is deprecated and will soon end its life in most if not all browsers. It is highly recommended to port all WebVR implementations to WebXR.

### Migrating from the VR Experience helper

If you used our [VR experience helper](/features/featuresDeepDive/cameras/webVRHelper) remove the VR initializer and add the XR experience helper. So this:

```javascript
var scene = new BABYLON.Scene(engine);
var vrHelper = scene.createDefaultVRExperience();
```

turns to this:

```javascript
var scene = new BABYLON.Scene(engine);
var xrHelper = scene.createDefaultXRExperienceAsync();
```

The XR helper has full controller support per default, including interactions with the scene meshes, pointer events and more. Read more about the [XR Experience helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers).

### Migrating controller support

Since WebXR controllers are no longer considered to be Gamepads the architecture is a bit different.

The most important feature that was added is the full pointer events support for the controllers. The controllers support all pointer events, so you can use [Pointer interactions](/features/featuresDeepDive/scene/interactWithScenes#pointer-interactions) just like you use to controller mouse interactions in your scene.

It is also important to note that it is now possible to query what features the controller has and act accordingly.

Here are the **VR** controllers' most used functions, and how to get them to work in **XR**:

```javascript
// On new controller attached:

// WebVR:
webvrCamera.onControllersAttached = (vrController) => {
  // fun with the new controller, which is a gamepad!
};

// WebXR:
const webXRInput = xr.input; // if using the experience helper, otherwise, an instance of WebXRInput
input.onControllerAddedObservable.add((xrController /* WebXRInputSource instance */) => {
  // more fun with the new controller, since we are in XR!
  inputSource.onMotionControllerInitObservable.add((motionController) => {
    // get the motionController, which is similar to but NOT a gamepad:
  });
  // xr supports all types of inputs, so some won't have a motion controller
  if (!xrController.gamepad) {
    // using touch, hands, gaze, something else?
  }
});

// From this point we assume we have two variables: vrController and xrController.
// We also assume motionController is present!

// main button

// WebVR:
controller.onMainButtonStateChangedObservable.add((newState /* ExtendedGamepadButton */) => {
  // is the button pressed?
  if (newState.pressed) {
    // Do something
  }
});

// WebXR:
// get the main component (decided by the controller's vendor!)
const mainComponent = xrController.motionController.getMainComponent();
// or get the trigger component, if present:
const mainTrigger = xrController.motionController.getComponent(WebXRControllerComponent.TRIGGER);
mainComponent.onButtonStateChanged.add((component /* WebXRControllerComponent */) => {
  // check for changes:
  // pressed changed?
  if (component.changes.pressed) {
    // is it pressed?
    if (component.changes.pressed.current === true) {
      // pressed
    }
    // or a different way:
    if (component.pressed) {
      // component is pressed.
    }
  }
});

// thumbpad / touchpad

// in WebVR - you had to check what controller is being used, but in general this would work:
vrController.onPadValuesChangedObservable.add(function (stateObject) {
  console.log(stateObject); // {x: 0.1, y: -0.3}
});

// in webXR you can check if it is present and work accordingly:
const thumbstick = xrController.motionController.getComponent(WebXRControllerComponent.THUMBSTICK);
if (thumbstick) {
  // Huzza! we have a thumbstick:
  thumbstick.onButtonStateChanged; // will trigger when the thumbstick is PRESSED or touched!

  thumbstick.onAxisValueChanged; // will trigger when axes of the thumbstick changed
}

// touchpad

// in WebVR we had "pad" concept which was for both thumbstick and touchpad
controller.onPadValuesChangedObservable.add(function (stateObject) {
  console.log(stateObject); // {x: 0.1, y: -0.3}
});

// in WebXR, it is much much better:
const touchpad = xrController.motionController.getComponent(WebXRControllerComponent.TOUCHPAD);
if (touchpad) {
  // Finally, a controller with a touchpad
  touchpad.onButtonStateChanged; // will trigger when the touchpad is touched or pressed

  touchpad.onAxisValueChanged; // will trigger when axes of the touchpad changed
}
```

Read more about the [XR Controllers system](/features/featuresDeepDive/webXR/webXRInputControllerSupport).

### Legacy support

Thou we always encourage backwards compatibility **We recommend using WebXR directly** and stop using the WebVR experience helper. However:

The latest WebVR Experience helper has a new flag in its init options - `useXR`. This will check for XR support and will launch the VR session in WebXR, if possible. A working example can be found in <Playground id="#TAFSN0#323" title="WebVR Check for WebXR" description="Simple example of the WebVR -useXR check to create a VR session using WebXR instead."/>

```javascript
var createScene = function () {
  // Create scene
  var scene = new BABYLON.Scene(engine);

  // Create simple sphere
  var sphere = BABYLON.Mesh.CreateIcoSphere(
    "sphere",
    {
      radius: 0.2,
      flat: true,
      subdivisions: 1,
    },
    scene,
  );
  sphere.position.y = 3;
  sphere.material = new BABYLON.StandardMaterial("sphere material", scene);

  // Lights and camera
  var light = new BABYLON.DirectionalLight("light", new BABYLON.Vector3(0, -0.5, 1.0), scene);
  light.position = new BABYLON.Vector3(0, 5, -2);
  var camera = new BABYLON.ArcRotateCamera("camera", -Math.PI / 2, Math.PI / 4, 3, new BABYLON.Vector3(0, 3, 0), scene);
  camera.attachControl(canvas, true);
  scene.activeCamera.beta += 0.8;

  // Default Environment
  var environment = scene.createDefaultEnvironment({
    enableGroundShadow: true,
    groundYBias: 2.8,
  });
  environment.setMainColor(BABYLON.Color3.FromHexString("#74b9ff"));

  // Shadows
  var shadowGenerator = new BABYLON.ShadowGenerator(1024, light);
  shadowGenerator.useBlurExponentialShadowMap = true;
  shadowGenerator.blurKernel = 32;
  shadowGenerator.addShadowCaster(sphere, true);

  // Enable VR, use XR when possible
  var vrHelper = scene.createDefaultVRExperience({
    createDeviceOrientationCamera: false,
    useXR: true, // This will enable XR if supported
    floorMeshes: [environment.ground],
  });

  // Runs every frame to rotate the sphere
  scene.onBeforeRenderObservable.add(() => {
    sphere.rotation.y += 0.0001 * scene.getEngine().getDeltaTime();
    sphere.rotation.x += 0.0001 * scene.getEngine().getDeltaTime();
  });

  // GUI
  var plane = BABYLON.Mesh.CreatePlane("plane", 1);
  plane.position = new BABYLON.Vector3(0.4, 4, 0.4);
  var advancedTexture = BABYLON.GUI.AdvancedDynamicTexture.CreateForMesh(plane);
  var panel = new BABYLON.GUI.StackPanel();
  advancedTexture.addControl(panel);
  var header = new BABYLON.GUI.TextBlock();
  header.text = "Color GUI";
  header.height = "100px";
  header.color = "white";
  header.textHorizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER;
  header.fontSize = "120";
  panel.addControl(header);
  var picker = new BABYLON.GUI.ColorPicker();
  picker.value = sphere.material.diffuseColor;
  picker.horizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER;
  picker.height = "350px";
  picker.width = "350px";
  // This will work in XR, since we are using native pointer events!
  picker.onValueChangedObservable.add(function (value) {
    sphere.material.diffuseColor.copyFrom(value);
  });
  panel.addControl(picker);

  vrHelper.onAfterEnteringVRObservable.add(() => {
    // This callback will still work! Would be better to use the XR native observables.
  });

  return scene;
};
```

The color picker works since it is using the pointer architecture. If XR is present, XR will be used. Otherwise, it will use WebVR as a fallback.

Note that some features will not work correctly or will not work at all. For example, camera gaze will not work at all. Controller will work, but since the interaction architecture is different, it is highly likely you will need to adjust a few observers in order to get it to work, especially if there are VR-specific callbacks.

We recommend using the WebXR polyfill instead.



The WebXR Camera is an extension of the `FreeCamera` that includes an update loop from an XRFrame and the ability to create one or more rig cameras that will be used to render the XR (both VR and AR) session.

If you are not using the [XR Experience helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers), you will need an [XR Session Manager](/features/featuresDeepDive/webXR/webXRSessionManagers) to construct a new WebXR Camera:

``` javascript
const xrCamera = new WebXRCamera("nameOfCamera", scene, xrSessionManager);
```

Notice that as opposed to the free and target cameras, the constructor does not accept an initial position. This has a few reasons, the main one being - the first frame will populate the camera's position with the real-world information which cannot be queried when creating the camera. In the next section(s) you will see how to get an initial position from a non-XR camera and how to control the user's position correctly.

## How positioning works

Just like any Babylon camera, the WebXR camera is an object with its own position and rotation that can be updated between frames. The WebXR camera's transformation is the current position of the device rendering the XR session (such as the HUD or the phone rendering the session) relative to the reference space currently used in the [Session Manager](/features/featuresDeepDive/webXR/webXRSessionManagers).

The camera's position will update every frame from the information sent to it by the XRFrame. The XRFrame contains a general position of the device, and, if available, the poses for each screen (or eye) that are used to update the position of the rig cameras (which are the ones actually rendering the scene).

Just like any other camera you can get its direction:

``` javascript
const direction = xrCamera.getDirection(Axis.Z);
```

Or a position in front of the camera in a certain distance:

``` javascript
// get the position 2 meters in front of the camera
const frontPosition = xrCamera.getFrontPosition(2);
```

If the camera's position changes between frames (manually by you, or by the [WebXR teleportation feature](/typedoc/classes/babylon.webxrmotioncontrollerteleportation)) a new reference space is being calculated (compensating for the position change). This reference space will now be used to update the camera's position.

The camera's Rig-system is the one responsible for rendering. The Rig-Camera's pose is always calculated based on the pose coming from the XR-Frame and **NOT** the WebXR Camera's pose. The main camera's pose is only a reference of the base transformation and is not directly influencing the rendered cameras.

## Current user's height

The WebXR camera can provide you with the height of the current user, unrelated to the ground. This is an important piece of information when ground level is not at 0. When ground level is 0, the user height and the camera's `position.y` will be the same.

To query the user's height from the real ground use the `realWorldHeight` of the camera:

```javascript
const userHeight = xrCamera.realWorldHeight;
```

This getter will return the height of the user or 0 if it's not available.

Note that the user height depends on the type of reference space you chose to your experience. Reference space type `local-floor` will deliver the user's height, but `viewer` (for example) will deliver what you defined as a height compensation in the [XR Session Manager](/features/featuresDeepDive/webXR/webXRSessionManagers).

## Updating from a Non-VR Camera

When entering XR you might want to duplicate the Non-XR camera's position and use it as the base position of the XR Camera. To do that you will need to find the absolute transformation of the old camera and apply it to the new one. Babylon allows you to do that with a single function of the XR Camera:

``` javascript
// if scene.activeCamera is still the non-VR camera:
xrCamera.setTransformationFromNonVRCamera();

// Otherwise, provide the non-vr camera to copy the transformation from:

xrCamera.setTransformationFromNonVRCamera(otherCamera);

// If you want XR to also reset the XR Reference space, set the 2nd variable to true:

xrCamera.setTransformationFromNonVRCamera(otherCamera, true);
```

When executing this function the XR camera will copy the position of the old camera, and its rotation around the Y axis. The rest will be provided by the XR Frame. It is important to know that the height provided by the XR Frame (relative to the reference space) will be added to the position as a compensation for the position reset. Which means that it is possible that the XR camera's Y position will not be exactly equal to the other camera's Y position.



## Basic scene

This is a step-by-step guide on how to add XR features to a basic scene

## Basic Scene with XR Support

Here we just add an environment, a sphere, and XR support

```javascript
const xrHelper = await scene.createDefaultXRExperienceAsync();
```

<Playground id="#9K3MRA" title="Basic Scene With WebXR Support" description="Simple example of a basic scene with WebXR support."/>

### Adding teleportation

To get teleportation enabled, we want to provide the experience helper with an array of floor meshes:

```javascript
const xrHelper = await scene.createDefaultXRExperienceAsync({
  // define floor meshes
  floorMeshes: [environment.ground],
});
```

<Playground id="#9K3MRA#1" title="Basic Scene With Teleportation" description="Simple example of a basic scene with teleportation enabled." isMain={true} category="WebXR"/>

### Adding a color picker to the basic scene

Add a color picker (from our GUI library) and use it to change the sphere's color.

Notice that no changes were made in the XR code, and that the scene works perfectly well outside VR as well.

```javascript
// GUI
var plane = BABYLON.Mesh.CreatePlane("plane", 1);
plane.position = new BABYLON.Vector3(1.4, 1.5, 0.4);
var advancedTexture = BABYLON.GUI.AdvancedDynamicTexture.CreateForMesh(plane);
var panel = new BABYLON.GUI.StackPanel();
advancedTexture.addControl(panel);
var header = new BABYLON.GUI.TextBlock();
header.text = "Color GUI";
header.height = "100px";
header.color = "white";
header.textHorizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER;
header.fontSize = "120";
panel.addControl(header);
var picker = new BABYLON.GUI.ColorPicker();
picker.value = sphere.material.diffuseColor;
picker.horizontalAlignment = BABYLON.GUI.Control.HORIZONTAL_ALIGNMENT_CENTER;
picker.height = "350px";
picker.width = "350px";
picker.onValueChangedObservable.add(function (value) {
  sphere.material.diffuseColor.copyFrom(value);
});
panel.addControl(picker);
```

<Playground id="#9K3MRA#2" title="WebXR Color Picker" description="Simple WebXR color picker example." isMain={true} category="WebXR"/>

## AR demo
![playground](/img/how_to/xr/ARPortal.jpg)
<Playground id="#M3G924" title="AR Portal" description="WebXR AR Portal" isMain={true} category="WebXR" image="/img/how_to/xr/ARPortal.jpg"/>

## Other demos

<Playground id="#PPM311#148" title="Goalkeeper Training" description="Goalkeeper Training" isMain={true} category="WebXR"/>
<Playground id="#B922X8#19" title="Legacy Physics Playground" description="Physics Playground" isMain={true} category="WebXR"/>

<Playground id="#F41V6N#139" title="A cylinder object is child of a controller" description="A cylinder object is child of a controller"/>

<Playground id="#1FTUSC#37" title="Simply grabbing objects by controllers" description="Simply grabbing objects by controllers"/>

## Babylon.js scenes with XR support

<Playground id="#JA1ND3#161" title="Mansion" description="Mansion Demo"/>
<Playground id="#TJIGQ1#3" title="Hill Valley" description="Hill Valley"/>
<Playground id="#ja1nd3#942" title="Espilit" description="Espilit"/>

## WebXR with Vite

Step-by-step project setup using Vite and Babylon.js ES6 modules for
WebXR development.

### Install Vite

```bash
## Setup vite
npm create vite@latest
##> Project name: babylon-webxr-test
##> Select a framework: vanilla
##> Select a variant: vanilla-ts
```

### Install @babylonjs ES6 packages

```bash
npm install @babylonjs/core
npm install @babylonjs/loaders
```

### Enable HTTPS dev server

HTTPS is required by most VR devices. Create or modify `vite.config.ts`:

```javascript
import { defineConfig } from "vite";

// https://vitejs.dev/config/
export default defineConfig({
  server: {
    port: 3443,
    https: true,
    // Uncomment to allow access from network
    // (or use `npm run dev -- -- host=0.0.0.0`)
    //host: '0.0.0.0',
  },
});
```

### Set up a basic WebXR scene

Modify the `main.ts` file:

```typescript
import "./style.css";

import { ArcRotateCamera } from "@babylonjs/core/Cameras/arcRotateCamera.js";
import { Color3 } from "@babylonjs/core/Maths/math.color.js";
import { Engine } from "@babylonjs/core/Engines/engine.js";
import { EnvironmentHelper } from "@babylonjs/core/Helpers/environmentHelper.js";
import { HemisphericLight } from "@babylonjs/core/Lights/hemisphericLight.js";
import { Mesh } from "@babylonjs/core/Meshes/mesh";
import { MeshBuilder } from "@babylonjs/core/Meshes/meshBuilder.js";
import { Scene } from "@babylonjs/core/scene.js";
import { StandardMaterial } from "@babylonjs/core/Materials/standardMaterial.js";
import { Vector3 } from "@babylonjs/core/Maths/math.vector.js";
import { WebXRDefaultExperience } from "@babylonjs/core/XR/webXRDefaultExperience.js";

// Required for EnvironmentHelper
import "@babylonjs/core/Materials/Textures/Loaders";

// Enable GLTF/GLB loader for loading controller models from WebXR Input registry
import "@babylonjs/loaders/glTF";

// Without this next import, an error message like this occurs loading controller models:
//  Build of NodeMaterial failed" error when loading controller model
//  Uncaught (in promise) Build of NodeMaterial failed: input rgba from block
//  FragmentOutput[FragmentOutputBlock] is not connected and is not optional.
import "@babylonjs/core/Materials/Node/Blocks";

// Create a canvas element for rendering
const app = document.querySelector<HTMLDivElement>("#app");
const canvas = document.createElement("canvas");
app?.appendChild(canvas);

// Create engine and a scene
const babylonEngine = new Engine(canvas, true);
const scene = new Scene(babylonEngine);

// Add a basic light
new HemisphericLight("light1", new Vector3(0, 2, 0), scene);

// Create a default environment (skybox, ground mesh, etc)
const envHelper = new EnvironmentHelper(
  {
    skyboxSize: 30,
    groundColor: new Color3(0.5, 0.5, 0.5),
  },
  scene,
);

// Add a camera for the non-VR view in browser
const camera = new ArcRotateCamera("Camera", -(Math.PI / 4) * 3, Math.PI / 4, 10, new Vector3(0, 0, 0), scene);
camera.attachControl(true);

// Add a sphere to have something to look at
const sphereD = 1.0;
const sphere = MeshBuilder.CreateSphere("xSphere", { segments: 16, diameter: sphereD }, scene);
sphere.position.x = 0;
sphere.position.y = sphereD * 2;
sphere.position.z = 0;
const rMat = new StandardMaterial("matR", scene);
rMat.diffuseColor = new Color3(1.0, 0, 0);
sphere.material = rMat;

// Setup default WebXR experience
// Use the enviroment floor to enable teleportation
WebXRDefaultExperience.CreateAsync(scene, {
  floorMeshes: [envHelper?.ground as Mesh],
  optionalFeatures: true,
});

// Run render loop
babylonEngine.runRenderLoop(() => {
  scene.render();
});
```

Corresponding git repo: [kaliatech/babylon-docs-vite-webxr](https://github.com/kaliatech/babylon-docs-vite-webxr)

### Run server and verify

Start vite dev server and make it accessible from the network:

```bash
npm run dev -- --host 0.0.0.0
```

Browsing to `https://<your-server-ip>:3443` on a desktop machine should show the basic scene. If
the [WebXR API Emulator](https://github.com/MozillaReality/WebXR-emulator-extension) is enabled, you
should also see Babylon.js's default VR headset mode icon

If viewing from within a headset, the controller models should correspond to what is available in the global
registry for your device.

To check TypeScript types and build:

```bash
npm run build
```

Using the setup above, vite reports a vendor.js size of ~2.4MB (520k gzipped) as of babylon-5.0.0-beta.8.




## Introduction

In order to simplify development for WebXR we offer a WebXR helper that will initialize WebXR environment automatically. This environment allows you to develop your AR and VR experiences in a easy to use, flexible and fully configurable manner.

We recommend using the [Default Experience Helper](#webxr-default-experience-helper) for ease of use, as it offers a full XR experience, including: UI, teleportation and more. In contrast, the basic set of functionalities is offered via [Basic Experience Helper](#webxr-basic-experience-helper), it offers more control when it comes to configuring the specifics that better fit your use-case.

*Note that you are not forced to use these experience helpers in any way. You can always use your own implementation instead.*

## WebXR (Basic) Experience Helper

The basic experience helper will:

* Initialize the XR scene;
* Create an XR Camera;
* Initialize the features manager;
* Offers help functions to enter and exit the XR session.

The basic experience helper offers direct access to the following:

* An initialized [Session Manager](/features/featuresDeepDive/webXR/webXRSessionManagers);
* A configured [XR Camera](/features/featuresDeepDive/webXR/webXRCamera) that will have the non-VR camera's position when entering XR;
* Initialized [Features Manager](/features/featuresDeepDive/webXR/webXRFeaturesManager).

### How to use (Basic) Experience Helper

The experience helper is promise-based, which technically means its help-functions are mostly asynchronous.

To use the WebXR Experience Helper, first Create one using the static factory:

``` javascript
const xrHelper = await WebXRExperienceHelper.CreateAsync(scene);
```

If the browser does not support XR an exception will be thrown and will you will need to catch it:

``` javascript
try {
    const xrHelper = await WebXRExperienceHelper.CreateAsync(scene);
} catch (e) {
    // no XR support
}

// or:
WebXRExperienceHelper.CreateAsync(scene).then((xrHelper) => {
    // great success
}, (error) => {
    // no xr...
})
```

### Entering XR

After initializing the XR helper, it is possible to enter an XR session, for example in immersive VR mode:

``` javascript
const sessionManager = await xrHelper.enterXRAsync("immersive-vr", "local-floor" /*, optionalRenderTarget */ );
```

To read more about session modes ( `immersive-vr` in this example), and reference type modes ( `local-floor` ), please read the [WebXR specs](https://immersive-web.github.io/webxr/). The most common scenario is VR in local floor mode, which is the one we are showing here.

If there is an error while creating the experience helper, the console will show it.

### Exiting XR

To exit, simply call the exitXRAsync function:

``` javascript
await xrHelper.exitXRAsync();
```

### Observables

#### The state observable

In this example, the session manager will be initialized if the XR session is ready to use. Alternatively, you can ignore the `await` and use the **state observable** of the xr helper:

``` javascript
xrHelper.onStateChangedObservable.add((state) => {
    switch (state) {
        case WebXRState.IN_XR:
            // XR is initialized and already submitted one frame
        case WebXRState.ENTERING_XR:
            // xr is being initialized, enter XR request was made
        case WebXRState.EXITING_XR:
            // xr exit request was made. not yet done.
        case WebXRState.NOT_IN_XR:
            // self explanatory - either out or not yet in XR
    }
})
```

#### Pose initialized observable

The `onInitialXRPoseSetObservable` will trigger all observers when the pose of the camera was set and before the first frame was rendered.

It can be used to define, for example, a height offset (if the ground is not at 0, or if the user starts in a different level):

``` javascript
xrBasicHelper.onInitialXRPoseSetObservable.add((xrCamera) => {
    // floor is at y === 2
    xrCamera.position.y = 2;
});
```

## WebXR Default Experience Helper

The default experience helper will:

* Create a [Basic WebXR Experience Helper](#webxr-basic-experience-helper) and initialize it;
* Create an HTML UI button to enter XR;
* Init the input source class which will initialize controllers;
* Enable the pointer selection and teleportation features.

### How to use the Default Experience Helper

The simplest way to use the default experience is calling the help function of `scene` :

``` javascript
var defaultXRExperience = await scene.createDefaultXRExperienceAsync( /* optional configuration options */ );
```

This is equivalent to:

``` javascript
var defaultXRExperience = await WebXRDefaultExperience.CreateAsync(scene, /* optional configuration options */ )
```

After calling this and assuming the browser supports XR, you will see the Enter-XR GUI on the bottom right of the screen unless the browser **does not support XR**. The GUI will not be displayed. To check if the default experience initialized correctly, make sure the `baseExperience` variable was created:

``` javascript
var defaultXRExperience = await scene.createDefaultXRExperienceAsync( /* optional configuration options */ );
if (!defaultXRExperience.baseExperience) {
    // no xr support
} else {
    // all good, ready to go
}
```

### Configuration options

The default xr experience helper takes an options map for configuration. Everything, including the options itself is optional:

``` javascript
class WebXRDefaultExperienceOptions {
    /**
     * Floor meshes that will be used for teleporting
     */
    public floorMeshes ? : Array < AbstractMesh > ;

    /**
     * Enable or disable default UI to enter XR
     */
    public disableDefaultUI ? : boolean;

    /**
     * optional configuration for the output canvas
     */
    public outputCanvasOptions ? : WebXRManagedOutputCanvasOptions;

    /**
     * optional UI options. This can be used among other to change session mode and reference space type
     */
    public uiOptions ? : WebXREnterExitUIOptions;

    /**
     * Disable the controller mesh-loading. Can be used if you want to load your own meshes
     */
    public inputOptions ? : IWebXRInputOptions;

    /**
     * Should teleportation not initialize. defaults to false.
     */
    public disableTeleportation ? : boolean;
}
```

Check the code documentation for further details.

### Pointer selection and teleportation

The default experience initializes both pointer selection (laser pointer and interaction with the meshes in the scene) and teleportation features automatically. If floor meshes are provided in the options, the teleportation will include it when created. If not you can add meshes later or disable it.

To read more about both features, check the [Features Manager](/features/featuresDeepDive/webXR/webXRFeaturesManager)

To enable / disable the teleport and pointer selection:

``` javascript
// assuming xrHelper initialized correctly!

// disable:
defaultXRExperience.teleportation.detach();
defaultXRExperience.pointerSelection.detach();

// (re)enable:
defaultXRExperience.teleportation.attach();
defaultXRExperience.pointerSelection.attach();
```

### Configuring the GUI

The GUI is the right way to enter and exit the XR experience. Otherwise you can directly use the method, offered by [Basic Experience Helper](#webxr-basic-experience-helper), to enter and exit XR as mentioned [above](#entering-xr). It defaults to a VR immersive session in local-floor reference space.

You can change the `uiOptions` to setup the **AR** session instead of the **VR** session:

``` javascript
var defaultXRExperience = await scene.createDefaultXRExperienceAsync({
    uiOptions: {
        sessionMode: 'immersive-ar'
    }
});
```

### Available Observables

Apart from those you can find in the [Basic Experience Helper](#webxr-basic-experience-helper) the [Default Experience Helper](#webxr-default-experience-helper) does not offer any observables.

### Further features

The default experience offers direct access to the following:

* `input` - initialized [WebXRInputSource]()
* `enterExitUI` - initialized [GUI]()
* `renderTarget` - The default WebXR render target



## Why is it needed

The Features manager, our XR plugin system, was born out of a simple need - stay backwards compatible, but still deliver cutting edge APIs in a production system.

Since APIs like [Hit Test](https://github.com/immersive-web/hit-test) and [anchors](https://github.com/immersive-web/anchors) are constantly changing, and currently still have different support in different browser versions, there was a need for the “versioning” of the current development to keep up with API modifications over time.

If you are a bit impatient you can check the [selected WebXR features](/features/featuresDeepDive/webXR/WebXRSelectedFeatures) section, but it is recommended to at least read this section quickly.

## How to use

### Construct a new features manager

If you are using the [base WebXR experience helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers) a features manager will be created for you and will be available in `xrBaseHelper.featuresManager`. If not, you only need to provide an XR session manager object to initialize a new one:

``` javascript
const fm = new FeaturesManager(xrSessionManager);
```

Note that even before creating the features manager you could call its static methods (check availability, register a new feature, etc.).

### What is available

To check what feature is available, use the features manager's static `GetAvailableFeatures` function, which will return an array of strings corresponding to specific features:

``` javascript
const availableFeatures = WebXRFeaturesManager.GetAvailableFeatures(); // in plain JS, use BABYLON.WebXRFeaturesManager

// availableFeatures = ["xr-hit-test", "xr-pointer-selection", ...]
```

To find if a specific feature is available use this code:

``` javascript
const availableFeatures = WebXRFeaturesManager.GetAvailableFeatures();
// using indexOf, but you can use any other method - includes, find, findIndex, in, etc'
if (availableFeatures.indexOf(WebXRFeatureName.POINTER_SELECTION) !== -1) {
    // Pointer selection is available
}
```

`WebXRFeatureName` will always contain the list of possible features.

When using ES6, make sure to import the feature you want to use, otherwise it won't be registered:

``` javascript
import {
    WebXRControllerPhysics
} from '@babylonjs/core/XR/features/WebXRControllerPhysics';

// The physics controller support will now be available and can be enabled:

fm.enableFeature(WebXRControllerPhysics.Name, "latest");
```

### Versioning

Just like any plugin system, the features are versioned numerically. The version is a number. Higher the number, the newer the version (pretty simple).

Two extra definitions available - `stable` for the latest stable version of this feature, and `latest` which is always updated with the plugin with the highest version number. Keep in mind that the `latest` definition will always be available, but the `stable` might not be — for features that are not yet considered "stable".

To get the available versions use the `GetAvailableVersions` static method. It will return an array of available versions, for example:

``` javascript
const availableVersions = featuresManager.GetAvailableVersions(WebXRFeatureName.POINTER_SELECTION);

// availableVersions = ["latest", "stable", "1", "2"];
```

This means that you can ask for version 1, but also for the stable version which will be one of the 2 - "1" or "2", depends on our definition. The "latest" version will load "2".

### Enable and disable a feature

Enabling a feature means that the feature is ready to be used in a working session. When a feature is enabled, it is ready to be attached, which technically means - influence the scene actively.

As an example, **enabling** the teleportation feature will register the `onControllerAdded` and `onControllerRemoved` observers. **Attaching** this feature will register all observers required for each controller to work properly, while **detaching** it will remove such observers. Furthermore, **disabling** the teleportation feature will make sure that the meshes are invisible and the `onControllerAdded` and `onControllerRemoved` observers will be removed. Finally, **disposing** of the feature will release everything, including all associated meshes.

To enable the feature use a constructed features manager's `enableFeature` function with the name and version you wish to load:

``` javascript
// get the features manager
const fm = xr.baseExperience.featuresManager;

// disable an already-enabled feature
fm.disableFeature(WebXRFeatureName.POINTER_SELECTION);

// enable the latest version of the "Hit Test" feature
const xrHitTestLatest = fm.enableFeature(WebXRFeatureName.HIT_TEST, "latest");

// enable the specific version of the "Hit Test" feature. This will disable an older implementation and enable the new one
const xrHitTest1 = fm.enableFeature(WebXRFeatureName.HIT_TES, 1);
```

### Configuring the feature

Every feature (as of now) has a different configuration options object that can be provided when enabling the feature. Each feature has an options interface. For example, Version "1" of the "Hit Test" feature has the following options available:

``` javascript
interface IWebXRHitTestOptions {
    /**
     * Only test when user interacted with the scene. Default - hit test every frame
     */
    testOnPointerDownOnly ? : boolean;
    /**
     * The node to use to transform the local results to world coordinates
     */
    worldParentNode ? : TransformNode;
}
```

To use it:

``` javascript
// get the features manager
const fm = xr.baseExperience.featuresManager;

// enable the latest version of the "Hit Test" feature with a configuration object. The old "enabled" version will be disposed of
const xrHitTestLatest = fm.enableFeature(WebXRFeatureName.HIT_TEST, "latest", {
    testOnPointerDownOnly: true
});

// enable the specific version of the "Hit Test" feature with default configuration. The old config will be lost
const xrHitTest1 = fm.enableFeature(WebXRFeatureName.HIT_TES, 1);
```

### Attach and detach a feature

Once you have a feature enabled, you can synchronously attach (or detach) it to the scene:

``` javascript
// enable version "1" of the "Hit Test" feature
// Feature will be auto-attached once the session starts
const xrHitTest1 = fm.enableFeature(BABYLON.WebXRFeatureName.HIT_TEST, 1);

// detach the feature, but keep it enabled (for reattachment). This also disables auto-attach, if not already attached
xrHitTest1.detach();

// re-attach
xrHitTest1.attach();
```

Enabling a feature will attach it automatically when an XR session starts. If you want to only enable but not attach a feature (so you have full control of when it starts working), set the `attachIfPossible` argument to `false` (defaults to `true`):

``` javascript
// get the features manager
const fm = xr.baseExperience.featuresManager;
const xrHitTestLatest = fm.enableFeature(WebXRFeatureName.HIT_TEST, "latest", {
    testOnPointerDownOnly: true
}, false /* prevent attaching automatically */ );
```

## ES6 passive loader

When using the ES6 module loader you will notice that no features apart from transportation and pointer selection are available. You'll have to import other features (and their versions) by yourself. This ensures that resulting bundle will include only the code that your experience actually needs to load, when it needs it. Wonderful for the "tree shaking"!

So this won't work in ES6

``` javascript
const featuresManager = giveMeMyFeaturesManagerSomehow();
// The "Hit Test" feature is not available
const xrHitTest1 = featuresManager.enableFeature("xr-hit-test", 1);
// Error thrown, "feature not found"
```

Adding an import will make it work:

``` javascript
// other imports...
import {
    WebXRHitTestLegacy
} from "@babylonjs/core/Cameras/xr/features/WebXRHitTestLegacy";

const featuresManager = giveMeMyFeaturesManagerSomehow();
// feature will now be enabled
const xrHitTest1 = featuresManager.enableFeature(WebXRFeatureName.HIT_TEST /* Same as "xr-hit-test" */ , 1);
```

## Writing a new feature

If you want to add your own feature here are a few guidelines:

### IWebXRFeature

The feature should implement the `IWebXRFeature` interface, which looks like this:

``` javascript
interface IWebXRFeature extends IDisposable {
    /**
     * Is this feature attached
     */
    attached: boolean;
    /**
     * Should auto-attach be disabled?
     */
    disableAutoAttach: boolean;
    /**
     * Attach the feature to the session
     * Will usually be called by the features manager
     *
     * @param force should attachment be forced (even when already attached)
     * @returns true if successful.
     */
    attach(force ? : boolean): boolean;
    /**
     * Detach the feature from the session
     * Will usually be called by the features manager
     *
     * @returns true if successful.
     */
    detach(): boolean;
}
```

In order to ease this process you can, instead, extend the `WebXRAbstractFeature`, which has a few extra help-functions and implements most functions needed for a working feature.

After creating the feature, you will need to register it with the features manager, so it can be enabled.

To register it, use the static `AddWebXRFeature` function:

``` javascript
const nameOfFeature = "awesome-name";
const version = 1;
const isTheFeatureStable = true
WebXRFeaturesManager.AddWebXRFeature(nameOfFeature, (xrSessionManager, options) => {
    return () => new MyNewFeature(xrSessionManager, options);
}, version, isTheFeatureStable);
```

This way you can enable the feature using its name whenever you wish to use it.



One of the major differences between WebVR and WebXR was the support for different types of controllers. While WebVR supported the non-standard extended gamepad API and a few selected controllers, WebXR already supports a lot of different types of inputs, including touch screens, motion controllers, and hands.

## Some terms and classes to clear things up

An XR controller comprises a lot of components that we at Babylon.js sometimes name differently. It is also important to know the terms themselves to be able to use what you actually need. I would recommend reading the [XR Input section](https://www.w3.org/TR/webxr/#input) of the WebXR proposal draft.

An XR Session controls the input source of the current session. Every new input source connected to this session will be registered in the `inputSources` array of the native XR Session and will also trigger the `inputsourceschange` event with the new input source.

An Input source has one of three target Ray modes - `tracked-pointer` for gamepad-like controllers, `screen` for touch-screen oriented inputs, and `gaze` for gaze-based inputs (input sources like google cardboard that has no proper way for user input).

The Babylon's [WebXRInput](/typedoc/classes/babylon.webxrinput) class is responsible for coordinating the addition and removal of input sources. It creates **WebXR Input Sources** classes and disposes of them automatically for you.

The Babylon's [WebXRInputSource](/typedoc/classes/babylon.webxrinputsource) class is the container for all user input related objects. It is created automatically by the WebXRInput class for every controller. It is in charge of attaching the motion controller which, in turn, is in charge of attaching the components and load the model.

An input source has two important [reference spaces](https://www.w3.org/TR/webxr/#xrspace):

1. The `targetRay`, which represents the pointer's position and direction. Think - the edge of your finger.
2. The `gripSpace`, which is the base of a handheld device connected to this input source. Think - the base of your hand. The `gripSpace` is optional and is only available when a motion controller is connected.

Although sometimes the same, the grip and target ray spaces can have different transformations.

A controller has an attached **motion controller** if the input source is a gamepad-like device (the Oculus Touch or the windows motion controllers are a good example for that). In turn, each controller has different **components** (buttons, triggers, thumbpads, touchpads), which have their state updated on each frame.

Note that input sources with [`targetRayMode`](https://www.w3.org/TR/webxr/#dom-xrinputsource-targetraymode) of [`"screen"`](https://www.w3.org/TR/webxr/#dom-xrtargetraymode-screen) are constantly created and removed when touch starts and ends. This is called [transient input](https://www.w3.org/TR/webxr/#transient-input).

## WebXRInput

Class [WebXRInput](/typedoc/classes/babylon.webxrinput) has little to no public members, you won't typically interact with it. An instance of that is usually created by the WebXRDefaultExperience. It has two observables that can be helpful:

### onControllerAddedObservable

This observable will be triggered when a new input source was detected and its corresponding [WebXRInputSource](/typedoc/classes/babylon.webxrinputsource) was created.

### onControllerRemovedObservable

This one will be triggered when a controller leaves the experience, right before its [WebXRInputSource](/typedoc/classes/babylon.webxrinputsource) instance is disposed.

## WebXRInputSource

[WebXRInputSource](/typedoc/classes/babylon.webxrinputsource), created by the [WebXRInput](/typedoc/classes/babylon.webxrinput), represents an input source (controller). It has a few important members.

## Mixing hands and controllers

Starting Babylon 6.42.0 it is possible to mix different types of controllers, as long as the underlying system supports it. For example, you could support both a hand and a controller at the same time.
To try that just enter any XR demo with your controllers and put one of them down, or enter with both hands and pick up one controller.

### Public methods

#### getWorldPointerRayToRef

As mentioned before, an input source has two reference spaces: `grip` and `target` (which we call `pointer`). Each of those spaces is represented by a mesh, positioned in the orientation and position that is provided by the XRFrame (and the XRSession). If you want to query current transformation of the user's hand or the direction the user is pointing at, you can use the `getWorldPointerRayToRef` method:

```javascript
// Get the pointer direction ray.
const pointerRay = new BABYLON.Ray();
xrInputSource.getWorldPointerRayToRef(pointerRay);

// Try to get the grip direction ray. If that's not available,
// the pointer direction ray will be given.
const gripRay = new BABYLON.Ray();
xrInputSource.getWorldPointerRayToRef(gripRay, /* gripIfAvailable = */ true);
```

In most scenarios, you will need the pointer (not the grip) direction ray.

### Observables

#### onMotionControllerInitObservable

This will be triggered when a motion controller, if available, was initialized and its profile loaded.

#### onMeshLoadedObservable

This is a helper observable, it is the same as `xrController.motionController.onModelLoadedObservable`. Because `motionController` is created asynchronously, using this one may require less code:

```javascript
// async, async, async
xrInput.onControllerAddedObservable.add((inputSource) => {
  inputSource.onMotionControllerInitObservable.add((motionController) => {
    motionController.onMeshLoadedObservable.add((model) => {});
  });
});

// a little cleaner
xrInput.onControllerAddedObservable.add((inputSource) => {
  inputSource.onModelLoadedObservable.add((model) => {});
});
```

#### onDisposeObservable

Will be triggered right at the end of the `dispose()` function of the input source.

## Motion controllers

In most cases when starting a VR session, the user will have handheld devices, called here motion controllers. A motion controller will be automatically loaded, if available. A motion controller has a profile, containing its different components and their positions in the buttons and axes array, but Babylon.js takes care of this for you so you don't have to know this to interact with the motion controller. You can see the different profiles in the [WebXR Input Profiles repository](https://github.com/immersive-web/webxr-input-profiles)

### Controller components

Each motion controller has different components, those are described in its profile. Available component types:

- Button
- Trigger
- Squeeze
- Thumbstick
- Touchpad

It also has a **unique** component id, which correlates to the actual component. For example, the **A button** on the Oculus Touch has the **type** `button`, and the **id** `a-button` .

#### Getting available components

To get a list of the components available, use the `getComponentIds` function. This will return an array of strings containing the IDs of the different components in this motion controller:

```javascript
const ids = motionController.getComponentIds();
// ids = ["a-button", "b-button", "xr-standard-trigger", .....]
```

You can also get all of the available components using the `components` public member of the Motion Controller class. It is a map of all `WebXRControllerComponent` elements in this motion controller.

#### Get components

To get a component you need to either know its ID or type. When more than one of that type exists, the id would be better. The IDs can be found in the profile.

To get a component according to its ID:

```javascript
const triggerComponent = motionController.getComponent("xr-standard-trigger");
if (triggerComponent) {
  // found, do something with it.
}
```

To get a component of a specific type:

```javascript
const squeezeComponent = motionController.getComponentOfType("squeeze");

// get the first registered button component
const buttonComponent = motionController.getComponentOfType("button");
```

to get all components of a specific type:

```javascript
// get all button components
const buttonComponents = motionController.getAllComponentsOfType("button");
if (buttonComponents.length) {
  // some were found
}
```

#### The main component

Each controller has a main component, defined by the vendor. In most cases it is the trigger component type that is the main component. To get the main component (as defined in the profile):

```javascript
const mainComponent = motionController.getMainComponent();
// mainComponent always exists!
```

#### Events and changes of a controller component

A component is updated on each frame with values provided by the gamepad object of the session's input source. Each button has 2 states - `touched` and `pressed`, and a value from 0 to 1 (0 being not pressed at all, 1 being fully pressed). Some components can only have the values 0 and 1 (like the button component).

Some types of components also have axes values (like a thumbstick or touchpad). The axes have values from -1 to 1.
1 in the X-axis means right, and 1 in the Y-Axis means down (towards the user).

To know what the components support:

```javascript
if (component.isButton()) {
  // we have a value
}
if (component.isAxes()) {
  // we have axes data
}
```

To get the component values at the current frame:

```javascript
let value = component.value;
if (value > 0.8) {
  // do something nice with this value
}
if (component.pressed) {
  // the component is pressed, meaning value === 1
}

if (component.touched) {
  // fingers are on the component, might be half-pressed or moved
}
```

To access the axes data:

```javascript
let axes = component.axes;
if (axes.x > 0.8) {
  // do something nice with the x-axis value
}
```

The component can also return the changes compared to the last frame. Changes are only populated when they exist, otherwise the changes map will be empty:

```javascript
// maybe nothing happened between this and last frame
if (!component.hasChanges) {
    return;
}
let changes = component.changes;
if (changes.pressed) {
    // pressed state changed
    const isPressedNow = changes.pressed.current;
    const wasPressedInLastFrame = changes.pressed.previous;
}
if (changes.value) {
    // value changed! let's get the delta
    const delta - changes.value.current - changes.value.previous;
}
```

The components have two observables that can be used to fetch changes (and avoid checking value changes on each frame):

```javascript
component.onButtonStateChangedObservable.add((component) => {
  // something changed, check the changes object
});

component.onAxisValueChangedObservable.add((values) => {
  console.log(values.x, values.y);
});
```

Here is a simple example of controllers input.  
By using Oculus Quest 2 controllers, component ids and controller buttons are mapped as follows.

On Oculus Quest 2 controllers:

![Quest 2 controller mappings](/img/how_to/xr/xr-quest2-controllers_ids_mapping.jpg)

On the Playground sample:

![ids mappings](/img/how_to/xr/xr-quest2-ids-mapping-playground.jpg)

The simple example, changes the scaling of each 3DCG object when each button is pressed.

```javascript
const xr_ids = motionController.getComponentIds();
let triggerComponent = motionController.getComponent(xr_ids[0]); //xr-standard-trigger
triggerComponent.onButtonStateChangedObservable.add(() => {
  if (triggerComponent.pressed) {
    Box_Right_Trigger.scaling = new BABYLON.Vector3(1.2, 1.2, 1.2);
  } else {
    Box_Right_Trigger.scaling = new BABYLON.Vector3(1, 1, 1);
  }
});
```

Playground for a simple VR controllers input: <Playground id="#28EKWI#37" title="WebXR_motion controller input" description="Simple code for motion controller input on WebXR."/>

### How to get a model

#### The input-profile online repository

As part of the (successful!) attempt at forcing standards to WebXR, the > at the [WebXR Input Profiles](https://github.com/immersive-web/webxr-input-profiles) GitHub repository created an online repository that holds models and definition of visual reference definitions for most (if not all) motion controllers available today.

The repository provides a useful tool as [WebXR Input Profile Viewer](https://immersive-web.github.io/webxr-input-profiles/packages/viewer/dist/index.html). You can easily check each id, state, button, and axis of XR controllers.

Babylon.js natively supports this repository and currently uses it as the default model delivery method for XR controllers.

There is little to no action required on your end - this is automated when not changing the configuration.

#### Babylon local controller definitions

Before the input-profile repository was published, Babylon had support for different types of controllers - Oculus Touch (1 and 2), Vive, Windows Motion Controllers and more. Since we still wanted to offer local support for those devices, we have decided to not only relay on the online profile repository, but to also deliver babylon-based controller classes for those that are already developed.

Babylon offers local definitions for the following:

- Windows Motion Controllers
- Oculus Touch 1 and 2
- Vive
- Generic-Button controller

To use them, import them to your project, while not forgetting to prioritize them or disable the online repository:

```javascript
// import the ones you want to use
import { WebXRMicrosoftMixedRealityController } from "@babylonjs/core/XR/motionController/webXRMicrosoftMixedRealityController";

// prioritize the local classes (but use online if controller not found)
WebXRMotionControllerManager.PrioritizeOnlineRepository = false;
// or disable the online repository
WebXRMotionControllerManager.UseOnlineRepository = false;

// now, if loading a microsoft motion controller it will use the local class
```



Babylon's [WebXRSessionManager](/typedoc/classes/babylon.webxrsessionmanager) class is your direct contact to the browser's native XR session. Each XR experience has a corresponding session that holds the entire XR functionality. A session is initialized with a session mode (the default is `vr-immersive`) and a reference space mode (default is `local-floor`) that decides how the scene calculates the user's location and what functionality is available.

Usually, the Session Manager will be initialized by the [experience helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers), but if you don't use the experience helper, you can create a session manger yourself. The session manager, along with the XR camera, are the only requirement on Babylons's side to implement an XR experience.

## Basic usage and initialization

To construct a new session manager, initialize it with a Babylon scene:

```javascript
const sessionManager = new WebXRSessionManager(scene);
```

The session manager is now ready and can be initialized to be used with XR.

Before activating the session, you can check if XR is available and the session mode is supported by the browser.

You can do this using the static function `IsSessionSupportedAsync` or the `isSessionSupportedAsync` function of the session manager itself:

```javascript
// Static:
const supported = await WebXRSessionManager.IsSessionSupportedAsync("immersive-vr");
if (supported) {
  // xr available, session supported
}

// or:

const supported = await sessionManager.isSessionSupportedAsync("immersive-vr");
if (supported) {
  // xr available, session supported
}
```

After making sure that XR is available and that the session is supported, you can initialize the session and prepare it for rendering:

```javascript
sessionManager.initializeSessionAsync("immersive-vr" /*, xrSessionInit */);
```

This function will initialize the native session. Without calling this function, no session is available and the XR experience will not work.

Right after, you will need to initialize the reference space of this session, which will define the coordinate system that the xr experience will use:

```javascript
const referenceSpace = sessionManager.setReferenceSpaceTypeAsync(/*referenceSpaceType = 'local-floor'*/);
```

The only thing that is left right now is to prepare the render target and the [XR WebGL Layer](https://developer.mozilla.org/en-US/docs/Web/API/XRWebGLLayer):

```javascript
const renderTarget = sessionManager.getWebXRRenderTarget(/*outputCanvasOptions: WebXRManagedOutputCanvasOptions*/);
const xrWebGLLayer = renderTarget.initializeXRLayerAsync(this.sessionManager.session);
```

The session manager is now ready to render the scene using the XR session.

## Start rendering

To start rendering the scene use the `runXRRenderLoop` :

```javascript
sessionManager.runXRRenderLoop();
```

The scene will now render to the XR Device instead of the browser canvas.

To stop rendering, end the session:

```javascript
await sessionManager.exitXRAsync();
```

## Reference Space management

### Managing yourself

WebXR coordinate system is based on a reference space object that defines the user's current transformation in the scene. This object calculates the current HUD and controllers transformation, and is also required for all AR features.

Babylon manages the reference space changes for you when you use the teleportation, so you don't have to deal with it yourself. If, however, you wish to access and change the reference space on your own, you have full access to three reference space objects that will help you:

1. `xrSession.referenceSpace` is the current reference space, the one being used to render the scene.
2. `xrSession.baseReferenceSpace` is the initial reference space generated using the `ReferenceSpaceMode` you chose (like `local-floor`).
3. `xrSession.viewerReferenceSpace` is the initial reference space in the `viewer` reference space mode, compensated with the `defaultHeightCompensation` value.

Using the `getOffsetReferenceSpace` function on the Reference Space object you can change the reference space yourself. Don't forget to update the new reference space in the session manager, otherwise it will not be used in the current scene:

```javascript
// move the player 2 units (meters) up:

// height change - move the reference space negative 2 units (up two units):
const heightChange = new XRRigidTransform({
  x: 0,
  y: -2,
  z: 0,
});
// get a new reference space object using the current reference space
const newReferenceSpace = xrSession.referenceSpace.getOffsetReferenceSpace(heightChange);
// update the session manager to start using the new space:
xrSession.referenceSpace = newReferenceSpace;
```

Read more about [`getOffsetReferenceSpace`](https://www.w3.org/TR/webxr/#dom-xrreferencespace-getoffsetreferencespace).

### Using a new reference space mode

To get a reference space object using a different `XRReferenceSpaceType`, use the `xrSession.setReferenceSpaceTypeAsync` with the requested reference space type. This will reset the viewer and base reference space as well. This function accepts a new reference mode and is promise based:

```javascript
// Session was created using the `local` reference space mode, let's move to `local-floor` :
const newReferenceSpace = await xrSession.setReferenceSpaceTypeAsync("local-floor");

// The session will take care of updating all reference space objects (base, viewer, and the current one).
```

## Observables available

### onXRFrameObservable

Observers registered here will be triggered on each XR frame rendered. Can be compared to the `beforeRender` of the scene, but executed only when the scene is rendered in XR.

### onXRSessionInit

Will be triggered when the session was initialized, before a frame was rendered and before any transformation was applied.
In XR terms, this is called right after `requestSession` was called and returned with a successful result (a new XRSession).

### onXRSessionEnded

Will be triggered after the XRSession has ended and Babylon finished with clean-ups. It is triggered one frame before the regular scene render function is triggered.

### onXRReferenceSpaceChanged

Will be triggered every time a new reference space was applied to the current scene. This is a good way of finding our if the user transported to a new location.

## Further functionalities

### Manually start rendering the XR session

When creating the experience yourself and not using the [WebXR Experience helper](/features/featuresDeepDive/webXR/webXRExperienceHelpers) you will need to tell the session manager to start rendering. To do that, call the `runXRRenderLoop` function.

The `runXRRenderLoop` will update the render function with the required XR rendering mechanism and run the engine's render loop.

Note that the FPS rate will probably change, as most if not all XR devices render in more than 60 FPS.

### World scale

If your scene is not set in meters, or you want to change the player's size in your scene you can use the world scaling factor in the session manager.

To do that, change `worldScaleFactor`:

```javascript
xeSessionManager.worldScalingFactor = 2;
```

This will make the world 2 times bigger than it is.

So, if, for example your scene is in feet instead of meters, you can set the world scale to:

```javascript
xeSessionManager.worldScalingFactor = 3.2808;
```

The value can be changed on each frame, meaning you can also animate it. If you want to know when the world scale has changed you can use the observable:

```javascript
xrSessionManager.onWorldScaleFactorChangedObservable.add((payload) => {
  // old value:
  payload.previousScaleFactor;
  // new value
  payload.newScalefactor;
});
```



# WebXR

The WebXR API enables developers to create VR and AR experiences for the web. XR (Extended reality) has unified both Augmented Reality features and Virtual Reality features into a single API that works across all supported devices.


While the Babylon team strives to maintain API backward compatibility, unfortunately it was not possible to have back compat *and* adopt modern tech for the V2 Babylon Viewer. One of the primary challenges is that V1 used [Handlebars](https://handlebarsjs.com/) which allows attributes and elements to be used interchangeably. For example, all of the following are equivalent with Handlebars:

```html
<babylon camera.behaviors.auto-rotate="0" />

<babylon>
  <camera behaviors.auto-rotate="0" />
</babylon>

<babylon>
  <camera>
    <behaviors auto-rotate="0" />
  </camera>
</babylon>
```

Handlebars was an OSS tech that predated [Web Components](https://developer.mozilla.org/en-US/docs/Web/API/Web_components/Using_custom_elements), the web standard that covers custom elements. As part of both modernizing the Babylon Viewer and shrinking bundle sizes, we wanted to move to web standards that are already supported by the browser. However, Web Components Custom Elements requires attributes to be statically declared, so the Handlebars attribute/element model cannot practically be retained.

This means some manual steps are required to migrate from Babylon Viewer V1 to V2. We don't currently have a migration guide or tooling, but if you run into problems migrating, please reach out on the [Babylon forum](https://forum.babylonjs.com/c/questions).
