
## Introduction

The [DeviceSourceManager](/typedoc/classes/babylon.devicesourcemanager) is a class that manages connections for various user input devices and provides methods for querying those devices for their current state.  
This class supports several methods of input:

- **Keyboard** _(DeviceType: BABYLON.DeviceType.Keyboard, Inputs: number)_
- **Mouse** _(DeviceType: BABYLON.DeviceType.Mouse, Inputs: BABYLON.PointerInput)_
- **Touch** _(DeviceType: BABYLON.DeviceType.Pointer, Inputs: BABYLON.PointerInput)_
- **DualShock Gamepad** _(DeviceType: BABYLON.DeviceType.DualShock, Inputs: BABYLON.DualShockInput)_
- **Xbox Gamepad, One or 360** _(DeviceType: BABYLON.DeviceType.Xbox, Inputs: BABYLON.XboxInput)_
- **Switch Gamepad, L+R JoyCon Grip or Pro Controller** _(DeviceType: BABYLON.DeviceType.Switch, Inputs: BABYLON.SwitchInput)_
- **Generic/Other Gamepad** _(DeviceType: BABYLON.DeviceType.Generic, Inputs: number)_

Here's an example of the DeviceSourceManager in use.
<Playground id="#C7PM2B#18" title="DeviceSourceManager Example" description="Simple example showing how to use the DeviceSourceManager in your scene." image="/img/playgroundsAndNMEs/pg-C7PM2B-17.webp"/>

To use the DeviceSourceManager, first create an instance of it. You will need to provide an engine object.

```javascript
const deviceSourceManager = new BABYLON.DeviceSourceManager(scene.getEngine());
```

Within your scene's render/game loop, you can query the DeviceSourceManager for the current state of a specific input. First, you will need to get the DeviceSource object. With this object, you can then query for a specific input's status.

```javascript
// If the device has been registered in the DeviceSourceManager
if (deviceSourceManager.getDeviceSource(BABYLON.DeviceType.Xbox)) {
  // And the A button was pressed
  if (deviceSourceManager.getDeviceSource(BABYLON.DeviceType.Xbox).getInput(BABYLON.XboxInput.A) === 1) {
    // Do something
  }
}
```

You can also use optional chaining to fit the checks into a single line.

```javascript
if (deviceSourceManager.getDeviceSource(BABYLON.DeviceType.Xbox)?.getInput(BABYLON.XboxInput.A) === 1) {
  // Do something
}
```

## Events and Observables

You can use the following Observables to work with a given device's identifiers.

```javascript
// When a device is registered in the DeviceSourceManager
onDeviceConnectedObservable.add((device) => {
  // You can get the device type by using device.deviceType
  // You can also get the device slot by using device.deviceSlot (default is 0 for keyboard and 1 for mouse)
  // "device" is of Type DeviceSource do you can also access the getInput function or add an observable for onInputChangedObservable
});

// When a device is unregistered in the DeviceSourceManager
onDeviceDisconnectedObservable.add((device) => {
  // You can get the device type by using device.deviceType
  // You can also get the device slot by using device.deviceSlot (default is 0 for keyboard and 1 for mouse)
  // "device" is of Type DeviceSource
});
```

Here's a demonstration of how these Observables work.
<Playground id="#Y4YWCD#1" title="Basic DeviceSourceManager Demo" description="Simple demonstration of how the DeviceSourceManager observables work" image="/img/playgroundsAndNMEs/pg-Y4YWCD-1.webp"/>

For keyboards and pointers, you can use an event-based system to get the current and previous input when an input is activated.

```javascript
deviceSourceManager.getDeviceSource(BABYLON.DeviceType.Keyboard).onInputChangedObservable.add((eventData) => {
  // eventData will contain the event data that you'd find in something like a PointerEvent or KeyboardEvent
  // eventData also has an additional value inputIndex
});
```



## Gamepad Manager

First create a GamepadManager:

```javascript
const gamepadManager = new BABYLON.GamepadManager();
```

Listen to the manager's gamepad connection and disconnection events:

```javascript
gamepadManager.onGamepadConnectedObservable.add((gamepad, state) => {});
gamepadManager.onGamepadDisconnectedObservable.add((gamepad, state) => {});
```

## Gamepad events

Once a gamepad is available, event listeners can be added to handle the gamepad's inputs.

```javascript
gamepadManager.onGamepadConnectedObservable.add((gamepad, state)=>{
    gamepad.onButtonDownObservable.add((button, state)=>{
        //Button has been pressed
        console.log(button)
    })
    gamepad.onleftstickchanged((values)=>{
        //Left stick has been moved
        console.log(values.x+" "+values.y)
    })
}
```

## Gamepad types

Depending on the type of controller, handle any of its unique properties:

```javascript no-code-variants
gamepadManager.onGamepadConnectedObservable.add((gamepad, state)=>{
    if (gamepad instanceof BABYLON.Xbox360Pad) {
        gamepad.onButtonDownObservable.add((button, state)=>{
            console.log(BABYLON.Xbox360Button[button])
        })
    }
    if (gamepad instanceof BABYLON.PoseEnabledController) {
        gamepad.onMainButtonStateChangedObservable.add((button, state)=>{
            console.log(button.value)
        })
        const position = gamepad.devicePosition
        const rotation = gamepad.deviceRotationQuaternion
    }
}
```

## Check a gamepad's current state

At any time, a gamepad's current state can be checked with the gamepad's properties:

```javascript
scene.registerBeforeRender(function () {
    if(gamepad instanceof BABYLON.Xbox360Pad){
        if(gamepad.buttonA){
            sphere.position.y+=0.05
        }
        sphere.position.x+=gamepad.leftStick.x*0.05
    }
}
```

<Playground id="#U3XJTB#38" title="Gamepad Example" description="Simple example of enabling gamepad support to your scene."/>



## VirtualJoystick

Create virtual joysticks for the left and right sides of the screen.

```javascript
const leftJoystick = new BABYLON.VirtualJoystick(true);
const rightJoystick = new BABYLON.VirtualJoystick(false);
```

Get the joystick value.

```javascript
leftJoystick.deltaPosition.x;
leftJoystick.deltaPosition.y;
```

<Playground id="#PRQU53#7" title="VirtualJoystick Example" description="Simple example showing how to add a VirtualJoystick to your scene." image="/img/playgroundsAndNMEs/divingDeeperVirtualJoystick1.webp"/>

Note: This will create an overlay canvas on top of the scene canvas. This disables interaction with Babylon GUI elements and scene pointer events. To avoid this, the overlay joystick canvas's z-index can be modified to toggle between scene interaction and joystick input, as shown in the playground above. Another option is to create a custom joystick using Babylon GUI, as described below.

## Custom joystick

To create a custom virtual joystick and modify it to suit specific use cases, the Babylon GUI can be used.

<Playground id="#C6V6UY#5" title="Custom Joystick Example" description="Simple example showing how to add a Custom Joystick to your scene." image="/img/playgroundsAndNMEs/divingDeeperVirtualJoystick2.webp"/>

As seen in this example, no overlay canvas is used, so other GUI elements will continue to function and the visuals can be modified/positioned if needed.

You can also create custom input for a camera. For example, here is a combined input suitable for FPS games, with a movement joystick on the left and a swipe-rotation area on the right:

<Playground id="#MQ9B0X#3" title="Custom Joystick Input Example" description="Simple example showing how to add a Custom Joystick Input to your camera." image="/img/playgroundsAndNMEs/divingDeeperVirtualJoystick2.webp"/>



# Inputs

In addition to the usual keyboard, mouse, and touch input methods, you can also use a gamepad, device orientation, and a virtual joystick.


## What is MRTK for Babylon.js

MRTK stands for the Mixed Reality Toolkit, a set of features, controls, and components designed to ease and accelerate development for VR and AR applications.

Documentation currently exists only for MRTK's [Unity](https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/) and [Unreal](https://docs.microsoft.com/en-us/windows/mixed-reality/develop/unreal/unreal-mrtk-introduction) implementations, so comparisons here refer to those versions.

This page goes over each MRTK-relevant control and feature present in Babylon.js, showing how each can be used. Many of the controls can also be viewed on the [3D GUI page](/features/featuresDeepDive/gui/gui3D), and relevant pages are linked where applicable.

## GUI Controls

The following Babylon.js controls have roots in MRTK features. As they extend the `Control3D` class, they all support the basic capabilities that `Control3D` objects have.

More on the `Control3D` class, and 3D GUI elements in general, can be found [here](/features/featuresDeepDive/gui/gui3D).

The following example showcases several of the controls listed, including `TouchHolographicButtons`, `TouchMeshButtons`, and the `NearMenu`. This scene is designed for use in XR and supports `WebXRNearInteraction`:
<Playground id="#24DLJ4#7" title="Near Interaction Button Scene Demo" description="Demo showcasing different Control3D objects with Near Interaction support."/>

#### Holographic Button

The `HolographicButton` class is the standard, familiar button that can be placed and pressed in a 3D environment. It is based on the MRTK `ButtonHoloLens1` and supports a title, image, and tooltip.

#### Touch Holographic Button

The `TouchHolographicButton` is akin to MRTK's `PressableButtonHoloLens2`. It supports everything that the `HolographicButton` supports, but has more depth to it, allowing for visual feedback as it is being interacted with. As part of this added depth, it now responds to near interactions by having the button surface depress as the user's hand/motion controller presses it.

#### Touch Mesh Button

While the `TouchMeshButton` doesn't have a direct representation within MRTK, it is a button that offers custom visuals and support for near interactions. This allows for any object to react to near interaction, if desired, which aligns with MRTK's experimental `Hand Physics Service`.

Aside from the customization of its visuals, this button behaves in the same way as the `TouchHolographicButton`.

#### Holographic Slate

The `HolographicSlate` is a flat panel that can float in a 3D environment, used to display 2D content in XR experiences.

It consists of a titlebar and content pane. The titlebar has room for a custom title, plus a follow button and close button. The follow button will toggle whether the slate follows the user around or not, and the close button will destroy the slate. The content pane hosts an `AdvancedDynamicTexture`, which is [explained in more detail here](/features/featuresDeepDive/gui/gui).

See it in action here:
<Playground id="#43YQHC#2" title="Slate Scene w/ Label" description="A showcase of different uses for HolographicSlates."/>

#### Near Menu

The `NearMenu` is a `HolographicMenu` with an attached `FollowBehavior` that tries to stay tethered a set distance from the user at all times.

It is a menu designed for use in XR experiences and by default stays about half an arm's length away. It also rotates to try to stay in the user's field of view.

The `NearMenu` also has a pin button that, when pressed, freezes the menu in space. The menu can also be grabbed around the edges to move it (it comes with an attached `SixDofDragBehavior`), at which point it becomes pinned to maintain its new position.

#### Hand Menu

The `HandMenu` is a `HolographicMenu` that is anchored to the user's hand using a `HandConstraintBehavior`.

Its default position lies on the outside of the user's left hand, such that when their palm is up, the menu is facing them and easily accessible by their right hand.

The menu can be customized to appear by default on the right hand or either hand, and can also be anchored to a different quadrant of the hand. Only one menu will be visible at a time though, even if both hands are up. More information on this can be found in the `HandConstraintBehavior`'s section [here](/features/featuresDeepDive/gui/mrtk#hand-constraint-behavior).

#### Slider 3D

The `Slider3D` is a customizable slider with a 3D presence. The default range of the slider is from `0` to `1`, though that can be customized, along with the step count and precision.

The `Slider3D` can be created with a backplate, which is determined on creation. After creation, the `minimum` and `maximum` values can be adjusted, as well as the `step` size.

See it in action here:
<Playground id="#2PKNJB#7" title="Slider Color Changer" description="A basic example showing how 3D sliders can affect their surroundings."/>

## Behaviors and Gizmos

Some behaviors and Gizmos are also shared with MRTK components. In MRTK terminology, these are usually referred to as Solvers, Bounds Control, or Object Manipulators. Behaviors and Gizmos are attached to `Nodes`, `TransformNodes`, and `Meshes` in order to provide additional functionality.

More information on `Gizmos` can be found [here](/features/featuresDeepDive/mesh/gizmo), while behaviors are covered in more detail [here](/features/featuresDeepDive/behaviors/meshBehaviors).

#### SixDof Drag Behavior

The `SixDofDragBehavior` is also known as MRTK's `Object Manipulator` component. By creating and attaching it to an object in the scene, users can grab, move, and rotate the object. Interacting with a ray-cast input allows the object to be moved in 3 dimensions, while near interaction adds the ability to rotate the object in place.

```javascript
const sixDofDragBehavior = new BABYLON.SixDofDragBehavior();
```

To attach a `SixDofDragBehavior` to a mesh in the scene, simply call the mesh's `addBehavior` function:

```javascript
mesh.addBehavior(sixDofDragBehavior);
```

#### Follow Behavior

The `FollowBehavior` tries to maintain a set distance between the `TransformNode` it is attached to and the `followedCamera` target (which defaults to the main camera).

There are many aspects to this behavior that can be customized. These are the core properties:

| Property            | Type   | Description                                                                                        |
| ------------------- | ------ | -------------------------------------------------------------------------------------------------- |
| followedCamera      | Camera | The camera the behavior will try to follow                                                         |
| defaultDistance     | float  | The starting distance to the camera, in meters                                                     |
| minimumDistance     | float  | The minimum distance before the behavior starts to move the attached `TransformNode` further away  |
| maximumDistance     | float  | The maximum distance before the behavior starts to move the attached `TransformNode` closer        |
| verticalMaxDistance | float  | The maximum height offset allowed between the camera's position and the `TransformNode`'s position |

#### Hand Constraint Behavior

The `HandConstraintBehavior` is a behavior that constrains the attached mesh to a specific location on the user's hand. The visibility of the attached mesh can be toggled based on the rotation of the hand and how centered it is in the user's field of view, by setting the `handConstraintVisibility` property.

This behavior takes advantage of the `WebXREyeTracking` feature, allowing the `handConstraintVisibility` property to check whether the user is actively looking at their hand or the mesh. As a fallback, if no eye data is available, it uses the direction the device is facing instead.

The following are the core properties that can be set to customize the behavior:

| Property            | Type                      | Description                                                                                                                                          |
| ------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| targetZone          | HandConstraintSide        | An enum that determines which side/quadrant of the hand the mesh is anchored to and offset from                                                      |
| zoneOrientationMode | HandConstraintOrientation | Determines whether the attached mesh's position should rotate with the hand                                                                          |
| nodeOrientationMode | HandConstraintOrientation | Determines whether the attached mesh should rotate with the hand                                                                                     |
| handedness          | XRHandedness              | Should the attached mesh only stick to the left hand, or right hand? Defaults to none, allowing it to stick to either hand as they become available. |

#### Bounding Box Gizmo

The `BoundingBoxGizmo` is also known as MRTK's `Bounds Control` component. This creates small objects that surround the attached mesh, which can be interacted with to rotate, scale, and [translate](/typedoc/classes/babylon.transformnode#translate) the attached mesh.

## Background Features

These are features that work in the background to enhance the capabilities available to a user.

General XR features are covered in detail [here](/features/featuresDeepDive/webXR/WebXRSelectedFeatures), while AR-specific features are covered in detail [here](/features/featuresDeepDive/webXR/webXRARFeatures).

#### Hand Tracking and Near Interaction

The Hand Tracking feature, `WebXRHandTracking`, is an enhancement to hand inputs for XR devices that support it.

This feature provides articulated joint tracking, hand meshes, and enables the use of touch interactions using near interaction.

The `WebXRNearInteraction` feature is enabled by default on the Babylon playground, and allows for touch interactions when using hands or motion controllers. Touch-supported objects will be able to dynamically react to 3D input, creating a more immersive experience than what basic buttons can provide.

If the `WebXRHandTracking` feature is not enabled, then the near interaction model used will be a visible orb that floats in front of the hand (or motion controller). This orb will only be visible while near a component that supports Near Interaction.

To provide support for Near Interaction to an arbitrary `AbstractMesh`, simply set the `isNearPickable` and/or `isNearGrabbable` properties. `isNearPickable` will enable near interaction in the form of touch input, whereas `isNearGrabbable` will allow the object to be grabbed. The main difference here is that touch input triggers a press on touch, but grabbing the object requires touch _and_ a trigger of some kind (pulling the trigger on a controller, or performing a pinch gesture with a hand). At most one of these options should be enabled for a given object at any given time, to prevent unintended interactions.

#### Eye Tracking

Disclaimer: The eye-tracking feature is only supported with BabylonNative for now.

The Eye Tracking feature, `WebXREyeTracking`, is an opt-in feature for supported XR devices that captures the direction the user's eyes are facing, and passes that along for developers to use.

To use it, simply enable the Eye Tracking feature when setting up a project in BabylonNative.

You then have the option to either listen to an observable, or directly query the component for the latest data. Eye tracking data is always provided as a `Ray`.


Input

Coming next

Virtual Joysticks

Learn about virtual gamepad input support in Babylon.js.

Gamepads

Learn about gamepad input support in Babylon.js.

Device Source Manager

Learn about managing input devices with the Device Source Manager.