WebXR extension / Gesture recognition

Video Overview

Capabilities of the WebXRHandTracking wrapper:

Introduction

User hand gesture recognition can make an application feel more native and allow for controller-free interaction. To enable recognition, gestures need to be captured and prepared in advance, and later recognized within the application.

Typical usage

Capturing

To capture hand shapes, you can use the captureGestureTemplate and captureWristsAngles functions after entering immersion mode.

But in most cases, capturing and configuring gestures is more convenient using the utility: WebXR utility

Nevertheless, it is possible to use the functions in IHandTrackingXRFeatEnv to capture gesture templates.

Recognition

After capturing shapes and constructing combinations, you can use JSON files to describe them. For this, the functions of the IHandTrackingXRFeatEnv interface are useful.

export interface IHandTrackingXRFeatEnv extends IXRFeatEnv<WebXRHandTracking> {
	
	// Repository of gestures (Handshapes)
	readonly gesturesRepo:IGesturesRepo;
	
	// Repository of combinations
	readonly combinationsReposMap:Map<string, ICombinationsRepo>;
	
	// Combinations flow
	readonly combinations$:Observable<TCombinationInfo>;
	
	// Start \ stop recognition
	startRecognize(combinationsRepoName?:string):void;
	stopRecognize(combinationsRepoName?:string):TFigureRecognizeReport | undefined;
	
	// Options setters
	setRecognizeOptions(options:Partial<TGestureMatchOptions>):void;
	
	// ... other functions
}

Example code for recognition using template files:

const options:TXRServiceOptions = {
	mode:'immersive-vr',
	features:{
		HAND_TRACKING:<HandsTrackingConfig>{/* config here */},
    }
} 

const xrService =   new XRService();
xrService.init(options);

const isActive  =   await this._xrService.activateXR();
if (!isActive) throw ('somthing went wrong');

const ftHandEnv =   xrService.extractFeatureEnv('HAND_TRACKING');

// Loading templates
const jLoader       =   new JsonLoader();
const fistTemplate  =   await jLoader.load('/figures/fist-outside.json');
const palmTemplate  =   await jLoader.load('/figures/palm-out.json');

// Add templates to repository
if(
    fistTemplate
    && palmTemplate
){
    // Add to feature
    ftHandEnv.gesturesRepo.set('fist-outside', fistTemplate);
    ftHandEnv.gesturesRepo.set('palm-out', palmTemplate);
}

// Combination
const activateComb  =   await jLoader.load('/combinations/activate.json');

const activateCombRepo  =   new CombinationsRepo();
activateCombRepo.set('activate', activateComb as TGestureCombination);

// Add to feature
ftHandEnv.combinationsReposMap.set('activate-deactivate', activateCombRepo);

// Start listening
ftHandEnv.combinations$.subscribe((comb)=>{
    if(
        (comb.type           ==  'COMBINATION')
        && (comb.repoName    ==  'activate-deactivate')
    ){
        if(comb.descriptor.combinationName == 'activate'){
            // ACTIVATE SOME
        }
    }
});

// Start recognition
ftHandEnv.startRecognize('activate-deactivate');