Media Device API

[saghul]

Ahoi!

(Somewhat related to #10)

I've been trying to do something which sounds simple like selecting the audio / video device I want to use before calling getUserMedia. I didn't know I was in for that much nonsense.

AFAIS there are at least a couple of ways browsers do it, navigator.mediaDevices.enumerateDevices being the new and shinny preferred way.

rtcninja provides getMediaDevices, which is a wrapper around, MediaStreamTrack.getSources, alas this doesn't work on Firefox nor on Chrome >= 47 (current Canary).

The problem is that depending on how the devices are enumerated the constraints need to be specified differently, so rtcninja would also need to accommodate for that.

This is something I can probably work on and provide a patch, but there are a couple of things that need clarifying first:

• do we want to shim navigator.mediaDevices, or just provide a custom getMediaDevices API function?
• since we are here, should it return a Promise? (I see promises are supported after FF 38 (latest ESR release) and Chrome 32, so we're fine)

Maybe this is a good start to implement the promise based API?

[ibc]

Let separate promises (#10) and this issue. I agree that we need to decide the exact API for "get media devices". The W3 spec has changed a lot along the time, but still Chrome is the only browser implementing it, and it implements an old version of the spec...

So this is very hard.

The problem is that depending on how the devices are enumerated the constraints need to be specified differently, so rtcninja would also need to accommodate for that.

May you detail it a bit more? If you mean "getUserMedia constraints", then yes, that is a paint and I'm pretty sure that the current Chrome implementation is not based on the latest spec.

[saghul]

The W3 spec has changed a lot along the time, but still Chrome is the only browser implementing it, and it implements an old version of the spec...

Not really. FF >= 40 (haven't tested previous versions) already have the new navigator.mediaDevices.enumerateDevices API. Chrome 45 doesn't, but has MediaStreamTracks.getSources, which rtninja exposes. Chrome 47 (current canary) has both.

The syntax to select what device one wants seems to be different:

(old)

        { audio: { optional: [{sourceId: audioDeviceId}] },
          video: { optional: [{sourceId: videoDeviceId}] },
        },

(new)

{ audio: true, video: { deviceId: videoDeviceId } }

(I haven't tested the new constraints mode yet though)

[mwittig]

Despite the deviceId attribute of the MediaDeviceInfo interface there a few more differences, which must be taken into account. I am referring to the the latest public TR published April 2015:

• the type of the kind attribute is different. It is a MediaDeviceKind enum with the following values: `"audioinput", "audiooutput", "videoinput".
• attribute groupId has been added which is also part of the constraints' MediaTrackConstraintSet.

Apart from this, it should be noted the label attribute may be an empty string if "none of the local devices are attached to an active MediaStreamTrack in the current browsing context, and if no persistent permission to access these local devices have been granted to the page's origin, let filteredList be a copy of resultList, and all its elements, where the label member is the empty string." As far as I can tell this is exactly the behaviour implemented with FF42 which is a bit awkward for WebRTC application implementors.

Now, the question is, whether or not a shim should also normalize the device info object. If normalized, one would need a helper function to create an appropriate (browser-specific) constraint object, as @saghul pointed out the constraint syntax has been changed, too. As the application probably wants to add more constraint attributes (e.g. constraints regarding video dimensions) the helper function would also need to deal with this and the whole constraint may get rather complex. I think it is better to let the application do the normalization to keep the complexity of rtcninja as low as possible.

Following this, I think rtcninja should shim enumerateDevices separately from getMediaDevices. Regarding Promises I think rtcninja should provide "promise-neutral" interface for now to support legacy applications. Thus wrapper for the new interfaces like navigator.mediaDevices would be nice.

[ibc]

Thanks for the input. This will be done once a globla refactor is done in the lib.

I think rtcninja should provide "promise-neutral" interface for now to support legacy applications.

That is not the idea. Instead rtcninja will just offer the newest API to the app, while it will be ready to handle older API browser implementations. I strongly think that is the way to go.

[mwittig]

That is not the idea. Instead rtcninja will just offer the newest API to the app, while it will be ready to handle older API browser implementations. I strongly think that is the way to go.

Ok. If enumerateDevices and getMediaDevices will be separate functions in rtcninja I agree with you. This way, it is probably good enough, to provide navigator.mediaDevices through rtcninja, if the browser supports it, and there are no differences accross browsers. The latter may be too optimistic - I have not tested Chrome 47 yet.

[ibc]

rtcninja will provide rtcninja.mediaDevices and standard API functions within it. Whether such API can be the native or a polyfill depends on the browser.