Virtual instrument cluster (second) display question

[grajen3]

Hi!

(this not exactly a feature request, but more of a question about feasibility)

Would it be possible to modify Man-in-the-middle approach, so that it announces support for second display (ref https://developer.android.com/training/cars/testing/dhu#instrument-cluster for what I'm talking about), such that aa-proxy would continue to send main display, events, sound etc normally and just "capture" second display video stream and "just" pipe it elsewhere (like some socket, file handle or something like that)? And then have separate custom app consume it and ... do something with it?

Extra question - when Android Auto is connected wirelessly - can other devices connect to aa-proxy device/wifi? Seeing https://github.com/aa-proxy/aa-proxy-rs?tab=readme-ov-file#embedded-web-interface

Use a different device, such as a laptop or another phone, to connect to the device’s Wi-Fi network.

it seems like a "yes", but maybe there is nuance here specific to web interface

I'm mostly looking for some insights here. Currently I have almost 0 context on inner working of AA protocols and contracts between devices and I just wonder if this idea has any feasibility before I potentially waste a lot of time trying to learn stuff needed to possibly make this happen. I would not expect feature like this to be included here because it's very specific and niche (unless you would be interested in it if I manage to get something to work ... eventually hah).

If you do have any specific pointers - I of course would be interested in those, but please don't waste too much time on those :)

Some background for motivation behind my question in case my questions above make 0 sense:
There is this project https://github.com/OneB1t/VcMOSTRenderMqb that allow to "stream" video/images to instrument cluster of Volkswagen group cars (well, certain generation of them that is). I played with it with attempt to re-create Android Auto navigation there with some success ( OneB1t/VcMOSTRenderMqb#2 some examples heh). This was accomplished with basically recreating Google Maps view with custom Android App using google's Navigation SDK ( https://developers.google.com/maps/documentation/navigation/android-sdk ), screenshotting it and then streaming that to instrument cluster. This kind of works, but it's far from ideal:

• I never found great way to grab actual's Google Map navigation route and I've been using trip sharing to separate google account, which my app was later pulling from (using same Google Maps private API that https://github.com/costastf/locationsharinglib is using). That worked, but android auto seems to allow sharing with single account (as opposed to sharing from the phone normally), so I could never share with anyone else. And I had to have to manually start sharing each time with that account which was annoying
• I needed messy setup when using AAWireless to be able to both be able to use wireless AA and stream to instrument cluster. It relied on using "experimental" feature of AAWireless 1 of using 3rd party Wifi (~client mode). This experiment seems like was removed with some AAWireless update, so I did lose this option recently when I updated my AAWireless device and the only way I can continue to use it now is either using usb connection for AA + wifi for instrument cluster streaming or maybe try to downgrade my AAWireless firmware.
• My solution was ... just not very efficient - with doing so much stuff inefficient stuff on the phone, I very often was getting into overheating territory on my phone and things would just randomly start getting killed, so this is rather ... frustrating experience on long trips ( while I could just do this on different device than my main phone which would run AA, I still would need some mobile device to use Navigation SDK - I don't really have spare and buying one just for that other than extra cost also would come with its own set of problems related to battery with either that phone dying or my car's batter dying when I would not use car for few days and this is still overall a "workaround" rather than nice solution and trying to make use of AA builtin second display seems generally like much better approach to me)

[korhojoa]

Hi, I'm currently trying to do exactly that. (and add exlap/openpilot fetching for some ev data so I can free up the wican)

I can stream any of the displays with my fork and am trying to get adding the cluster or navigation view to the advertised services so the HU doesn't notice anything.

I've been looking at doing this so I can stream it to VcMOSTRenderMqb, just to put it in the VC :)
It should be possible to decode the video fine on the HU, but it might also be possible to stream the video directly to the VC, as it is h264 and that's what the VC accepts from the mib anyway...

ps. Are you in the aa-proxy discord or the VcMOSTRenderMqb telegram chat?

[grajen3]

It should be possible to decode the video fine on the HU

When I was playing with it some time ago I came to (possibly very wrong) conclusion that might be tough, because software decoding might be just too much to ask of HU's cpu and for accelerated decoding I didn't find any useful material for APIs/SDKs that could be used, so I did abandon this track. But as I said - maybe I was just wrong, but stuff like this is really just not my area of expertise so don't let my answer here discourage you from exploring it!

but it might also be possible to stream the video directly to the VC, as it is h264 and that's what the VC accepts from the mib anyway...

Oh interesting, I didn't know that, but for what I want to do it might not be enough to stream as-is, because I do want to overlay some extras there (seeing you mention exlap, that's part of extras I already did add in my custom android app - oil temp/fuel consumption for me, plus some other integration - like showing if my garage door is opened/closed and I wanted to preserve all that), so that's why I wanted to capture the AA second video stream on aa-proxy device - do some extras there (seperately) and then send it all to VC.

But after you mentioned this and I thought about for a moment - I do recall that windowing system maybe had overlays so maybe instead map view could be send as-is and my extras would be seperate "window" on top 🤔

--edit:

Main thing I appreciated when doing the custom android app is that I could test it all in front of my computer - so I wanted to minimize need to tinker with car itself as much as possible because I do know that VcMOSTRenderMqb works and I can test a lot of stuff this way - so doing most of the work on aa-proxy device would allow me to continue doing that. It's not very fun for me that much anymore to have to go to my car to test and iterate. It might not be the most efficient final solution, but as long as it would work - that would be good enough for me and much easier to do shorter tinkering sessions this way

ps. Are you in the aa-proxy discord or the VcMOSTRenderMqb telegram chat?

Ah, no. Too hard for me to keep track of things. I think I joined some telegrams like that in the past (not sure if those concrete ones), but for the most part there was just a lot of "user support" spam there and almost none of development chats heh so they were distracting me more than helping.

Plus I only have time to tinker with this stuff during some weekends. In fact I didn't touch this stuff in few months heh because life got in the way and it's not like this is priority in my life - it's more to satisfy my inner-tinkerer while also adding some minor quality of life improvement to my car heh

[manio]

Sure - let’s see what comes out of all this. For now, @korhojoa has managed to get the stream working to VLC, which is awesome for me :)

Either way, you’re welcome to join us on Discord - it’s definitely a more casual and direct way to chat, but of course, totally up to you 🙂