Virtual Pilot's Hand - Mouse Pointer as a Cockpit Device

If you are using an Head-Tracker or VR, you will have experience the issue of trying to operate a Clickable-Cockpit Switch or Dial while trying to hold you head still enough to allow you to put your mouse pointer on the switch or dial and hold your head steady long enough for you to operate said switch or dial. This can be a very challenging task, especially in a combat environment, in which you have very little time to complete such activities, and your attention is also required elsewhere.

 

With many Head-Tracking configurations you have a function to freeze head-tracking, so that your view is locked in position, allowing you to manipulate the mouse pointer within the display without having to coordinate with your head's motion. From my experience, this can lead to head-tracking becoming maladjusted for which you have to re-center your head-tracker. All of this are necessary steps you must contemplate and carry out, which can often be required at moments, when your attention is also required for other important activities.

 

My request is for a Virtual Pilot's Hand which is moved by the mouse, but only relative to its position in the cockpit. in other words, where you are looking with VR or Head-Tracking is irrelevant. Only mouse movement controls the VPH, which would allow the Virtual Pilot to easily position the VPH at a switch or dial or whatever to operate it, while being able to glance anywhere he wishes, without losing the VPH'es position.

For starters you're doing it wrong if you have to click around during combat. Fence in properly and switch everything you need in time. Either put everything you need into your HOTAS config or don't use it in the heat.

VPH is good idea. Actually there are working virtual gloves already.

But there are possible problems. Much work to do if you want it modeled realistically, with limits of the reaching hand (not only the reach of the hands and switching hands for left and right panels, but also G limits and disconnecting the controls this hand operates while clicking, but what if you don't have RL HOTAS setup?) and proper graphics for pilot body and moving hands.

Edited July 16, 2018 by draconus

Okay, madly trying to hit a switch in combat is probably a bit over-dramatic :D

 

None-the-less having a large number of switches available is not a solution. It's also a question of ergonomically applying them, and that across multiple aircraft. Just trying to harmonize switches between the P-51 and the Spitfire is problematic. I'm already using Modifier "Switches" which have their own issues, to be able to ergonomically assign controller switches to cockpit switches.

 

It has been noted for years that players are unhappy with the mouse pointer flying around with the virtual pilots view. It's a real issue, and not player friendly.

 

Virtual gloves, when they finally work properly in DCS, will offer a solution to the issue, but at what cost... no literally COST as in $$$.

 

ED is partnered with CaptoGlove to develop their VR gloves for use in DCS. You can already purchase a pair... for $490(!!), but they don't work with DCS out-of-the-box yet. That's nearly as much a my VR HMD setup!

 

Here's a YT video from Jabbers, who tried them out at the E3:

 

 

 

So, whether VR gloves will actually be a general solution is still a big question.

i think a decent solution could be to have the mouse move on an invisible sphere around the pilot (the sphere moving with the aircraft, not the camera) and have a "pointer" projected to the outside from the position of the mouse.

 

interaction would be, where the "laser" pointer hits the cockpit/button etc.

 

so the mouse would not be fixed to the cockpit geometry, which would come with its own problems, but it would still be stabilized to the cockpit.

a little bit like using the laser-pointer functionality of the oculus controller in dcs...

 

by the way... i don't see that much potential in those glove solutions everyone is excited about, as they would need your sim-pit/hotas-setup to perfectly mirror the plane you are flying. otherwise you would constantly collide with your hotas, your table or rig/mount etc.

 

so i think having effortless pointing mechanics is more important than "true" touch integration (in vr)...

Edited July 16, 2018 by twistking

i think a decent solution could be to have the mouse move on an invisible sphere around the pilot (the sphere moving with the aircraft, not the camera) and have a "pointer" projected to the outside from the position of the mouse.

 

interaction would be, where the "laser" pointer hits the cockpit/button etc.

 

so the mouse would not be fixed to the cockpit geometry, which would come with its own problems, but it would still be stabilized to the cockpit.

 

That sounds like a suggestion I've heard before--not to take anything away from your suggestion, only noting that others think the same.

 

Basically, in my words, the mouse would move the focus from one cockpit instrument to the next ONLY. So the mouse would always have a defined focus, and that is really what the player requires. Then the switch or dial at the focus could be actuated with <LMB>, <RMB>, or <Mouse-Wheel>.

 

a little bit like using the laser-pointer functionality of the oculus controller in dcs...

 

lol I've never tried that, because wearing the Touch Controllers while operating the HOTAS will be kind of clunky, literally, but I think I'll have to try it once, just for fun :D

 

by the way... i don't see that much potential in those glove solutions everyone is excited about, as they would need your sim-pit/hotas-setup to perfectly mirror the plane you are flying. otherwise you would constantly collide with your hotas, your table or rig/mount etc.

 

That's true, but I think it might be solvable. One way would be to have DCS define exactly where your hands can be to operate throttle and joystick--the HOTAS Zones. As long as your hand is in the HOTAS Zones, it cannot interact with the cockpit. Move your hand away from the HOTAS Zone, and you interact with the cockpit. Only when your hand grasps the joystick or throttle again, is interacting with the cockpit turned off.

 

Ofc that will also have issues, like how do you reach switches "behind" your HOTAS Thottle/Joystick, or even more basic, what if objects around your chair block your physical hand from reaching cockpit switches.

 

Just to the left of my chair I have a set of drawers, on which my HOTAS Throttle sits. This set of drawers would likely block my hand from reaching a lot of switches and levers in the warbirds - - I'm glad I started this thread. The more we discuss solutions and their drawbacks, the more we learn :thumbup:

 

so i think having effortless pointing mechanics is more important than "true" touch integration (in vr)...

 

I'm really starting to think you are right about this.

 

I really love the idea of being able to reach out with my hand to operate something. The other day I was sitting in the P-51 and wanted to "turn the mirror on" (flip it down), and my first impulse was to reach up with my hand to flip it down (VR is awesome :D ). I think having that experience in DCS, that you can reach out with your real hand to operate something, would be fantastic.

 

But if using VR gloves in DCS begins with the instruction--like in Oculus setup--clear out a playing area... I think we're going to have some major issues. Maybe the only viable solution will be to, as you noted, require your joystick and throttle to be physically coordinated with their positions in the VR cockpit, and for nothing else to be in the way, and I don't think that would be possible for many, if not most VR pilots.

lol I've never tried that, because wearing the Touch Controllers while operating the HOTAS will be kind of clunky, literally, but I think I'll have to try it once, just for fun :D

You definitely should. It works quite well actually. of course in its current form, it is more of a gimmick, since it is neither practical nor immersive to cosntantly switch between hotas and vr-controller. I don't want to derail this thread, so i will make another post in the vr-subforum about this, but i actually plan to develop a small vr controller, that you could wear on your finger and wrist while using a hotas and that would enable simple pointing controls with simple gestures for "click" and "scroll".

 

I really love the idea of being able to reach out with my hand to operate something. The other day I was sitting in the P-51 and wanted to "turn the mirror on" (flip it down), and my first impulse was to reach up with my hand to flip it down (VR is awesome :D ). I think having that experience in DCS, that you can reach out with your real hand to operate something, would be fantastic.

But if using VR gloves in DCS begins with the instruction--like in Oculus setup--clear out a playing area... I think we're going to have some major issues. Maybe the only viable solution will be to, as you noted, require your joystick and throttle to be physically coordinated with their positions in the VR cockpit, and for nothing else to be in the way, and I don't think that would be possible for many, if not most VR pilots.

I agree with all of this. I also think that "true" touch interaction would be the gold standard, but it comes with so many problems, that i don't see it getting fully realized for "normal" home users.

To be honest i also suspect that ED are interested in this glove technology more as a "casual" alternative to hotas systems and less as an upgrade for enthusiasts.

However i think some true touch interaction could be implemented per aircraft. F.e. mirrors, canopy cranks, ejection seat handles... single elements that are physically big and in positions where they are unlikely to collide with your real world setup.

Switches and buttons would still be operated with the "laser-beam" you could shoot from your finger-tip.

This would of course require some sort of lightweight vr-controller, be it a glove or a small device that simply tracks your hand/index finger and allows for simple gestures...

Edited July 17, 2018 by twistking

Good points here and hopefully ED are paying attention and picking up some potentially good ideas and solutions from this thread.

There is so many problems and questions that may actually explain why this kind of cursor haven't been done yet...

 

Making the cursor being able to stay in place within a bubble only solve a single issue which is X and Y axis rotation of the camera, it could also be projected into Z axis, but where it start to become complicated is axis translation, if your cursor is over "S" switch and you transition the camera in a lateral way, if the cursor is projected into a sphere, it will simply move away and now be over another switch.

The solution would be to project it against the 3D cockpit, but it cause two issues, the first is now, do the player really meant to put it here against the seat, or should we consider it as exactly over that switch 3 pixel away (then the player move and the cursor is in a radically different position) and also projecting the coordinate of the cursor into a 3D object will be way more complex than using a simple coordinate system, you would need to use a method close to ray tracing.

Tho it is only for a single cursor and the performance hit would be impossible to notice.

Also what if the cursor follow the cockpit, but we move from left to right making where it was now hidden under another object, the joystick, another switch, the pilot body, etc etc...And should the cursor track the general control location or should it stay on the same place (placing it on the top or on the base of a switch).

 

The cursor lock also cause the same location issue, tho it can be a little helped by binding the cursor location on screen to the location of the control (the same can also be used for cockpit locking the cursor but we now need to address it over a polygon) which is not as simple to address as a control depending on how DCS is coded...But now should we have cursor locked over control and fixed otherwise or locked and who stay in the same place in the cockpit.

Because when hovering a control, if turned on a label with the name of the control appear, but now if we consider the mouse to be either cockpit or screen locked, it either undesirably lock over control we didn't wanted because of a slight movement of your mouse caused by your desk & joystick making your cursor to be gone further from where you excepted it to be, or if screen locked as now, it will cast unwanted label when getting over controls...

 

Also cockpit fixed cursor cause the issue of loosing the cursor, we should not have to remember where the cursor was, so what if it was in a weird location the last use and we try to find it by moving it left and right, but since it may be on the pilot's back, the cursor move but is nowhere to be found...

 

Also how should the cursor act against the interface now, etc etc, so many issues...

I personally want to to this :

http://www.instructables.com/id/DIY-Data-Glove-V2/

Having a left click button, a right click button, a scroll wheel and a button to basically switch the cursor ON/OFF, (so when I'll put my hand on my HOTAS, the cursor won't move like crazy on the VR screen, which would be so annoying), also a simple calculation between mouse location and headset attitude may with some maths allowing for some quick script on Glovepie or little software in C or VB.NET for example to having it on a fixed place in the cockpit, this is my personal solution for this, and it also fix the issue of having to deal with a mouse while in VR.

 

But for now, my proposal for how the cursor should act :

Fixed over the same part of the cockpit until it leave the field of view (avoiding both loosing the cursor or the annoying cursor moving everywhere causing labels to flash), and then, when moving the mouse, the cursor appear on the centre of the screen and again is cockpit locked, except of course if hovering over an interface element (Briefing, escape menu, radio when they will be clickable etc).

Also, if you put, lets say, the cursor against the cockpit floor, but move your head against the canopy in a position where another element get over it, the lock location will always be switched, except if the cursor was over a control like a button or rotary switch for example.

I use a head tracker and also have the "bobble head" problem, so I have a switch that will toggle freeze/unfreeze the head movement.

Remember that the mouse cursor is already projected on the cockpit in a 3D position. This is apparent when playing in VR. In my opinion the cursor position in 3D place is not a issue. The only issue I personally have is with the mouse moving with the viewport.

 

The mouse position could easily be tracked by having an arrow at the edge of the screen pointing towards the cursor when outside the screen. The arrow would disappear just like the cursor right now when the mouse remains stationary for a couple of seconds.

 

 

I do agree with the problem of VR gloves versus the physical positioning of HOTAS controls. To expand on it, the problem is not only where they are physically... repositioning is - depending on the setup - "relatively easy". BUT. Different aircraft have different ranges of motion and types of motion for their controls... they might have Significantly larger movement ranges (especially throttles!) or different types of hinging (two single axis hingepoints in stead of a single ball gimbal)... So even if you were able to reposition the controller bases to a proper position, the handles themselves may move in a completely different manner to the real deal.

 

 

Regards,

MikeMikeJuliet

I've already addressed some of these issues. First off, it's not a mouse cursor, it's mouse pointer. A cursor is the marker used in text editors etc.

 

The problem is, the mouse is a Windows object, and as a Windows object it stays in it's position within the Window (your entire monitor(s) when in full-screen mode), regardless of what's going on graphically behind the Mouse Pointer.

 

So if turning your view, either through moving your head while wearing a VR HMD or TrackIR "moves" the cockpit around the Window, the Mouse Pointer remains in it's position relative to the Window.

 

The only way to fix this is by having an entity, independent of the Window, which follows the cockpit, like a Virtual Hand. If you don't move the Virt. Hand, it remains in it's position relative to the cockpit.

 

You could make the Virt. Hand free moving, which means you would have to move it to the switch or dial, much the same as you would move your own hand. Or you could have it jump from one switch or dial to the next, always being locked onto one or the other, which might be easier to use, but ultimately it would be a question of preference.

 

There are no easy solutions. And we are talking about two different things; mouse pointers and VR-Gloves, both of which have their own issues and solutions.

Does anyone know how to adjust the mouse pointer brightness? Night landings are a trick with that bright glowing orb in the middle of your view!

As long as the solution does not incorporate disembodied hands...

Best way to do it is the BMS way IMO. You move the mouse slowly around a clickable spot and it "latches" to it, if you keep moving the mouse it unlatches and moves. As a result its much easier to hit switches and buttons without needing to be perfectly still. Pretty simple solution.

Best way to do it is the BMS way IMO. You move the mouse slowly around a clickable spot and it "latches" to it, if you keep moving the mouse it unlatches and moves. As a result its much easier to hit switches and buttons without needing to be perfectly still. Pretty simple solution.

 

If I understand correctly, until the mouse pointer gets close enough to a 'clickable spot' it's just free to move anywhere. Is that right?

 

My question would be, is there any purpose for it to ever move freely? After all, if it's not latched to a 'clickable spot', it basically has no function at all. It's just floating about.

 

I'm just learning the AV-8B currently, and a lot of switches etc are extremely difficult to get the pointer onto, especially on the extreme far left and right. This is partly due to being in VR, in which is is very difficult to even turn your head around and down to see the dang switches. It's such a friggen PITA to get the mouse just right over them, when you just want to set them quickly an be done with it.

 

For cryin' out loud!! There's got to be a better way!

In addition, I wish mouse cursor would rather jump (optionally) from a switch to switch etc than how it is gliding now. So for example, you move the mouse right and the cursor jumps to whatever closest button/switch/etc at the right of where the original position is. Just like a magnet :). And then stays there regardless of a head tracking.

That's kind of what I think would work best. It would be very easy to use.

 

In fact, because in some cockpits, switches and dials are not necessarily easy to even find, being hidden behind throttles, in dark corners etc., have a frame defined around each switch. build the frames together into a checker board framework, in which sometimes the 'squares' are not equal sized and evenly placed, because they follow the positions of the switches and dials, which are also not always placed in even rows and equal intervals.

 

The frame around the active switch could be yellow, and around immediately neighboring switches they could be white (or what ever variation), making it easy to find the latched on switch and how to get to the next switch.

 

Life in any DCS cockpit could be made so much easier :thumbup:

any progress in this area? VR pointing device support in clickable cocpits or at least "freeze the blue cross head-tracking " to help use mouse or any other substitution?

I've never heard ED even acknowledge that there is an issue, so unless I've completely missed it, I know of no progress with this in any way.

"Progress" being questionable . I like very much the current system , which allows very fast ufc entries , fine-tuning of cursor position with head movement , and a disappearing cursor when when i don't need it .

 

I don't want disembodied hands , or lasers in my cockpit .

I would simply like the option to turn off the head tracked pointer and have it solely operated by the mouse. For instance once you have located say a rotary control you don't have to keep looking at it to operate it.

"Progress" being questionable . I like very much the current system , which allows very fast ufc entries , fine-tuning of cursor position with head movement , and a disappearing cursor when when i don't need it .

 

I don't want disembodied hands , or lasers in my cockpit .

 

I have no idea how the current system allows for very fast anything. What? because turning your head and moving your hand in the same direction causes the pointer to move twice as fast? Twice as fast and twice as uncontrolled so that you then have to stop and start to adjust the pointer position before you get near the switch you need to operate. Moving the pointer with the mouse alone would be far quicker, because far more controlled.

 

It is exponentially (literally) more difficult to use the current system, because you move the pointer with both your hand and your head simultaneously, and pointing with your nose is completely unnatural and leads to neck tension, which is unhealthy. And as you have already attested, you can only fine tune your input by freezing the movement of either your head or your hand, which will always be a possible cause for error and attests to using one or the other is by far superior than both.

 

No one said anything about a virtual hand (a la Thing from the Adam's Family) creeping around your cockpit (although I'm sure there are some who read this and are already thinking - KEWL!! lol).

 

My thoughts are more along the lines of a frame or highlighting around the switch being addressed with a dead zone so that it doesn't easily jump to another switch unintentionally. It would only be moved with the mouse (NOT through head movement).

 

There could be many solutions to it not being in view the entire time. Have it fade out after a few seconds of inuse. Drag it onto the stick or throttle to simulate grasping one of those and then the carrot blinking out.

 

So many simply solutions; all 100 times better than the acrobatic mouse games we have now.

I would simply like the option to turn off the head tracked pointer and have it solely operated by the mouse. For instance once you have located say a rotary control you don't have to keep looking at it to operate it.

 

The problem is that DCS uses windows to control mouse movement, and windows is only concerned with the mouse pointer coordinates within the video frame (ie your monitor). Windows doesn't care if there are virtual buttons anywhere on the screen, or the DCS is slewing those buttons across the monitor, controlled by your head movement.

 

So to have a pointer relative to its position within the virtual world cockpit, the pointer must be inside the virtual world cockpit ONLY, and be completely unconcerned with the windows mouse pointer.

The problem is that DCS uses windows to control mouse movement, and windows is only concerned with the mouse pointer coordinates within the video frame (ie your monitor). Windows doesn't care if there are virtual buttons anywhere on the screen, or the DCS is slewing those buttons across the monitor, controlled by your head movement.

 

So to have a pointer relative to its position within the virtual world cockpit, the pointer must be inside the virtual world cockpit ONLY, and be completely unconcerned with the windows mouse pointer.

Ok but the pointer moves with my mouse and with my head movement. I would like the option to switch off the link to head movement. When I click on the mouse button to bring up the pointer it is always in the last place it was (usually bottom right as that is where the "Fly" button sits in the UI). So not centred. I just want it to stop moving with my head.

Ok but the pointer moves with my mouse and with my head movement. I would like the option to switch off the link to head movement. When I click on the mouse button to bring up the pointer it is always in the last place it was (usually bottom right as that is where the "Fly" button sits in the UI). So not centred. I just want it to stop moving with my head.

 

I'm with you. That's basically what I'd like. I'm just asking for a but more comfort added to the basket.

 

Basically, I don't need a pointer. I need a carrot (that's when the input focus can be caused to jump from one input field to the next, like filling out an online forum and pressing <Tab> to jump to the next field). I don't need to be able to put the pointer on some random point between switches or somewhere in the cockpit. I mean, I can't scratch my knee with it, and it is of no use unless it's on an input field. No one is in the cockpit with me, looking over my shoulder for whom I need to point things out to. I need to operate switches and dials and things. I need to designate which of those switches and dials and things I'm operating at any given time, and using a mouse is very flexible for designating, but only if you can control what it's pointing to easily.