No true headphone 3D audio in games!

RELATED THREADS:

AMD TrueAudio - ED Forums

No headphone-dedicated mixing mode - ED Forums

Example: OpenAL HRTF implementation

A quick primer:

1. True 3D audio (3D positional audio, 3D spatial audio) allows blind location of the sound source in 3D space, so e.g. 'the sound is comming from my 8 o'clock, 60 degrees high'
   
2. True 3D audio is achievable on regular stereo headphones!
   
3. You can enable true 3D audio for any game using OpenAL right now!
   
4. Vast majority of games offer fake 3D, i.e. not 3D, implementations like 5.1, 7.1 surround, which is essentially a 2D panning. Even implementations such as Dolby Headphone or Creative CMSS-3D are inferior
   
5. 
   

To put it bluntly we're being shafted with poor in-game sound implementations inferior even to some technologies available (and widely used!) in late 90's :)

The main articles on the subject

http://matthewkerswill.com/i-want-hrtfs-in-my-games/

An article with examples

http://steamcommunity.com/discussions/forum/10/864980278184895534/

Guides on how to enable HRTF true 3D positional audio in some games

http://steamcommunity.com/id/wildcat/myworkshopfiles/?section=guides

I higly recommend starting with Amnesia The Dark Descent :Sly:

Reddit link:

A quick example:

lqqiIR017SQ

Or, somethinh like this is possible in-game:

u163wC6mP2A

What happened to 3D audio from the 90's?

One of the key elements was this - Creative suit vs Aureal3D

So you might reconsider your future soundcard purchase :sly:

Dolby Headphone is not just "2D panning" It uses head related transfer function.

In the example vid the effect is way to pronounced and the sound behind the listener sounds wierd and muffled. Dolby Headphone actually does it better.

Dolby Headphone is not just "2D panning" It uses head related transfer function.

 

In the example vid the effect is way to pronounced and the sound behind the listener sounds wierd and muffled. Dolby Headphone actually does it better.

I didn't say DH is 2D panning. As for disadvantages of DH - refer to the article I linked.

Well, DH is not intended to put you in the game environment. Instead it puts you in a simulated room with a simulated speaker system. It does this by stereo channel crossfeed, reverb and HRTF. That can be an disadvantage I guess but at the same time it has advantages too.

The disadvantage: Does not put your head in the environment of the game

The advantage: Works with anything (not just games) that outputs multichannel sound.

I have heard both good and bad implementations of DH. For example on my laptop with crappy soundcard there is a checkbox for DH on/off. This is the bad implementation. The good one is on my desktop with Esscence STX soundcard and HD650 headphones. The soundcard has DH built into it's drivers and it allows for custom DH settings, not just on/off. This means I can tell the DH software how many channels it should use, what kind of room I want to sit in etc. When used right, this setup can be very convincing without actually doing much unnecessary damage to the sound.

A game itself has to be encoded for multi-channel information.

5.1 or 7.1 is actually pretty good, if the game implements it's advantages fully.

But the sad truth of the matter is you will find that it's very far from most developers objectives to have a "true 3d" soundfield, not at the very least because it would require much computing overhead. It's not only angle of incidence that gives audial spacial awareness cues, but the reverberant environment of the "world" itself.

A game itself has to be encoded for multi-channel information.

I think it's not really constructive to spit anything one has in mind re in-game audio. The original post says multi-channel is not required for true 3D and that multi-channel rarely provides true 3D.

OpenAL HRTF / Rapture3D uses in-game actual positions of sound sources etc. and it does not require much computing power.

So I suggest at least a 5 minutes read before posting.

But the reverberation of the in-game world does require more computing time.

It's very simple to pan a sound to one of two speakers in stereo, and say the sound is coming from it. Everything that happens in between the two speakers is an illusion (http://en.wikipedia.org/wiki/Phantom_center)

You are right to say headphones can provide a "true 3d" soundfield (what is that defined as to you, btw?). Headphones do not have a phantom center, and when a sound is panned to one side, only one ear will hear it, and this is not a normal hearing sensation unless something is very close to the ear.

The "true 3d" you are talking about only mimics the HRTF (http://en.wikipedia.org/wiki/Head-related_transfer_function) for an anechoic environment (http://en.wikipedia.org/wiki/Anechoic_chamber), what the ears will hear from a sound some distance with no outside acoustical influences.

This does provide directional cues to locate audial events, but that's it.

Any type of reverberation added after this should be able to simulate the environment that the viewer is seeing. Reverberation can make locational cues less accurate. Simply simulating the reverberant conditions of one room is not walking around in a world full of sound.

@camsr

Please don't get me the wrong way. The current situation is that in vast majority of games audio sucks as far as positioning / true 3D audio goes because they don't even employ OpenAL HRTF. I've started this thread to shed some light on the subject and I simply refuse to extensively discuss OpenAL HRTF shortcomings. It's like criticising virtual reality goggles for not providing sense of smell when we don't even see VR goggles adopted yet.

IMO it's blurring the subject. People still think they have 3D sound in their games or that they get the best sound the current technology allows. Let's change that first.

By "True 3D" I mean implementations providing elevation information (Z coordinate/height) using in-game information on sound source locations, not some gimmicks. So at least OpenAL HRTF or equivalent.

If you don't believe the situation is bad go ahead and look up 'AMD TrueAudio' and look how many gaming journalists actually discuss whether we need it. Ridiculous.

From the HRTF wiki:

"The perceived accuracy of the result depends on how closely the HRTF data set matches the characteristics of one's own ears."

IOW, one size HRTF does not fit all, absolutely. Does OpenAL's program make any considerations for this?

From the HRTF wiki:

"The perceived accuracy of the result depends on how closely the HRTF data set matches the characteristics of one's own ears."

 

IOW, one size HRTF does not fit all, absolutely. Does OpenAL's program make any considerations for this?

Yes. You can 'calibrate' to better match your head. Please see the OP update, the reddit link.

Yes I read the links and the OpenAL manual.

It's very interesting, the whole idea of positional audio, even if I think it is fundamentally flawed by lack of proper acoustical reverberation.

When I listened to the Rapture3d video, it was off by a mile. Not the actual positioning, but the frequency balance was all wrong (to me). I guess that is the pinnae "filters" tuned wrong for my ears. If I wanted to spend an hour or so listening to the pre-compiled set of HRTFs at the OpenAL website, I could probably find something close.

The flaw here is their notes on each HRTF test lack the proper data to choose by numbers, and use the power of observation.

Yes I read the links and the OpenAL manual.

It's very interesting, the whole idea of positional audio, even if I think it is fundamentally flawed by lack of proper acoustical reverberation.

When I listened to the Rapture3d video, it was off by a mile. Not the actual positioning, but the frequency balance was all wrong (to me). I guess that is the pinnae "filters" tuned wrong for my ears. If I wanted to spend an hour or so listening to the pre-compiled set of HRTFs at the OpenAL website, I could probably find something close.

The flaw here is their notes on each HRTF test lack the proper data to choose by numbers, and use the power of observation.

Yeah, the documentation is poor in some areas, so I've heard.

A particular implementation also shouldn't discard a given 3D technology. For example the latest Thief game featuring AMD's TrueAudio sounds highly unimpressive in its current state, with clear reverb errors etc.

It could be that in-ear type headphones are required for the best experience. Sound bypassing the pinna this way should make the not-so-accurate HRTF more bearable.

It could be that in-ear type headphones are required for the best experience. Sound bypassing the pinna this way should make the not-so-accurate HRTF more bearable.

Since the driver sits coaxially to the ear canal in most closed headphones, you shouldn't have too much influence by the pinna anyway. But depending on what HRTF you are using, in ear speakers might give you best results, especially if using your own HRTFs that were taken with in-ear mics.

Edit: So c0ff, if you ever happen to incorporate binaural spatialization, please make HRTFs customizeable, i do happen to have mine laying around. :)

Since the driver sits coaxially to the ear canal in most closed headphones, you shouldn't have too much influence by the pinna anyway. But depending on what HRTF you are using, in ear speakers might give you best results, especially if using your own HRTFs that were taken with in-ear mics.

 

Edit: So c0ff, if you ever happen to incorporate binaural spatialization, please make HRTFs customizeable, i do happen to have mine laying around. :)

You mean you've prepared your own custom HRTF profile? How does one get to it?

You mean you've prepared your own custom HRTF profile? How does one get to it?

I made an internship at a research institution in vienna that has the equipment to take them, and they do take the chance to measure every intern for their HRTF database. They also took fotos because they have a project running to create HRTFs from 3d models of heads in term generated from (2d) pictures (it sounds crazy but it works).

You can see the facility in the two pictures at the bottom.

http://www.kfs.oeaw.ac.at/index.php?option=com_content&view=category&layout=blog&id=67&Itemid=391&lang=de

It incorporates a loudspeaker array that is used for elevation angle. Azimuth angle is accomplished by turning the chair the subject is sitting on in small increments. So for every azimuth angle, the response is measured for all elevation angles with frequency sweeps. The response is measured by in ear micrphones inserted into the ear canals. Additionally they use a magnetic head tracking system to ensure the head is orientated correctly in regard to the chair and the loudspeaker array respectively.

To speed up the process they measure the response by sweeping several elevation angles at the same time (but out of phase, so sources can still be identified correctly).

You sneaky bastard! :) Thanks for the info. Really interesting. So you're an audio professional?

Professional or not if you know some gaming audio trends please drop a line or two here http://forums.eagle.ru/showthread.php?p=2036600

You sneaky bastard! :)

If i ever work there again, i'll set you up for a measurement. :) Maybe for my master thesis.

Thanks for the info. Really interesting. So you're an audio professional?

I am a student of electrical and audio engineering. Professionally i work in the automotive sector but yes, i have a background.

Professional or not if you know some gaming audio trends please drop a line or two here http://forums.eagle.ru/showthread.php?p=2036600

Will do.

The problem with reverberation being dynamically generated in a video game is a tradeoff between accuracy and computation. The reverberant characteristics of a 6 sided box are relatively straight forward, but add an interface to another 6 sided box (like a doorway between rooms), or a large profiled surface, and the problem gets deeper. There are also material considerations like what the room is constructed of, and honestly, even level designers don't usually know :lol: they aren't being paid to architect real-world-functional environments, they are paid to make game-world-functioning ones.

So what I am trying to get at is positional audio is good, but dry at the same time and in many ways unnatural, being anechoic.

This is a problem similar to all disciplines that deal with fields. Once the boundary conditions differ from the most simple problems, the field can not be solved analytically. The sound field of a run off the mill room with furniture in it is way too complex to solve analytically, and the statistical methods are too slow to apply them in real time, at least for now. You just can't calculate realistic impulse responses based on the geometry of the room in games right now, even les if they are to differ with your position in the room.

Greetings. I'm the author of the article, guides, and the first of the two Youtube videos that the topic starter posted. I found this forum when I was looking at my website stats. I'm very interested in this topic so I thought I'd come and join the discussion and respond to some of the comments.

Dolby Headphone is not just "2D panning" It uses head related transfer function.

Indeed, Dolby Headphone does use head-related transfer functions but I would argue that the accuracy of its sound localization is analogous to a two dimensional plane. Consider the following:

• 3D is perceiving left-right, front-behind, and above-below.
  
• 2D is perceiving left-right, and front-behind.
  
• 1D is perceiving just left-right.
  

A surround sound system (eg: 7.1 or 5.1) has speakers placed around the listener at head height but there are no speakers above or below. This means that these popular surround sound systems actually provide 2D audio because sounds can only be localized on a flat plane at head height.

Dolby Headphone takes a 7.1 or 5.1 channel audio stream as its input and converts it to a 2 channel audio stream for headphones. Therefore, you won't be able to hear the difference between a sound coming from above or below with Dolby Headphone. That's why I consider it 2D.

Without the use of Dolby Headphone, most games effectively provide 1D audio for headphones because they don't implement the psychoacoustic cues that provide the sensation of front-behind and above-below.

... not at the very least because it would require much computing overhead. It's not only angle of incidence that gives audial spacial awareness cues, but the reverberant environment of the "world" itself.

The "true 3d" you are talking about only mimics the HRTF (http://en.wikipedia.org/wiki/Head-related_transfer_function) for an anechoic environment (http://en.wikipedia.org/wiki/Anechoic_chamber), what the ears will hear from a sound some distance with no outside acoustical influences.

This does provide directional cues to locate audial events, but that's it.

Any type of reverberation added after this should be able to simulate the environment that the viewer is seeing. Reverberation can make locational cues less accurate. Simply simulating the reverberant conditions of one room is not walking around in a world full of sound.

I disagree that sound localization is reliant on environment reverberations. "Spacial awareness" is different from 'sound localization'. Providing the player with an awareness of the proportions and shape of the environment is the purview of reverberation effects. It would be nice to have more sophisticated reverberation calculations but that's not really relevant to the subject of 'sound localization'.

The use of HRTFs is all that is needed to provide a more accurate sensation of a specific sound's direction.

Also, the implementation of HRTFs is well within the boundaries of what can be achieved on current computer hardware. This has been proven by excellent implementations like Rapture3D and OpenAL Soft.

...I simply refuse to extensively discuss OpenAL HRTF shortcomings. It's like criticising virtual reality goggles for not providing sense of smell when we don't even see VR goggles adopted yet.

Hehe, I really like this analogy.

When I listened to the Rapture3d video, it was off by a mile. Not the actual positioning, but the frequency balance was all wrong (to me). I guess that is the pinnae "filters" tuned wrong for my ears.

Rapture3D provides 6 different HRTF modes. These are provided so that you can choose the one that sounds best to you.

That video was the first of several demo videos that I created. At that time, I chose the 'Green' mode but subsequently decided that I preferred the 'Red' mode. Perhaps you should listen to some of my other videos and see if any of them sound better to you.

iikd_hucZbY

(Examples with both 'Green' and 'Red' mode. Music clips and sound effects are used.)

zx4k8qewC_8

(Only 'Red' mode is used. Demo in S.T.A.L.K.E.R.)

f4PJ68LHU-I

(Only 'Red' mode is used. Demo in Minecraft.)

I've recently discovered that Youtube may have a very subtle degrading effect on the quality and accuracy of the HRTF, so just be aware of that possibility.

Red mode sounds decently better. What's the difference?

Some critisism of your videos....

Disable music in the game shots.

Don't turn around constantly like a one-footed drunk. Point and wait a bit.

Red mode sounds decently better. What's the difference?

Different HRTF data sets were used for these two modes. The 'Green' mode uses a processed form of the CIAIR HRTF data set (http://www.sp.m.is.nagoya-u.ac.jp/HRTF/database.html) whilst the 'Red' mode uses a processed form of the MIT KEMAR data set (http://sound.media.mit.edu/resources/KEMAR.html).

Some critisism of your videos....

Disable music in the game shots.

Don't turn around constantly like a one-footed drunk. Point and wait a bit.

Thanks - I'll take this into account if I make any more videos in the future. However, there's a strong possibility that I won't ever make any more (at least not for Youtube) because I have reason to believe that Youtube's audio compression messes with the phase of the sound and detracts from the interaural time differences that form a part of the HRTFs.

If you do make any more, please do share them :)

Youtube will destroy audio mercilessly. It's not only Joint Stereo Compression that does it, but the codec itself. WAV or FLAC would be a better choice.

I don't know how to encode videos with lossless audio, so you may have your work cut out for you.

This is a problem similar to all disciplines that deal with fields. Once the boundary conditions differ from the most simple problems, the field can not be solved analytically. The sound field of a run off the mill room with furniture in it is way too complex to solve analytically, and the statistical methods are too slow to apply them in real time, at least for now. You just can't calculate realistic impulse responses based on the geometry of the room in games right now, even les if they are to differ with your position in the room.

Real time calculation of complex boundary conditions is not yet feasible.

I gotta admit, maybe I bit off more then I can chew, because I am uncertain it would even be an augment to the gaming experience.

Thomas N. gives a good synopsis of the importance of sound reflection in his music studios.

http://prorecordingworkshop.lefora.com/reply/25934260/FrontToBack-Acoustic-Technique#reply-25934260