32 bit normal maps

[twistking]

@Taz1004 I want to make a dedicated bug report in the dcs main bug section, but want to gather some further knowledge on texture management in realtime 3d.

 

How is the texture quality setting in the options expected to work? When tested on the A-10c II (stable version) i did not see any difference between high and medium settings. Would you expect the engine to reduce original textures when loading the mission, or is this done done on the fly during runtime, or would one expect different sizes pre-authored in the game files?

 

I want to do some more testing: For example in the p-51D the visual difference between high and medium textures is significant (medium looking completely blurry already), while in the A-10C is see no difference at all and did not notice and performance difference neither.

[Taz1004]

@Taz1004 I want to make a dedicated bug report in the dcs main bug section, but want to gather some further knowledge on texture management in realtime 3d.

 

How is the texture quality setting in the options expected to work? When tested on the A-10c II (stable version) i did not see any difference between high and medium settings. Would you expect the engine to reduce original textures when loading the mission, or is this done done on the fly during runtime, or would one expect different sizes pre-authored in the game files?

 

I want to do some more testing: For example in the p-51D the visual difference between high and medium textures is significant (medium looking completely blurry already), while in the A-10C is see no difference at all and did not notice and performance difference neither.

 

I don't have SDK so I could be wrong. I could dig a bit deeper by doing some tests but will take some time. So I'm only speculating based on the files that I can see.

 

For some things like terrain texture, they do have smaller "min" version. Cockpits do not.

So for cockpit texture, I believe they're using mip levels to adjust texture quality. DDS files basically store multiple resolution of the texture in single file and these are called mip level. You can view these mip levels if you open the DDS files with Visual Studio and change mip levels.

 

Then the game (or GPU) chooses the mip level based on the resolution, distance, AA, etc. So Medium and Low cockpit texture setting in DCS probably is forcing lower mip level.

Downside of doing this over using smaller file version like terrain is that this method will save bandwidth but will not save VRAM.

Upside of course is not having to manage multiple file versions.

 

This is rather long but good read at Nvidia developer diary regarding how mip level works and how to optimize.

https://developer.nvidia.com/gpugems...el-measurement

 

"Do your users ever get to see your highest mip level?" (Cebenoyan 2004). If the answer is no, performance can be enhanced and memory can be saved by swapping those unused mipmap levels out of video memory.

 

The method in that Nvidia dev diary (false-color) however is rather time consuming and I doubt ED (or any developer) really does this. Most just use common sense and estimate. But good read for explaining mip nonetheless.

 

 

And I want to reiterate this before it gets off track too far.

Lowering diffuse texture is more of personal preference. Some may never see fine prints on side of Maverick and they would want lower diffuse. Some making 4K video footage might want those detail. I want to reiterate that original post of this thread was about Normal maps. Not Diffuse. And about them being 32 bit.

[twistking]

Thanks for the insight.

 

ps: I forgot to state in my question that i was specifically asking because of the Normal maps, but i assume they are handled the same way as Diffuse (I agree with you on the matter of diffuse texture size btw.)

[davidp57]

Hi Taz, thanks for all the work you do for us on DCS ! I specifically love your trees ^^

 

Is there a way we can lobby ED into working on this texture optimization ? I don't know, mass mail them, make a list of all users who would like them to work on this and send it to them ?

If there is, I'm in !

[Taz1004]

1 hour ago, davidp57 said:

Hi Taz, thanks for all the work you do for us on DCS ! I specifically love your trees ^^

 

Is there a way we can lobby ED into working on this texture optimization ? I don't know, mass mail them, make a list of all users who would like them to work on this and send it to them ?

If there is, I'm in !

 

I doubt it.  If a line of code causes 1GB of VRAM, it'd be considered a bug.  But if textures do it, then it's a "Wishlist"

[Knock-Knock]

59 minutes ago, Taz1004 said:

 

I doubt it.  If a line of code causes 1GB of VRAM, it'd be considered a bug.  But if textures do it, then it's a "Wishlist"

Well put. I cant believe they havent jumped on this yet.

[Pikey]

Hi Taz! @Taz1004
I'd like to get behind getting this to the right people in Eagle Dynamics but there are some barriers! One of them is the technical part, once we go through google translate, technical detail tends to get messed up. What we need is to draw the link between:

Lower bit textures == more DLC sales == reason to invest money to change something

 

The link involves showing the direct link between good performance and textures being too large and consuming too much memory.

There's a lot of folks that can happily jump on the bandwagon and point their fingers at ED, but I'll bet not many of them truly understand what this thread is about. My own understanding is super super low. Even if this is the cause of stutter in VR, how much? How do you tell texture artists that they must spend another 5 (number pulled out of arse) man days per year per person creating optimised textures, if we cannot articulate the benefits of these changes, we stand much less chance on the English language forum. What I am mostly afraid of is asking for this and having a texture artist answer back, "We provide Low quality cockpit textures already" and I dont know what to say because I dont understand how to talk this language.

People love to leap to conclusions. I think there is a lot of interesting data here but I can't say quantitively myself, what performance anyone would expect to see. In 2D I get more FPS than I need. In VR is this more important? I think so but I couldnt explain it. How many people can explain it convincingly? None, apparently, that I've met

I need help. Whats the bottom line? What data could we show ED? Why does the F-14 Tomcat seem to kill my VR? Quantitatively and empirically.

This is a wishlist item because its not a defect, its an optimisation. A defect is the texture is missing, it's working fine, this is wishlist because its performance based. I can increase my VRAM in the mission by introducing new units, is that a bug too? It's not of course. I doubt the Russian texture artists pour over these forums, why does everyone seem to think they do? How many people read to the end of this post? Not many. It's easy to moan. It's hard to convey what is really needed and its even harder to persuade people to part with money for no apparent gain, so don't be too surprised if your hard work isn't put in shining lights Be reaslistic, lets take the good work, make a simple message to ED, provide the data for the conclusion and present it? I am up for that, but I'm screwed if I had a conversation on it because I do not understand the true value of what you suggest. What could we do?

[Taz1004]

5 minutes ago, Pikey said:

A defect is the texture is missing, it's working fine, this is wishlist because its performance based. I can increase my VRAM in the mission by introducing new units, is that a bug too?

 

Hey Pikey.  That is not a good example because you're trading a feature for VRAM.  In this case, no feature whatsoever.  Just wasted.

Like I've said before, I've never even heard of using 32bit for normal maps.  I'm seeing this on their old textures and I'm seeing this in their new textures.  If anything at all, they need to stop doing this.

 

The time thing, I don't agree with either because it just took me about 1 hour to optimize majority of the useless textures.  Including ground units and terrain.  This is not difficult thing to do.  And dev's know exactly what those textures are.

 

My system is on the lower end of spectrum at Core i7-4790K and 1080 (not Ti) with 32GB Ram.  I'm playing VR and have no stutter with these optimized textures even in Syria.  At 45fps.  Dips to 30fps at Beirut but still no stutter.

I don't play multiplayer but I'm willing to bet this is big part of the issue.  Because whenever someone enters the multiplayer server with different plane, you're loading gigabytes of their plane's texture.  And from what I can see, are not released when they leave the server.

[Taz1004]

This is good simple explanation of the difference.  The swords in the middle of the article is good example.  Are the differences big enough to sacrifice gigabytes of VRAM?  And that example is comparing 8bit to 16 bit.  Not even 32 bit.  And it is on very reflective chrome material with no diffuse texture.  Once you put diffuse textures (like the bottom sword), these bumps for the most part will disappear and only work as supplement to provide specular highlights.  There is growing need to use 32 bit for diffuse texture.  But not for normal.

 

https://polycount.com/discussion/148303/of-bit-depths-banding-and-normal-maps

[Lurker]

We need someone fluent in Russian and English and reasonably technically proficient who would be able to present this info to Eagle Dynamics. 

[Pikey]

Hi @Taz1004 You will have to forgive me as I dont understand a lot of the background or technology about what you are saying. I'd like to take this up with ED.
Whats the feature that we are losing? I'm not sure I understand still.
Can you elaborate on not hearing of <someone> using 32bit normal maps? is this in all gaming or some other sector that uses textures? Any examples? I don't really study this type of thing so it's good to share your experience.
What are the pros and cons of ED doing this? There must be a con, right? Is this for free? I heard you said something about adjusting these and costing 1hr. Is this the true cost? are we missing something? So if I said, it would cost a maximum of 3 (?) days to optimise all the games textures then this is something that could be true?

You said you dont get stutter with your setup. But you didnt explicitly say with only your textures or what your baseline config is. Is that only Cockpit textures, can you be clear about what it takes you to be without stutter and what other changes do you make on your system? I assume its the same but im not confident making an assumption as you dont provide your baseline configurations.
Releasing or loading textures is a definite thing, can we measure that process on graphics cards? Would I see a jump in texture memory use in MP if a Tomcat joined? And... what really happens when you run out of memory, texture load is one source of stutter, right? Disk>GPU. How does it unload? I have always lived in 100% full territory and I use Medium Cockpit and Low terrain texture. If the root cause of stutter is texture load/swap, then the hypothesis of High texture usage is good if it causes stutter, but I dont see it all well end to end.

[Taz1004]

 

Quote

There must be a con, right?

 

Con of using 8 bit normal maps?  No.  I already said there's no benefit to using 32 bit normal maps.

 

Quote

Whats the feature that we are losing?

 

I didn't say anything about losing feature?  Perhaps you misunderstood.  You compared this issue with trading VRAM with new ground unit.  I said this is different because you're losing "VRAM" for nothing.  Not feature.

 

Quote

Is that only Cockpit textures

 

I already said including ground units and terrain textures.  And you don't need my base configuration.  This is compared "With everything else being equal" on my first post.  My configuration is irrelevant.

 

This is not difficult subject and most of your questions, as I pointed out, I've already answered.  You can read the article I linked.  That is the simplest explanation I found.

This is not something I have to convince someone.  This is that obvious.

You may not understand it but dev's definitely understand.  I have to believe this is oversight.  Oversight that has been piling on for years.

 

Fixing it is easy.  List textures by size.  You can immediately tell which are 32bit normals by file size.  Open them all in Photoshop.  Save.  That's it.  You can batch process it and it will all be done automatically.  Time you spend calculating the time into cost will actually take longer.

Edited January 19, 2021 by Taz1004

[Snakedoc]

 

 

Quote

A defect is the texture is missing, it's working fine, this is wishlist because its performance based. I can increase my VRAM in the mission by introducing new units, is that a bug too?

Quote

Hey Pikey.  That is not a good example because you're trading a feature for VRAM.  In this case, no feature whatsoever.  Just wasted.

Quote

Whats the feature that we are losing?

 

 

I think what Taz meant is that when you load more units you are indeed increasing the VRAM usage, but you're asking for more units to be present to make the mission more interesting/alive/more obj/etc ( = feature ). 

 

 

 

In the case of 32 bit normal maps you are increasing the VRAM usage but with no benefit gained from it compared to the 8 bit, only with a disadvantage of losing available VRAM 

 

 

[Pikey]

 

1 hour ago, Taz1004 said:

 

 

Con of using 8 bit normal maps?  No.  I already said there's no benefit to using 32 bit normal maps.

 

 

I didn't say anything about losing feature?  Perhaps you misunderstood.  You compared this issue with trading VRAM with new ground unit.  I said this is different because you're losing "VRAM" for nothing.  Not feature.

 

 

I already said including ground units and terrain textures.  And you don't need my base configuration.  This is compared "With everything else being equal" on my first post.  My configuration is irrelevant.

 

This is not difficult subject and most of your questions, as I pointed out, I've already answered.  You can read the article I linked.  That is the simplest explanation I found.

This is not something I have to convince someone.  This is that obvious.

You may not understand it but dev's definitely understand.  I have to believe this is oversight.  Oversight that has been piling on for years.

 

Fixing it is easy.  List textures by size.  You can immediately tell which are 32bit normals by file size.  Open them all in Photoshop.  Save.  That's it.  You can batch process it and it will all be done automatically.  Time you spend calculating the time into cost will actually take longer.

 

Hey guys and @Taz1004, i spoke to a Texture artist about this, who spoke to someone at Heatblur who had spoken to one of the artists at ED. Since they didnt speak directly to me I will withold names and minor details.

It turns out this is a deliberate decision and one made with findings that differ from the data asserted in this post. The reason given was compession artifacts and banding the lower the normal maps go, especially on clean and glossy or metallic surfaces.

I have no idea if that is true or not, or why, I don't understand this enough to comment, but I can state that there are multiple points of view on this topic and at least as far as the layman is concerns, the very forceful assertaion that:

 

"This is not something I have to convince someone.  This is that obvious."

 

is unfortunately not true in this case, apparently you do have to convince Eagle Dynamics.

It's not for me to decide, I do have a personal point of view that prefers the lower bit textures and accepts artefacts, but im not the PM and I dont know if there is a reasonable solution. I don't know how so many textures could coexist in the installation, but for the cockpit textures, the choice of both would be fantastic as there is most likely a large performance benefit.

[Taz1004]

30 minutes ago, Pikey said:

It turns out this is a deliberate decision and one made with findings that differ from the data asserted in this post. The reason given was compession artifacts and banding the lower the normal maps go, especially on clean and glossy or metallic surfaces.

 

That is just misinformation.  If that in fact did come from ED or HeatBlur, they're more ignorant than I assumed.  I thought this was just oversight.  But I'm gonna assume the person you talked to talked to talked to doesn't know what he/she's talking about.

 

As shown on the sword example, yes it is true you will see some artifacting on reflections of HIGHLY reflective material.  There are no such materials in DCS.  There are no reflections in DCS.  People are complaining about fake baked reflections and they're using 32 bit normal maps for glossy, metallic surfaces?  Something that doesn't exist in the game?  Normals maps in DCS are used for bevels and bezels and rivets.

 

But even if that is true, perhaps 16 bit.  Not 32 bit.  That article clearly shows there's no difference whatsoever between 16 and 32 bit in quality EVEN ON GLOSSY METALLIC SURFACES.  And if this was by design, why not all the normal maps?

 

I don't have to convince Eagle Dynamics.  Eagle Dynamics has to convince their customers why they're wasting valuable resources if this in fact is the conscious decision.

Edited January 19, 2021 by Taz1004

[Pikey]

 

5 minutes ago, Taz1004 said:

 

That is just misinformation.  If that in fact did come from ED or HeatBlur, they're more ignorant than I assumed.  I thought this was just oversight.

As shown on the sword example, yes it is true you will see some artifacting on reflections of HIGHLY reflective material.  There are no such materials in DCS.  There are no reflections in DCS.  Normals maps in DCS are used for bevels and bezels.

 

But even if that is true, perhaps 16 bit.  Not 32 bit.  That article clearly shows there's no difference whatsoever between 16 and 32 bit in quality. 

 

I don't have to convince Eagle Dynamics.  Eagle Dynamics has to convince their customers why they're wasting valuable resources.

 

I'm sorry that you appear upset about this, I'm sure you could still do good work in helping others tune the textures in their offline installations to work better. It would be nice if we had a "lite" choice in DCS.
There's not a rapid rate of change with Eagle Dynamics, they do need help from the community in knowing where to put their energies, but its an absolute shame that you aren't willing to help them understand what is for the best.
At least you got some good results though!

[Taz1004]

12 minutes ago, Pikey said:

 

I'm sorry that you appear upset about this, I'm sure you could still do good work in helping others tune the textures in their offline installations to work better. It would be nice if we had a "lite" choice in DCS.
There's not a rapid rate of change with Eagle Dynamics, they do need help from the community in knowing where to put their energies, but its an absolute shame that you aren't willing to help them understand what is for the best.
At least you got some good results though!

 

Why would I be upset about this?  It is frustrating however because someone who claiming to not know anything about the subject keeps trying to debate about this citing unnamed source.

And as I said, they know the issue.  You didn't read that article did you?  You should read the whole article but I'll quote one line.

 

Quote

What about in game? 16 and 32 bit file formats are simply not supported (or at least not commonly used due to texture memory constraints) by most game engines. So, at the end of the day you most likely need to output a 8 bit file.

 

This simply is something that is not practiced in game industry.  You learn this in school.  If they hired texture artist for games, they KNOW.

Edited January 19, 2021 by Taz1004

[Pikey]

Just now, Taz1004 said:

 

Why would I be upset about this?  It is frustrating however because someone who claiming to not know anything about the subject this keeps trying to debate about this.

And as I said, they know the issue.  Trust me.  You didn't read that article did you?  You should read the whole article but I'll quote one line.

 

 

This simply is something that is not practiced in game industry.  If they hired texture artist for games, they KNOW.  Trust me.

I'm debating nothing, I reported the answers. Have a good evening and try to relax more.

[Taz1004]

3 minutes ago, Pikey said:

I'm debating nothing, I reported the answers. Have a good evening and try to relax more.

 

I think you're reading posts assuming the tones based on how YOU are feeling?  I use bolds and CAPS to emphasize points.  I'm not yelling.

Edited January 19, 2021 by Taz1004

[RuskyV]

I'm wondering with the DCS install size now approaching a rather large size would actually save some space if you start to include other objects that could also benefit from Taz's discovery. Seems to me that there are some fairly considerable gains to be had not only in performance but possibly in storage size? 

 

Not directly related but when I started digging around in the DCS install I noticed a fair few legacy hold overs from the very early DCS days, things like unused icons and old GUI interface art that should have really been cleaned out along time ago. I'm wondering if all this could really do with refresh and some old files being deleted and trimmed out. probably wouldn't save much but at least it would be a little less messy

[MiG21bisFishbedL]

It'd be really nice if ED actually did an optimization pass that included this.

[twistking]

i want to add to the discussion between @Taz1004 and @Pikey:
i am not a fulltime professional 3d artist, but i do some simple 3d work for my job and also do 3d modelling as a hobby. so, i'm not "properly" trained on this, but have experience with the subject nevertheless.

for me, it's unbelievable that ED has still not done anything about it. the texture sizes are a joke honestly and while it would be fine and dandy to have those extreme sized textures as an optional download for people with 12gb+ vram cards, having those as standard is really not acceptable. i have a very long thread in the bug section about vram bottlenecking that describes in detail how dcs cloggs up gpus with less than 8gb of vram. after single core cpu performance, gpu vram limtiation seems to be the main factor for vr performance problems and the reason why you efectively need more than 8gb of vram for vr. now look at the gpu market: 8gb is still the standard with nvidia and going higher comes with a serious price tag.

 

most importantly i want to add to @Pikey 's question about what reason there might be for the current situation. i think taz is right when he said earlier, that the current size is just how the textures got exported. there has simply not been any optimization at all. i think @Taz1004 makes it a little bit too easy when he says that batch processing all files would be sufficient, because if done properly, you would maybe want to check every texture in game to make sure that it stll looks fine and that it is as low as possible (some textures may look good with even higher compression/size reduction, depending on reflectivness detail, sharpness etc.)

 

but even if this whole process would take one dev multiple days (which i do not believe) it would still mean huge increases in performance on low vram cards, noticeable improvments in vr even on mid/high vram cards and smaller install size for dcs.

[Taz1004]

8 hours ago, twistking said:

most importantly i want to add to @Pikey 's question about what reason there might be for the current situation. i think taz is right when he said earlier, that the current size is just how the textures got exported. there has simply not been any optimization at all. i think @Taz1004 makes it a little bit too easy when he says that batch processing all files would be sufficient, because if done properly, you would maybe want to check every texture in game to make sure that it stll looks fine and that it is as low as possible (some textures may look good with even higher compression/size reduction, depending on reflectivness detail, sharpness etc.)

 

Yes, this is very obvious even for non experts.  I can't imagine how ED or experienced texture artist wouldn't understand this.  It's like having to explain to engineers at General Motors why regular oil change is important.  Which is why I said this has to be an oversight.  If this was intentional, then all normal maps should be 32 bit.  But it's always a portion of a full texture pack.  I see them in Viper too.  But I do not see them in HeatBlur or Ugra textures.

I almost want to guess that someone at ED has wrong export setting as default.  It's obvious that few textures got exported with wrong setting and they didn't think it mattered.  And it didn't matter.  But over the years, those got piled up.  And still getting piled on.

 

I want to clarify it tho that when I said this is easy to do with batch process, I meant converting 32bit normal maps to 8bit.  Not resolution.  And for that, you don't even need to test them.

 

I did optimize resolution of textures for myself too and yes, that is slightly more involved process because that does affect visual quality and you have to look at what you're actually resizing.  Whether if it's for instrument gauges or seat belts and seatbacks you never look at.  That is why I did not suggest optimizing "resolution" on this thread.  Only 32bit normal maps.

Edited February 21, 2021 by Taz1004

[Pikey]

@twistking The reason was provided by a texture artist from Heatblur. Choosing not to believe that is a valid reason is fine. But I would ask everyone that doesnt beleive their reason is valid to consider why, not only ED, but most of the third party developers are doing the same with the textures. Have they all got group hallucinations? Why would something so unbelievable only be unbelievable for people who aren't creating modules for DCS?
For me,  I never tested how much the distortion is really apparent, others here claim its not a factor, so for me, its like choosing between several companies that have people working on this or someone that makes a counter claim. I pick the majority until I become able to do better for myself, but I realise that is an uneducated position. Still, I don't claim any of this, from either side, is unbelievable. Nothing is unbeleivable and that word keeps going around.

[Taz1004]

46 minutes ago, Pikey said:

but most of the third party developers are doing the same with the textures. Have they all got group hallucinations?

 

Which 3rd party module?  I haven't checked all 3rd party modules but I've seen none in F14 or Syria.  Tell me which and I'll check them out on next free trial.

 

Or perhaps have that Heatblur texture artist actually chime in here for productive discussion?

 

What you're saying is entire game industry other than ED are hallucinating.

Edited February 22, 2021 by Taz1004