Source of performance problems

Warning: Task manager does not tell you about the threads.

 

 

What the video demostrates is that a single thread can due to OS thread scheduler normal activity jump around multiple-cores at a high rate, too fast for a human to be able to notice and loss of accuracy due to sampling interval and averaging of the results, creating the same/similar kind of data/visuals (illusion) as if there were more threads using up those cores, (eg each thread per core as it's the usual assumption by the average user). Infact that 1 thread kept in practice only using up 1 core, it never gained any boost whatsoever by jumping between cores, infact if the jump to another core includes a lot bigger distance onboard the chip it self like the AMD's CCX then there will be a latency penalty due to having to copy the cache of the previous core to a new core, (don't quote me on this one right now, it's something like this) and even a greater latency penalty if the move involves having to go to another CCD, and in the future we will be seeing a lot more of these new chiplet designs where more smaller dies will be used, but this really shouldn't worry an average player that much.

 

The thread has a poor title, it seems like it's general, but it's a specific case and VR. Anyway I'll see what it looks like here but I'm 2D not VR.

 

---------------------

 

If it is the AI, in context usually meaning only the "AI logic" which shouldn't be that costly compared to the rest, there may be AI calculations going on those parked AIs, or someone's radar/sensors interacting with them. I'll see what I can do, but keep your tests very identical people, you need to snap a screenshot at exactly the same place when comparing, it'll take some more effort setting a test case up but it's vital for accurate readings.

Edited March 9, 2020 by Worrazen
removed text about CPU 4 cores, mixup

Warning: Task manager does not tell you about the threads.

 

The OP's CPU is a 4 core with 4 hyperthreads so those are not actual 3 cores being used,

 

The thread has a poor title, it seems like it's general, but it's a specific case and VR.

 

--------------------

 

There may be AI calculations going on those parked AIs, or someone's radar/sensors interacting with them. Seems like something's going on, doesn't make sense that it's purely a GPU cost, I'll see what I can do, but keep your tests very identical people, you need to snap a screenshot at exactly the same place when comparing, it'll take some more effort setting a test case up but it's vital.

 

I find your post quite extraordinary. Of all the posts on these forums you criticise the testing methodology of the OP? His post is already superior to the overwhelming majority simply because it includes screenshots and numbers.

 

We are all just users of a game volunteering our time for free on this. I'm not quite sure what you're expecting.

 

...

 

The OP's CPU is a 4 core with 4 hyperthreads so those are not actual 3 cores being used,

I'll take the liberty of responding on behalf of the OP for this one. Yes, he knows.
Edited March 8, 2020 by Hippo
Some posters make it very difficult to comply with rule 1.2

^^ That was a general comment, if necessary, I indicated that with dashed lines, if he already knows it the better. (Extraordinary? I don't think you seen many of my posts around here ;) [btw I don't endorse some of it, wall of text written in a hurry is a bad habit]

 

Ofcourse the time is apprectiated and the info provided is good enough for the first steps, I wouldn't want to see you guys frustrated and confused with some of the readings so that's why I don't hesitate to get to the point.

 

However, it is very important you need to stay at the same exact point/camera angle/distance when doing screenshots. But screenshot is also not a video.

 

Troubleshooting/Performance testing is a big big pain in the butt, it's not that easy, it seems my talk is harsh, I'm doing you a favor by rather telling you what's ahead, but at the same time I'm here to help however I can, this was just the initial post as I'm readying up to test.

 

The above video is sort of a constant reminder, for my self as well, it's good to know that phenomenon ahead, or else it can lead to confusion down the line.

 

EDIT: Also, you can't compare just between maps and base the conclusion there, they have different levels of tech behind them, different CPU behavior and threading, compare NTTR to NTTR.

Edited March 8, 2020 by Worrazen

Hmmm, i'm running an i9 10920X with hyperthreading disabled (12 real cores). After watching DCS's load on the CPU i used process lasso to restrict DCS affinity on these 3 cores. Later i used the extended FPS view in DCS to see whats going on in the "Main:" line, which lead me to the conclusion that the value behind "Simulation" maybe the source of the problem. So i came to the conclusion, that something that is calculated on the main thread is increasing CPU frametime making the GPU wait by taking too long. So there is definitely no more cost on the GPU (indicated by low usage).

 

@Worrazen ???

 

In the multiple testing sessions i used the Huey to circle around those parked planes and then turned away from them to watch the FPS go up again as soon they are out of my FOV. I even build a replica in the mission editor to see the same happening on the other 3 maps as well.

 

However, it's not my job to analyse deeply whats going on there. The info should be enough for ED to know where to look for.

Edited March 9, 2020 by Alec Delorean

Thanks to Minsky https://forums.eagle.ru/showthread.php?t=265686

for making USSR_Rik aware.

Minsky's info is much better, in his screenshots you can see low drawcalls vs. an increase of "Simulation".

Edited March 9, 2020 by Alec Delorean

Hmmm, i'm running an i9 10920X with hyperthreading disabled (12 real cores).

 

Yeah I mixed it up watching some other pictures, sorry about that I was going to look up your CPU and correct the post, I realized I was wrong shortly after posting it, but forgot before I went to sleep, it was evening and I had to prevent again from spilling into late nights, I'm a bit of exhausted after all the previous testing so sorry about that.

 

After watching DCS's load on the CPU i used process lasso to restrict DCS affinity on these 3 cores.

 

Heh, usually it's expected to leave things default when doing an initial test/report, if there's just one, and then with modifications to compare, you might have told us about that earlier, but no biggie because 3 cores is right about where DCS should work fine except disk I/O reads which apparently can around 10 threads in some cases (depends on the MAP, I have a thread about that pending so I won't go into it now)

 

Later i used the extended FPS view in DCS to see whats going on in the "Main:" line, which lead me to the conclusion that the value behind "Simulation" maybe the source of the problem. So i came to the conclusion, that something that is calculated on the main thread is increasing CPU frametime making the GPU wait by taking too long. So there is definitely no more cost on the GPU (indicated by low usage).

 

You are thinking into the right direction yes.

 

But don't do conclusions, perf/troubleshooting is like one of those things that it's good to have "current opinion" that can change the more you look and dig into it.

 

 

In the multiple testing sessions i used the Huey to circle around those parked planes and then turned away from them to watch the FPS go up again as soon they are out of my FOV. I even build a replica in the mission editor to see the same happening on the other 3 maps as well.

 

Yes depending on the subject it may or may not be map specific (to a certain degree) looks like this one is not, but I haven't done any tests outside NTTR yet.

 

Map terrain graphics code and disk loading thing is the more map specific thing I see for example.

 

However, it's not my job to analyse deeply whats going on there. The info should be enough for ED to know where to look for.

 

Sure sure, that's why I'm here, this isn't for everybody but it's a lot of fun for me for some reason, I also want to know for myself anyway, and why not share my findings, no that developers desperately need it but also as a community thing, so you should stick around to at least get an idea so when you have something to report later on you'll know the procedure how to do a good report, this is all in the FAQs anyway so I'm not saying anything new, most people that report don't produce any video or picture, I'll probably do a tutorial on how to make a good video, the problem is running OBS to snap a video interferese with CPU usage which can affect DCS if the user has only 4 cores or a slow CPU in general, some of the stuff I had to go back to using a digital camera to record because of this. In the end it is still helpful to get closer to it than not I think, even for the developers. And it's not that things aren't known, new people come by here and do a report and they're off back to playing again so it's just that the tools aren't well known out there and I'm upset at my self it took me this long to figure this out how deceptive the numbers and simple utilities can be.

 

 

 

Thanks to Minsky https://forums.eagle.ru/showthread.php?t=265686

for making USSR_Rik aware.

Minsky's info is much better, in his screenshots you can see low drawcalls vs. an increase of "Simulation".

 

Good, except tho I don't know why the sysinfo keeps being a bit broken, videomem either showing something else or now it's not working at all, but the term simulation is also a bit vague, there's probably a huge load of stuff in that moniker.

 

I don't know where you see drawcalls there, they're not showing in the DCS SYSInfo, but yes that's the usual suspect most of the time.

Edited March 9, 2020 by Worrazen

The number of objects, my hyperintelligent friend...

Thanks to Knock-Knock, this shows the problem at best: https://forums.eagle.ru/showpost.php?p=4238222&postcount=188

I'm not using infantry , or AI , and still find the performance dismal with 2.56 .

https://forums.eagle.ru/showthread.php?t=265709

 

Okay, generally for all of my posts, if there does sound like I have an attitude then I apologize it wasn't meant, it's partially due to my skewed english writing when I'm in a hurry, but I try to do best.

 

Sorry about my sort of rant on the Task Manager because I again recently got into this whole thing about it while learning the tools I discovered only recently, much late than I should have.

 

The number of objects is not it sorry.

 

Drawcalls range from 300-5000 in DCS roughly, if you have a lot more units those maximums will be higher ofcourse, you'll get the maximums particularly in the case this thread is about, and on the F10 Awacs view in ALT mode under certain zoom conditions. The minimums are ofcourse when viewing the clear sky and/or sea, there's not much going on there, and object (airplane) can have hundreds of draw calls.

 

Draw Calls Numbers Detailed:

 

 

Pretty big change after they spawn, and when you view them up close. How much optimization room while using the same API there is that's what I can't tell you, or whether it shouldn't take that much and there's a bug somewhere.

 

A draw call can cost different amount of GPU time, as we seen with Black Smoke, that's all in 1 draw call (1 for each engine, disregarding parallel drawindexed ones), tho the CPU cost of these draw calls is I think the same irrespective of the time it takes for the GPU to finish a draw call but don't hold me on that one.

 

 

 

 

Additionally. I've uncovered another potentially huge thing while doing these tests, related but not limited to this, a general thing across the board, so the thread title fits well and I've posted it right here.

 

Occlusion Culling is either broken or not implemented?

 

 

https://docs.unrealengine.com/en-US/Engine/Rendering/VisibilityCulling/index.html

 

What does this mean, everything behind the object that is blocking your view is still being rendered, for nothing.

 

I have noticed this in the past but it was extremely subtle but never kinda thought more or went looking deeper (that there's some rendering going on that isn't on the screen), I noticed more of that when doing SpeedTree LOD testing but since I didn't finish that completely and there was a bunch of things going on I didn't get to that.

 

This may be the same for trees, this means you're spending drawing all those trees that are behind the first layer of trees ...

 

Also, I have yet to test (if I even can) whether the objects are being rendered fully, even the far sides of that that you can never see in any situation anyway, hopefully that's not the case but should be made sure.

 

 

NO CULLING (Result: Probably every PC would lag out)

 

 

VIEW FRUSTUM CULLING

 

 

OCCLUSSION CULLING

Edited March 15, 2020 by Worrazen

Additionally, you can see the relationship of GPU Speed vs. CPU Usage in this separate general test using the same scenario/mission. The more frames the GPU can push the better CPU it needs, even if there is practically no extra activity going on in DCS, the view is still and there's nothing whatsoever happening in the background or any other area on the map that would require logic on the CPU to be calculated, the graphics thing is so costly on the CPU in these circumstances of this 2.5.6 OpenBeta build (+2 hotfixes).

 

 

 

Please note this camcorder video is not completely in sync stats wise with the WPA analysis below, the CPU numbers will not match (higher), but they show exactly the same thing, because there was 25% CPU activity in the background from another process I forgot to close so the whole camcorder thing was kinda all for nothing, I use a digital camera whenever I want to avoid any CPU interference from software recoding utilities, however because the CPU cost was only on one thread and always constant (taking 1 full thread at all times) I think the demo is still valid because the proportions of the drops and spikes should still be the same so stuck with it because I just didn't want to redoit yet again it's very tedius and timeconsuming to setup the camera (angle, height, position, not hit it and ruin the shot) ... I want to get the hell out of the house to some spring weather already :)

 

 

 

 

 

You can see, going from slowest GPU speed to the highest, CPU cost is ~50%,

 

 

 

 

 

So people who invest in a much better GPU over the CPU are wasting a lot of that GPU because it needs the CPU to feed it, ofcourse if you wait long enough until Vulkan API rendering then I guess okay.

I haven't rea... erm, binged the thread yet, but I noticed one thing lately when I was flying the Tomcat. As I was using the mirrors to check my wing sweep, I noticed that the fps didn't drop at all when switching them on. Usually they eat up like 30-50fps of my rendering performance, hence I usually keep them off. In 256 however it seems the performance drop of having them on is always there, even if they're off. However it made a difference when I toggled them while I was over the sea with land at least like 30nm away, literally completely invisible in a CAVOK day thanks to the hefty DCS "everfog" haze.

@Worrazen

 

If you add DCS's own performance stats with Ctrl+Alt+Pause (twice for full) you might understand what i mean with "number of objects" aka draw calls and "Main: Simulation" etc. it even has a polygon counter. You also may read the whole thread again.

 

If you really want to deeply analyze the problem, check this .miz file: https://forums.eagle.ru/showpost.php?p=4238222&postcount=188

 

(might be pretty interesting on the graphs!)

Edited March 17, 2020 by Alec Delorean

Well I've just shown you that objects are not draw calls. An object can have 1 or 100s of draw calls, or even none, there is no context whether those objects are drawn or not or if they're even visible or not, since there's is no occlussion culling I can only speculate that it shows all the rendered ones regardless if they're visible to our eyes, or they may have some internal meaning.

 

Simulation doesn't mean much to me either, it's an internal counter, it may be equivalent to something general but I avoided the issue by simply bypassing and ignoring it in my tests, so I never use or do comparisons with that counter.

Edited March 19, 2020 by Worrazen

Actually it's the other way around, 1 draw call can be one object, a predefined group of objects or just a per pixel color change render pass that adds 33% red on every pixel on the entire screen. Usually a significant number of dynamic objects is used for assembling a interactive 3D world, so the CPU to GPU calls matter especially on objects, because the more are called the more significant their impact will be on performance. Thats why the object count is displayed in the FPS info.

 

About the simulation value, well it's the point of the whole thread that you are missing completely. Again, if you really want to analyze something, check this mission with your fancy graphs: https://forums.eagle.ru/showpost.php?p=4238222&postcount=188

And look for the connection between simulation value and massive increase of frametime.

"Simulation" and all the other integrated counters may be useful they just weren't for me in what I was trying to figure out, it's not logged and there's no histogram to compare things easily with the other things so I didn't bother with it. I wasn't able to enable any more debug/perf/console in DCS despite configuring the options and lua files, it may not be possible in public builds.

 

 

And a picture of what we're actually talking about:

 

 

Also the counters move so fast you can barely see it and for just eyeballing things is quite hard, slowing down video is necessary, but then again, you can use software that interferes with CPU usage and the stats, I made many false assumptions and conclusions in the past, most of which I figured out before I posted anything thankfully.

 

I have a Nikon D3200 but video is only 720p and it's barely usable for this type of recording, can't even get half the screen on with good focus.

 

If there's some key clue that the Simulation counter can help us I'd love to see it.

Edited March 21, 2020 by Worrazen

Dunno about you guys, but after updating to the latest 2.5.6.45317, my performance problems are resolved. I'm now having roughly the same fps as in the 2.5.5.

@Minsky

 

Some issues got fixed yes. The spawning of AI groups in the FPS AI mission doesn't tank performance anymore. But the bomber formation test scenario: https://forums.eagle.ru/showpost.php?p=4243909&postcount=202

still tanks my performance extremely. The simulation value shoots up to over 80 and my frames drop from 45 to 10 or lower.

Do you have wake turbulence enabled?

 

In some missions it tanks performance on my PC.

I was having serious issues with missions in the F-18 Aggressors campaign. This has thankfully now been fixed:

 

Campaigns

 

F-5E Aggressors BFM Campaign, F-5E Aggressors ACM Campaign, F-15C Aggressors BFM Campaign and F/A-18C Aggressors BFM Campaign by Mapple Flag:

 

Updated some units to current models

Removed unused units and replaced them with static models

Corrected parking positions to match 2.5.6 changes

But, and it's a huge but, it appears to have been fixed by having to make considerable changes to all of the missions.

 

Thoughts / inferences:

 

Would they really have gone to all this trouble, if it was an engine issue which was going to be fixed imminently?

 

The previous versions of the missions would still tank the frame rate (I tried to test this but those missions wouldn't run for me in the new version).

 

This latest version of the engine will be going to stable, and many missions, etc, will have to be reworked to avoid the issue, whatever it is.

 

It really would help if ED would just explain what's going on. I reckon something's inadvertently been broken and only been discovered at a very late stage. Now it's so much effort to go back and lose all the work, and they're trying to kludge it. Very, very concerning.

Edited March 22, 2020 by Hippo

"Removed unused units and replaced them with static models"

 

I guess that's the simple "fix" for the Aggressors campaign missions. Those parked jets aren't idle AI's anymore.

@some1

 

Just tested this, disabling wake turbulence helps a little, but not that much. As soon the simulation value shoots over 18, frametimes rise extremely and FPS drop hard.

Good benchmark scenarios are those WWII bomber formation attack missions.

I guess that's the simple "fix" for the Aggressors campaign missions. Those parked jets aren't idle AI's anymore.

 

I'm not sure how simple it was for poor old Maple Flag to have rework, test, etc many of their campaigns. Hopefully, it wasn't too much of an effort. Still, the simple fact that the effort was made suggests that this puppy's going into stable with whatever the issue is.

Edited March 22, 2020 by Hippo

With "simple" i meant the approach of fixing the problem by simply removing assets from the mission. The problem still exists.

Just checked the F5E Agressors campaign and the performance gain is only marginal. The initial parking position is changed, but as soon you are facing a couple of the other parked planes, performance starts to drop heavily.

With "simple" i meant the approach of fixing the problem

Sorry, yes I did get that, but couldn't resist twisting your words to make my point. ;)

 

I actually flew the Familiarisation F-18 mission last night, and was able to fly it without issue. With my (very) low settings I was able to start up, taxi and takeoff. Frame rates were mostly 40 fps (ASW), with some dips to 25 fps - not ideal but perfectly usable. I was down to a constant 5 fps (totally unusable) with the previous version. For reference, once away from Nelllis, it's 80 fps throughout.