Major Camera Movement Stuttering Performance Problem

As I am investigating a bigger issue into the whole dynamic allocation system, this issue seems to be separate and on-top, so I can split it up and cover it separately.

Graphics settings maxed out. My analysis shows this issue has nothing to do with GPU Perofrmance. The term "stuttering" is used very unprofessionaly out there, what I mean here is exactly what the word means, the word does not carry any indication of what may be the cause.

This may be one of the things that many people report as stuttering, but usually think it's a FPS issue. I don't know how those cases are connected.

The major facts:

• - FPS is 70-90
  
• - No asset loading from storage
  
• - Camera movement causes a ton of write events
  
• - RAMDisk doesn't help
  

The writing to Temp is not as big as it is frequent, as you can see, the game's writing for every N coordinate change.

The Temp location is normally on an SSD.

But it gets even weirder, the stuttering does not go away nor it is diminished when the TempDCS director is put on a RAMDisk.

So I'm out of ideas, and now, seeing other videos, if this was a wide, It could be only with my PC hardware configuration as well as Win7.

-------------------

I'm posting this in general since it could be wider issue, not just to this release, certaily I know it was also the same on 1.5.5 Beta. Now I probably will have to download Alpha just to test this if it's indeed a local issue or not.

Confirmed on 1.5.5 Release and 1.5.6 Release. But this is been going on for a long time, I just thought it had to do with loading things, but it's not. I will create a separate thread for anything that has to do, that would be a way bigger wall of text, please don't speculate because I don't want to derail the thread as I am positive this is an unrelated issue.

This is as big of a problem as if I had a crappy PC, certainly my FPS is higher and a lot more playable with a new GPU now, but this one alone is killing it.

NOTE: This was originally titled as "F2 Viewing Lag" however it also affects Cockpit half the time, so I just started to call it CameraMovementTempWriteLag

Interesting investigation, let's hope it brings somewhere

UPDATE: Major Discovery

The lag is confined to camera MOUSE movements, that's why there is no lag in cockpit view when you normally look straight and the world rotates around as you fly the aircraft through normal mission gameplay.

This eliminates the idea that draw calls or some kind of terrain/model processing is the culprit for the camera stuttering.

UPDATE2: Major Discovery 2

The ~TR0000 TempWrites are what appears to be part of an input logger. It is writing down all input device data such as mouse and keyboard. Probably used by the debriefing stats system and for crash debugging.

(It also means it's a keylogger so I wouldn't be typing any passwords:music_whistling:)

The mouse produces a lot more of these write events than keyboard keys. I accidentially discovered this while testing a pure F2,F3,F4 views, without moving mouse, I saw a big spike when increasing the engine throttle, after that I crashed the airplane into the terrain and watched my keyboard actions correlate to these TR0000 write events.

I believe that the inefficiency of the implementation is causing lag, not the logging it self. The strain on the disk/ssd is not that much in terms of data, it's not a lot amount of data, but it possibly is a lot of small bits, and random access on SSDs isn't that good compared to how sequential speed is versus HDDs.

However I do not have any evidence to support that disk strain is the problem, earlier I have shown the RAMDisk test, which did not help. Which is more clues showing it's the game's fault, but it may not be intended behavior, it could be an exaggeration caused by this particular computer and OS configuration that I just happen to have.

----------------------

Unrelated: There is still some stuttering visible in external F2F3F4 views, which is not the focus of this thread, that is the asset loading lag (probably can't be obvious on these videos), that's the thing I will cover in a separate thread so don't mix it with this.

If someone wants to try it out on their own, but doesn't know how to replicate what I did, let me know on PM and I will help with windows/utility configuration.

Would help if I get a number of people to report their findings, so developers have more cases not just one. Basically the game is rather practically unplayable so this is the highest priority for me now.

It seems as if it's a small issue, but it's very frustrating and just lowers the experience, infact in older versions it wasn't so pronounced, after some months and new hardware, this effect has multiplied itself.

For A-10C, when cockpit camera movement has to be used frequently, this is even more important.

 

The ~TR0000 TempWrites are what appears to be part of an input logger. It is writing down all input device data such as mouse and keyboard. Probably used by the debriefing stats system and for crash debugging.

 

First and foremost used for replay tracks I would guess, as all devices input records are required for track saving and later playback.

First and foremost used for replay tracks I would guess, as all devices input records are required for track saving and later playback.

Oh ofcourse, I didn't thought about that, thanks for the heads-up.

But, knowing that, it still doesn't make it okay to be like that, should still be looked into.

I will try to see if I can turn off the replay feature.

More thoughts:

Even if the disk write stuff isn't significant, I would have a 2 versions of replay logging, one that would be enabled by default and would be less brutal on disk, it would put the input logs into a cache on memory, and only write to the HDD in a larger chunk every 5 or every 10 seconds.

In an event of a crash, then you would lose those 5-10 seconds, but that is a rare case.

In the options there should be an option to "Enable Replay Logging", or vice versa you would have to untick "Enable Friendly Replay Logging" which would be ticked by default.

Not everyone may have an SSD as their first disk, but I know this is not a game practical for the low-ends and there's no need to spend too much effort to attempt some support.

The other reason is, it would be laxer of SSD lifetime.

UPDATE3: I have to pull back a bit

Unfortunately as it usually goes, after the whole thing there comes a twist.

I have fiddled with this issue for more than a week, I thought I gave it enough time. But it's usually that I figure things out after I post the thread.

The recording utility for what happens under the hood is causing this lag effect to be multiplied. I am sorry, but the real lag in effect is lower, not as pronounced, I have been testing for 2 weeks now on other issues, that's why I uncovered this one I guess, because the nature of it makes it so boosted when monitoring.

Hey on another note, if this effect wasn't multiplied, it may have not alerted me to look into it.

It does not mean it is completely gone, not a chance, it is more subtle tho, while I was monitoring it was really unplayable, but it's a lot better when I stopped monitoring. I will have to make future videos without the use of monitoring so much.

I was simply testing so much and had these utilities running, but that other issue I was testing for is definitely not affected by this monitoring utility, that one is the reason I started testing in the first place.

So, back above I said how "it got worse as I got newer hardware and later version" that should now be disregarded, it should be the same how it always was, not as strong as it was in these cases when I was monitoring, I could hardly move the camera where I wanted.

Topic name should be changed from "Major" to "Minor" ... but still frustrating.

EDIT: All other things still hold, the effect is just less frequent, when it hits it feels like the camera is draggin a lot of weight and kinda moves slow and moves skewed, I want to go down but it kinda goes to the side a bit, like it's old can crampy, yes it does happen in cockpit, everywhere where camera movement with mouse is involved.

Well, now I'm thinking, with my investigation hat on again, what if, the lag that happens when there's a lot of units, AI's, missles, projectiles, movement etc, what if 20-30 % of that lag that people usually assing to "AI-Computation Lag" (CPU bottleneck) is actually caused by replay feature recording all the AI's movements and everything else... now that's a, jeez that's something to think about right?

These are all barebone tests, with only 1 player, only 10 units that sit still on ground on caucus videos, no units on nevada videos.

AI's don't have camera mouse movement, yes they don't count the same, but 50 of them, 200, they add up, airplanes more than ground units, or are the individual bullets from AA guns also tracked ?

So like 10-20 AI planes could have the same strain on the tracking system as 1 player moving his camera half the time, I'm just speculating for example, to explain the idea.

Now that's an eye opener. First of all it just needs to be optional, then, see if it can be optimized for people who really love to use it. Cause, I'm not playing seriously until 2.5 stable hits, I'm just messing around and doing some custom textures, I don't need replays now, but I don't have any hate against them. (I actually enjoy this investigative adventure)

I'd really like to see what happens if I turn the replays off, how much of a difference it is when having AI units. all the missiles, that's a lot of tracking to do for the replay system, and if the replay system is not even handling one single airplane perfectly ... whooh!

Here's the problem of why testing this and similar scenarios around replay tracking probably wouldn't be perfectly accurate, because the "unrelated" thing is in the way, well it becomes related-to-the-topic, but not directly, because it's blocking us from getting genuine pristine test, that "unrelated" thing is a different type of lag, the asset allocation one.

For this to be tested on it's own, that variable has to be eliminated, at least for testing purposes if not in practise and I have a big idea and still writing that thread.

It is crucial to understand that there are many types of lag all blended between each other in practise, it's necessary for me to go into such long explanations because it has to be separated and breaken down for it to be understood, for me myself as well and to document it, half a year later I will forget this and I need the thread to remind me. But I'm not making any earth-shattering discoveries for the developers, I'm rather pointing it out, they might have already detected and already worked on it for 2.5 for all we know.

UPDATE4: Now it gets convoluted

All of the following test videos were run on freshly booted state with zero additional programs.

First, here's a comparison of the aircraft F2 view when using the monitoring utility.

A lot of the F2 and Cockpit viewing lag was unfortunately caused by the monitoring utility.

-------------------------------------

Now, the area which I mostly saw this happen, is the ground view after crash or landing.

But, at least in nevada, it's more laggy in a large city than outside.

However the FPS is still stable around 40-60 over the city.

Second video should offer more answers.

• Clean boot, no monitoring utilities
  
• The mouse camera movement lag is more heavy in city scenes
  
• Frame rate still acceptable and should not be the cause
  
• flying fast and moving the world view does not produce this kind of lag, only mouse movement make it
  

So, if you are in F2 view and you fly over the city you'll get this amount of camera jitter with mouse movements, it's quite significant, even tho I made an alarm before and pulled back, yes it's not as laggy outside city area, but it's still an issue throught long-term, the laggies happen every 20-40 seconds, about 1 or 2 a minute or so, along with the rest, this is the thing that I noticed that got me looking into this in the first place, so someone may say it's not that bad, but looking at it from a bigger point of view, considering all the rest, it's really only minor things that are holding the beast of hardware back, these are not genuine requirements, no hardware will magically overpower things like this.

Found 2 other threads way back to 2015 and later

https://forums.eagle.ru/showthread.php?t=153411

https://forums.eagle.ru/showthread.php?t=157579

Looks like the replay reliability is also not good from what these people have to say. Quickest and best workaround is to just make it optional from now on, fix it later.

https://forums.eagle.ru/showthread.php?t=151050&highlight=track+replay

https://forums.eagle.ru/showthread.php?t=181467

https://forums.eagle.ru/showthread.php?t=78246

This issue is many years old.

I believe ED changed the input handling and introduced a bug where mouse movements that are smaller that one unit per frame are not accumulated properly. That's why the stuttering effect is most visible when you have high frame rate and pan the view slowly. It's also most pronounced with high polling rate mice ( basically the better ones or gaming models).

Quite hard to believe that, how does everyone just deal with it, so I guess people just want to throw the hardware at it, obviously it's not working, or let's say it's inefficient but helping some.

Why on earth is the replay feature mandatory in the first place.

how does everyone just deal with it

Stick with it and think that without DCS there's NO other modern flight combat sim out there.

There's been a few posts since the release of 1.5 pertaining to inputs clashing with performance..

The mouse camera movement is one that i am afflicted with.

As Snaut noted though, some of the newer mice are configured for 500/1000hz rate naturally.

Set this to 125hz and the problem moves aside.

The overhead of 1000hz never bothered dcs 1.2, regardless how high/low the framerate got.

Stick with it and think that without DCS there's NO other modern flight combat sim out there.

True,

Still tho, I would wait and see what ED can do when the engine merges and gets some polishing. I bought a 1070 a few months back and 2.0 is better on performance as it's a different engine to 1.5 running the Caucasus region.

We don't know what the new Caucasus map and merge 2.5 will bring to the table soon. ED hasn't really pressed and optimized all they can out of this edge engine by a long way, I would like to see my 1070 maxed out too.

This will be never ending for ED I guess, tweaking optimizing the new engine, and Perhaps then ED with feed the cards even more later with DX12 API's added? https://blogs.msdn.microsoft.com/directx

I'm going to sit back for now and see what happens with 2.5.

Stick with it and think that without DCS there's NO other modern flight combat sim out there.

That's not going to be problem before 2.5, but it would after 2.5. I'm already only testing and not really playing until 2.5 stable is out. But if this doesn't make into 2.5 stable only then will I seriously mind it, when I'm going to play the campaigns.

Update:

I finally took the time to test this not with different settings but with a completely different mouse, as some people were suggest

And they were right.

I have a secondary PC lying around and I got a 15 euro mouse for it, a logitech M100, while my main mouse is RAT5 from Saitek/MadCatz.

So, I disabled the Saitek/MadCatz driver and profiler utility from starting at boot, and then I rebooted the PC, disconnected the mouse before it rebooted, and put the M100 in, and sure enough there is no camera lag bug anymore.

I'll figure out later if it's the poll-rate or what, but still, gaming mice should be supported by a game. I can't be with a cheap mouse like that all the time.

Since we still don't have a an option to disable replay tracking, I can't be sure if it's really replay tracking causing this or not, yes I was kinda 99% sure, but I never want to say I'm 100%.

I think I have stressed this point many times in many threads: USB-Polling, it#s a nightmare !

When you set your mouse to 1000polls/sec you basically occupy the USB bus very much.

Try lowering the desponsivness to 250 or 500 polls/sec and see if that works.

Might be the #1 reason why some have this stupid stutter and jitter and all sorts of problems, people underestimate the power of USB when it comes to screwing up a whole system to a point where it totally mises the point...because of a high end Laser Gaming Mouse.

Logitech has a warning in their software, telling you that 1000/sec may hog your CPU !

AHh yes, USB is software driven = CPU hog.

Gimme a Thunderbolt Mouse !!!! Hardware driven, keep the CPU for better stuff !!!!

Meh, I would have to do some ehm special medicine to get this working ...

http://madcatz.kayako.com/index.php?/Knowledgebase/Article/View/276

My mouse drivers don't have an option for manually adjusting poll rate. But I am doing a number of tircks I've been at it all day long, I found some newer drivers, reinstall completely, clean registry, and then I'll also put the keyboard to another port so it's on another hub, so only 1 input device per hub. And then i'll limit the rate to 500hz and see what happens, hopefully i can stay with 500.

The polling rate is how often the mouse reports data back to the PC

 

whereas some mice on the market have the option to manually set the polling rate of the mouse. The R.A.T. range instead adopts a dynamic polling rate where we report new data on an event driven basis, as per USB spec and Win/Mac bandwidth usages to minimise unnecessary communication, plus minimising latency. This dynamic behaviour is why we state “Up to 1000Hz” on the packaging

 

Ultimately though, provided you’re operating over 125Hz you’re unlikely to truly be able to notice much of a change when you start ramping up the polling rate from there – remember that 125Hz means that the mouse is communicating with the PC 125 times per second. With normal movement speeds that we use when using Windows, we’re likely to be generating polling rates between 300 and 500Hz; the rapid movements you’ll see used in gaming situations easily generate 1000Hz polling rates

 

There are many other factors that influence this such as monitor refresh rate, V-Sync and how different games work (some use polling themselves and others are event driven), but the net result is that the dynamic polling rate on the R.A.T.s should not be considered a limiting factor as a lot of the ‘unused’ data with fixed polling rate mice is simply discarded by the PC’s OS.

 

So the answer is: No. R.A.T. mice report data changes automatically, depending on mouse acceleration and speed. This auto-reporting ensures the best positional performance.

Early testing has surprisingly proved promising at 500hz max poll rate.

But I still think if it can be fixed from the game, it should. This is far from an official solution, even tho for me I'm used to dealing with them. I'm basically using 3rd-party community hacky drivers to force my mouse to work at a lower poll rate.

If this isn't really that prevalent, and it's really the bad driver, and other gaming mouses that are also 1000hz don't have this problem then I have to eat some crow for early assumptions, I kinda never glossed over the idea of too high poll rate, I started asking some programmer channels and hardware people and they were pointing the finger at mouse drivers and the whole windows USB environment, so I went into that direction.

Now that I'm going to completely wipe everything about the current drivers and device ids, I'm writing down all the info, I'll post it here just incase it comes handy if devs want to replicate or whatever:

Version until later today:

Mad Catz R.A.T. 5 Mouse

 

Driver Date: 20.9.2012

Driver Version: 7.0.20.0

Software Version: 7.0.27.13

SaiU1705.sys (USB)

SaiK1705.sys (HID)

Second Note:

I have for the past days been talking how it's important to realize there are many kinds of stuttering and how they overlap and can be very tricky to pinpoint and thus can be falsely attriubted.

Some of the stuttering when moving the mouse was caused by the fact that when mouse moves as the track replay system write events are created, are then displayed in the monitor tool and over the game's viewport, because of the sheer amount of the events it would be lagging the system as the game is running and the Windows desktop explorer.exe probably has to be involved, when I have another window on top it's probably not running in real-fullscreen, probably something in between, and DirectX and WDDM which is ver 1.1 on Win7 is also the stuff that defines what kind of support there is and I'm familiar not full acceleration/speed is supported in some of these intermediary modes between exclusive fullscreen and desktop rendering, not exactly sure but it's not surprising that it's making the game slowdown a bit when the GUI is screeching as it tries to display so many events from the tracking writes, this event caputring and GUI display could be better on Win10, I would be testing that first when I get a Win10 test setup up. Ofcourse, it may also pertain to the monitoring program it self and it's capabilities, the monitoring program it self may be getting loaded so it's taking a lot of CPU to display.

And the type of the stutter that does happen is more of a slowdown type, smoother, it's not at all the same kind that the high-poll rate mouse does, it's unique, but the most important thing is that I know this now and I will be more cautious and more careful when future testing to make sure I count this in or try to avoid it.

I'll also keep in mind that when I'm testing DCS for the other asset loading stuff, I'll put the poll-rate down to 125hz just to be on the safe side and make extra sure.

But still, for any kind of testing we would still be better with replay tracking temporairly disabled, just to get rid of a variable and rule that out, even if it never had anything to do with the mouse issue. My apologies for any false associations.

Auuuuu a little ooops. It's not exactly but very close ....

Okay, wow, I actually had a little bit of testing setup going on with clipmaps from Nevada being disabled (part of the testing I was doing for this thread) , and it made it look good, so I thought there was no lag anymore.

But this little too fast as usual, actually helped otherwise I wouldn't figure it out even better.

No, polling-rate doesn't completely eliminate camera lag ... going from 1000hz to 125hz it makes a 95% difference in F2 view (outside views) and around 30-40% improvement in F1 Cockpit view.

However, However, However, ... it all depends whether or not there is other asset loading going on, with Clipmaps,Scenes,Roads folder disabled (so it doesn't load any textures for NTTR) even the 1000hz poll rate doesn't make that much mouse camera lag in F2 view. none, it does however make mouse camera lag in F2 cockpit view, but 3 times less than with Clipmaps enabled.

This discovery now shifts the blame back to I/O, right now I/O in general, not anything specific, I would need more testing to maybe figure if there's certain assets from NTTR or not.

SSD is fast but ... remember I tried putting ~tr0000 temporary track replay files onto RAMDISK and it did not help, it could mean that the game might just be getting jammed with I/O and possibly hitting CPU Max.

So if this is tied to the storage I/O, and maybe hitting game's own internal capabilities, no wonder high-speed storage users just quickly load all the assets and the tracking file has enough room to do it's job ... that would fit, the game it self is hitting a bottleneck of I/O then, because the track file and installation of assets are on different disks and they shouldn't be botlenecking the game, as the streaming textures shouldn't make the engine to stop, the textures if they can't get loaded because of slow storage they would just take longer and you would see blurred textures for a longer period of time, that's what the symptom already is, but the tracking write events which is again the game's responsibility is kinda messing with the engine in such a way that it affects the camera responsivness, or maybe it's just hitting CPU max.

FACTS:

• - These test this post is based was the latest RAT5 drivers only, 7.0.60.6 - January 2017.
  
• - 125hz case tested both with M100 ordinary logitech mouse and the RAT5 with the tweak-adjusted setting to 125hz (additional driver filter), the 1000hz case was tested with no tweak installed.
  
• - No Joystick drivers, No Programming Profile Software
  
• - I was cleaning the registry for about 4-5 hours, prior, of all references to all Saitek or MadCatz things, shining clean.
  
• - I even completely removed the Mouse and Joystik OEM drivers for Saitek that ship with Windows 7, completely deleted them out of FileRepository too (backups ok), so they wouldn't be able to interfere in any way.
  

One thing is for sure, the replay tracking system should be caching things into RAM and writing a bigger chunk every 20 seconds or something, yes that would mean something could get lost if there was a crash but the game's core may be operational for some moments as it writes crashlog, not sure but i've seen things happening under the hood for 10 secs after windows shown a crash message, in that time the first thing it should to is to flush the replay tracking cache so it attempts writing the last bits it had. Secondairly an option could be made for "agressive replay logging" to log things in the behavior that's similar to this existing one.

The Saga Continues, I'm actually quite enjoying the challenge and the journey!

PS/2 !!!

I still have a combo to install for example Win7 onto Ryzen via DVD and bypass this damn USB30-Win7 issue.

Sometimes, when you have your account password protected, you can come to a very stubborn STOP-Sign without PS/2 keyboard at hand.

This whole USB BS is nasty !!! I am glad I dont suffer this ( yet ).

*edit>: I wish there was an OPTION to turn off recording, it doesnt work reliably anyway.

The Main Bottleneck is PCH and DMI. The bandwidth is anything but great !

*edit>: I wish there was an OPTION to turn off recording, it doesnt work reliably anyway.

 

The Main Bottleneck is PCH and DMI. The bandwidth is anything but great !

I didn't actually mean to say it's bottlenecking the USB. I meant somewhere along the lines of tiny SSD write operations, but still a lot of unknown so I can't really point any fingers specifically.

Referr to this thread: https://forums.eagle.ru/showthread.php?t=189175

I don't think it's tied to the camera as I originally titled it. It's something with the write I/O or just I/O handling in the engine, without would be then tied to the CPU as well

I hope I won't get down to my CPU being too slow, I mean I get 200 FPS max and average 70, and when it stutters, the fps does NOT drop.