OMG: Mission Editor and F10 could be a lot less demanding! - ED Forums
 


Notices

Reply
 
Thread Tools Display Modes
Old 06-07-2017, 06:51 PM   #1
Worrazen
Senior Member
 
Join Date: Oct 2014
Location: Slovenia
Posts: 1,403
Exclamation OMG: Mission Editor and F10 could be a lot less demanding!

I have uncovered something that is responsible for choking the Mission Editor around the whole asset loading thing, which directly affects performance.

The thread is based on a clean install of DCS 2.1 (configs rebuilt) using NTTR map, on Windows 7 x64. HDD is WD10EZEX 64MB.

This mostly pertains to users which usually use HDDs as their DCS install location. But don't rule out some improvements to other setups as well.

THE REALIZATION:
The Mission Editor streams ALL data for ALL view modes at ALL times, depending on your movement and view position.

Let's give it a proper credit before I start peeling this, not negative in a whining way, just reality from HDD point of view :p

Yes, this way makes it so you can instantly have the stuff displayed when you choose to switch view mode, just that, it's a convenience thing (not saying they purposelly done this way)... AND if, if your storage would be fast enough to even get to all the data it's trying to load, so it turns into a problem and not a convenience in HDD cases.

But for high-speed storage users, throwning brute force at it, this is allright, if they can handle it, let them enjoy this convenience.

Data isn't being saved much, only slightly, only some data is being allocated to the GPU, slowly, most is thrown out as you move view out of the area, it has to be this way at least for the clipmaps which are huge and wouldn't fit all into VRAM anyway, but DCS doesn't yet really use all the resources it actually has at disposal but that's an overall memory management issue for another thread/post, it's a big subject and I need to break it up a bit into parts, can't spend 6 months making one giant thread or I forget things by the time I'm finished.

No surprise everything gets jammed, HDDs are well known to get "clogged" when you want it to load something else (on another part of disk) when it's already loading the current stuff at max. The lag, stuttering and just would be even greater on average HDDs which more fragmentation, not only I always get the high-end HDDs, but I also keep good maintenance of the PC and HDDs, which included good sorting, I don't have bazillon of boatware and random pictures or videos thrown around, the HDD that's used in this test is only for Games and extra program files (ones that I don't need on SSD) none of which are running when launching a game.


Mission Editor / F10 Map View Modes, and some of their data associations:
  • MAP ::::: -> RasterCharts (2 GB)
  • SAT ::::: -> Clipmaps (26 GB), Surface (4.36GB), Roads (645 MB), Models (1.1GB, fully textured building tops, static objects), vfstextures (minor amount)
  • ALT ::::: -> Surface (4.36 GB), SurfaceDetails (3 MB), Scenes (675 MB, minor, some tree indicators), Models (1.1 GB, minor, structure top outlines only)

Size numbers indicate total folder size, unless I specify details, assume it's going to need all of the data at some point (not all at once, unless it's purposelly preloaded by design, just it's possible it may need this or that but if you use the game fully then it'll eventually need the rest too)

Update: SAT Needs both Surface and Clipmaps to function, otherwise it's mostly blackness.

How did I figure this out: Example folder view



There's still some smaller things left which are harder to pin point if they're needed for a certain ViewMode, mostly "Models" and "vfstextures" folder have things in that may or may not be needed and is crossingover with the 3D stuff, and also, they can't be as easily disabled because some of them contain LUA files, the game will crash if it can't find certain LUA files it wants, so I need more time to test that because I have to recreate the ZIP files so they only contain LUA files, without the actual textures.

Yes after some more looking it appears there is some textures in there that are needed in either ALT or SAT view, some gray marks around airfields, for example, however the total size of vfstextures is less than 300 MB and only a fraction is needed , still I need to be sure if it's loading more than it actually needs. However this may be more tedious and a bit timeconsuming thing and I don't think it's that important especially when developers can, if they decide to, figure this out faster and easier, I would assume they use special tools for asset overivew and/or blueprints.


Here is a video example of MAP view mode trying to display to display RasterCharts which were disabled, you can see Nevada.surface5 events as the game's trying to load the Surface assets which are NOT needed for MAP view mode.



-------------

Modded to display Mission Editor ALT View Mode blazing fast:



Reminder: I'm not some protester against all streaming things, yes there is Nevada.surface5 streaming going on, necessary for ALT View Mode, it's not fully preloaded nor it stays preloaded I think (not checked that yet), however Windows Standby Memory would definitely cache this file so it would work even better as session time moves on.
So all the blue background you see it's the area which has yet to be streamed in, but the most important thing is, it is not freezing the engine or viewport, I can scroll and move with the mouse smoothly!!!
So the proof of concept is a mission success!

This ALT video was also shot with a COLD BOOT, with empty Standby Memory.

-------------------------

Here's the videos for those other modes, just to check if there's any leftover assets they display:







-These 4 videos are with vfstextures enabled and Models disabled.
-Tops of buildings and hangars are not seen in ALT view so I need more fine-tuning, somewhere in "Models"
-And SAT view doesn't have all it's stuff removed, you can see the city textures, those are somewhere in "vfstextures".

---------------------
---------------------

Better experience for everyone especially HDD uses on this subject would be achieved with:
  • - creating a proper loading screen/section for the Mission Editor, so that the assets that meant to be loaded at the beginning are loaded before the engine tries to display the map partially, it's more clean and doesn't feel as if you're trying to strain a steel cable. Let the files load and the engine relaxes and then when it has everything for the default view, only then it tries rendering, and only then additional assets get streamed in after the user moves the viewport.
  • ------------------------------------------
  • - creating a Mission Editor pre-config menu before launching any loading, I don't have Normandy so I don't know how maps are selected right now, but I'm just going to explain with my draft idea, doesn't mean it's the best or the only way, just what I had first in mind, a pre-config menu would also include a map selection section, and from there further options.
  • ------------------------------------------
  • - Adding a Map View Mode Selection for the default start, the user would be able to select the View Mode he wants to use first or use all 3 modes, only the assets of that view mode would start loading and/or streaming later. Each of the modes has a minimal preload with details later streaming in, some modes are small enough that they can be loaded fully. But even if the user selects (tickbox) more than one or all 3 modes, they would only be loaded minimally and only the current viewing mode would be streaming in as the user moves the viewport around the map, so the base behaviour would also be changed, UNLESS for extreme users, they enable (another tickbox) an option to stream ALL modes simultaneously (no limitations), only then they will achieve a behavior equivalent to the current one right now.
  • ------------------------------------------
  • -- when selecting other View Modes the user would receive a warning message that a new set of data has to be first-time loaded, unless it was included in the pre-config menu screen, the user would have an option to append the new map data or replace (throw the data previously loaded for the existing view mode out of memory)
  • ------------------------------------------
  • - The FileStructure update that the overall game needs also could to be made inside terrains, to better separate assets that are specifically needed for the F10 Map/Mission Editor view vs the in-game 3D view, and the rest of them which are shared, putting them into their own subfolder, this would all work well with my idea of the global asset manager index system, the more the individual files to sort, the more maintenance heavy will the index be.



Files that appear to be streamed as necessary but could be fully loaded at beginning - they are small enough:
  • Map\Nevada.sup5
  • DCS World 2 OpenAlpha\Sounds\wnd_sample.dat
  • DCS World 2 OpenAlpha\Sounds\EditorMusic\Default MainMenuLoop.ogg
... more may be added to the list if necessary


------------------
Some minor items are being searched in a lot of folders (this is all I/O that is wasted)
and it find the file after some time.

Found it here:
E:\DCS World 2 OpenAlpha\MissionEditor\themes\main\images\logotypes-v1.png
E:\DCS World 2 OpenAlpha\MissionEditor\themes\main\images\base-menu-window.png
E:\DCS World 2 OpenAlpha\Mods\terrains\Nevada\Theme\icon_select.png
E:\DCS World 2 OpenAlpha\MissionEditor\themes\main\images\MainMenulogoComp.png
E:\DCS World 2 OpenAlpha\MissionEditor\themes\main\images\DCS-plate.png


Each of these files it tries to find the ones with _EN suffix but it doesn, so when it tries without EN it finds them, but whatever it's doing it's wasting resources. Some of these files are reloaded several times, they're tiny common GUI items, this should all be in memory without any need of accessing storage for any queries or readings. These are the types of assets I would reclassify as core and they would be loaded at boot as usual.
__________________
Temporairly off for PC Maintenance - DCS not installed - Specs will change
Specs: Win10 x64 1607/14393, 1440p@75"32 - CPU: Intel Core i7 3820 @ 4.0GHz - GPU: Radeon RX 480 8GB - RAM: 32 GB, DCS SSD Samsung 860 EVO 250GB, Saitek Cyborg X/FLY5 joystick.
Modules: A-10C, F/A-18C, Mig-21Bis, M-2000C, AJS-37, WW2AP, CA, FC3, SuperCarrier.
Terrains: NTTR, Normandy, Persian Gulf

Last edited by Worrazen; 06-08-2017 at 10:59 AM.
Worrazen is offline   Reply With Quote
Old 03-21-2020, 10:49 PM   #2
Worrazen
Senior Member
 
Join Date: Oct 2014
Location: Slovenia
Posts: 1,403
Default

Update March 2020:

Something else may also be a factor in the laggyness while moving/zooming the F10 AWACS view, and it's not asset loading:
https://forums.eagle.ru/showthread.php?t=264333


Why update this old thread from the legacy version ... precisely because it's still a hefty post that goes in depth and explains a lot, some of it still relevant and some people may come across it in the future.
__________________
Temporairly off for PC Maintenance - DCS not installed - Specs will change
Specs: Win10 x64 1607/14393, 1440p@75"32 - CPU: Intel Core i7 3820 @ 4.0GHz - GPU: Radeon RX 480 8GB - RAM: 32 GB, DCS SSD Samsung 860 EVO 250GB, Saitek Cyborg X/FLY5 joystick.
Modules: A-10C, F/A-18C, Mig-21Bis, M-2000C, AJS-37, WW2AP, CA, FC3, SuperCarrier.
Terrains: NTTR, Normandy, Persian Gulf
Worrazen is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

All times are GMT. The time now is 06:16 AM. vBulletin Skin by ForumMonkeys. Powered by vBulletin®.
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.