Jump to content

New GME software


sedenion

Recommended Posts

i also noticed JSGME is faster, and, i think i know why. The copy function provided by Windows seem faster than the legacy C "fwrite" function, at least, in the MinGW implementation...

problem solved but not solved... in fact, i guess JSGME always use the WinAPI CopyFile function. However, the WinAPI CopyFile works in some asynchronous way, so the program send a bunch of calls with quick returns, but the HDD still working for long time after function returns. In the other hand, the standard ANSI fwrite function is synchronous, and waits until all HDD operation finished to return... the real HDD work time is the same for both, but in the first case, the quick return of function give the illusion it's faster. For the second case, when the function return you are sure that all data is copied and ready for new operation.

 

I now understand why under Windows, the HDD still working for long time for apparently no reason... crap asynchronous implementation, a false good idea...


Edited by sedenion
Link to comment
Share on other sites

sedenion, you're doing a fantastic job improving this new GME every day now. I haven't seen it mentioned yet -- might you be able to add a 'save/load mod profile' function similar to what JSGME has? IMO this is one of the most useful features in JSGME, especially with all the regular updates that DCS receives. That way you can disable all your mods with one click, update DCS, then re-enable all the same mods you had running before without having to individually select them.

Link to comment
Share on other sites

Great !!!!!!!!!!!!!!!!!!!

 

Have a beer on me or whatever you like :)

 

Will test asap....

 

 

robocopy is a robust MS tool to copy reliably, lots of arguments. You might wanna look into that. It's similar to rsync in some ways.

Gigabyte Aorus X570S Master - Ryzen 5900X - Gskill 64GB 3200/CL14@3600/CL14 - Asus 1080ti EK-waterblock - 4x Samsung 980Pro 1TB - 1x Samsung 870 Evo 1TB - 1x SanDisc 120GB SSD - Heatkiller IV - MoRa3-360LT@9x120mm Noctua F12 - Corsair AXi-1200 - TiR5-Pro - Warthog Hotas - Saitek Combat Pedals - Asus PG278Q 27" QHD Gsync 144Hz - Corsair K70 RGB Pro - Win11 Pro/Linux - Phanteks Evolv-X 

Link to comment
Share on other sites

sedenion, you're doing a fantastic job improving this new GME every day now. I haven't seen it mentioned yet -- might you be able to add a 'save/load mod profile' function similar to what JSGME has? IMO this is one of the most useful features in JSGME, especially with all the regular updates that DCS receives. That way you can disable all your mods with one click, update DCS, then re-enable all the same mods you had running before without having to individually select them.

 

Hmm, one profile per game would be sufficient ? ( please, say yes, i don't want to create a new bunch of dialog window :p )

Link to comment
Share on other sites

robocopy is a robust MS tool to copy reliably, lots of arguments. You might wanna look into that. It's similar to rsync in some ways.

 

Robocopy is a shell command while CopyFile() or fwrite() are C libraries functions... so to say, Robocopy is just another compiled program who (i am pretty sure of that) uses the legacy WinAPI CopyFile(), CreateFile(), CreateDirectory() functions... so there is no benefit using Robocopy (this is like calling a taxi while you have your own car), i can code algorithm directly into OvGME, and this is what i do.

 

Anyway, i modified algorithm to improve file copy performances and memory usage when the mod is not a zip archive, it is now comparable to JSGME.

Link to comment
Share on other sites

Hmm, one profile per game would be sufficient ? ( please, say yes, i don't want to create a new bunch of dialog window :p )

 

Absolutely! One profile would be enough, at least for me, and I would imagine for most users.

Link to comment
Share on other sites

Robocopy is a shell command while CopyFile() or fwrite() are C libraries functions... so to say, Robocopy is just another compiled program who (i am pretty sure of that) uses the legacy WinAPI CopyFile(), CreateFile(), CreateDirectory() functions... so there is no benefit using Robocopy (this is like calling a taxi while you have your own car), i can code algorithm directly into OvGME, and this is what i do.

 

Anyway, i modified algorithm to improve file copy performances and memory usage when the mod is not a zip archive, it is now comparable to JSGME.

 

Ahhh OK

 

I am admin as you can see and no coder LOL, thanks for the clarification

Gigabyte Aorus X570S Master - Ryzen 5900X - Gskill 64GB 3200/CL14@3600/CL14 - Asus 1080ti EK-waterblock - 4x Samsung 980Pro 1TB - 1x Samsung 870 Evo 1TB - 1x SanDisc 120GB SSD - Heatkiller IV - MoRa3-360LT@9x120mm Noctua F12 - Corsair AXi-1200 - TiR5-Pro - Warthog Hotas - Saitek Combat Pedals - Asus PG278Q 27" QHD Gsync 144Hz - Corsair K70 RGB Pro - Win11 Pro/Linux - Phanteks Evolv-X 

Link to comment
Share on other sites

Be informed the 1.2 version is on the way... current changelog is (aproximately):

 

Improvements:

- New file mod-apply algorithm for non-zipped mod (sightly faster, takes less memory)

- Progress bar logic improved (now work correctly for all operations)

 

New features:

- Files tree snapshot and compare.

 

I will work on the "profile" feature tomorow...

Link to comment
Share on other sites

excellent sedenion !

 

will try when back from client...it's always when things get hot I have tons of work and no free time grrrhhhhh.

 

BF was almost a red vicory and all I could do was watch the BF GUI while on the train..going YAYAYAYA !!!!! ..they looked at m,e a bit strange HAHA

Gigabyte Aorus X570S Master - Ryzen 5900X - Gskill 64GB 3200/CL14@3600/CL14 - Asus 1080ti EK-waterblock - 4x Samsung 980Pro 1TB - 1x Samsung 870 Evo 1TB - 1x SanDisc 120GB SSD - Heatkiller IV - MoRa3-360LT@9x120mm Noctua F12 - Corsair AXi-1200 - TiR5-Pro - Warthog Hotas - Saitek Combat Pedals - Asus PG278Q 27" QHD Gsync 144Hz - Corsair K70 RGB Pro - Win11 Pro/Linux - Phanteks Evolv-X 

Link to comment
Share on other sites

Version 1.2 released... well, version number increase fast, but i think we are now very close to a first good final release.

 

 

Changelogs for 1.2:

 

- Added game snapshot feature (CRC-32 based).

- Added Mods profile feature.

- New threaded method for mod apply/restore to avoid "Not Responding" during heavy operations.

- Overall mod apply/restore algorithm optimized and reinforced (faster, less memory usage).

- Main menu logic improved (unavailable commands are grayed).

- Overall memory management optimization.

- Progress bar logic improved (now work correctly).

- Cleaned code & several optimizations.

- Help updated

 

 

The "no enough memory" problem discussed few posts ago, is now a past history, even for the 32 bits version and even with zipped archives. The "mod apply" algorithm is, not totally, but much rewrited.

 

I also have made a lot of write/read tests and benchmarks, even tested the famous (and boring) File Memory Mapping technic, i think we can't do better than what it is currently.

 

So ok, i have to admit : on ~40000 entry file tree with about 20 Gio data, the Snapshot takes 11 minutes to be completed with the Microsoft file memory mapping, instead of 12 minutes with the standard read function...well... but since this pure Microsoft "chef d'oeuvre" API which use 4 functions for one thing with a bunch of "HANDLE", Flags, and things that you don't even know why that exists, is to me, as beautiful as rubbish scattered on the ground... i opted for the standard read... and "so be it" for the "one minute" better.

Link to comment
Share on other sites

As long as it works, your choice is welcomed!

 

After all features are complete... and tested by many... maybe that net thing is next? :D

 

having mods manually updated and most important manually checked for updates on forums instead of pushed in by an app is a pain in the back.

 

For example I keep my skins also as mods. If one author updates the skins... I need to do the research again and all the download... unrar... copy... rename mod folder again.

 

With an integrated automatic updates pushing things would be much better and a lot of organizing would be made easier... you can add categories for mods, auto change versions and release dates (I change my mods folders names to include always the type of mod, author, version as author published and date of install) etc.


Edited by zaelu

[sIGPIC][/sIGPIC]

I5 4670k, 32GB, GTX 1070, Thrustmaster TFRP, G940 Throttle extremely modded with Bodnar 0836X and Bu0836A,

Warthog Joystick with F-18 grip, Oculus Rift S - Almost all is made from gifts from friends, the most expensive parts at least

Link to comment
Share on other sites

As long as it works, your choice is welcomed!

 

After all features are complete... and tested by many... maybe that net thing is next? :D

 

having mods manually updated and most important manually checked for updates on forums instead of pushed in by an app is a pain in the back.

 

For example I keep my skins also as mods. If one author updates the skins... I need to do the research again and all the download... unrar... copy... rename mod folder again.

 

With an integrated automatic updates pushing things would be much better and a lot of organizing would be made easier... you can add categories for mods, auto change versions and release dates (I change my mods folders names to include always the type of mod, author, version as author published and date of install) etc.

 

 

The problem of what you want, is that all designs i can imagine needs web server...

There is two ways i see this kind of feature:

 

1) The centralised one:

You have a web server, with a database that deals both with moders and users.

 

2) the decentralised way:

- Moders (must) have their own web server, or at least, a shared space with, for example, a formatted text in any URL which can be parsed by OvGME.

- The user choose to subscribe on the moders's server he want by adding the proper url in OvGME

- OvGME check all url, parse data, and give some "need update", and maybe (depending many things) a download.

 

 

The clear thing is that i will NEVER develops the centralised way, even if i could. For several good reasons (to me).

- I am against centralised things on the web (for a gouvernement it's ok, but not for MY Internet...).

- I don't want to host and manage the server (i could) and i don't want the software to be linked to a particular server.

- A centralised serveur, means: If the server is down, if a day, the server's owner decide to stop, the feature stops working... (this is the problem of centralized things).


Edited by sedenion
Link to comment
Share on other sites

At least for teams, a local/ftp/ssh/webdav/wtf reachable directory could be used to synced mods. Checksums or special tags could be used to determine different versions. At least for my team this feature would help to simplify the mod management.

Link to comment
Share on other sites

At least for teams, a local/ftp/ssh/webdav/wtf reachable directory could be used to synced mods. Checksums or special tags could be used to determine different versions. At least for my team this feature would help to simplify the mod management.

 

what kind of "sync" are you looking for ?

Link to comment
Share on other sites

Very nice! Eases the complications with multiple DCS/JSGME installs and reduces my mod folder size considerably with the zip support. I'll be recommending this to my friends.

 

Here's some feedback from what I've experienced so far:

- When 'run as administrator' is required but not granted, no files will be copied but OvGME will show the mod as successfully installed anyway. I think JSGME might also have this problem; not sure if there's anything you can do.

- Comparing snapshots could be improved with a resizable window or with word wrap off - see pic below.

 

These are very minor points, though, and overall I'm very satisfied with your work :thumbup:

 

Thanks!

1.png.8a8864b3f935b1d5c62e818c4397dcc7.png

Link to comment
Share on other sites

what kind of "sync" are you looking for ?

 

Let me think about it....

 

It would be good define a master mod repository for each game/profile, this could be local/remote. I guess local one is easier in Windows.

(In our team we are using a Owncloud installation, integrated into the explorer to manage our mods.)

Then GME could check this special folder for differences between the special folder and the mod folder. If they are differences, GME could display an option to copy/sync/symlink the new version into the mod folder.

 

Up to now, we need to remove the mods from the dcs mod folder and copy the new one every time someone changes the mod and NOTIFYs us. This notification sometimes is not given and we are ending up in mod conflicts on our weekly session...

Link to comment
Share on other sites

- When 'run as administrator' is required but not granted, no files will be copied but OvGME will show the mod as successfully installed anyway. I think JSGME might also have this problem; not sure if there's anything you can do.

 

I guess the files are not copied due to access permission (files permission is hell under Windows). The Mod appear as "enabled" because OvGME probably successfully created backup data, and when there is backup, OvGME takes it as "mod enabled"... I could sure create a function to crc-checksum all mod's files to verify it is correctly applied in the game tree... but this kind of operation will increase considerably the time needed for mod's installation.

 

EDIT: Well in fact, i forget the very simple thing: i could just verify if the CopyFile function fail... what i don't do currently... If CopyFile fail, this would throw you an error and, there will be no more ambiguity.

 

- Comparing snapshots could be improved with a resizable window or with word wrap off - see pic below.

 

The resizable window is hell... but i can avoid the line wrap. Also, think that you can copy-paste this output, the "wrap" is only a cosmetic for the text entry... if you copy-paste in a text editor, the line returns will be correct... also, the output "table" uses tab character, so it can be easily parsed by an Excell-like software.


Edited by sedenion
Link to comment
Share on other sites

Let me think about it....

 

It would be good define a master mod repository for each game/profile, this could be local/remote. I guess local one is easier in Windows.

(In our team we are using a Owncloud installation, integrated into the explorer to manage our mods.)

Then GME could check this special folder for differences between the special folder and the mod folder. If they are differences, GME could display an option to copy/sync/symlink the new version into the mod folder.

 

Up to now, we need to remove the mods from the dcs mod folder and copy the new one every time someone changes the mod and NOTIFYs us. This notification sometimes is not given and we are ending up in mod conflicts on our weekly session...

 

Ok, you and zaelu speaks about two different features...

Link to comment
Share on other sites

Version 1.2.1 fix available (uninstall previous before installing new version... you know...)

 

Features / Changes (1.2.1) :

 

- Fixed two ugly memory leaks.

- New write error check for mod apply and backup.

- Write performances slightly improved.

- Auto text wrap disabled for snapshot output log.

- New snapshot cheksum method based on xxHash (faster).

- Source code: most functions renamed for better naming logic.

- Help corrected.

 

For the snapshot, i replaced the CRC-32 by the xxHash, the fastest open-source hash algorithm presently existing... the fact is that xxHash is, by my tests, 3x faster than CRC32 in term of pure data processing... However, the main bottleneck for snapshots still the hard drive, so, no miracle... but people using SSD will may be seeing a good difference (you'll tell me).

Link to comment
Share on other sites

Great! :thumbup:

However, the main bottleneck for snapshots still the hard drive, so, no miracle... but people using SSD will may be seeing a good difference (you'll tell me).

 

With a DCS folder of 30.2 GB, 35,960 files, 4,036 folders:

 

OvGME 1.2

------

Snapshot 1: 3m50s

Snapshot 2: 3m59s

 

OvGME 1.21

------

Snapshot 1: 3m27s

Snapshot 2: 3m23s

Link to comment
Share on other sites

Great! :thumbup:

 

With a DCS folder of 30.2 GB, 35,960 files, 4,036 folders:

 

OvGME 1.2

------

Snapshot 1: 3m50s

Snapshot 2: 3m59s

 

OvGME 1.21

------

Snapshot 1: 3m27s

Snapshot 2: 3m23s

 

Well, this is not spectacular, 10% gain... the spectacular thing is that where i need 9mn, you only need 3mn...


Edited by sedenion
Link to comment
Share on other sites

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...