[TOOL] Cynagen's Universal Intelligent Pre-Loader [4-21-15]

[Suedenim]

I would guess that the only thing that would change in game files is the save/progress data and when patches are applied. Perhaps a fast dump for shutdown or something would be possible.

[Cynagen]

... move originals to safety deposit, copy originals into RamDrive...

I've covered this guys. I never move the originals straight to the RamDrive. That's asking for all sorts of trouble. Please re-read my post and make a numbered list for everything I mention, so you can see step by step how this operates.

[Zencyde]

Haha, it's 3x4GB + 3x2GB so I can maintain triple channel. ;D It's not THAT much extra performance, but I'd rather have it than not.

[Downtown1]

This tool does nothing of the sort. No code injection, no intra-process redirection. There is only the NTFS junction (redirect) which is seamless to the user and games run on the machine. The process is relatively simple: Figure out which files to pre-load, generate a pre-calculated RamDisk based on the max disk size permitted and the total size of files to be loaded, move originals to safety deposit, copy originals into RamDrive, junction the files on the RamDrive back into their original places on the HDD/SDD, launch the game. Upon close of the game, unload the RamDrive (freeing up your RAM for normal operation), and then restore files to their original locations from safety deposit. Pretty simple (for those technically inclined), it's just the fully automated management is what I'm providing. Technically you can all do this now, I'm just making it a no-brainer and readily available for every user to take advantage of. Not everyone is comfy running all the command line tools required for this operation to be completed successfully.

That sounds pretty elegant in its simplicity :). I guess I was expecting people to go completely overboard with their architecture nowadays. Question though, what happens if the power goes out or the computer locks up? When you reboot is there anything to clean up the junctions back to their former files (safety deposit)?

Also it goes without saying you wouldn't be able to do partial file IO (e.g. compressed files) ram backing this way since you can't junction part of a file (can you?) afaik. I'm actually more of a Linux dev myself so feel free to correct me if I'm wrong.

There was also talk in the IRC chat of why this tool was not written up and released years ago... well there was 2 reasons. First, RAM was not cheap like it is today. Second, most machines before the Phenom/i5 series could not address more than 8GB of RAM, the "cap" on desktop computers has recently been lifted to allow greater and greater quantities to be installed. Yes, server hardware has traditionally had a much higher "cap" however these components were usually much more expensive than desktop, and thus prohibitive to enthusiast gamers by being outside their price range.

I wanted to further update everyone, I will have a 5-day weekend coming up here shortly, 6-day if I don't get called in on Labor Day, and plan on getting a beta out sometime on the weekend. I look forward to everyone's feedback.

Yes, yes, this makes perfect sense. Up till now the only reason I had to have a lot of RAM was to run a bunch of virtual machines, with this tool I could see more RAM being a lot more useful. Gotta remember to go up to 32 or 64 at my next upgrade.

[Cynagen]

In this Windows implementation I cannot address parts of files (compressed or otherwise) with this tool. I had a startup check implemented to check the safety deposit and make sure there weren't any stragglers, and if there were, check if a link existed with the original name, and remove it, then move out of safety deposit to original location before starting the application.

[Cynagen]

So uhh, yeah, I planned on having a beta out this weekend, that clearly didn't happen. Brief explanation, Diablo III is what happened, and no, I didn't play the game, I was testing the tool on the game Diablo III and well, lo and behold, the last accessed timestamp never changes unless patched, (when accessed by the loader), so I can't track which files are being read and load those ones up into RAM. However, the game IS small enough that if you have a computer fitted with 16GB of RAM, you could just load all of Diablo III into RAM, but alas, that doesn't solve the problem for those with less. I'm looking into other ways to track this information and implement them so I can make sure this tool works as advertised...

I'm back at the drawing board for the usage detection methods, so I'll keep everyone posted on what's going on. I don't work tomorrow but I have to visit my sister, she's been ragging on me and wants me dead if I don't make an appearance at her place every 3 months.

UPDATE: I talked with Hayden in IRC earlier, and he pointed out that the behavior I was expecting from windows was no longer valid and hasn't been since Vista. I'm reworking my strategy and looking at making use of some free tools in order to accomplish the goals set forth. We'll see how this turns out. Hopefully I can have something for you all next weekend.

[Suedenim]

Filemon - run it whilst playing and see what pops up although going this route will leave you with the problem of either having to play the entire game or guess at what files later levels might be using. Perhaps this is one of the cases where you might have to rely on the community to provide the necessary config. Might be worth considering the option to load an entire directory to ramdisk as well as individual files (if you haven't already) - sub-directory recursion as well?

[Haldi]

Update on UIPL :P




So About Process Write/Read Monitor.

Sysinternals "Process Monitor"

The Problem! you'd need a pre configured "configurationFile.pmc" which sets Filter. But when you have that its kinda easy. You Log From Start to Stop. Terminate the tool, extract the .csv from the logfile. And Voilà you just need to create a statistic from this MASS of data which files are read/wrote the most.

Here is what i've done with Smite. (Procmon.exe is direclty in C:)

Code: Select all

   Run(@ComSpec & " /c " & 'C:/Procmon.exe /LoadConfig C:smite.pmc /quiet /backingfile C:smite.PML /minimized /acceptEula', "", @SW_HIDE)

   MsgBox(0, "Test", "Pause, Press OK when game is finished")

   Run(@ComSpec & " /c " & 'C:/Procmon.exe /terminate', "", @SW_HIDE)

   WinWaitClose("Process Monitor")

   Sleep(2000) ; Security time to make sure File write is over.

   Run(@ComSpec & " /c " & 'C:/Procmon.exe /Openlog C:smite.PML /saveas C:smite.csv', "", @SW_HIDE)

Sadly i have no idea how the .pmc file is built. At least i cant edit with with an simple Editor. IF you somehow can change that, you'd just need to change the "Process name" in the Filter from "Smite.exe" to "Gamename.exe" to make in universal.

In the Attachements there is this smite.csv just startet up the launcher and then went into Login Screen, nothing more!

Maybe you could tweak the filter from Process Monitor to filter more...



P.S make sure you set "Drop filtered events" or else it will log ALL process but only show filtered!

Attached files

smite.csv (1.5 MB)

[Cynagen]

I love how you've latched onto the same idea that we all have and went so far as providing a sample file, this will go a LONG way, I can completely hook and automate the interface for procmon while completely hidden from user view, making it just a utility tool. Same concept, 'cept without running it 3 times, just one shall suffice.

I found a way to get WAY smaller files which give me the summarized data that I need without actually sorting any of it myself in UIPL, huzzah! Now to write up some automation and make sure it work.

[Haldi]

Tell me when you got a *working* version and need some results from different games :) i'd be happy to test it out.

[Zencyde]

I'm calling it now, this project is going to get out of control at some point as performance hungry gamers realize how easily this will take load times to nil.


Also calling in early bets for Valve to contact you possibly in regards to integrating it into Steam within the next 5 years.

[Cynagen]

Haha, Valve can contact me all they want, this is nothing new. I'll be more than happy to work with them to implement this should they wish to (and I don't see why not with Steam going to Linux, that makes this all wayyy easier, and with the actual Steam client in the way, it could do everything I'm doing for us.)

I hope this takes off and people use it, but I've got to do this RIGHT. No half-assing this... whole-ass or no-ass.

[Haldi]

IMHO it's a shame for all game developers that we have to resort to Tools like this! The Engine itself should realise how much RAM aviable is an then directly put all the big/often used files into the RAM.

But most games are developped for 256mb(PS3? Right?) RAM and quickly portet to PC to save expenses.

Well, lucky we got a Community to fix such flaws ;)

[takaides]

3. If you have 16GB of RAM, your MAX RamDisk size will be 10GB, leaving 6GB (more than enough for most 64bit games (most are 32bit and can't directly address larger than 3.5GB due to addressing limits) and your OS (64bit Windows)) to operate in normally. If you have 32GB, this cap will be lifted to 26GB. The RamDisk estimated (these numbers are NOT final and may be adjusted as needed based on user performance reports) max disk sizes based on total RAM are as follows:
6GB (Minimum total RAM supported): 2GB RamDisk (4 Reserved)
8GB (Average 'gamer' total): 3GB RamDisk (5 Reserved)
10GB: 4GB RamDisk (Full 6 Reserved)
12GB: 6GB RamDisk
14GB: 8GB RamDisk
16GB: 10GB RamDisk
32GB: 26GB RamDisk

There's a method to the madness, a science to the math. ;-)

Are you planning on putting in a setting to override the defaults? I have 32GB of ram, but I would only want a 20GB RAM disk. Also, some games are huge and may use all of that, but some games are tiny and would be swimming in 4-8GB of RAM disk...

[Zencyde]

Ah yes, a whole ass, sir!

It's time to put Kim Kardashian to shame.

[Cynagen]

Overrides for the cap will be implemented, and the RAMDisk will intelligently resize to only take as much RAM as you cap up to, or only what's needed. So say you load up an older game that only takes about 9GB of space, provided you have a maximum cap of 9 or greater, it will only generate a RAMDisk of 9GB to preload the data.

Also, I would like to publicly apologize for the lack of updates recently, had some life issues, my roommates totaled their car this weekend, so I'm back to using the bus system which means I get 2 maybe 3 hours at home before I need to be in bed for work the next day instead of the 5-almost-6 when we had the car. So yeah, development is going to be a little bit slower, but I'm making progress, so hopefully I'll have something soon. Real life sucks sometimes.

[Wijkert]

Also, I would like to publicly apologize for the lack of updates recently, had some life issues, my roommates totaled their car this weekend, so I'm back to using the bus system which means I get 2 maybe 3 hours at home before I need to be in bed for work the next day instead of the 5-almost-6 when we had the car. So yeah, development is going to be a little bit slower, but I'm making progress, so hopefully I'll have something soon. Real life sucks sometimes.

I hope nobody was injured?

[Cynagen]

Nobody injured, nobody arrested, no damage to the other vehicle.

[Suedenim]

Came across the start of a discussion on RAM disks here http://forums.anandtech.com/showthread.php?t=2270087

[Cynagen]

I read through that discussion and still have faith that my tool will provide some valuable gains to some if not most games. The discussion towards the end of the thread as it stands currently focuses on the multiple random reads that some games can do, like Diablo 3 and SWTOR do, this is definitely what this tool is targeted for as I've beta tested a script something like what they put up, on SWTOR's art assets, and experienced a largely noticeable performance gain when loading multiple player and non-player characters in a new environment. (Think the name floating over the head but not being able to see them/invis.) I'm still moving forward with development. There's a market here, I know there is, and they just need a tool to manage it for them and they'll see there's some games that will benefit greatly. You just have to find out which ones and which settings work best is all. I'm providing the tool, it's up to you, the end users to figure out it's best application.