[TOOL] Cynagen's Universal Intelligent Pre-Loader [4-21-15]

[Cynagen]

I know I'm necro'ing again. This time, I have better news however. Effective after next week (April 13th), I don't have a normal 8-5 that I have to go to so I'm gonna have plenty of time to get this finished. It's been on my mind for a while, and after seeing that DIMMDrive debacle on Steam, I figured I might want to push to get this done sooner than later. The software I see out there still is either locked to certain manufacturer's hardware, or is otherwise dumb and just loads whatever it can into RAM, so I still feel there's a niche I can fill with this tool once it's finished. I still have the source code, and I've got some more coding knowledge from playing around with small bits of code at work, so I hope to make some major actual progress this time around, and with no job to get in the way now, I'm very optimistic. I ask for everyone who's followed this thread's forgiveness for my touch and go, it's been a weird couple years.

[Haldi]

Long time no see. How's going? xD

I'm all ears if you have new test versions ready.

[Cynagen]

Long time no see. How's going? xD

I'm all ears if you have new test versions ready.

It's going, working out my last week at the office, mail off some money, and I'm home free for a couple months. When a test version comes out, I'll keep you posted.

[HaYDeN]

this probably sounds really negative, but more interested than anything as there's some interesting problems that arise - would love to see some proof of concept, I also saw Dimmdrive on Steam and rolled my eyes tbh.

I'd find it difficult to believe that any real tangible [and not negligible] benefit would be seen over an SSD, which is obviously _significantly_ cheaper dollar/gb than system ram, a decent SSD/HDD steam library manager would be more beneficial IMO? something I've thought about starting a few times but not gotten around to, the available ones are a little basic and unintelligent for my liking.

The other thing to consider is that most engines are pretty smart these days in terms of what content to keep around in memory, looking at just the (first) load time of a level is fairly redundant considering you'd have to move that data to memory to start with to see any acceleration - it's also fairly common for any decent engine to preload content on the fly before it's required as your roaming the level, I can't remember the last time I saw significant stuttering in any games, and I typically don't bother having the majority of my steam library on my SSD (infact the majority lives on a 2TB 5400rpm Samsung HD204UI)

Implementing it (properly) would require a very low-level hook of the read/write which certainly is not trivial in terms of unexpected behavior or more appropriately a kernel mode/custom filter driver, then you get into having to get it signed by Microsoft (cost money) etc etc, any other approach I can think of would set off VAC and various other anticheat systems.

[Cynagen]

HaYDeN, there's a lot of valid points you bring up, but there's a couple things I'd like to reply to directly.

There is a tangible difference, load times are practically instantaneous since it's a direct RAM-to-RAM transfer meaning there's near-0 load time for even large chunks of data. I remember quoting 12.8GBps throughput on RAM compared to the measily ~250MBps that you get from a standard HDD (at best), and ~600MBps from SSD (again, at best).

Most newer engines are getting smarter about memory management and you're right, this tool will eventually not be needed. That being said, however, this tool is still needed/usable today as there are still a number of games very poorly optimized (W_D and SWTOR for example) that could benefit from this directly. (I've played SWTOR on HDD, SSD, and RAM, and by far, the RAM is the best when encountering new players as textures have to be called and modified on the fly before being shunted off to GPU to render a player, and the game will not render the other player AT ALL until their assets are fully loaded, meaning with a RAM disk, they pop into existence on your screen LONG before you appear on theirs, giving you a tactical advantage if it's an enemy in contested territory).

There is already a toolkit called ImDisk that has signed drivers by Microsoft, which emulate a hard drive out of allocated RAM, so that's taken care of, and the files are simply "symlinks" (using a Linux term here, in Windows it's actually called a "Junction") back to the RAM once copied over, redirecting all requests aimed at the HDD/SSD to the RAM disk we create. None of this has ever set off any anti-cheat system as it's all built-in stuff, and signed third party drivers for other uses on the computer that we're just playing with for our purposes.

DIMMDrive was a crappy, half-baked version of my idea, and I'm not surprised somebody else tried, just surprised they were stupid and failed miserably to make anything worth a damn.

[Haldi]

well yeah, if a Ramdisk is needed or not anymore is a good question.

At least for Star Citizen (Alpha 1.0) there is no big benefit. (CPU Limit)


But hey, how can we say we don't need it if we don't test it? Might still make a big deal in some games. Yet i'm to lazy to move all my games / or everytime i play a game, onto the SSD or RamDisk, my 250gb 840 Pro is full with other stuff and my games drive with over 2TB is idle most of the time when i'm not playing.
I'd be totally in for a inelligent loader tool.

Btw, i think the read/write monitoring was done via procmon.exe which works.... well... works. I've been playing around with that tool for a while but i lack the competence to evaluate the validity of the result ^^

[HaYDeN]

HaYDeN, there's a lot of valid points you bring up, but there's a couple things I'd like to reply to directly.

There is a tangible difference, load times are practically instantaneous since it's a direct RAM-to-RAM transfer meaning there's near-0 load time for even large chunks of data. I remember quoting 12.8GBps throughput on RAM compared to the measily ~250MBps that you get from a standard HDD (at best), and ~600MBps from SSD (again, at best).

Most newer engines are getting smarter about memory management and you're right, this tool will eventually not be needed. That being said, however, this tool is still needed/usable today as there are still a number of games very poorly optimized (W_D and SWTOR for example) that could benefit from this directly. (I've played SWTOR on HDD, SSD, and RAM, and by far, the RAM is the best when encountering new players as textures have to be called and modified on the fly before being shunted off to GPU to render a player, and the game will not render the other player AT ALL until their assets are fully loaded, meaning with a RAM disk, they pop into existence on your screen LONG before you appear on theirs, giving you a tactical advantage if it's an enemy in contested territory).

There is already a toolkit called ImDisk that has signed drivers by Microsoft, which emulate a hard drive out of allocated RAM, so that's taken care of, and the files are simply "symlinks" (using a Linux term here, in Windows it's actually called a "Junction") back to the RAM once copied over, redirecting all requests aimed at the HDD/SSD to the RAM disk we create. None of this has ever set off any anti-cheat system as it's all built-in stuff, and signed third party drivers for other uses on the computer that we're just playing with for our purposes.

DIMMDrive was a crappy, half-baked version of my idea, and I'm not surprised somebody else tried, just surprised they were stupid and failed miserably to make anything worth a damn.

IMO.

The throughput of the ram whilst high is still severely bottle necked by the CPU.

The symlink/junction approach is not a good approach and is the main reason why these tools are poorly designed IMO, dynamic interception on a per block basis is the way to approach it, thus you can begin serving from RAM before the file has even completed copying, realtime, and infact you could serve only parts of a file... eg, you intercept the read, copying it to ram and serving it from ram rather than HDD, negating the need for multiple redundant read operations, effectively adding (near) zero overhead on the initial load... this is the only way I would consider approaching it.

[Cynagen]

New patch to fix the "long monitor" bug. Please download from the URL on the original post once again and see if you can report any further bugs.