The easiest way to determine what hex values you need for a certain resolution is to go
here and check out Tanuki's post.
The second easiest way is to use the Windows Calculator program. Make sure it's on scientific mode (view menu) and then type in a number (say, 1024) and click the "hex" bubble on the left side of the calculator to change to hexadecimal mode. This will change the number to 400, which is the unformatted hexadecimal version of 1024. Without getting too
technical, all you have to do to change it to its proper format is put a zero in front (0400) and then take the last two numbers and put them in front (00 04). That's the code for 1024. Do the same with 768 and you'll come up with 00 03. So 1024x768 would be 00 04 x 00 03.
mjrpes explains it in more detail.
As for the method of finding the locations of the hex values in a program? Well it's simply a matter of searching for the hex values of resolutions that you know are supported by the game. Sometimes you get lucky and both codes are right next to each other - other times they're pretty far apart, making it trickier to spot them. Sometimes you only need to change one, sometimes you need to change a dozen that are scattered around the executable. Sometimes the codes aren't even
in the executable, but in one of the other game files (like a dll or config file).
And just because you find 00 04 or whatever doesn't mean it affects the resolution, either. It's also a matter of trial and error. You back up the executable, change the code, try the game. Maybe it works, maybe it crashes the game, or maybe it has no discernible effect at all.
In short, it can be very tedious.
And once you're done with the resolution, that doesn't necessarily mean you're done with the fix, either. Some games change the FOV (Field of View) automatically, but others don't, and if you can't find a way to change it, the game is just going to stretch, which is no good.
Luckily Rollercoaster Tycoon here has a rendering method where the game resolution automatically determines the amount of game area rendered (more pixels, more game area), so stretching isn't an issue.
Now you should be able to use 1280x768 which is fairly close to your native resolution. In order to do this you would have to change "20 03" to "00 05" and "58 02" to "00 03". Go to my
original thread for instructions on how to do so. Click the hex codes for pictures of what they'll look like in the hex editor.
Let us know if you need any help.