Recently ModelSim was recommended to me as a (somewhat) slimmer, and possibly nicer alternative to Xilinx's built-in tools. I checked the webpage, and hey, it displays some pretty graphics - so I decided to check it out.
Unfortunately, this is yet another hardware-design tool which isn't compatible with recent Windows versions because of sheer laziness. It turns out, the incompatibilites are only in the InstallShield installer, which is from 1998. After debugging the unpack and install process a bit, I managed to get it working:
(P.S. This is the student edition because they blocked me from downloading an evaluation of a better version...seriously, what kind of company does that?)
The issue is that the installation process expects to use SendMessage with hWnd = HWND_BROADCAST (actually, it uses -1, but it doesn't matter). Because of <some reasons I don't care about>, this causes the install to hang forever. My workaround for this is to nop any calls to SendMessage which have hWnd == -1. This can be accomplished with the following windbg breakpoint:
bp USER32!SendMessageA ".if (dwo(@esp+4) == 0xffffffff) { r @eip=USER32!SendMessageA+0x47; gc } .else { gc }"
SendMessageA+0x47 is the first ret in the function on my version of user32.
A walkthrough of the install process is:
modelsim-pe_student_edition.exe -> unpacks to path like %temp%\pft8E21~tmp\.
The self-extracting .exe might be hanging around in the background, you can just kill it.
The pftXXXX~tmp directory contains the InstallShield installer. Start Setup.exe in windbg and enable attaching to child processes. Apply the above breakpoint to each child process. There will be _ISDel.exe launched first, then another temp directory will be created (with a path like %temp%\_ISTMP1.DIR\), and _INS5576._MP will be executed from there. With the patch applied to these 3 processes, the install will proceed until it expects to broadcast the exit notifications (I assume). You can just kill all the executables at this point and start using ModelSim. Then send an incredulous email to Mentor asking if they're personally really using OSs which this installer runs properly on.
N.B. Yes I realize I could have just extracted the InstallShield cab files, but I wanted it to install properly and complete any component registration or other actions which may only be done by the installer. Also, no "compatibility mode" solved the issue.
Or - the cloud
I've also started experimenting with EDA Playground, which is a decent online simulator, even including a ModelSim backend and an online waveform viewer (along with a slew of other cool features)! Definitely an alternative worth considering...