We've spent some time investigating on our own and mostly just ignoring the issue and going through the tedious process of RDP-ing into the server and MMC-ing into each Servers Services Panel and restarting each instance and verifying they come back online - staggering them throughout a period of time until all cluster members were synced. This sucks.
Some of the proposed solutions included:
1.) Using ActiveMQ and a CF Gateway to Broadcast a 'reload' message and have the cluster members stagger their execution of it so they all don't reload at once (taking the app offline) via a thread sleep randomly based on the JVM instance etc.
2.) Appreloading via the url X times until you think you've HIT them all
3.) Use JRun's internal Web Server
I figured I'd give number 3 a try as I knew it would be the quickest and easiest to implement if it worked. Turns out it does.
What would happen is this: each CF instance has it's own dedicated IP and port that is running the application that can be accessed via a browser or over http (which can get automated but we'll get into that later).
The IP:Port is normally used if you wanted to get to a particular cluster members cfide to do some configurations etc i.e. 127.0.0.1:8303/cfide/administrator/index.cfm. However, if you browse to just 127.0.0.1:8303, you'll notice a nice directory structure that comes form your installation instance directory i.e. C:\JRun4\servers\{instancename}\cfusion.ear\cfusion.war if on Jrun.
So what we'll do is modify your jrun-web.xml file by adding the virtual root path to your application AND make sure if you use a separte jvm.config for your instance, that the default jvm.config is the same (more on that later).
Navigate to C:\JRun4\servers\{instancename}\cfusion.ear\cfusion.war\WEB-INF and open jrun-web.xml (if you don't have one you can try borrowing one from the default cfusion instance).
Add the following to the file and save:
<virtual-mapping>
<resource-path>/</resource-path>
<system-path>C:/webroot/approot</system-path>
</virtual-mapping>
<resource-path>/</resource-path>
<system-path>C:/webroot/approot</system-path>
</virtual-mapping>
This will make JRUN's internal web server root be your application's root directory.
Restart your CF instance.
If you are like most shops and use some kind of query parameter to execute your application reload procedure i.e. ?reloadsite=true or the like, then you can execute it via the ip:port?param and reload each cluster member by changing your ip:port.
Advanced Note: if you're like us and have a separate jvm configured for your CF instances (different than the stock jvm.config found in c:\JRun4\bin) then you'll need to make sure that
a.) either the jvm.config is the same as your CF Instance jvmconfig file
b.) make the jvm.config (default) the same as the CFInstance jvmconfig file, and then reconfigure that instance to use the 'jvm.config' file. This is key, as the internal JRun webserver will use the default jvm.config on startup, so if you have separated these jvm's out, reloading your app from ip:port will do nothing, as you have only just setup 2 separate jvm's running your application, and the one has NO knowledge of the other.
This is one method anyway - just make sure you have tested this thoroughly, especially if you have environment specific configurations that get loaded up depending on the url - as the IP address block may load up an unexpected set of configuration data from your application.
We'll be trying this out over the next few weeks and likely integrate it into the application in some kind of threaded reload routine if some configuration needs to be done mid-flight.
Enjoy!
