Clustered Application Needs Re-Loading? Here's How

A question came up at last years cfObjective in Mike Brunt's presentation about reloading your application/model/ColdSpring config in mid-flight on a clustered application (your application has 2 or more CF instances running it).
We've spent some time investigating on our own and mostly just ignoring the issue and going through the tedious process of RDP-ing into the server and MMC-ing into each Servers Services Panel and restarting each instance and verifying they come back online - staggering them throughout a period of time until all cluster members were synced. This sucks.

Some of the proposed solutions included:
1.) Using ActiveMQ and a CF Gateway to Broadcast a 'reload' message and have the cluster members stagger their execution of it so they all don't reload at once (taking the app offline) via a thread sleep randomly based on the JVM instance etc.
2.) Appreloading via the url X times until you think you've HIT them all
3.) Use JRun's internal Web Server

I figured I'd give number 3 a try as I knew it would be the quickest and easiest to implement if it worked. Turns out it does.

What would happen is this: each CF instance has it's own dedicated IP and port that is running the application that can be accessed via a browser or over http (which can get automated but we'll get into that later).

The IP:Port is normally used if you wanted to get to a particular cluster members cfide to do some configurations etc i.e. 127.0.0.1:8303/cfide/administrator/index.cfm. However, if you browse to just 127.0.0.1:8303, you'll notice a nice directory structure that comes form your installation instance directory i.e. C:\JRun4\servers\{instancename}\cfusion.ear\cfusion.war if on Jrun.

So what we'll do is modify your jrun-web.xml file by adding the virtual root path to your application AND make sure if you use a separte jvm.config for your instance, that the default jvm.config is the same (more on that later).

Navigate to C:\JRun4\servers\{instancename}\cfusion.ear\cfusion.war\WEB-INF and open jrun-web.xml (if you don't have one you can try borrowing one from the default cfusion instance).

Add the following to the file and save:

<virtual-mapping>
<resource-path>/</resource-path>
<system-path>C:/webroot/approot</system-path>
</virtual-mapping>

This will make JRUN's internal web server root be your application's root directory.

Restart your CF instance.

If you are like most shops and use some kind of query parameter to execute your application reload procedure i.e. ?reloadsite=true or the like, then you can execute it via the ip:port?param and reload each cluster member by changing your ip:port.

Advanced Note: if you're like us and have a separate jvm configured for your CF instances (different than the stock jvm.config found in c:\JRun4\bin) then you'll need to make sure that
a.) either the jvm.config is the same as your CF Instance jvmconfig file
b.) make the jvm.config (default) the same as the CFInstance jvmconfig file, and then reconfigure that instance to use the 'jvm.config' file. This is key, as the internal JRun webserver will use the default jvm.config on startup, so if you have separated these jvm's out, reloading your app from ip:port will do nothing, as you have only just setup 2 separate jvm's running your application, and the one has NO knowledge of the other.

This is one method anyway - just make sure you have tested this thoroughly, especially if you have environment specific configurations that get loaded up depending on the url - as the IP address block may load up an unexpected set of configuration data from your application.

We'll be trying this out over the next few weeks and likely integrate it into the application in some kind of threaded reload routine if some configuration needs to be done mid-flight.

Enjoy!

Separating the JVM config files - A Cluster Conversation

I had an interesting discussion with a fellow who was looking to solidify his stand with his boss on jvm configuration separation and the need for it. Below is the IM conversation the ensued - I think it helps to show some of the various points that could be made arguing FOR having the separation of the config files per logical JVM (names changed to protect the innocent and guilty)

[09:20] cluster_help: Hello, i was just talkin to Jared about some jvm clustering stuff and he said you were way smarter than him on that stuff
[09:20] kpenny: sure ok -
[09:20] cluster_help: we're setting up a few clustered instances for a large app on a single vmware box
[09:21] kpenny: cool - 32 bit?
[09:21] cluster_help: i know i should have each instance running its own jvm config so we (yeah, 32) can lower the max heap etc so all the instances aren't trying to use ALL the available memory
[09:22] cluster_help: i can find lots of posts and articles on HOW. Do you know of any that mention WHY?
[09:22] cluster_help: need to help my infrastructure guy convince his boss
[09:22] kpenny: so 'why' separate the instances?
[09:22] cluster_help: load balance and failover
[09:23] cluster_help: its a custom installation of our main app for a large, high traffic customer
[09:23] cluster_help: we're isolating them from all our "regular" customers
[09:23] kpenny: yup - both excellent reasons - keep the app UP -
[09:24] kpenny: so you'll have x vm's
[09:24] cluster_help: yea
[09:24] cluster_help: and a gig of RAM to allocate among them
[09:25] kpenny: i also recommend if you have say 8 processor box - not to give each vm virtual procs that add up t o 8
[09:25] kpenny: as the ESX host for example would need cpu 2
[09:25] cluster_help: right, it is an ESX host
[09:25] kpenny: k - we noticed that when we dedicated 2 cores per vm
[09:25] kpenny: we were cpu strapped -
[09:26] kpenny: also you have to think about memory utilization - if you have 4g ram on each vm
[09:26] kpenny: and only a single jvm - likely the jvm max is 1.2G -
[09:26] kpenny: and the os isn't going to use 2.8g ram
[09:26] cluster_help: u know of anything published anywhere that describes WHY its a good idea to lower the max (allocate memory judiciously) across each instance
[09:26] kpenny: so you're also leveraging your hardware better
[09:26] cluster_help: right
[09:27] kpenny: describe - lower the max -
[09:27] cluster_help: when you create new instances, they all use the same jvm.config, so they all get allocated whatever the default is
[09:27] kpenny: right - separating them gives you more control that's all
[09:28] kpenny: and also allows you to do custom class paths
[09:28] kpenny: etc, as your apps grow -
[09:28] cluster_help: so you create a sep jvm.config for each instance, and lower the max heap size, so they all aren't trying to use all the available
[09:28] kpenny: and/or as you allocate resources - you have more granular control
[09:28] kpenny: well even though you're using the same jvm.config
[09:28] kpenny: its still a separate unique jvm
[09:28] kpenny: that they are running w/in -
[09:28] kpenny: which can be a misconception -
[09:29] kpenny: the jvm.config just configures the values for each of the individual jvms in use -
[09:29] kpenny: by separating out the configs, then you're able to have more control over 1 jvm out of the 4 say
[09:29] kpenny: vs. one change in one file for 'all' jvms
[09:30] cluster_help: roger that. but other than the obvious, if you have 8 instances and 8 x maxheap ends up being more than the available memory on the host, don't you risk having a problem?
[09:31] kpenny: yes
[09:31] kpenny: that's when you'd need either 64 bit - or multiple machines - etc -
[09:31] kpenny: but yeah you never wanna allocate 2 much
[09:31] cluster_help: so why my infrastructure guy's boss is looking for is a technote or a blog post or something that says you SHOULD lower the max in the .config
[09:31] kpenny: but with separation of the configs you can control that -
[09:32] kpenny: well really the max should be determined by the app
[09:32] cluster_help: yeah. Like I told my boss, I know HOW to do it. (I did it yesterday on my laptop!)
[09:32] kpenny: so if you have large mem requirements -
[09:32] kpenny: then it should be higher - i guess i'd say there's no set rule - just
[09:32] kpenny: that depending on performance tsting what's best for the app at a given time -
[09:33] kpenny: i.e. if you have multiple applications using frameworks and cs etc -
[09:34] cluster_help: here's the background on my company - the dev manager and the infrastructure manager constantly bicker and fight little political battles. the infrastucture guy and me have about 10 years of exp in each of our areas of expertise
[09:34] cluster_help: but both our bosses want us to prove via external sources what we recommend
[09:34] kpenny: ok -
[09:35] cluster_help: and I am saying, and my infrastructure guy concurs, that we need sep config files for each instance in the cluster, (for granular control) of the max memory allocations
[09:35] kpenny: whats your max set to now - ? and is there a memory problem i.e. using 2 much memory
[09:35] cluster_help: but we need to show them WHY we recommend that
[09:35] cluster_help: the cluster isn't even in production yet, this is a first time set up
[09:35] cluster_help: we're actually clustering the staging env first for load testing
[09:36] cluster_help: then we'll do it in production
[09:36] cluster_help: we've never clustered or had very good front end load balancing here
[09:36] kpenny: i guess one more good point to throw in the mix is this
[09:37] kpenny: some apps perform better on different jvm versions - heck even sun vs. ibm etc -
[09:37] kpenny: so - with that - w/o granular control you're stuck using the same version that all other apps are using
[09:37] kpenny: in a single config -
[09:37] kpenny: vs what we went through were were we had an app that uses coldspringn and
[09:37] cluster_help: right
[09:38] kpenny: benefited from the bug fix in 1.6.v10
[09:38] kpenny: so we were able to just change that config and run the app
[09:38] kpenny: others can use stock or whatever
[09:38] cluster_help: this is going to be 3 or 4 instances of a single app deployed as an .ear
[09:38] kpenny: so 3 jvm's
[09:38] kpenny: clustered?
[09:39] kpenny: i guess i'd argue for it having the separation b/c it's easier to do it from the start -
[09:39] kpenny: and you really can't predict how things will be configured 5 months from now -
[09:39] kpenny: it's not a big deal to separate them -
[09:39] kpenny: and you can always do it at a later date if needed 2
[09:40] cluster_help: yeah
[09:40] cluster_help: but as far as you know, there isn't anything out there in a kb article or a tech note that says WHY you should do it that way, its just kind of a common knowledge/common sense thing?
[09:41] kpenny: well i know for dev its essential
[09:41] kpenny: for reasons like control over debugging ports
[09:41] kpenny: and you're likely working with multiple applications each with their own requirements
[09:42] kpenny: so for dev i'ts a no brainer - but if you have production separated out where x apps are all on y vm's etc.
[09:42] kpenny: and the other ones are on separate ones and they never cross over
[09:42] kpenny: then you can likely get away with stock setup
[09:42] kpenny: however -
[09:42] kpenny: we have some 'maintenance' things that happen with the app
[09:42] kpenny: and we found it most beneficial to have this as a separate instace - even though tis the same app -
[09:43] kpenny: but run schedules through it -
[09:43] kpenny: for batch processes etc - off on it's own - which
[09:43] kpenny: has it's own requirements for mem etc -
[09:43] kpenny: and we benefited from having them separate configs
[09:43] kpenny: also -
[09:43] kpenny: if you wanted to setup a scheduler 'instance' where all it did is kick off
[09:43] kpenny: cf schedules to your app
[09:43] kpenny: you don't wanna allocate 512 mb ram for that guy
[09:44] kpenny: likely small 128 via config
[09:44] kpenny: and run it alongside or whatever -
[09:44] kpenny: so there are many use cases - and they all depend on your preference
[09:44] kpenny: debugging locally with multiple instances w/o having separate jvm configs is impossible
[09:44] kpenny: so that's #1 reason I started doing it -
[09:45] kpenny: then after that i noticed all these other possibilities
[09:45] cluster_help: right, but if we're doing the multiple instances so we can have load balance and failover, each instance should be set to the max we can afford given the machines total available RAM and taking into account the OS's needs and stuff like that?
[09:45] cluster_help: in a production environment?
[09:46] kpenny: if you're app performs best with the max at that level -
[09:46] kpenny: some apps won't -
[09:46] kpenny: and typically if you allocate over what 1.8gb?
[09:46] kpenny: the service won't start?
[09:46] cluster_help: so really the load testing we're going to do will let us know if the settings will work
[09:46] kpenny: yup
[09:46] cluster_help: i think its 1.2gb.
[09:46] kpenny: right ok -
[09:47] kpenny: yeah try it at various levels -
[09:47] cluster_help: cool - hey, thanks for the info man, it really helps
[09:47] kpenny: np - ! best of luck -
[09:47] cluster_help: thanks, i appreciate it