Thursday, October 30, 2008

Separating the JVM config files - A Cluster Conversation

I had an interesting discussion with a fellow who was looking to solidify his stand with his boss on jvm configuration separation and the need for it. Below is the IM conversation the ensued - I think it helps to show some of the various points that could be made arguing FOR having the separation of the config files per logical JVM (names changed to protect the innocent and guilty)

[09:20] cluster_help: Hello, i was just talkin to Jared about some jvm clustering stuff and he said you were way smarter than him on that stuff
[09:20] kpenny: sure ok -
[09:20] cluster_help: we're setting up a few clustered instances for a large app on a single vmware box
[09:21] kpenny: cool - 32 bit?
[09:21] cluster_help: i know i should have each instance running its own jvm config so we (yeah, 32) can lower the max heap etc so all the instances aren't trying to use ALL the available memory
[09:22] cluster_help: i can find lots of posts and articles on HOW. Do you know of any that mention WHY?
[09:22] cluster_help: need to help my infrastructure guy convince his boss
[09:22] kpenny: so 'why' separate the instances?
[09:22] cluster_help: load balance and failover
[09:23] cluster_help: its a custom installation of our main app for a large, high traffic customer
[09:23] cluster_help: we're isolating them from all our "regular" customers
[09:23] kpenny: yup - both excellent reasons - keep the app UP -
[09:24] kpenny: so you'll have x vm's
[09:24] cluster_help: yea
[09:24] cluster_help: and a gig of RAM to allocate among them
[09:25] kpenny: i also recommend if you have say 8 processor box - not to give each vm virtual procs that add up t o 8
[09:25] kpenny: as the ESX host for example would need cpu 2
[09:25] cluster_help: right, it is an ESX host
[09:25] kpenny: k - we noticed that when we dedicated 2 cores per vm
[09:25] kpenny: we were cpu strapped -
[09:26] kpenny: also you have to think about memory utilization - if you have 4g ram on each vm
[09:26] kpenny: and only a single jvm - likely the jvm max is 1.2G -
[09:26] kpenny: and the os isn't going to use 2.8g ram
[09:26] cluster_help: u know of anything published anywhere that describes WHY its a good idea to lower the max (allocate memory judiciously) across each instance
[09:26] kpenny: so you're also leveraging your hardware better
[09:26] cluster_help: right
[09:27] kpenny: describe - lower the max -
[09:27] cluster_help: when you create new instances, they all use the same jvm.config, so they all get allocated whatever the default is
[09:27] kpenny: right - separating them gives you more control that's all
[09:28] kpenny: and also allows you to do custom class paths
[09:28] kpenny: etc, as your apps grow -
[09:28] cluster_help: so you create a sep jvm.config for each instance, and lower the max heap size, so they all aren't trying to use all the available
[09:28] kpenny: and/or as you allocate resources - you have more granular control
[09:28] kpenny: well even though you're using the same jvm.config
[09:28] kpenny: its still a separate unique jvm
[09:28] kpenny: that they are running w/in -
[09:28] kpenny: which can be a misconception -
[09:29] kpenny: the jvm.config just configures the values for each of the individual jvms in use -
[09:29] kpenny: by separating out the configs, then you're able to have more control over 1 jvm out of the 4 say
[09:29] kpenny: vs. one change in one file for 'all' jvms
[09:30] cluster_help: roger that. but other than the obvious, if you have 8 instances and 8 x maxheap ends up being more than the available memory on the host, don't you risk having a problem?
[09:31] kpenny: yes
[09:31] kpenny: that's when you'd need either 64 bit - or multiple machines - etc -
[09:31] kpenny: but yeah you never wanna allocate 2 much
[09:31] cluster_help: so why my infrastructure guy's boss is looking for is a technote or a blog post or something that says you SHOULD lower the max in the .config
[09:31] kpenny: but with separation of the configs you can control that -
[09:32] kpenny: well really the max should be determined by the app
[09:32] cluster_help: yeah. Like I told my boss, I know HOW to do it. (I did it yesterday on my laptop!)
[09:32] kpenny: so if you have large mem requirements -
[09:32] kpenny: then it should be higher - i guess i'd say there's no set rule - just
[09:32] kpenny: that depending on performance tsting what's best for the app at a given time -
[09:33] kpenny: i.e. if you have multiple applications using frameworks and cs etc -
[09:34] cluster_help: here's the background on my company - the dev manager and the infrastructure manager constantly bicker and fight little political battles. the infrastucture guy and me have about 10 years of exp in each of our areas of expertise
[09:34] cluster_help: but both our bosses want us to prove via external sources what we recommend
[09:34] kpenny: ok -
[09:35] cluster_help: and I am saying, and my infrastructure guy concurs, that we need sep config files for each instance in the cluster, (for granular control) of the max memory allocations
[09:35] kpenny: whats your max set to now - ? and is there a memory problem i.e. using 2 much memory
[09:35] cluster_help: but we need to show them WHY we recommend that
[09:35] cluster_help: the cluster isn't even in production yet, this is a first time set up
[09:35] cluster_help: we're actually clustering the staging env first for load testing
[09:36] cluster_help: then we'll do it in production
[09:36] cluster_help: we've never clustered or had very good front end load balancing here
[09:36] kpenny: i guess one more good point to throw in the mix is this
[09:37] kpenny: some apps perform better on different jvm versions - heck even sun vs. ibm etc -
[09:37] kpenny: so - with that - w/o granular control you're stuck using the same version that all other apps are using
[09:37] kpenny: in a single config -
[09:37] kpenny: vs what we went through were were we had an app that uses coldspringn and
[09:37] cluster_help: right
[09:38] kpenny: benefited from the bug fix in 1.6.v10
[09:38] kpenny: so we were able to just change that config and run the app
[09:38] kpenny: others can use stock or whatever
[09:38] cluster_help: this is going to be 3 or 4 instances of a single app deployed as an .ear
[09:38] kpenny: so 3 jvm's
[09:38] kpenny: clustered?
[09:39] kpenny: i guess i'd argue for it having the separation b/c it's easier to do it from the start -
[09:39] kpenny: and you really can't predict how things will be configured 5 months from now -
[09:39] kpenny: it's not a big deal to separate them -
[09:39] kpenny: and you can always do it at a later date if needed 2
[09:40] cluster_help: yeah
[09:40] cluster_help: but as far as you know, there isn't anything out there in a kb article or a tech note that says WHY you should do it that way, its just kind of a common knowledge/common sense thing?
[09:41] kpenny: well i know for dev its essential
[09:41] kpenny: for reasons like control over debugging ports
[09:41] kpenny: and you're likely working with multiple applications each with their own requirements
[09:42] kpenny: so for dev i'ts a no brainer - but if you have production separated out where x apps are all on y vm's etc.
[09:42] kpenny: and the other ones are on separate ones and they never cross over
[09:42] kpenny: then you can likely get away with stock setup
[09:42] kpenny: however -
[09:42] kpenny: we have some 'maintenance' things that happen with the app
[09:42] kpenny: and we found it most beneficial to have this as a separate instace - even though tis the same app -
[09:43] kpenny: but run schedules through it -
[09:43] kpenny: for batch processes etc - off on it's own - which
[09:43] kpenny: has it's own requirements for mem etc -
[09:43] kpenny: and we benefited from having them separate configs
[09:43] kpenny: also -
[09:43] kpenny: if you wanted to setup a scheduler 'instance' where all it did is kick off
[09:43] kpenny: cf schedules to your app
[09:43] kpenny: you don't wanna allocate 512 mb ram for that guy
[09:44] kpenny: likely small 128 via config
[09:44] kpenny: and run it alongside or whatever -
[09:44] kpenny: so there are many use cases - and they all depend on your preference
[09:44] kpenny: debugging locally with multiple instances w/o having separate jvm configs is impossible
[09:44] kpenny: so that's #1 reason I started doing it -
[09:45] kpenny: then after that i noticed all these other possibilities
[09:45] cluster_help: right, but if we're doing the multiple instances so we can have load balance and failover, each instance should be set to the max we can afford given the machines total available RAM and taking into account the OS's needs and stuff like that?
[09:45] cluster_help: in a production environment?
[09:46] kpenny: if you're app performs best with the max at that level -
[09:46] kpenny: some apps won't -
[09:46] kpenny: and typically if you allocate over what 1.8gb?
[09:46] kpenny: the service won't start?
[09:46] cluster_help: so really the load testing we're going to do will let us know if the settings will work
[09:46] kpenny: yup
[09:46] cluster_help: i think its 1.2gb.
[09:46] kpenny: right ok -
[09:47] kpenny: yeah try it at various levels -
[09:47] cluster_help: cool - hey, thanks for the info man, it really helps
[09:47] kpenny: np - ! best of luck -
[09:47] cluster_help: thanks, i appreciate it

No comments: