Separating the JVM config files - A Cluster Conversation

I had an interesting discussion with a fellow who was looking to solidify his stand with his boss on jvm configuration separation and the need for it. Below is the IM conversation the ensued - I think it helps to show some of the various points that could be made arguing FOR having the separation of the config files per logical JVM (names changed to protect the innocent and guilty)

[09:20] cluster_help: Hello, i was just talkin to Jared about some jvm clustering stuff and he said you were way smarter than him on that stuff
[09:20] kpenny: sure ok -
[09:20] cluster_help: we're setting up a few clustered instances for a large app on a single vmware box
[09:21] kpenny: cool - 32 bit?
[09:21] cluster_help: i know i should have each instance running its own jvm config so we (yeah, 32) can lower the max heap etc so all the instances aren't trying to use ALL the available memory
[09:22] cluster_help: i can find lots of posts and articles on HOW. Do you know of any that mention WHY?
[09:22] cluster_help: need to help my infrastructure guy convince his boss
[09:22] kpenny: so 'why' separate the instances?
[09:22] cluster_help: load balance and failover
[09:23] cluster_help: its a custom installation of our main app for a large, high traffic customer
[09:23] cluster_help: we're isolating them from all our "regular" customers
[09:23] kpenny: yup - both excellent reasons - keep the app UP -
[09:24] kpenny: so you'll have x vm's
[09:24] cluster_help: yea
[09:24] cluster_help: and a gig of RAM to allocate among them
[09:25] kpenny: i also recommend if you have say 8 processor box - not to give each vm virtual procs that add up t o 8
[09:25] kpenny: as the ESX host for example would need cpu 2
[09:25] cluster_help: right, it is an ESX host
[09:25] kpenny: k - we noticed that when we dedicated 2 cores per vm
[09:25] kpenny: we were cpu strapped -
[09:26] kpenny: also you have to think about memory utilization - if you have 4g ram on each vm
[09:26] kpenny: and only a single jvm - likely the jvm max is 1.2G -
[09:26] kpenny: and the os isn't going to use 2.8g ram
[09:26] cluster_help: u know of anything published anywhere that describes WHY its a good idea to lower the max (allocate memory judiciously) across each instance
[09:26] kpenny: so you're also leveraging your hardware better
[09:26] cluster_help: right
[09:27] kpenny: describe - lower the max -
[09:27] cluster_help: when you create new instances, they all use the same jvm.config, so they all get allocated whatever the default is
[09:27] kpenny: right - separating them gives you more control that's all
[09:28] kpenny: and also allows you to do custom class paths
[09:28] kpenny: etc, as your apps grow -
[09:28] cluster_help: so you create a sep jvm.config for each instance, and lower the max heap size, so they all aren't trying to use all the available
[09:28] kpenny: and/or as you allocate resources - you have more granular control
[09:28] kpenny: well even though you're using the same jvm.config
[09:28] kpenny: its still a separate unique jvm
[09:28] kpenny: that they are running w/in -
[09:28] kpenny: which can be a misconception -
[09:29] kpenny: the jvm.config just configures the values for each of the individual jvms in use -
[09:29] kpenny: by separating out the configs, then you're able to have more control over 1 jvm out of the 4 say
[09:29] kpenny: vs. one change in one file for 'all' jvms
[09:30] cluster_help: roger that. but other than the obvious, if you have 8 instances and 8 x maxheap ends up being more than the available memory on the host, don't you risk having a problem?
[09:31] kpenny: yes
[09:31] kpenny: that's when you'd need either 64 bit - or multiple machines - etc -
[09:31] kpenny: but yeah you never wanna allocate 2 much
[09:31] cluster_help: so why my infrastructure guy's boss is looking for is a technote or a blog post or something that says you SHOULD lower the max in the .config
[09:31] kpenny: but with separation of the configs you can control that -
[09:32] kpenny: well really the max should be determined by the app
[09:32] cluster_help: yeah. Like I told my boss, I know HOW to do it. (I did it yesterday on my laptop!)
[09:32] kpenny: so if you have large mem requirements -
[09:32] kpenny: then it should be higher - i guess i'd say there's no set rule - just
[09:32] kpenny: that depending on performance tsting what's best for the app at a given time -
[09:33] kpenny: i.e. if you have multiple applications using frameworks and cs etc -
[09:34] cluster_help: here's the background on my company - the dev manager and the infrastructure manager constantly bicker and fight little political battles. the infrastucture guy and me have about 10 years of exp in each of our areas of expertise
[09:34] cluster_help: but both our bosses want us to prove via external sources what we recommend
[09:34] kpenny: ok -
[09:35] cluster_help: and I am saying, and my infrastructure guy concurs, that we need sep config files for each instance in the cluster, (for granular control) of the max memory allocations
[09:35] kpenny: whats your max set to now - ? and is there a memory problem i.e. using 2 much memory
[09:35] cluster_help: but we need to show them WHY we recommend that
[09:35] cluster_help: the cluster isn't even in production yet, this is a first time set up
[09:35] cluster_help: we're actually clustering the staging env first for load testing
[09:36] cluster_help: then we'll do it in production
[09:36] cluster_help: we've never clustered or had very good front end load balancing here
[09:36] kpenny: i guess one more good point to throw in the mix is this
[09:37] kpenny: some apps perform better on different jvm versions - heck even sun vs. ibm etc -
[09:37] kpenny: so - with that - w/o granular control you're stuck using the same version that all other apps are using
[09:37] kpenny: in a single config -
[09:37] kpenny: vs what we went through were were we had an app that uses coldspringn and
[09:37] cluster_help: right
[09:38] kpenny: benefited from the bug fix in 1.6.v10
[09:38] kpenny: so we were able to just change that config and run the app
[09:38] kpenny: others can use stock or whatever
[09:38] cluster_help: this is going to be 3 or 4 instances of a single app deployed as an .ear
[09:38] kpenny: so 3 jvm's
[09:38] kpenny: clustered?
[09:39] kpenny: i guess i'd argue for it having the separation b/c it's easier to do it from the start -
[09:39] kpenny: and you really can't predict how things will be configured 5 months from now -
[09:39] kpenny: it's not a big deal to separate them -
[09:39] kpenny: and you can always do it at a later date if needed 2
[09:40] cluster_help: yeah
[09:40] cluster_help: but as far as you know, there isn't anything out there in a kb article or a tech note that says WHY you should do it that way, its just kind of a common knowledge/common sense thing?
[09:41] kpenny: well i know for dev its essential
[09:41] kpenny: for reasons like control over debugging ports
[09:41] kpenny: and you're likely working with multiple applications each with their own requirements
[09:42] kpenny: so for dev i'ts a no brainer - but if you have production separated out where x apps are all on y vm's etc.
[09:42] kpenny: and the other ones are on separate ones and they never cross over
[09:42] kpenny: then you can likely get away with stock setup
[09:42] kpenny: however -
[09:42] kpenny: we have some 'maintenance' things that happen with the app
[09:42] kpenny: and we found it most beneficial to have this as a separate instace - even though tis the same app -
[09:43] kpenny: but run schedules through it -
[09:43] kpenny: for batch processes etc - off on it's own - which
[09:43] kpenny: has it's own requirements for mem etc -
[09:43] kpenny: and we benefited from having them separate configs
[09:43] kpenny: also -
[09:43] kpenny: if you wanted to setup a scheduler 'instance' where all it did is kick off
[09:43] kpenny: cf schedules to your app
[09:43] kpenny: you don't wanna allocate 512 mb ram for that guy
[09:44] kpenny: likely small 128 via config
[09:44] kpenny: and run it alongside or whatever -
[09:44] kpenny: so there are many use cases - and they all depend on your preference
[09:44] kpenny: debugging locally with multiple instances w/o having separate jvm configs is impossible
[09:44] kpenny: so that's #1 reason I started doing it -
[09:45] kpenny: then after that i noticed all these other possibilities
[09:45] cluster_help: right, but if we're doing the multiple instances so we can have load balance and failover, each instance should be set to the max we can afford given the machines total available RAM and taking into account the OS's needs and stuff like that?
[09:45] cluster_help: in a production environment?
[09:46] kpenny: if you're app performs best with the max at that level -
[09:46] kpenny: some apps won't -
[09:46] kpenny: and typically if you allocate over what 1.8gb?
[09:46] kpenny: the service won't start?
[09:46] cluster_help: so really the load testing we're going to do will let us know if the settings will work
[09:46] kpenny: yup
[09:46] cluster_help: i think its 1.2gb.
[09:46] kpenny: right ok -
[09:47] kpenny: yeah try it at various levels -
[09:47] cluster_help: cool - hey, thanks for the info man, it really helps
[09:47] kpenny: np - ! best of luck -
[09:47] cluster_help: thanks, i appreciate it

Lowering the Number of Verity Threads on Servers with Large Number of Collections

With our 55+ collections - we have some suggested ways to better manage the amount of threads that Verity has to manage. Below is the suggestion direct form Adobe Tier 3 ColdFusion/Jrun support consultants.

So, the suggested workaround to prevent the issue of collectionnames being ‘switched’ is to lower the number of threads assigned to each collection in Verity. Verity support had indicated this is an issue they are familiar with. They see it when the ratio of (total number of threads/CPU) gets high. Your total threads is about 165 now (55 collections * 3 threads). They suggest lowering the threads per collection to 2. Note, it is generally not recommended you use less than 2 threads per collection. One thread per collection may work for collections that are not heavily used.

You can update the threads per collection by running Verity RCAdmin. I have pasted below the output from my updating a “linuxarchive” collection. You will run the rcadmin from your verity_root\k2\_nt40\bin directory in a DOS window.
1. type in rcadmin
2. at rcadmin prompt type in indexattach
3. type in collectionname for index alias
4. enter ‘c’ for collection
5. enter the search server alias (ColdFusionK2_server1)
6. modify type is update = 0
7. index state is online = 2
8. Threads is the update you are making. All collections have 3 threads currently, which you should change to 2.
9. save changes=y
You should do this for each of your collections. Once you have updated all the collections you will need to stop and restart the Verity search service.

I have recently done this on our beta environment and the numbers fell from about 270 threads (task manager - Processes - view - Select columns - thread count) to 210 or so. We'll see how this effects overall management of the collections. This is not necessary on servers with lower number of managed collections.

64 Bit Microsoft Windows ODBC Drivers (Excel)

If you've moved from a 32 bit environment to a 64 bit one and are using the Microsoft ODBC drivers for things like Excel (reading a worksheet) and Access, I'm afraid you're out of luck (this includes even text files etc).

I just found out the hard way that code written to pull data using the 'jdbc:odbc:Driver={Microsoft Excel Driver (*.xls)}' no longer works in Windows Server 64 bit edition. Microsoft apparently feels that they don't need to provide 64 bit drivers at all for Office connectivity. Which I guess means that we as developers and software designers should no longer support Office, which is OK by me.

I've hated working with Excel to import/export data to / from, and have had nothing but trouble either direct to the Database with OPENDATASOURCE and having to move uploaded files to the database server, or simply on the server through a jdbc odbc call.

I'm fine with Eliminating Excel as a valid import option - I just need to find / create a fast alternative that will do CSV or other formats (not using CF to do the heavy work with those files as they can be large). There is one ray of hope - in an opencsv format found here.

Generic ColdSpring Gateway Service Code

I've been digging into ColdFusion's Gateway's of late and created a simple generic Gateway that works great with coldspring (GatewayService.cfc). The beauty is the simplicity. With this process, you can send any 'message' to the gateway and have the application run it asynchronously. The message you send will be any unit of work in your coldspring model. For example if you have an XmlManager object that has a method of 'generateRSSFeeds', you can pass it a Structure of data as an argumentcollection, as well as the object name and method name and have coldspring execute that for you w/in your Gateway.

<cffunction name="onIncomingMessage" output="false" access="public" returntype="void" hint="Generic Method that will execute any method on the Service Factory - Struct: beanname (coldspring Bean to call), methodname (method to call on cs bean), argumentcollection (data passed as argument collection to the method)">
<cfargument name="CFEvent" type="struct" required="yes">

<cfset var data = arguments.CFEvent.data/>
<cfset var errorMessage = ''/>

<cfif not structkeyexists(data,"args")>
<cfset data.args = StructNew()>
</cfif>
<!---
Structure:

beanname
methodname
args: argumentcollection

--->
<cftry>
<cfinvoke component="#application.serviceFactory.getBean(data.beanname)#"
method="#data.methodname#"
argumentcollection="#data.args#"/>
<cfcatch type="any">
<cfset errorMessage = cfcatch.Message/>
</cfcatch>
</cftry>
<cfif structkeyexists(data.args,"debug")>
<cflog application="true" file="cfmlgateway" text="CF Gateway Called: bean: #data.beanname# method: #data.methodname# data: #structKeylist(data.args)# #errorMessage#">
</cfif>

<cfreturn />
</cffunction>

There is also a 'server.init.cfm' feature that sets up this gateway through the ColdFusion Admin API and makes sure that it exists in the server that's running your application - and any changes to the gateway it will determine and recreate it when needed. The gateway is a CFML gateway and is given a name that uniquely identifies it (just like a DSN). Note: You must call the login method of the cfide.adminapi.administrator before you are allowed access to do this.

<cfset Local.gatewaypath = expandpath('.') & '\model\gateway\GatewayService.cfc'/>
<cfset Local.eventGateway = createObject( "component", "cfide.adminapi.eventgateway")/>
<cfset Local.gatewayFound = false/>

<!--- query the gateway Instances to find a match and if not create it --->
<cfset Local.tmpArray = Local.eventGateway.getGatewayInstances()/>

<cfloop from="1" to="#arraylen(Local.tmpArray)#" index="Local.i">
<cfif Local.tmpArray[Local.i].gatewayid eq 'mygatewayname'>
<cfset Local.gatewayFound = true/>

<!--- check and see if the cfcpath is the same- if not delete it and recreate it --->
<cfif Local.tmpArray[Local.i].cfcpaths[1] neq Local.gatewaypath>
<cfset Local.eventGateway.deleteGatewayInstance(Local.tmpArray[Local.i].gatewayid)>
<cfset Local.gatewayFound = false/>
</cfif>
<cfbreak/>
</cfif>
</cfloop>


<cfif not Local.gatewayFound>

<!--- Create gateway instance --->
<cfset Local.tmpArray = ArrayNew(1) />
<cfset Local.tmpArray[1] = Local.gatewaypath/>

<cfset Local.eventGateway.setGatewayInstance('mygatewayname','CFML',Local.tmpArray,'','auto') />
<cfset Local.eventGateway.startGatewayService() />

<cfset Local.eventGateway.startGatewayInstance('mygatewayname') />
<cflog log="Application" text="Gateway mygatewayname Created" type="information"/>
</cfif>

There is also a 'gatewaymanager.cfc' that simply is the API to all your gateway service calls, in which you can add on other things like debugging and error handling etc if needed.

<cffunction name="sendMessage" output="false" access="public" returntype="boolean" hint="Returns true of false if the message was successfully sent to the gateway">
<cfargument name="beanname" type="string" required="true"/>
<cfargument name="methodname" type="string" required="true"/>
<cfargument name="args" type="struct" required="false" default="#StructNew()#"/>

<cfset var boolReturn = SendGatewayMessage('mygatewayname',arguments)>

<cfreturn boolReturn />
</cffunction>

With great power come great responsibility - please use wisely!

Object Refactoring into Structures for Performance

Lately I've been tasked with refactoring some of our batch processes that are taking entirely too long to run - especially with large datasets (10-25K records). I've noticed one thing in common with each procedure that I've had to rewrite - they have heavy reliance on objects to do the work for them i.e. I'm doing some work on a 'job' object, for each job i'll call 'getJobById' which returns a populated job bean from the db in which I manipulate it and then call the 'save' method on it - and repeat 10-25 thousand times. With cf8.01 we're seeing heap size of the JVM being used up after a period of time crunching through these methods. They become slow to the point of unresponsive and ultimately bring the jvm down or cause the heap size to become too large and the process halted.

Here are some of the highlights that I've done in refactoring these processes:

1.) Only query for the data I need (i.e. NOT selecting all records when all i need are the id field and a location field)
2.) Verity that all local variables are var scoped.
3.) remove any calls to 'getJobById' in favor of a simple reusable JobStructure.
4.) populate the job Structure from the query directly and then pass this simplified structure into an alternate method that accepts the structure and does any massaging it needs to with the data (returning it by reference).
5.) Create a specific 'update' method in my DAO that accepts this Structure and saves ONLY the data that is needed vs. the entire bean object.

The Results??? Extremely fast execution and stable memory throughout the entire process - regardless of how long the entire process may take. We went from roofing the JVM to maintaining a stable memory footprint on these executions, while also staying true to good programming techniques - and even 'introspecting' a single Object before the run and exposing it's internal 'metadata' to the job Structure as a base for the operation - and then proceeding from there - Appending data and overwriting the Structure each time through the loop.
i.e. StructAppend(jobStruct,newStruct,'true')

CreateObject('component','job').init().getInstance()
We have all our instance data in a 'variables.instance' structure w/in the object. We also create a 'getInstance' method which returns that structure. That means we can use this structure of the objects data instead of using a new object each time - NOR using the bean each time. We're still able to change/modify the Bean metadata and still have that available to the other calling methods that require it.

I.e. We can call the 'getBeanConfig()' method on the job object which returns a structure of metadata (structure of structures) about things like field size, and nullability etc. of each field in our bean.

<cffunction name="getBeanconfig" output="false" access="public" returntype="Struct" hint="returns the entire bean config structure">
<cfreturn variables.beanconfig />
</cffunction>

<cffunction name="setBeanConfig" access="private" returntype="void" output="false" displayname="category config" hint="I setup the configuration for the bean category.">
<cfscript>
variables.beanconfig = structnew();
variables.beanconfig.title = StructNew();
variables.beanconfig.title.isnullable=0;
variables.beanconfig.title.maxlength = 95;
</cfscript>
</cffunction>

When working with thousands of beans in a process - originally it was a lot easier to get the thing working by just calling 'getBeanById()' and then manipulating it and calling 'Save()' on it. Over time I've learned that it's not sustainable and may need to be refactored. As long as you're being careful to var scope all your local variables in your cfcs and use a lowest common denominator approach to large long running batch requests, you will see a large performance gain by extracting your object structure data and working with a single copy of it vs. populating your entire bean each time through a loop and saving the entire bean once you're done with it. That plus leveraging the CFML Gateway to thread your executions, you'll be off the ground flying once again!

Learn Java with Video Tutorials

Excellent reference for all ColdFusion Developers to extend their knowledge base.

Learn Java with Video Tutorials (free)

Free video screencam tutorials for Eclipse and Java. Includes "Eclipse and Java for Total Beginners", "Using the Eclipse Workbench", "Introducing Persistence", and "Using the Debugger". Intended for beginning and intermediate users and programmer

Learn the Eclipse IDE for Java with Video Tutorials showing the features of the language and IDE together in a very clear and concise manner from Mark Dexter.

Enjoy - thank you Mark!

http://sourceforge.net/projects/eclipsetutorial/

ColdFusion Gateway Data passed by Reference?

At our company tech summit just held the past few days (where all us remote developers get together and discuss all things technical) I had the opportunity to show off some of the cool new CFML Gateways functionality that ships with CF 8. I had created a generic object that could be called at any point in the application that would allow asynchronous threading of method executions where we needed it. I would execute

SendGatewayMessage(gatewayid, structure)

call which would then go through the cfml gateway and exeute the method that ColdSpring new about (code to come).

This was all great and fun, but what I wanted to know is this:

'What if I have an object that has it's data changed from inside the gateway. Will the calling page know about that change, and does the calling page still have reference to the object?'.

Objects and Structures are passed by 'reference' in ColdFusion. But what if that object is being manipulated in an Asynchronous process that is executing in a CFML Event Gateway in a separate thread from the original? That was my 'Poll the audience' question in which I asked our developers to think about and come up with their answer -

A - The Object would Change
B - The Object would not Chanage

I put together a CFC and Gateway call that tested this theory.

1.) Create an Object and assign a value through an accessor method i.e. setTitle('Bank Foreclosure Professional')

2.) Append that object to the data 'structure' that I am passing to the 'SendGateWayMessage(gatewayid, structure)

3.) Dump Data of the Object to a cfm page

4.) Call the Gateway and pass the structure containing the object

5.) Within the gateway call, do a thread sleep for 2.5 seconds and then call a setTitle('I have changed') on the Object.

6.) Meanwhile back on the .cfm page, right after my call to the 'sendgatewaymessage' function i will dump the value of the object again to the page (as this dump will happen immediately after the sendgatewaymessage call is executed asynchronously)

7.) On the .cfm page I will do a thread sleep for 5 seconds to make sure the gateway has enough time to change the value of the object.

8.) Finally, once I feel I've waited long enough, I'll dump the results of the Original Object to the page....

The Results????

You may be surprised at the results - and in the room of 9 people, we split the room in half with 'No Change' and 'Change' votes - the tie breaker coming from our Project Manager who (like myself) guessed 'NO Change' (i.e. there is no reference to the original object once it get's passed into the gateway and executes).

Who was right?? Please provide your comments and I'll let you know what I found out and who won the Poll (I'll post the code for those who want to see it and prove it on their own)