Deploying Cold Fusion to Google's App Engine and Big Table

At the ColdFusion User Group in Minneapolis Wednesday November 3, 2011 I will be presenting on the topic of "Deploying Cold Fusion to Google's App Engine and Big Table".


Learn the basics of app engine, the deployment methodologies required to develop a Cold Fusion application with Open Bluedragon on gae, and the nuances of working with 'big table', Googles distributed object data store.


Visit ColderFusion.com to RSVP!


Looking forward to seeing you there


Update:
Those interested in the files (pdf, some code minus the jar files) can download it here.  If you're looking for the magic bits for the working OpenBD version, see the pdf link for the enlist project link.

(Minneapolis) Application Architect / Developer Job - Jobs2Web, Inc.

Location: Minnetonka, MN, USA

Are you looking for an opportunity to maximize your career with one of Minnesota’s fastest growing internet startups? Jobs2Web is one of the few venture backed Minnesota startups which made the Inc 500 list in 2009 and 2010 for our fast growth, and now we’re looking to expand our team of passionate professionals in the human capital software marketplace.

Jobs2Web helps employers maximize their interactive recruiting strategy using our recruitment marketing platform. The Jobs2Web platform helps major employers to attract, capture, and communicate with quality talent all while measuring recruiting effectiveness using our patent-pending solutions which dramatically reduce dependency on expensive and less effective sourcing mediums.

Working within our company will put you close to our award winning technology that combines the best of recruiting and marketing solutions including Search Engine Marketing (SEO), SEM (Pay-Per-Click), Social Networks and Social Media Marketing, and Email Marketing.

The Architect / Developer enhances and maintains the Jobs2Web Recruitment Marketing Platform and associated applications.

Specifically, the Application Architect / Developer role will have impact through these responsibilities:

• Create and maintain the application architecture for the Jobs2Web platform, provide application architecture guidance to the development group and other Jobs2Web departments, and engage in hands-on development as required.
• Participate in a dynamic team environment, including frequent team feedback, collaboration, training and code review.
• Professionally communicate with our internal clients, including Client Services and Sales.
• Demonstrate ability to manage multiple priorities in a fast-paced environment.

Desired Qualifications and Competencies:

• Bachelors Degree or equivalent work experience, with at least 7 years of application development experience required.
• Experience with Java required. Desired areas of expertise include
  • Java EE 6
  • Apache Solr/Lucene
  • JMS
  • JPA/Hibernate
  • GlassFish 3
  • JSF
  • EJB 3.1
• Experience in creation and maintenance of enterprise-grade application architecture, including SOA and messaging systems is required
• Experience in SQL language and database technologies required.
• Application of OO principles and use of best practices in web development required, with a strong background in application architecture.
• ColdFusion MX experience desired, with experience using application frameworks and ColdFusion components.
• Exposure to Adobe Flex 3 applications desired.
• Competent in use of HTML, XML, CSS, and AJAX technologies.
• Experience working in an Agile Scrum environment is desired.
• Proficient in use of source control, defect management, and release management tools.
• Consistently demonstrates strong team and individual performance.
• Effective written and oral communication skills as well as strong interpersonal skills are essential.
• Builds positive and effective relationships with individuals both internal and external to the organization.
• Staffing/HR industry knowledge a plus.

Jobs2Web is an entrepreneurial employer that challenges and rewards all employees to succeed at every opportunity! Your talents are fully utilized as we are building this rapidly growing business. Our competitive market driven package includes a total rewards package of competitive pay, bonus, health insurance coverage, 401(k), career development, a very exciting marketplace and a really great work culture.

Nearest Major Market: Minneapolis
Job Segments: Architecture, ColdFusion, Computer, Creative, Database, Developer, Engineer, Engineering, Information Technology, Interactive Marketing, Java, Legal, Marketing, Patent, Product Development, Programmer, QA, Quality, Quality Assurance, Research, SEO, SQL, Technology, Web Design, XML

Apply Now 

mxUnit Testing with Hudson CI Server @ CFUG Minneapolis

I'll be speaking Wednesday Nov 3, 2010 @ the ColdFusion User Group in St. Paul/Minneapolis on mxUnit Testing with Hudson Continuous Integration Server.

I'll be talking about basic Assertions, as well as going into some advanced mocking topics with the new 2.x release of the mxUnit Framework.
The second part of the evening we'll look at the Hudson Server to help execute our Unit Tests and provide jUnit reports that show execution trends as well as some special surprises.

See you there!

cf_PropertyInspector Bolt Extension Released!

Version 1 of the cf_PropertyInspector ColdFusion Builder/Bolt Extension for your IDE has been released on riaforge (pending).

Features Include:
For ease of Flex Integration, CFProperty tags are needed to have your methods available to ActionScript classes. Also complex webservices need this metadata defined for them. With existing components already created, having to go back in and add in all these cfproperty tags for all your 'accessors' is a pain. How about just right clicking -> 'Inspect CFC' and be done with it?? Best used if components exist under your webroot.

Features:
- Searching subdirectories
- Alpha sorting properties
- Wverwriting/keeping existing properties
- Lower case properties
- Exclusion of Methods (i.e getInstance, getBinaryNull etc)


Version 2 of this application will include the following:
- Backups of cfc's before modification
- Fine Tune properties before wizard completes
- Mapping support if your components are not located under your webroot.


Requirements:
ColdFusion 8+
ColdFusion Builder (Beta 2+) Build: 240677

Download NOW Available on riaforge.





















Final Result!

Disable CFC Type Check Proof

I usually question a lot of the things I hear, and this one I just had to set the record straight. I can stand being corrected on things when I'm proven wrong so I'm hoping this will come across as simply a clarification and not any kind of 'I told you so'.

But in the CFADmin in CF8 'Settings' there is a 'Disable CFC type Check' option that allows you to turn ON the ability to ignore the argument 'types' of CFC's. For example, if I have a component of type 'com.adobe.reactor' and I pass that to a method expecting a 'com.adobe.transfer', if this option is ON, then there would be NO type checking and the argument would be successfully passed in (I'd assuming something else would fail after that however).

At cfObjective this year it was stated at a session that if this option was turned ON, a datatype like 'numeric' would be able to accept a 'string' value as there would be no type checking. This is simply not true

Proof:

Visit your cfadmin and turn ON the 'Disable CFC Type Check' option and restart your server.
Run the following script:

<cffunction name="test" output="false" access="public" returntype="any" hint="">
<cfargument name="num" type="com.adobe.transfer" required="true">
<cfreturn>
</cfreturn>

<cfset x="test('hi')">
<cfdump var="#x#">

You'll notice that this works just fine. Notice that I'm passing a string of 'hi' into the test method that is expecting a 'com.adobe.transfer' object. This works just fine with Disable CFC Type Checking ON as it simply treats the argument type as an 'ANY' type.

The CFAdmin states:

"When checked, UDF arguments of CFC type is not validated. The arguments are treated as type "ANY". Use this setting in a production environment only."

In this case 'com.adobe.transfer' is a UDF or user defined argument - I defined it in a directory of com/adobe/transfer.cfc.

However, when I change my test method to accept a 'numeric' value instead of this UDF value, you will see it will throw an exception:

<cffunction name="test" output="false" access="public" returntype="any" hint="">
<cfargument name="num" type="numeric" required="true">
<cfreturn>
</cfreturn>

"The NUM argument passed to the test function is not of type numeric."

This proves that ONLY UDF arguments are changed to 'ANY' types when 'Disable CFC Type Check' is turned ON in cfadmin (oh no I've gone cross eyed)

Imagine the security holes created by such an option if indeed the native datatypes were all treated as 'ANY'? SQL Injection could occur if you had typed the cfargument as a 'numeric' datatype. A malicious user could simply pass in a string value and bypass your initial line of attack - that being your cfargument data type. This would not be a good situation (more information on security and cfml see Jason Dean's blog at 12Robots.com).

The fact is, regardless of whether or not 'Disable CFC Type Check' is ON or OFF in your CFAdmin Setttings, native Datatypes still need to be adhered to, or an exception will be thrown.

A ColdFusion Developers Keyboard

How can you tell what kind of developer you are by your Keyboard??

Check out my keyboard - Notice the 'C' key is all but worn right off - This has been from about 2 years of programming on this external keyboard - My previous laptop was the same way!

Just for fun, shoot me a picture of your keyboard - whether ColdFusion Programmer, .Net Programmer or dba (would your 'S' key be worn down from all the SELECTS?).

Enjoy

Query of Queries is Inconsistent (Adobe CFML Runtime)

I've never been a big fan of a half baked solution, but I will acknowledge that coldfusion's query of queries has come a long way from it's beginning and does do a good job on simple queries to get data from an existing query.

However, there are times when it simply sucks at doing what you think it should do, especially when it comes to multiple tables and joins, as well as converting datatypes to be alike.

I'll show a few examples of it's inconsistencies and attempt to point out some problems with it's implementation.

Example 1: Q-of-Q (Query of Queries) and Proper variable scoping inside a cffunction.

When inside a cffunction you always want to fully qualify your variables as best practice. If it's an arguments, prepend the value with 'arguments.' etc. The same goes for Q-of-Q. But I will show you where q-of-q forces you to break it's own best practice advice by not allowing you to do this simple task.

I will be referencing the following 2 queries:

<!--- Test Query 1 --->
<cfset q = QueryNew('id,name','Integer,varchar')>
<cfset queryaddrow(q,1) >
<cfset querysetCell(q,'id',1,1) >
<cfset querysetcell(q,'name','test',1) >

<!--- Test Query 2 --->
<cfset q2 = QueryNew('id,age','Integer,Integer')>
<cfset queryAddRow(q2,1) >
<cfset querySetCell(q2,'id',1,1) >
<cfset querySetCell(q2,'age',33,1) >

This is a simple query that creates a single row of data




What I'm going to do is merge the results of these 2 queries (as one may have come from a cfdirectory, and the other from a cfsearch, or cfquery etc).

My method takes 2 query arguments and joins them - but for now I'll just return the value of the 1st argument 'qry'.

<cffunction name="merge" output="false" access="public" returntype="query" hint="">
<cfargument name="qry" type="query" required="true"/>
<cfargument name="qry2" type="query" required="true"/>
<cfset var Private=StructNew()>

<cfquery name="private.q" dbtype="query">
select *
from arguments.qry
</cfquery>

<cfreturn private.q />
</cffunction>

Resulting in a simple query returned - nothing special.



Now when i wish to merge the 2 queries - I would assume that I could do something like this then if 'arguments.qry' worked to reference the qry argument.

<cffunction name="merge" output="false" access="public" returntype="query" hint="">
<cfargument name="qry" type="query" required="true"/>
<cfargument name="qry2" type="query" required="true"/>
<cfset var Private=StructNew()>

<cfquery name="private.q" dbtype="query">
select *
from arguments.qry, arguments.qry2
where arguments.qry.id = arguments.qry2.id
</cfquery>

<cfreturn private.q />
</cffunction>

WRONG!
Query Of Queries syntax error.
Encountered "arguments . qry .. Incorrect conditional expression, Expected one of [like|null|between|in|comparison] condition,

Ok fine - I won't use proper best practices and just reference the query w/o the arguments proper fully scoped value.

<cffunction name="merge" output="false" access="public" returntype="query" hint="">
<cfargument name="qry" type="query" required="true"/>
<cfargument name="qry2" type="query" required="true"/>
<cfset var Private=StructNew()>

<cfquery name="private.q" dbtype="query">
select *
from qry, qry2
where qry.id= qry2.id
</cfquery>

<cfreturn private.q />
</cffunction>

Results: works fine



Example 2: Using QofQ (query of queries) to reference a column from the table using dot notation

The ColdFusion LiveDocs states the following:

Example

If a structure named A contains a field named B, which contains a table named Products, you can refer to the table with dot notation, as follows:

SELECT tape_ID, length
FROM A.B.Products;

The 'structure' in my case is 'arguments', the field name is 'id', and the table name is 'qry'. With that in mind:

A=arguments
B=id
Products=qry

Therefore you could deduce that the following is true:

SELECT id, name
FROM arguments.id.qry

Results:

Query Of Queries syntax error.
Encountered ". Incorrect Select List, Incorrect select column, arguments.id cannot be followed by '.'


However, you would normally reference this data as 'arguments.qry.id'.

Results:

Query Of Queries syntax error.
Encountered ". Incorrect Select List, Incorrect select column, arguments.qry cannot be followed by '.'

The kicker is - Neither of these work.

Manipulating CFC Properties through getMetaData

I started toying with the idea of creating a generic utility that would populate a cfc with ‘property’ as in cfproperty data and I have something work now – (this would make the compatibility between flex and the model that much better and more ‘well defined’)

Would this be of benefit to us – i.e. running our model through it which would then create all cfproperty tags for a given object based on the getter methods w/in the object – and determine datatype of string, numeric etc – based on the getter methods data etc –

What sort of things do I have to be careful of ??

Order of the cfproperty tags counts right – and Case sensitivity of the names obviously ??

So the process would either do it alpha – or by order that was returned by getMetaData on the object –

An interesting thing happens however when I’m doing this – I could actually build this into coldspring likely where these properties are autogenerated on the fly and never actually have to exist w/in the cfc (written back to the cfc)

The strange thing happened was this:

Once I do a create object on a cfc – I can dump it and it shows that there are NO properties defined in the meta data – which is good

So I run it through my process where I’m introspecting the cfc and create this ‘properties’ key and append the array to this key.

This works well and now a side by side before and after shows the left object with NO properties and the right with Properties – great.

However the next time I run it – the left side also shows that the object now HAS properties –

This is strange – b/c cfproperty tags don’t actually exist in the cfc – but yet I’m able to fake cf (caching likely) into thinking / assigning these properties to the object.

For Example:

Here my Job Object has NO cfproperty tags:

I’m going to make ColdFusion ‘think’ it has these properties defined on it by manipulating the metadata about the object.

Simple App/UI

There are No properties on the left – On the right below, shows the ‘manipulated’ meta data based on the getter methods in the object.

Now after I run this again – you see both left and right are identical (cached metadata about the object)

The left side ‘thinks’ that the object now has these properties available to it- which if called or introspected by Flex, they should be available.

This feels a bit like cheating, but if it doesn’t work – we can always complete this thought by having the process ‘write’ the cfproperty tags back to the object.

Interesting huh?

Separating the JVM config files - A Cluster Conversation

I had an interesting discussion with a fellow who was looking to solidify his stand with his boss on jvm configuration separation and the need for it. Below is the IM conversation the ensued - I think it helps to show some of the various points that could be made arguing FOR having the separation of the config files per logical JVM (names changed to protect the innocent and guilty)

[09:20] cluster_help: Hello, i was just talkin to Jared about some jvm clustering stuff and he said you were way smarter than him on that stuff
[09:20] kpenny: sure ok -
[09:20] cluster_help: we're setting up a few clustered instances for a large app on a single vmware box
[09:21] kpenny: cool - 32 bit?
[09:21] cluster_help: i know i should have each instance running its own jvm config so we (yeah, 32) can lower the max heap etc so all the instances aren't trying to use ALL the available memory
[09:22] cluster_help: i can find lots of posts and articles on HOW. Do you know of any that mention WHY?
[09:22] cluster_help: need to help my infrastructure guy convince his boss
[09:22] kpenny: so 'why' separate the instances?
[09:22] cluster_help: load balance and failover
[09:23] cluster_help: its a custom installation of our main app for a large, high traffic customer
[09:23] cluster_help: we're isolating them from all our "regular" customers
[09:23] kpenny: yup - both excellent reasons - keep the app UP -
[09:24] kpenny: so you'll have x vm's
[09:24] cluster_help: yea
[09:24] cluster_help: and a gig of RAM to allocate among them
[09:25] kpenny: i also recommend if you have say 8 processor box - not to give each vm virtual procs that add up t o 8
[09:25] kpenny: as the ESX host for example would need cpu 2
[09:25] cluster_help: right, it is an ESX host
[09:25] kpenny: k - we noticed that when we dedicated 2 cores per vm
[09:25] kpenny: we were cpu strapped -
[09:26] kpenny: also you have to think about memory utilization - if you have 4g ram on each vm
[09:26] kpenny: and only a single jvm - likely the jvm max is 1.2G -
[09:26] kpenny: and the os isn't going to use 2.8g ram
[09:26] cluster_help: u know of anything published anywhere that describes WHY its a good idea to lower the max (allocate memory judiciously) across each instance
[09:26] kpenny: so you're also leveraging your hardware better
[09:26] cluster_help: right
[09:27] kpenny: describe - lower the max -
[09:27] cluster_help: when you create new instances, they all use the same jvm.config, so they all get allocated whatever the default is
[09:27] kpenny: right - separating them gives you more control that's all
[09:28] kpenny: and also allows you to do custom class paths
[09:28] kpenny: etc, as your apps grow -
[09:28] cluster_help: so you create a sep jvm.config for each instance, and lower the max heap size, so they all aren't trying to use all the available
[09:28] kpenny: and/or as you allocate resources - you have more granular control
[09:28] kpenny: well even though you're using the same jvm.config
[09:28] kpenny: its still a separate unique jvm
[09:28] kpenny: that they are running w/in -
[09:28] kpenny: which can be a misconception -
[09:29] kpenny: the jvm.config just configures the values for each of the individual jvms in use -
[09:29] kpenny: by separating out the configs, then you're able to have more control over 1 jvm out of the 4 say
[09:29] kpenny: vs. one change in one file for 'all' jvms
[09:30] cluster_help: roger that. but other than the obvious, if you have 8 instances and 8 x maxheap ends up being more than the available memory on the host, don't you risk having a problem?
[09:31] kpenny: yes
[09:31] kpenny: that's when you'd need either 64 bit - or multiple machines - etc -
[09:31] kpenny: but yeah you never wanna allocate 2 much
[09:31] cluster_help: so why my infrastructure guy's boss is looking for is a technote or a blog post or something that says you SHOULD lower the max in the .config
[09:31] kpenny: but with separation of the configs you can control that -
[09:32] kpenny: well really the max should be determined by the app
[09:32] cluster_help: yeah. Like I told my boss, I know HOW to do it. (I did it yesterday on my laptop!)
[09:32] kpenny: so if you have large mem requirements -
[09:32] kpenny: then it should be higher - i guess i'd say there's no set rule - just
[09:32] kpenny: that depending on performance tsting what's best for the app at a given time -
[09:33] kpenny: i.e. if you have multiple applications using frameworks and cs etc -
[09:34] cluster_help: here's the background on my company - the dev manager and the infrastructure manager constantly bicker and fight little political battles. the infrastucture guy and me have about 10 years of exp in each of our areas of expertise
[09:34] cluster_help: but both our bosses want us to prove via external sources what we recommend
[09:34] kpenny: ok -
[09:35] cluster_help: and I am saying, and my infrastructure guy concurs, that we need sep config files for each instance in the cluster, (for granular control) of the max memory allocations
[09:35] kpenny: whats your max set to now - ? and is there a memory problem i.e. using 2 much memory
[09:35] cluster_help: but we need to show them WHY we recommend that
[09:35] cluster_help: the cluster isn't even in production yet, this is a first time set up
[09:35] cluster_help: we're actually clustering the staging env first for load testing
[09:36] cluster_help: then we'll do it in production
[09:36] cluster_help: we've never clustered or had very good front end load balancing here
[09:36] kpenny: i guess one more good point to throw in the mix is this
[09:37] kpenny: some apps perform better on different jvm versions - heck even sun vs. ibm etc -
[09:37] kpenny: so - with that - w/o granular control you're stuck using the same version that all other apps are using
[09:37] kpenny: in a single config -
[09:37] kpenny: vs what we went through were were we had an app that uses coldspringn and
[09:37] cluster_help: right
[09:38] kpenny: benefited from the bug fix in 1.6.v10
[09:38] kpenny: so we were able to just change that config and run the app
[09:38] kpenny: others can use stock or whatever
[09:38] cluster_help: this is going to be 3 or 4 instances of a single app deployed as an .ear
[09:38] kpenny: so 3 jvm's
[09:38] kpenny: clustered?
[09:39] kpenny: i guess i'd argue for it having the separation b/c it's easier to do it from the start -
[09:39] kpenny: and you really can't predict how things will be configured 5 months from now -
[09:39] kpenny: it's not a big deal to separate them -
[09:39] kpenny: and you can always do it at a later date if needed 2
[09:40] cluster_help: yeah
[09:40] cluster_help: but as far as you know, there isn't anything out there in a kb article or a tech note that says WHY you should do it that way, its just kind of a common knowledge/common sense thing?
[09:41] kpenny: well i know for dev its essential
[09:41] kpenny: for reasons like control over debugging ports
[09:41] kpenny: and you're likely working with multiple applications each with their own requirements
[09:42] kpenny: so for dev i'ts a no brainer - but if you have production separated out where x apps are all on y vm's etc.
[09:42] kpenny: and the other ones are on separate ones and they never cross over
[09:42] kpenny: then you can likely get away with stock setup
[09:42] kpenny: however -
[09:42] kpenny: we have some 'maintenance' things that happen with the app
[09:42] kpenny: and we found it most beneficial to have this as a separate instace - even though tis the same app -
[09:43] kpenny: but run schedules through it -
[09:43] kpenny: for batch processes etc - off on it's own - which
[09:43] kpenny: has it's own requirements for mem etc -
[09:43] kpenny: and we benefited from having them separate configs
[09:43] kpenny: also -
[09:43] kpenny: if you wanted to setup a scheduler 'instance' where all it did is kick off
[09:43] kpenny: cf schedules to your app
[09:43] kpenny: you don't wanna allocate 512 mb ram for that guy
[09:44] kpenny: likely small 128 via config
[09:44] kpenny: and run it alongside or whatever -
[09:44] kpenny: so there are many use cases - and they all depend on your preference
[09:44] kpenny: debugging locally with multiple instances w/o having separate jvm configs is impossible
[09:44] kpenny: so that's #1 reason I started doing it -
[09:45] kpenny: then after that i noticed all these other possibilities
[09:45] cluster_help: right, but if we're doing the multiple instances so we can have load balance and failover, each instance should be set to the max we can afford given the machines total available RAM and taking into account the OS's needs and stuff like that?
[09:45] cluster_help: in a production environment?
[09:46] kpenny: if you're app performs best with the max at that level -
[09:46] kpenny: some apps won't -
[09:46] kpenny: and typically if you allocate over what 1.8gb?
[09:46] kpenny: the service won't start?
[09:46] cluster_help: so really the load testing we're going to do will let us know if the settings will work
[09:46] kpenny: yup
[09:46] cluster_help: i think its 1.2gb.
[09:46] kpenny: right ok -
[09:47] kpenny: yeah try it at various levels -
[09:47] cluster_help: cool - hey, thanks for the info man, it really helps
[09:47] kpenny: np - ! best of luck -
[09:47] cluster_help: thanks, i appreciate it

Generic ColdSpring Gateway Service Code

I've been digging into ColdFusion's Gateway's of late and created a simple generic Gateway that works great with coldspring (GatewayService.cfc). The beauty is the simplicity. With this process, you can send any 'message' to the gateway and have the application run it asynchronously. The message you send will be any unit of work in your coldspring model. For example if you have an XmlManager object that has a method of 'generateRSSFeeds', you can pass it a Structure of data as an argumentcollection, as well as the object name and method name and have coldspring execute that for you w/in your Gateway.

<cffunction name="onIncomingMessage" output="false" access="public" returntype="void" hint="Generic Method that will execute any method on the Service Factory - Struct: beanname (coldspring Bean to call), methodname (method to call on cs bean), argumentcollection (data passed as argument collection to the method)">
<cfargument name="CFEvent" type="struct" required="yes">

<cfset var data = arguments.CFEvent.data/>
<cfset var errorMessage = ''/>

<cfif not structkeyexists(data,"args")>
<cfset data.args = StructNew()>
</cfif>
<!---
Structure:

beanname
methodname
args: argumentcollection

--->
<cftry>
<cfinvoke component="#application.serviceFactory.getBean(data.beanname)#"
method="#data.methodname#"
argumentcollection="#data.args#"/>
<cfcatch type="any">
<cfset errorMessage = cfcatch.Message/>
</cfcatch>
</cftry>
<cfif structkeyexists(data.args,"debug")>
<cflog application="true" file="cfmlgateway" text="CF Gateway Called: bean: #data.beanname# method: #data.methodname# data: #structKeylist(data.args)# #errorMessage#">
</cfif>

<cfreturn />
</cffunction>

There is also a 'server.init.cfm' feature that sets up this gateway through the ColdFusion Admin API and makes sure that it exists in the server that's running your application - and any changes to the gateway it will determine and recreate it when needed. The gateway is a CFML gateway and is given a name that uniquely identifies it (just like a DSN). Note: You must call the login method of the cfide.adminapi.administrator before you are allowed access to do this.

<cfset Local.gatewaypath = expandpath('.') & '\model\gateway\GatewayService.cfc'/>
<cfset Local.eventGateway = createObject( "component", "cfide.adminapi.eventgateway")/>
<cfset Local.gatewayFound = false/>

<!--- query the gateway Instances to find a match and if not create it --->
<cfset Local.tmpArray = Local.eventGateway.getGatewayInstances()/>

<cfloop from="1" to="#arraylen(Local.tmpArray)#" index="Local.i">
<cfif Local.tmpArray[Local.i].gatewayid eq 'mygatewayname'>
<cfset Local.gatewayFound = true/>

<!--- check and see if the cfcpath is the same- if not delete it and recreate it --->
<cfif Local.tmpArray[Local.i].cfcpaths[1] neq Local.gatewaypath>
<cfset Local.eventGateway.deleteGatewayInstance(Local.tmpArray[Local.i].gatewayid)>
<cfset Local.gatewayFound = false/>
</cfif>
<cfbreak/>
</cfif>
</cfloop>


<cfif not Local.gatewayFound>

<!--- Create gateway instance --->
<cfset Local.tmpArray = ArrayNew(1) />
<cfset Local.tmpArray[1] = Local.gatewaypath/>

<cfset Local.eventGateway.setGatewayInstance('mygatewayname','CFML',Local.tmpArray,'','auto') />
<cfset Local.eventGateway.startGatewayService() />

<cfset Local.eventGateway.startGatewayInstance('mygatewayname') />
<cflog log="Application" text="Gateway mygatewayname Created" type="information"/>
</cfif>

There is also a 'gatewaymanager.cfc' that simply is the API to all your gateway service calls, in which you can add on other things like debugging and error handling etc if needed.

<cffunction name="sendMessage" output="false" access="public" returntype="boolean" hint="Returns true of false if the message was successfully sent to the gateway">
<cfargument name="beanname" type="string" required="true"/>
<cfargument name="methodname" type="string" required="true"/>
<cfargument name="args" type="struct" required="false" default="#StructNew()#"/>

<cfset var boolReturn = SendGatewayMessage('mygatewayname',arguments)>

<cfreturn boolReturn />
</cffunction>

With great power come great responsibility - please use wisely!

ColdFusion Gateway Data passed by Reference?

At our company tech summit just held the past few days (where all us remote developers get together and discuss all things technical) I had the opportunity to show off some of the cool new CFML Gateways functionality that ships with CF 8. I had created a generic object that could be called at any point in the application that would allow asynchronous threading of method executions where we needed it. I would execute

SendGatewayMessage(gatewayid, structure)

call which would then go through the cfml gateway and exeute the method that ColdSpring new about (code to come).

This was all great and fun, but what I wanted to know is this:

'What if I have an object that has it's data changed from inside the gateway. Will the calling page know about that change, and does the calling page still have reference to the object?'.

Objects and Structures are passed by 'reference' in ColdFusion. But what if that object is being manipulated in an Asynchronous process that is executing in a CFML Event Gateway in a separate thread from the original? That was my 'Poll the audience' question in which I asked our developers to think about and come up with their answer -

A - The Object would Change
B - The Object would not Chanage

I put together a CFC and Gateway call that tested this theory.

1.) Create an Object and assign a value through an accessor method i.e. setTitle('Bank Foreclosure Professional')

2.) Append that object to the data 'structure' that I am passing to the 'SendGateWayMessage(gatewayid, structure)

3.) Dump Data of the Object to a cfm page

4.) Call the Gateway and pass the structure containing the object

5.) Within the gateway call, do a thread sleep for 2.5 seconds and then call a setTitle('I have changed') on the Object.

6.) Meanwhile back on the .cfm page, right after my call to the 'sendgatewaymessage' function i will dump the value of the object again to the page (as this dump will happen immediately after the sendgatewaymessage call is executed asynchronously)

7.) On the .cfm page I will do a thread sleep for 5 seconds to make sure the gateway has enough time to change the value of the object.

8.) Finally, once I feel I've waited long enough, I'll dump the results of the Original Object to the page....

The Results????

You may be surprised at the results - and in the room of 9 people, we split the room in half with 'No Change' and 'Change' votes - the tie breaker coming from our Project Manager who (like myself) guessed 'NO Change' (i.e. there is no reference to the original object once it get's passed into the gateway and executes).

Who was right?? Please provide your comments and I'll let you know what I found out and who won the Poll (I'll post the code for those who want to see it and prove it on their own)

Verity HotFix Received from Adobe

We have received Tier 3 support from Adobe and they have gotten us a patch for our Verity Collection documented in prior posts. The one thing that shocked me as they they took a look at our case and stated that "Ideally, Verity will run without incident efficiently on about 12 collections (especially if they are using CATEGORIES)." 12 collections? I've never seen this documented anywhere - (as of 9.21.2008) and I've read about everything there is to read on the subject.

So there may be a hotfix for this and other Verity issues coming out soon from Adobe - The entire process took a month to resolve but we're excited to implement the fix and I'll be sure to blog the results of the patch/hotfix.

Verity - Internal collection mappings

This is in response to my origional POST about verity returning incorrect data.

This seems to confirm that there is something WRONG with Verity’s internal mapping of collections – as this is what I had executed prior to restarting the verity service. (listing 1.0)

Listing 1.0
D:\CFusionMX\verity\Data\services\ColdFusionK2_indexserver1\log\status.log

2008-08-27 09:55:25 Status: OdkWrk1 service worker thread [0] executing request K2IdxCollOpen
2008-08-27 09:55:25 Error: [VDKCB ws=cf_jrpp-1383_workspace] Error E0-0401 (Vdb): Couldn't find vdb descriptor d:\CFusionMX\verity\collections\ameriprise\assists\00000002.wld
2008-08-27 09:55:25 Status: OdkWrk1 service worker thread [0] executing request K2IdxCollSubmitBif
2008-08-27 09:55:26 Status: [VDKCB ws=cf_jrpp-1383_workspace] Bulk insert operation into d:/CFusionMX/verity/collections/ameriprise (7) from d:/CFusionMX/verity/Data/services/ColdFusionK2_indexserver1/tmp/cf_jrpp-1383_workspace_BIF (0, 0)
2008-08-27 09:55:26 Status: [VDKCB ws=cf_jrpp-1383_workspace] Initializing dataset 00000007.ddd, index 00000007.did
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] Totals (154 documents): 770 para 3336 sent 88918 word (1392 Kb used)
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] Optimizing database layout
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] (1734 ms) Indexed 154 docs into d:/CFusionMX/verity/collections/ameriprise/parts/00000007
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] Writing partition index data
2008-08-27 09:55:28 Error: [VDKCB ws=cf_jrpp-1383_workspace] Error E0-0401 (Vdb): Couldn't find vdb descriptor d:\CFusionMX\verity\collections\ameriprise\assists\00000002.wld

Note that you can see the correct data 154 records being added to the collection – this is also likely why in the cfadministrator, it’s listed this collection as having 154 records.
However, as we know, when we ask for this collection, it seems to be returning

It’s interesting that each client was ‘one off’ according to this CollRef listing in D:\CFusionMX\verity\Data\host\admin\adminX.xml.
As in – ask for ‘ameriprise’, you’ll get ‘cingular’.

<CollRef alias="aigretirement" status="online" threads="3"/>
<CollRef alias="ameriprise" status="online" threads="3"/>
<CollRef alias="cingular" status="online" threads="3"/>

ColdFusion 64 Bit on ESX Virtual Server running Windows Server 2003 64Bit

We've successfully migrated our jobs2web application from 32 bit cf7 to 64 Bit ColdFusion 8!

We still have Verity separated out onto a separate server but with the new ColdFusion 8 installation I hope it solves some of the offline index issues we've seen in the past (turns out it wasn't even 12 hours into it before we saw verity handing back data from other collections again - VERITY!! arg)

We also have a separate instance that just handles maintenance/batch job requests that we have setup through the ColdFusion scheduler. This will mean that any issues with a long running batch will not bring down the production client sites.

On top of a major application update to lower our memory footprint with the application, we're running smooth as ice.

I'll discuss the application update in a separate blog post.

Yeah Team!

Dynamic cfapplication name - Don't do it!

What happens if you use a dynamic name for your cfapplication tag
name=X, is especially if you are using frameworks of any kind, the
more sites you load on top, the memory needs increase X fold. i.e.
coldspring's memory needs + fusebox memory needs + others = Mass
Memory when you have lots of these apps running. Simple right -

It wasn't until we started piling apps on it that this came to life -
the good news is the fix is not complex and doesn't have to mean HIGH
impact changes to your application.

1.) change your cfapplication name value to something static i.e. 'sitex'
2.) I created my site Object that holds the invividual settings for
each client (that was previously stored as it's own application data
object) and used coldspring to wire up that to a siteFactory object
that gives me site Beans based on the domain.
3.) on application start, the siteFactory creates a large structure of
all active site objects 1 time only. Then when the app asks for say
application.site.getName() - it will call the siteFactory, the site
factory gets the data by the domain matching the current domain, and
returns the site object.

With this in place - there was ZERO code changes in the application and the memory requirements have gone WAY down.

Verity - Collections Taken Offline

In doing some more digging into what makes Verity tick (the stand alone version that can be installed separately from the ColdFusion application sever on a separate physical/virtual machine) I uncovered yet another clue into collections being taken 'offline' at various times. We have multiple collections registered through Verity for our clients, and almost monthly now we'll notice that one or more indexes just decides to stop responding to requests (and looks like it's disappeared as it's not listed in the cfadmin).

The directory and files to look at are here: \{verityroot}\verity\Data\host\admin.
Here you will find 2 files -
admin.dat - This file looks like it tells you the 'ID' of the adminX.xml file that its currently using as it's data store
adminX.xml - where X is the value from admin.dat file.

AdminX.xml is a file which has all configuration settings for the K2 Index engine, including names of collections, ports, paths, memory data etc.

You will also find an interesting area near the bottom of the file:

<CollRefs>
<CollRef alias="col1" status="online" threads="2"/>
<CollRef alias="col2" status="online" threads="3"/>
<CollRef alias="col3" status="offline" threads="3"/>

Here you may find your missing indexes - as they have been turned 'offline' for some reason (there is little logging going on to determine why an index get's taken offline).

This may help you in debugging, as next time this happens, I will be going straight to this file and turning any indexes back 'online' and noting any ones that are marked as offline.

To be continued....

ColdFusion Debugger - Separate JVM Files Per Instance - Don't forget the PORT!

In attempting to have another instance of our application up and running during local development for 'back end' processing (fire off a request to a second Instance and have it process it) I made sure I separated the jvm config settings per instance as normal.

However, I had forgotten that the 'address' port in the jvm config was the same as the original, which would make the one service NOT want to start up if the other service was already running (due to port conflics).

Remember to go into your cfadmin (Debugging & Logging -> Debugger Settings -> Allow Line Debugging checked - and copy jvm config data to your jvm config) and get the address setting and specify it in the jvm configured to that instance. The port must be unique across all your instances

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxxx (then restart service)

Verity - Ask for a Collection and Ye Shall not Receive

Today is another one of those days that 'I hate Verity' (that is shipped with coldfusion).

Every few months we have a disaster occur in production where we have 50+ verity collections on a separate server (separate server install of the Verity version that is shipped with cf 7). Seems like randomly after a service restart of the Instance that mostly calls the Verity collections through cfsearch cfcollection and cfindex, Verity will simply return a resultset from a random collection.

So we'll send a

<cfsearch collection="CollectionA">

and we'll get back results from CollectionB, for no good reason.

Solution:
Restarting the ColdFusion MX 7 Search Service on the other machine 're-aligns' things and once again the application resumes normal processing.

This has occurred so many times, that I've decided to log a Bug with Adobe about the product.

We separated out the Verity Search service onto a separate machine as this was happening very more frequently when on the same machine (we'd ask for data from one collection, return data from another collection). It still happens now, although less frequent, but with our current dependency on Verity to do much of the work for string comparisons and categorization, when Verity goes, so does the application.

If anyone else has experienced this please post your response. I've even logged to a cflog exactly what we're passing cfsearch when this issue is occurring, and all the data is correct.

It's like the connection between the 2 machines get's severed and resuming that connection the collection mapping get's thrown off, and incorrect data is the result.

Arg - Can't wait for Lucene Integration in cf9, but I'm afraid it will be too late by then (Summer 2009)

ColdSpring - Using the Parent attribute to specify configuration settings like DSN

I am doing a bit of re factoring in our product, and starting to do a cleaner job of having configuration settings available to each Manager object that needs it (in the typical Manager/Gateway/DAO scenario).

I am looking at the undocumented (at this point) 'parent' attribute that can be specified on a Bean, which will then have the configuration settings injected into it, without the need to explicitly specify the config dependency. Here's an example as I've implemented it:

coldspring.xml definitions...

<!-- Regular Bean in need of some config settings (and specifies the
configGateway as the parent) -->
<bean id="regularBean" class="com.regularBean" parent="configGateway">

<property name="configSettings">
<ref bean="configSettings">
</property>
</bean>
<!-- Configuration Bean which holds config settings -->
<bean id="configSettings" class="com.configSettings">
<constructor-arg name="dsn">
<value>odbcdsn</value>
</constructor-arg>
</bean>
<bean id="nextBean" class="com.nextBean">
<property name="configSettings">
<ref bean="configSettings">
</property>
</bean>

Then in my 'RegularBean' I need the methods for configSettings i.e.:
<!--- Author: penny - Date: 7/30/2008 --->
<!--- getter and setter for configSettings --->
<cffunction name="getconfigSettings" access="public" output="false" returntype="com.configSettings">
<cfreturn>
</cfreturn>
<cffunction name="setconfigSettings" access="public" output="false" returntype="void">
<cfargument name="configSettings" type="com.configSettings" required="true">
<cfset configsettings="arguments.configSettings/">
</cfset>
</cfargument></cffunction></cffunction>

When calling Regular Bean -
application.serviceFactory.getBean('regularBean').getConfigSettings().getDSN()

This technique still requires the 'stubbed' getter/setter for the configSettings bean, but is less xml in the coldspring xml file. Here is an example of the alternative:

(BAD)
coldspring.xml file

<property name="configSettings">
<ref bean="configSettings">
</ref>
</property>
<!-- Configuration Bean which holds config settings -->
<bean id="configSettings" class="com.configSettings">
<constructor-arg name="dsn">
<value>odbcdsn</value>
</constructor-arg>
</bean>
<bean id="nextBean" class="com.nextBean">
</bean> <property name="configSettings">
<ref bean="configSettings">
</ref>
</property>

.. etc.

This will be very nice to use, and extend when new configuration settings are needed to be added to the application like 'verityroot' 'rootdirectory' or 'write_dsn' or 'read_dsn' or whatever other application configuration settings are needed for your application.

They say the documentation of this should be out soon at the coldspringframework.org/docs documentation site, so stay tuned for that.

Verity Separate Server Install Collections Fail to Create

Verity sucks some days, and today is one of those days.

We had another episode with Verity where we had a collection disappear. I am not sure where or why, but I was tasked with getting it back online. This is not the first time this has happened, but the solution should be easy: Re-Create the Index, however this is where things got UGLY.

I looked in the CFadmin to see if the collection was listed, which it was not. So I proceeded to attempt to create it and received:

Unable to create collection companyX.
An error occurred while creating the collection: com.verity.api.administration.ConfigurationException: Index exists. Cannot insert entry. (-6065)

So it isn't listing the collection, and yet it says it exists?? OK, so that's problem one.

Problem 2 is that when i execute a 'reindex' which does a
<cfcollection action="create" ..>
I received no errors from the application (things looked like they were successful)

However when I look into CFADMIN again, in Verity Collections, the collection was once again not listed.

Here's what I did to fix it:

In our installation we have the Verity Installer (ColdFusion 7) installed on a separate box from the web server. I told the web1 to use the Verity K2Server at the web1 address, instead of the web2 address that it was using. Then I recreated and populated the collection using the same server the application was using (Verity was installed on both servers, but is typically not running on web1). Once done, I specified the web2 IP address again in cfadmin and copied the collection directory of the collection that I wanted from the web1 to web2.

When I go into cfadmin I should NOT see the collection listed - but after I executed a <cfcollection action="create"> with that collection name, something behind the scenes worked that time, and the collection showed up in cfadmin WITH the correct amount of data in it.

This is strange for a number of reasons, but I had to repeat this process for another collection just after this, so I know this process works. Executing a 'action="create"' typically kills any data that is in the collection, which didn't happen when I used this approach.

Anyway, hope this helps, and I know it will help me again the next time it happens....because it WILL!