Search Engine Death Match - Solr Wins

It's now been over a month since we rolled out our Lucene / Solr powered search engine replacement for Verity in our application and let me sum it up this way: Zero maintenance.

Even though the OEM Verity licensing states 250K record max, we ran into a ceiling on a few of our machines when the number of Verity collections reached 82-88. We were doing constant rcadmin fixes to drop the thread counts for each collection down to 2 from 3 - but on each restart or at random points in time, the thread counts would return to 3. Instability and random collection switches were the main reasons we moved off the platform, and we'll never look back.

Solr took some configuration, but it was more than flexible enough for our needs, and with the delta updates and deletes, it made the maintenance near zero.

As far as parsing the resulting xml, that was done efficiently through coldfusion's xmlparse and xmlsearch to walk the nodes and return a 'like' verity recordset to the application. With ColdSpring we were able to swap out implementations easily and everything just worked. Great planning and execution resulted in a seamless rollout with Solr being run on RedHat on a dedicated machine.

I will most likely do a presentation at our next/upcoming CFUG (ColdFusion User Group) in Minneapolis/St.Paul in the near months about the transition and talk about integration points before ColdFusion 9 comes out and makes it all available with a single tag ;)

Long Live Open Source software

Query of Queries is Inconsistent (Adobe CFML Runtime)

I've never been a big fan of a half baked solution, but I will acknowledge that coldfusion's query of queries has come a long way from it's beginning and does do a good job on simple queries to get data from an existing query.

However, there are times when it simply sucks at doing what you think it should do, especially when it comes to multiple tables and joins, as well as converting datatypes to be alike.

I'll show a few examples of it's inconsistencies and attempt to point out some problems with it's implementation.

Example 1: Q-of-Q (Query of Queries) and Proper variable scoping inside a cffunction.

When inside a cffunction you always want to fully qualify your variables as best practice. If it's an arguments, prepend the value with 'arguments.' etc. The same goes for Q-of-Q. But I will show you where q-of-q forces you to break it's own best practice advice by not allowing you to do this simple task.

I will be referencing the following 2 queries:

<!--- Test Query 1 --->
<cfset q = QueryNew('id,name','Integer,varchar')>
<cfset queryaddrow(q,1) >
<cfset querysetCell(q,'id',1,1) >
<cfset querysetcell(q,'name','test',1) >

<!--- Test Query 2 --->
<cfset q2 = QueryNew('id,age','Integer,Integer')>
<cfset queryAddRow(q2,1) >
<cfset querySetCell(q2,'id',1,1) >
<cfset querySetCell(q2,'age',33,1) >

This is a simple query that creates a single row of data




What I'm going to do is merge the results of these 2 queries (as one may have come from a cfdirectory, and the other from a cfsearch, or cfquery etc).

My method takes 2 query arguments and joins them - but for now I'll just return the value of the 1st argument 'qry'.

<cffunction name="merge" output="false" access="public" returntype="query" hint="">
<cfargument name="qry" type="query" required="true"/>
<cfargument name="qry2" type="query" required="true"/>
<cfset var Private=StructNew()>

<cfquery name="private.q" dbtype="query">
select *
from arguments.qry
</cfquery>

<cfreturn private.q />
</cffunction>

Resulting in a simple query returned - nothing special.



Now when i wish to merge the 2 queries - I would assume that I could do something like this then if 'arguments.qry' worked to reference the qry argument.

<cffunction name="merge" output="false" access="public" returntype="query" hint="">
<cfargument name="qry" type="query" required="true"/>
<cfargument name="qry2" type="query" required="true"/>
<cfset var Private=StructNew()>

<cfquery name="private.q" dbtype="query">
select *
from arguments.qry, arguments.qry2
where arguments.qry.id = arguments.qry2.id
</cfquery>

<cfreturn private.q />
</cffunction>

WRONG!
Query Of Queries syntax error.
Encountered "arguments . qry .. Incorrect conditional expression, Expected one of [like|null|between|in|comparison] condition,

Ok fine - I won't use proper best practices and just reference the query w/o the arguments proper fully scoped value.

<cffunction name="merge" output="false" access="public" returntype="query" hint="">
<cfargument name="qry" type="query" required="true"/>
<cfargument name="qry2" type="query" required="true"/>
<cfset var Private=StructNew()>

<cfquery name="private.q" dbtype="query">
select *
from qry, qry2
where qry.id= qry2.id
</cfquery>

<cfreturn private.q />
</cffunction>

Results: works fine



Example 2: Using QofQ (query of queries) to reference a column from the table using dot notation

The ColdFusion LiveDocs states the following:

Example

If a structure named A contains a field named B, which contains a table named Products, you can refer to the table with dot notation, as follows:

SELECT tape_ID, length
FROM A.B.Products;

The 'structure' in my case is 'arguments', the field name is 'id', and the table name is 'qry'. With that in mind:

A=arguments
B=id
Products=qry

Therefore you could deduce that the following is true:

SELECT id, name
FROM arguments.id.qry

Results:

Query Of Queries syntax error.
Encountered ". Incorrect Select List, Incorrect select column, arguments.id cannot be followed by '.'


However, you would normally reference this data as 'arguments.qry.id'.

Results:

Query Of Queries syntax error.
Encountered ". Incorrect Select List, Incorrect select column, arguments.qry cannot be followed by '.'

The kicker is - Neither of these work.

Killer WDDX - Beware!

I came across a rather interesting 'feature' tonight while working on some logging and alerting processes. We have data in a structure that holds raw information about various processes that have occurred. In one of the structure keys there is an XMLObject (coldfusion parsed xml object). See the code snippet below and DO NOT run this on production equipment!!

<cfset stcStruct = StructNew()/>

<cfset xmlDoc = "<myxml><name>Kevin Penny</name></myxml>">
<cfset stcStruct.xml = xmlParse(xmlDoc)/>
<cfset stcStruct.id = 1>

<cfwddx action="cfml2wddx" input="#stcStruct#" output="tmp"/>

This takes a Structure, adds a key of 'xml' and places a parsed xml object in it's position. I'm then serializing the data into wddx via cfwddx as cfml2wddx.

I originally thought it was the date fields that I had in the structure, so I removed those, but the problem was the xml object's serialization.

Here's the error I get along with a partial dump:
java.lang.StackOverflowError

....
at java.beans.Introspector.instantiate(Introspector.java:1438)
at java.beans.Introspector.findExplicitBeanInfo(Introspector.java:410)
at java.beans.Introspector.(Introspector.java:359)
at java.beans.Introspector.getBeanInfo(Introspector.java:222)
at java.beans.Introspector.(Introspector.java:368)
at java.beans.Introspector.getBeanInfo(Introspector.java:222)
at java.beans.Introspector.(Introspector.java:368)
at java.beans.Introspector.getBeanInfo(Introspector.java:222)
at java.beans.Introspector.(Introspector.java:368)
at java.beans.Introspector.getBeanInfo(Introspector.java:222)
at java.beans.Introspector.(Introspector.java:368)
at java.beans.Introspector.getBeanInfo(Introspector.java:222)
at java.beans.Introspector.(Introspector.java:368)
at java.beans.Introspector.getBeanInfo(Introspector.java:222)
at java.beans.Introspector.getBeanInfo(Introspector.java:208)
at coldfusion.wddx.BeanSerializer.writeObject(BeanSerializer.java:48)
at coldfusion.wddx.WddxOutputStream.writeObject(WddxOutputStream.java:310)
at coldfusion.wddx.BeanSerializer.writeObject(BeanSerializer.java:120)
at coldfusion.wddx.WddxOutputStream.writeObject(WddxOutputStream.java:310)
at coldfusion.wddx.BeanSerializer.writeObject(BeanSerializer.java:120)
at coldfusion.wddx.WddxOutputStream.writeObject(WddxOutputStream.java:310)

I've submitted this as a bug to Adobe #75230.

Update to CFPROPERTY Inspector Utility

I implemented a new UI for the CFPROPERTY Inspector as well as small bug fixes etc.

I used the JQuery tab UI which helps the flow of the steps being that the process is just one large form. Here are the new screen shots.

CFProperty Inspector on RiaForge

I've released the CFPROPERTY Inspector on riaforge and google code for those interested in it.

This Utility is designed to introspect components and auto-create CFPROPERTY tags based on the function meta data.
Specify the mapping to your components, and choose which cfc's you want to have CFPROPERTY Tags defined on them as well as what properties of your components that you wish to have CFPROPERTY tags defined for. (for assisting in getting your components well defined for Flex integration etc.)

For Screen Shots and more checkout cfproperty.riaforge.org and to download the code visit Google Code here.









Let me know what you think.

Manipulating CFC Properties through getMetaData

I started toying with the idea of creating a generic utility that would populate a cfc with ‘property’ as in cfproperty data and I have something work now – (this would make the compatibility between flex and the model that much better and more ‘well defined’)

Would this be of benefit to us – i.e. running our model through it which would then create all cfproperty tags for a given object based on the getter methods w/in the object – and determine datatype of string, numeric etc – based on the getter methods data etc –

What sort of things do I have to be careful of ??

Order of the cfproperty tags counts right – and Case sensitivity of the names obviously ??

So the process would either do it alpha – or by order that was returned by getMetaData on the object –

An interesting thing happens however when I’m doing this – I could actually build this into coldspring likely where these properties are autogenerated on the fly and never actually have to exist w/in the cfc (written back to the cfc)

The strange thing happened was this:

Once I do a create object on a cfc – I can dump it and it shows that there are NO properties defined in the meta data – which is good

So I run it through my process where I’m introspecting the cfc and create this ‘properties’ key and append the array to this key.

This works well and now a side by side before and after shows the left object with NO properties and the right with Properties – great.

However the next time I run it – the left side also shows that the object now HAS properties –

This is strange – b/c cfproperty tags don’t actually exist in the cfc – but yet I’m able to fake cf (caching likely) into thinking / assigning these properties to the object.

For Example:

Here my Job Object has NO cfproperty tags:

I’m going to make ColdFusion ‘think’ it has these properties defined on it by manipulating the metadata about the object.

Simple App/UI

There are No properties on the left – On the right below, shows the ‘manipulated’ meta data based on the getter methods in the object.

Now after I run this again – you see both left and right are identical (cached metadata about the object)

The left side ‘thinks’ that the object now has these properties available to it- which if called or introspected by Flex, they should be available.

This feels a bit like cheating, but if it doesn’t work – we can always complete this thought by having the process ‘write’ the cfproperty tags back to the object.

Interesting huh?

JRun4 wsconfig Error Loading jvm.dll

Recently I attempted to patch our JRun4 servers with updater 7. Unfortunately after the fact, I found out that there is no 64 bit installer for it on windows. "64-bit Support – JRun4 Updater 7 provides 64-bit support on Solaris-SPARC. Other 64-bit operating systems including Windows 64-bit and Solaris-x86 64-bit are not supported."

So after a quick panic attack and reverting to the backed up copy (yes I did back up some recommended things) I felt that I was out of the woods. Twas not the case.

The next time I went to add a site to the Coldfusion instance through wsconfig command line - I received a dreaded "Error Loading jvm.dll". So I could no longer add sites and associate them to the webserver etc.

Here was the old version:
wsconfig.exe -ws IIS -site "jobs-jobs.com" -coldfusion -server "cfcluster" -v -ws64 -norestart
(Error Loading jvm.dll)

The new version of this has be using another technique using a java properties file instead. :
java -jar wsconfig.jar -f c:\j2w_IISconfig\jrun.properties

This called a jrun.properties file that had the arguments that i was executing directly to the wsconfig.exe prior

The Properties looks somehing like this:

ws=IIS
site=jobs-jobs.com
ws64=true
norestart=true
server=cfcluster
coldfusion=true

Bingo - Hope this helps some other soul out there - as it stopped me in my tracks for a bit.