Verity - Internal collection mappings

This is in response to my origional POST about verity returning incorrect data.

This seems to confirm that there is something WRONG with Verity’s internal mapping of collections – as this is what I had executed prior to restarting the verity service. (listing 1.0)

Listing 1.0
D:\CFusionMX\verity\Data\services\ColdFusionK2_indexserver1\log\status.log

2008-08-27 09:55:25 Status: OdkWrk1 service worker thread [0] executing request K2IdxCollOpen
2008-08-27 09:55:25 Error: [VDKCB ws=cf_jrpp-1383_workspace] Error E0-0401 (Vdb): Couldn't find vdb descriptor d:\CFusionMX\verity\collections\ameriprise\assists\00000002.wld
2008-08-27 09:55:25 Status: OdkWrk1 service worker thread [0] executing request K2IdxCollSubmitBif
2008-08-27 09:55:26 Status: [VDKCB ws=cf_jrpp-1383_workspace] Bulk insert operation into d:/CFusionMX/verity/collections/ameriprise (7) from d:/CFusionMX/verity/Data/services/ColdFusionK2_indexserver1/tmp/cf_jrpp-1383_workspace_BIF (0, 0)
2008-08-27 09:55:26 Status: [VDKCB ws=cf_jrpp-1383_workspace] Initializing dataset 00000007.ddd, index 00000007.did
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] Totals (154 documents): 770 para 3336 sent 88918 word (1392 Kb used)
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] Optimizing database layout
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] (1734 ms) Indexed 154 docs into d:/CFusionMX/verity/collections/ameriprise/parts/00000007
2008-08-27 09:55:27 Status: [VDKCB ws=cf_jrpp-1383_workspace] Writing partition index data
2008-08-27 09:55:28 Error: [VDKCB ws=cf_jrpp-1383_workspace] Error E0-0401 (Vdb): Couldn't find vdb descriptor d:\CFusionMX\verity\collections\ameriprise\assists\00000002.wld

Note that you can see the correct data 154 records being added to the collection – this is also likely why in the cfadministrator, it’s listed this collection as having 154 records.
However, as we know, when we ask for this collection, it seems to be returning

It’s interesting that each client was ‘one off’ according to this CollRef listing in D:\CFusionMX\verity\Data\host\admin\adminX.xml.
As in – ask for ‘ameriprise’, you’ll get ‘cingular’.

<CollRef alias="aigretirement" status="online" threads="3"/>
<CollRef alias="ameriprise" status="online" threads="3"/>
<CollRef alias="cingular" status="online" threads="3"/>

ColdFusion 64 Bit on ESX Virtual Server running Windows Server 2003 64Bit

We've successfully migrated our jobs2web application from 32 bit cf7 to 64 Bit ColdFusion 8!

We still have Verity separated out onto a separate server but with the new ColdFusion 8 installation I hope it solves some of the offline index issues we've seen in the past (turns out it wasn't even 12 hours into it before we saw verity handing back data from other collections again - VERITY!! arg)

We also have a separate instance that just handles maintenance/batch job requests that we have setup through the ColdFusion scheduler. This will mean that any issues with a long running batch will not bring down the production client sites.

On top of a major application update to lower our memory footprint with the application, we're running smooth as ice.

I'll discuss the application update in a separate blog post.

Yeah Team!

Dynamic cfapplication name - Don't do it!

What happens if you use a dynamic name for your cfapplication tag
name=X, is especially if you are using frameworks of any kind, the
more sites you load on top, the memory needs increase X fold. i.e.
coldspring's memory needs + fusebox memory needs + others = Mass
Memory when you have lots of these apps running. Simple right -

It wasn't until we started piling apps on it that this came to life -
the good news is the fix is not complex and doesn't have to mean HIGH
impact changes to your application.

1.) change your cfapplication name value to something static i.e. 'sitex'
2.) I created my site Object that holds the invividual settings for
each client (that was previously stored as it's own application data
object) and used coldspring to wire up that to a siteFactory object
that gives me site Beans based on the domain.
3.) on application start, the siteFactory creates a large structure of
all active site objects 1 time only. Then when the app asks for say
application.site.getName() - it will call the siteFactory, the site
factory gets the data by the domain matching the current domain, and
returns the site object.

With this in place - there was ZERO code changes in the application and the memory requirements have gone WAY down.

Verity - Collections Taken Offline

In doing some more digging into what makes Verity tick (the stand alone version that can be installed separately from the ColdFusion application sever on a separate physical/virtual machine) I uncovered yet another clue into collections being taken 'offline' at various times. We have multiple collections registered through Verity for our clients, and almost monthly now we'll notice that one or more indexes just decides to stop responding to requests (and looks like it's disappeared as it's not listed in the cfadmin).

The directory and files to look at are here: \{verityroot}\verity\Data\host\admin.
Here you will find 2 files -
admin.dat - This file looks like it tells you the 'ID' of the adminX.xml file that its currently using as it's data store
adminX.xml - where X is the value from admin.dat file.

AdminX.xml is a file which has all configuration settings for the K2 Index engine, including names of collections, ports, paths, memory data etc.

You will also find an interesting area near the bottom of the file:

<CollRefs>
<CollRef alias="col1" status="online" threads="2"/>
<CollRef alias="col2" status="online" threads="3"/>
<CollRef alias="col3" status="offline" threads="3"/>

Here you may find your missing indexes - as they have been turned 'offline' for some reason (there is little logging going on to determine why an index get's taken offline).

This may help you in debugging, as next time this happens, I will be going straight to this file and turning any indexes back 'online' and noting any ones that are marked as offline.

To be continued....

ColdFusion Debugger - Separate JVM Files Per Instance - Don't forget the PORT!

In attempting to have another instance of our application up and running during local development for 'back end' processing (fire off a request to a second Instance and have it process it) I made sure I separated the jvm config settings per instance as normal.

However, I had forgotten that the 'address' port in the jvm config was the same as the original, which would make the one service NOT want to start up if the other service was already running (due to port conflics).

Remember to go into your cfadmin (Debugging & Logging -> Debugger Settings -> Allow Line Debugging checked - and copy jvm config data to your jvm config) and get the address setting and specify it in the jvm configured to that instance. The port must be unique across all your instances

-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=xxxx (then restart service)

Verity - Ask for a Collection and Ye Shall not Receive

Today is another one of those days that 'I hate Verity' (that is shipped with coldfusion).

Every few months we have a disaster occur in production where we have 50+ verity collections on a separate server (separate server install of the Verity version that is shipped with cf 7). Seems like randomly after a service restart of the Instance that mostly calls the Verity collections through cfsearch cfcollection and cfindex, Verity will simply return a resultset from a random collection.

So we'll send a

<cfsearch collection="CollectionA">

and we'll get back results from CollectionB, for no good reason.

Solution:
Restarting the ColdFusion MX 7 Search Service on the other machine 're-aligns' things and once again the application resumes normal processing.

This has occurred so many times, that I've decided to log a Bug with Adobe about the product.

We separated out the Verity Search service onto a separate machine as this was happening very more frequently when on the same machine (we'd ask for data from one collection, return data from another collection). It still happens now, although less frequent, but with our current dependency on Verity to do much of the work for string comparisons and categorization, when Verity goes, so does the application.

If anyone else has experienced this please post your response. I've even logged to a cflog exactly what we're passing cfsearch when this issue is occurring, and all the data is correct.

It's like the connection between the 2 machines get's severed and resuming that connection the collection mapping get's thrown off, and incorrect data is the result.

Arg - Can't wait for Lucene Integration in cf9, but I'm afraid it will be too late by then (Summer 2009)

psexec remote batch file execution trailing slash problem solved

I've been working with a batch file that will execute an xcopy to transfer files to another server, then i execute a 'psexec' call to fire off a batch file located on the remote server.

Here are a list just some of the problems/solutions I ran accross:

Problem:
NT Scheduled Task Wouldn't complete -
Solution:
Created a domain user that had access to both machines to execute the batch file

Problem:
Working Directory was OFF when calling the remote batch file
Solution:
Specify -w or the absolute path on the remote server that this batch file is located in and should run/execute in.

Problem:
Error = [Microsoft][SQL Native Client]Unable to open BCP host data-file
Path was still not being rendered correct.
Solution:
Change the order of the -w argument to be after the remote computer but before the remote command
i.e. psexec \\xxx.xxx.xxxx -w "C:\mypath\pathtobatch" "c:\mypath\pathtobatch\batchfile.bat" (GOOD)
psexec \\xxx.xxx.xxxx "c:\mypath\pathtobatch\batchfile.bat" -w "C:\mypath\pathtobatch"(BAD)

Problem:
There seemed to be a path problem even after a working directory was specified
Solution:
Remove the trailing '\' Slash mark in the path
i.e. psexec \\xxx.xxx.xxxx -w "C:\mypath\pathtobatch" (good)
psexec \\xxx.xxx.xxxx -w "C:\mypath\pathtobatch\" (bad)

With persistence, it will work!