MXUnit Blog

MongoDB at Ria Unleashed

noreply@blogger.com (Marc Esher) — Thu, 26 Aug 2010 22:28:00 +0000

RIA Unleashed is back for a 2nd year, and it’s going to be a great one? Why? Because yours truly is speaking, naturally!

No no no, I’m not going to yap for yet another hour on unit test this, TDD that, automate blah blah blah. This time, MongoDB gets some lovin’.

See, a while back, Billy started poking about with Mongo. He wrote some blog posts and started CFMongoDB. As fate would have it, earlier this year I had a chance to work on a project using Mongo, and I took to it quite naturally. With Billy going back to school to join the neckbeards doing Ph.D. work, I’ve taken up the reigns of CFMongoDB for the timebeing. Meanwhile, Brian Rinaldi asked me if I’d like to speak at RIA Unleashed, and he graciously accepted a topic on Mongo. Here it is:

MongoDB is a popular open source Document-oriented database. Like other NoSQL datastores, it's fast and scalable. Unlike most others, it provides rich ad-hoc query capabilities which we ColdFusion developers know and love. During the presentation, I'll give a broad overview of the usual introductory topics: MongoDB's "solution space", installation, working with the shell, administration, and replication. Then I'll dive into using Mongo with ColdFusion, showing why I enjoy working with Mongo, the problems I've encountered, and how to solve them.
By session's end, attendees should understand what problems MongoDB was meant to address and how to use MongoDB as a datastore in their CFML projects.

RIA Unleashed is a kick-ass, reasonably priced, 2-day conference featuring both sessions and workshops by world-famous speakers such as Ray Camden, Jesse Warden, Bob “All Your Mocks Are Belong to Us” Silverberg, and the guy who created Peanuts!

During its inaugural year, teh twitterz were buzzing with compliments. This year will be even better!

CF9 ORM: Using the Secondary Cache to optimize for expensive columns

noreply@blogger.com (Marc Esher) — Sat, 17 Jul 2010 23:36:00 +0000

This is an experiment. It most likely speaks to my poor SQL skills and willingness to jump for a quicker code solution than to figure out how to write faster queries. So if you can tell me a faster way to do this in SQL, I’ll listen.

Still, this is mostly about using a combination of simple property mappings and the secondary cache for an interesting, perhaps unusual solution to a problem.

This is my story

I have a table. Its name is “N_Organizations”. It’s prefixed with “N_” because this is a blog post and I am experimenting. Like many organizations, an Org can be a parent of another Org: “Paint Shop” is a child of “Operations” is a child of “The Penthouse Suite”. In an untold number of places, in both the web application and in reports that are not web-driven, the “tree” for any given child organization has a simple requirement: Show the names of the parents as a single string:

"The Penthouse Suite –> Operations –> Paint Shop”

Ideally, The creation of this string would happen in one place, and then any place that needed it would not have to do the requisite looping to construct the string itself. If different formatting were needed, it’d be better to have the reformatter be a simple string reformatter than a parent org climber: simply replace the arrows with colons, for example.

So, when thinking of a typical query, you’d get something like this:

id | Name                  | ParentID | ParentName
1    The Penthouse Suite     null       
2    Operations              1          The Penthouse Suite
3    Paint Shop              2          The Penthouse Suite --> Operations

And so on.

To fetch this stuff, I created a view that uses a Common Table Expression (CTE) to recursively loop over the parents in order to derive the ParentName. It does some other foolishness with “level”, but ignore that. CTEs are kind of like recursing but without functions:

CREATE view [dbo].[v_organizations]   as

WITH cte_tree AS
		(
			SELECT id, organizationname, OrgNode.GetLevel() as Level, CAST('' AS varchar(500)) AS ParentName
			FROM dbo.n_organizations
			WHERE organizationtypecode = 'D'

			UNION all

			SELECT parent.id, parent.organizationname, OrgNode.GetLevel() as Level,
			CAST(
				cte_tree.parentName +
				case when cte_tree.parentname = '' then '' else ': ' end
				+  cte_tree.organizationname as varchar(500)
			)
			FROM dbo.n_organizations parent
			JOIN cte_tree ON parent.parentorganizationid = cte_tree.id
		)
		SELECT
			o.id, o.OrganizationName, OrganizationTypeCode,
				ParentOrganizationID, CreatedOn, ModifiedOn, 
				Level,
				case when Level <=3 then o.id else o.ParentOrganizationID end as GroupLevel,
			cte_tree.ParentName
		FROM cte_tree
		JOIN dbo.n_organizations o ON cte_tree.id = o.id

GO

There’s a big problem with this: it’s slow. The tables are as indexed as they can be, but you can’t index views with CTEs. And what’s worse, the query is slow regardless of what kind of criteria you add to it.

Select * from v_Organizations runs about 125 milliseconds. Select * from v_Organizations where id = 215 runs in about 90 ms. The reason is that it still has to do all that recursion. Thus, this is probably a terrible way to solve this problem.

And yet, we continue….

This is my Component

Organization.cfc

component persistent="true" accessors="true" table="N_Organizations" {
	property name="id" fieldtype="id" generator="native";
	property name="organizationName" type="string";
	property name="parentName" column="parentName" readonly="true" type="string"  table="v_organizations" joincolumn="id" cacheuse="read-only";
}

Notice how the parentName property points to “v_organizations” as the table? Pretty cool, huh? So now when I load this component, when I access the parentName property, it’ll pull it from v_organizations and all will be well. Hibernate will generate a query that simply joins table N_Organizations on v_Organizations where id = id. Yeah.

Except, that’s slow, too. Remember, as I said, querying v_organizations adds about 100ms no matter how many records I’m pulling back. Notice that “cacheuse=’read-only” attribute? Yeah… that doesn’t work. No matter how many times I ask for the parentName, it’s going to run that same query.

What If?

What if I just cached the Organization.cfc in Secondary cache? Wouldn’t that rock? No… most of the data in this component will change too frequently, so I can’t cache the entire component (I’m leaving off a bunch of other properties, obviously).

But… what if I could cache just that parentName property? That’s the slow one, after all. And that never changes. Or, it changes like 2x a year, and in that case, we just clear the cache and off we go.

How could you possibly achieve such awesomeness?

You need two components: 1) Your original Organization.cfc and 2) a new ParentName.cfc

Organization.cfc:

component persistent="true" accessors="true" table="N_Organizations"{
	property name="id" fieldtype="id" generator="native";
	property name="organizationName" type="string";
	property name="parentName" readonly="true" type="string" fieldtype="one-to-one" cfc="ParentName" constrained="true" cacheuse="read-only" ;
	
	//make it easy to get the parentName property
	function getParentName(){
		try { return parentName.getParentName(); }
        	catch(Any e) { return ""; }
	}

}

ParentName.cfc

component accessors="true" persistent="true" readonly="true" table="v_Organizations" cacheuse="read-only" cachename="ParentName"{
	property name="id" fieldtype="id";
	property name="parentName" type="string" ;
}

So what does this do, exactly?

First, what we’ve done is simply rolled that parentName field from v_organizations into a 4-line component. Notice that the “table” attribute on the component is v_organizations. Notice it specifies cacheuse=”read-only”. Notice that read-only is true.

Second, in Organization.cfc, we replaced the original parentName property with a new one that is a simple one-to-one relationship with this new ParentName property

Third,we add a function, getParentName(), so that we could simply call organization.getParentName() instead of organization.getParentName().getParentName(). It’s wrapped in try/catch because this will fail for the very top-level organization which has no parent organization.

The result: The first time getParentName() is called for any organization, the view will run (slowly, as usual). Subsequently, CF will fetch that property from the cache until that cache is cleared or the server restarts or whatever. So you do take an initial one-time hit of 90ms or so for each organization, but after that, you’re gold.

But wait, there’s more!

I mentioned this experiment to my boss, and his first reaction was, Could you just do a one-time load of all those ParentName organization instances, right on app start, so that you’d never see that 90-ms / per object penalty?

So I tried it, and this is where it gets even better. None of this is unsurprising, by the way. It all makes complete sense.

Simple math:

Doing it the ‘pay-as-you-go’ way: You have 2500 records. You hit the view one time for each record. 2500 * 90ms = 225000 ms = 225 seconds = almost 4 minutes total to load each of those records were you to do them one at a time over the lifespan of the application between server restarts.

However, if you load them all at once, remember, SQL-wise, it’s only taking about 125ms to load. So if you created all 2500 instances of ParentName.cfc, you’ve got the 125ms for the SQL and whatever time it takes to create those 2500 component instances. In my case, it’s about a second, total. To achieve such a performance improvement, you run this when your app starts:

EntityLoad(“ParentName”);

Since ParentName is a cached component, it’ll get stuck in whatever cache you have configured in Application.cfc, and then all subsequent calls to organization.parentName() will take however long it would take to fetch a simple string from the cache. In my experience this is anywhere from 1/3 to 1 ms doing unscientific averaging.

If you need to clear the cache, simply use:

ormEvictEntity("ParentName");

Conclusion

1) I was amazed at how easy it was to work with Hibernate’s second-level cache with ColdFusion ORM. It just worked.

2) Sometimes, when you’re not Joe Celko and you don’t want to spend time time figuring out how to write a better view, you say “you know what, I think I’ll see if there are fun and interesting bits in CF ORM that could solve this problem for me”. And then you go play, and you come up with some nifty things.

3) I have no idea whether I’ll ever use this, but perhaps you’ll find a use for such a solution and, when you do, thank the CF team for doing such a fine job of integrating Hibernate’s L2 cache into CF.

Good luck!

--Marc

How to set up a new Eclipse environment

noreply@blogger.com (Marc Esher) — Sun, 11 Jul 2010 11:06:00 +0000

Have you ever wanted to start using a new version of Eclipse but couldn’t justify it because of all the time you’ve spent getting your preferences and project settings just so? And you don’t want to type in all those Update site URLs again to get all your plugins? While there is currently no one-click easy way to completely migrate an existing Eclipse install to a new one, I do have some timesavers for you.

Always start with a new Workspace

I’ve seen too many problems with trying to reuse an existing workspace in hopes of saving some time moving from an old version of Eclipse to a new one. Don’t do it. When you fire up the new Eclipse, and it asks you for the workspace, give it a new location. I keep a directory named “Eclipse Workspaces”, and in it lay the corpses of a dozen or more old workspaces. I name my workspaces with this general pattern:

[Product]_[MajorVersionName]_[RCNumber]_[Purpose]

For example, for my work projects, I use ColdFusion Builder on Galileo. My workspace is Bolt_Galileo_Work. When I was fiddling with some of the Helios (Eclipse 3.6) RCs, I was using Bolt_Helios_RC2_Work.

For Eclipse Plugin development I use CFEclipse_Helios_PluginDevelopment.

I always reserve the right to break that pattern.

Getting all your plugins

I’ve seen some people suggest copying them from the old install, and perhaps that works just fine. Here’s how I do it: I Export the update sites I want to install into my new Eclipse, and then Import them in the new one. Then, I install the plugins using the tried-and-true software installer. Here’s how:

In your old Eclipse

Help – Install New Software
Click the “Available Software Sites” link
On the Available Software Sites screen, select the plugins that you want to install into your new Eclipse
Click the “Export” button, and go through the dialog of saving the XML file someplace sensible, like your Desktop

That screen looks like this:

In your new Eclipse

Following the same steps as above, get to the Available Software Sites screen
Click the “Import” button
Navigate to that XML file you saved previously and follow all steps
Now you have those update sites available to you. From there, get back to the Install software screen, select “All Software Sites” from the dropdown, and let Eclipse find and install all those plugins you know and love

Getting some of your Preferences

You can export some of your preferences from the old install to the new one, though you do have to beware that you’re going to import some preferences you don’t actually want. More on that in a bit.

In your old Eclipse

File – Export
Select “Preferences”
Follow the rest of the screens to save your prefs as an .epf file

In your new Eclipse

File – Import
Select “Preferences”
Follow the rest of the screens

As I said, it’s going to copy some preferences that you don’t want. In particular for me, I always get bitten by the “ANT Runtime” preferences. In that case, I let it do its thing, and then I go back into Window – Preferences and point the ANT runtime to the one that comes bundled with the new Eclipse. Even with that little annoyance, it’s worth it to export/import preferences because it brings over so much.

Getting all your projects

Perhaps the most time-consuming part of moving to a new Eclipse installation is getting all your projects back in there. Fortunately, this can also be the least time-consuming. Here’s what to do in your New Eclipse installation:

In CFEclipse

File – Import
Select “Existing Projects into Workspace”. Click Next
In the “Select Root Directory” field, type in the directory where you keep most of your projects. Hit Enter
Once it finds all the projects in that directory, choose the ones you want, and hit “Finish”

In ColdFusion Builder

Follow the same steps as above, but at Step #2, choose “ColdFusion – Import existing Projects”

NOTE: This will not copy All of the existing properties for the projects, such as the MXUnit cfcpath and remote facade URL. For ColdFusion Builder, it will carry over the server settings since they are stored in a file that lives inside that project and not outside of it like most project properties in Eclipse

Some Preferences I’m always sure to confirm

Once I’m finished with the above, I have a few preferences I always double-check. In Window – Preferences:

General – Show Heap
Quick Diff – Pristine SVN Copy
Local History – at least 180 days
Key Bindings – Alt-A for ANT view, Alt-S for CFEclipse Snip Tree View, and Alt-Y for Mylyn Task List view
Appearance – Colors and Fonts – Basic Text -- Consolas 12pt
CFEclipse / ColdFusion Builder Snippets path (I point it to a dropbox location)

If you have any quick tips for making migrating from one Eclipse installation to another less time-consuming, please share them.

MXUnit 2.0 Released

noreply@blogger.com (bill shelton) — Fri, 09 Jul 2010 17:02:00 +0000

Once again, Los Hombres at MXUnit.org are delivering a new batch of the crack cocaine of unit testing for ColdFusion--MXUnit 2.0. Yes, it's true. Once tasted, a life of uncontrollable and desperate automated testing will surely ensue. This particular batch has some especially addictive features:

New Open Source License: MIT. Changing from GPL to MIT effectively allows you to integrate MXUnit into any product with little restriction (other than honoring the No Whining clause).
New built-in mocking framework: Simply call the new mock() method to create a new mocked object and inject that into your component under test.
New integrated HTML view: This replaces the existing HTML and ExtJS view and is backward compatible.
New beforeTests() and afterTests() for component-level set up and tear down.
New and improved documentation and bug tracking at http://wiki.mxunit.org and http://jira.mxunit.org/secure/IssueNavigator.jspa?reset=true&mode=hide&pid=10000
Ant task improvement: now you can send arbitrary URL parameters to your custom runners.
New Ant build.xml examples and Hudson-friendly examples.
More Eclipse snippets!
More Dataprovider improvements.
Eclipse plugin data compare tool.
Multiple Github fork(s) (http://github.com/virtix/mxunit).
Office Hours every other Monday at 12:00pm US Eastern Time. Join in via Adobe Connect and bring any testing or other technical questions or topics: http://bit.ly/MXUnitOfficeHours.

MXUnit couldn't deliver the kind of product it is without the generous help of many talented and dedicated programmers. This particular release saw significant contributions from these chefs:

Tim Farrar

Peter Farrell

Patrick McElhaney

Randy Merrill

Bob Silverberg

John Whish

Cop the latest and greatest at http://mxunit.org/download.cfm and get involved http://groups.google.com/group/mxunit

Test and be Happy!
theguys at mxunit.org

Your CFML Wish List

noreply@blogger.com (Marc Esher) — Mon, 21 Jun 2010 14:39:00 +0000

For years, Adobe ColdFusion has added feature after feature that has dramatically improved productivity for CFML developers. With CF8, support for struct and array literals was added as well as improvements for operators (ability to use <= instead of LTE, ++ for incrementing, etc). With CF9, they added support for full-script CFCs. Well done, Adobe.

I’d like to hear from you: what other enhancements would you like to see to the core language? What drives you nuts that you wish you could change (var scope, cough cough). What is missing? What’s half-baked? I’m not talking about features or new tags or functionality … purely language-level stuff. As a coder, what do you wish you could do with CFML that you can’t do today? Perhaps you’ve worked in other languages and encountered semantic constructs that you miss when working with CFML?

This is strictly for my curiosity: I don’t work for Adobe, and I have exactly Zero Pull (tm). I have a list of enhancements I’d like to see made to CFML, and I’m curious how your wish list compares with mine.

Thanks!

A New Eclipse Timesaver for Flex, Java, and [insert language here] Developers

noreply@blogger.com (Marc Esher) — Mon, 21 Jun 2010 10:50:00 +0000

CFML developers using CFEclipse (and more recently ColdFusion Builder) have long enjoyed the benefits of Snippets and the Snip Tree View, a way to create customized, variable-driven chunks of code which you can drop into your source with a single keyboard combination. I wrote about them a long time ago, Matt Gifford recently wrote what has become the de facto guide to snippets, and lo many years back Mark Drew made this video which is still relevant today.

To add to the general awesomeness of snippets, you can share them amongst your computers or even other developers. This shows you how to do so with DropBox.

In my experience, once a developer becomes snippet-infected, they tend to bemoan their absence when editing other types of source code. Unfortunately, Snippet insertion never worked when editing other files like Flex, and even when it did work, you had to install the entire CFEclipse plugin which some developers don’t want to do. Until now.

Who’s the target audience?

Flex, Java, JavaScript, Groovy, Scala, [insert your language here] developers who use Eclipse and who do not have ColdFusion Builder installed.* It’s also important that you like to save time.

*ColdFusion Builder snippets should already work with other sources! Good job, CFB team.

How do I install it?

The Eclipse Update Site is at http://cfeclipse.org/snippets/update/. In Eclipse 3.5 and later, you’d use “Help – Install New Software”. Then click the “Add” Button. Then add that URL into the URL field. Follow through to the end.

What will this install into my Eclipse?

This will install a standalone Snippet Tree View and corresponding key binding into Eclipse. By default, the key binding is “Ctrl-Shift-I”. Note that this will be different from the binding mentioned in the above articles, although you can reassign it to whichever key combination you like. In FlashBuilder, I immediately reassigned mine to the familiar Ctrl-J sequence.

Who’s responsible for bringing such Joy unto the world?

We’d be remiss if we didn’t thank the CFEclipse founding fathers. Rob Rohan and Spike Milligan will most likely never know that the work they did still continues to improve the lives of developers around the world. Mark Drew has led CFEclipse for years and many credit him with helping make it a mature, stable platform on which to build CFML applications. And Robert Blackburn added the SnipEx support into the Snippet Tree View. Thanks to all you wonderful people!

For this release -- separating Snippets into their own plugin and refactoring so that they’d be useful for other languages on the Eclipse platform -- Denny Valliant (github) gets all the credit. Denny’s one of those guys you don’t hear much about, but he’s brought inestimable joy into developer lives through all the work he’s done on CFEclipse and other projects. Thank you, Denny.

What to do next?

Install the Snippets Plugin
Read one or more of the links above explaining how to use snippets.
Open up the Snippet Tree View and add some snippets (don’t forget the “trigger text”!)
Go into your editor and type the trigger text for one of the snippets you added. Hit Ctrl-Shift-I to insert the snippet
Repeat Steps 3 and 4 and rejoice in all the time this will save you

Creating a Hybrid ColdFusion – Java Project in Eclipse

noreply@blogger.com (Marc Esher) — Sun, 23 May 2010 23:23:00 +0000

On more than one occasion recently I’ve wanted to incorporate Java libraries into ColdFusion applications. You can do this in your CFML, but it usually involves the following:

CFML code littered with createObject(“java”,”whatever.Whatever”);
Zero content assist for the Java objects
Optionally, Putting jar files in your CF installation’s lib directories and deploying those to all servers (dev, test, staging, prod) and developer machines

1 and 2 are the worst problems because 3 is solved with Mark Mandel’s javaloader (more on that later).

Ideally, I’d be able to write Java code in Java source files, compile it and test it in Eclipse, and fairly seamlessly use it in my CFML… in the same project. In this post, I’ll show you how to set it up in ColdFusion Builder. Note that these same steps apply to CFEclipse as well. In a follow-up post, I’ll show a more real-world example

Create your CFML project

In CFBuilder or CFEclipse, either create a new project or use an existing one.
Create a directory named “java”
1. under there, create “src”, “bin”, and “lib” directories
2. Download javaloader from Mark’s site, and drag the “javaloader” directory into the root of your project (or put it in your webroot)
3. If you put it into your project, add this to your Application.cfc: this.mappings["/javaloader"] = getDirectoryFromPath(getCurrentTemplatePath());

By now, your project should look something like this:

Apply the Java Nature

Close the project: right click – “Close Project”
From the file system, open the project’s “.project” file in a text editor
In the “<natures>” element, add this: <nature>org.eclipse.jdt.core.javanature</nature>
In the “<buildSpec>” element, add the following:

<buildCommand>
	<name>org.eclipse.jdt.core.javabuilder</name>
	<arguments>
	</arguments>
</buildCommand>

Back in Eclipse, open the project (right click -- “Open Project”). Once you do that, it’s going to look very strange… something like this:

Configure the Java Build Path

Your project now looks funky because it’s treating everything as a Java source folder. Let’s fix that.

On the project, right click, select “Build Path”, then “Configure Build Path”
In the “Source” tab, remove any entries
In the same tab, select “Add Folder” and navigate to your project’s “java/src” folder
Down at the bottom, in “Default output folder” text field, browse to your project’s “java/bin” directory

Now, your project should look something like this:

That’s better.

Create a simple Java file

Note in that screenshot above that you have “java/src” with a package decorator… that’s where you’ll be working. Ignore the “java” directory that you see in that screenshot… we don’t care about it for now.

In the “java/src” directory, right click, select “New – Package”. Name it “hello”.
In that (now empty) package, right click, select “New – Java Class”. Name it “Hello”
This will open a new Hello.java in a java editor. Let’s create a simple “echo” method. For now, just use a “main” method to drive it in Eclipse… we don’t need to get fancy with unit tests just yet:

package hello;

public class Hello {
	
	public String echo(String input){
		return "Hello. You said: " + input;
	}
	
	public static void main(String[] args) {
		Hello h = new Hello();
		System.out.println(h.echo("marc"));
	}
	
}

If all goes well, you should be able to run this file and see output in the console. To run it, select “Hello.java” in the Project Explorer and hit the green “Play” button in the toolbar, or right click in Hello.java and select “run as – java application”, or hit “Alt-Shift-X, J"

Use it in your CFML

To use it in your CFML, you’ll want to use javaloader. Conceptually, you simply tell javaloader where your compiled classes are, where any additional .jar files live, and then use it to create objects. Here’s what it looks like in practice. For demo purposes, I just created a file in my project root; thus, all paths you see in here are relative to that file in the root.

<!--- set up javaloader --->
<cfset binDir = expandPath("java/bin")>
<cfset libDir = expandPath("java/lib")>
<cfset jars = directoryList(libDir)>
<cfset arrayAppend(jars,binDir)>
<cfset loader = createObject("component","javaloader.JavaLoader").init(jars)>

<!--- run it --->
<h1>Hello!</h1>

<h2>Echo example</h2>
<cfset hello = loader.create("hello.Hello")>
<cfoutput>#hello.echo("marc")#</cfoutput>

If you’ve done everything right, when you run this in your browser you should see the expected output. If you go back into your java file, make a change, and hit refresh in the browser, the changes you made should be reflected.

Final Thoughts

For years, folk have praised the ability to integrate Java into CFML. However, the workflow has always been complicated and the tooling nonexistent. CFMLers who’ve worked for years without content assist on their own Components most likely have no idea just how powerful an ally their IDE can be. So we suffer, typing “whatever = createObject(“java”, “some.Java”); whatever.NowWhatWasThatMethodAgain(AndWhatWereThoseArgumentsAgain)”, not realizing that if we were writing our java in a “real” editor, you’d get all the richness the IDE can offer. In other words, most CFMLers don’t know what they’re missing when they have to incorporate java into their projects.

I am not saying that when you create Java objects in your CFML code that you’re going to get content assist; rather, what I’m saying is that if you have more than a few lines of Java code in your CFML, then perhaps it’s best to put that code into its own Java sources – inside the same project, as I’ve described above – and simplify the work you have to do with Java without sacrificing workflow and deployment speed.

It also opens up some other nice possibilities: If you’ve a Groovy fan, or have a problem that would be better suited for Groovy – yes… even if the thing you want to do is simply more enjoyable in that language --- then you could do very similar steps to turn your CFML project into a hybrid Groovy project. Create your Whatever.g sources, and use Barney Boivert’s excellent CFGroovy to run your Groovy sources.

If you need to write some java of your own, and you need to incorporate it into your CFML projects, then this IDE setup can help you.

Coming soon: a real-world example.

A Fun Bug in VMWare Workstation Installer

noreply@blogger.com (Marc Esher) — Fri, 07 May 2010 22:51:00 +0000

I hit this fun bug this morning and wanted to share it. It’s quite unlikely that you’ll hit it yourself, so the value here is probably as much in the debugging as it is in helping folks down the road fix this should they find it via Googling.

The Problem

I clicked to install VMWare workstation, and was greeted with this:

Yes, you’re seeing that correctly. Rather than seeing a normal installer window, I was looking at a Disney Vacation site. At first, I thought I must’ve had a virus on my computer but ruled that out pretty quickly. If it were a virus, I’d have been seeing pr0n or warez or some other such thing, not a Disney site… especially not a “static” Disney site.

Debugging the problem

The programmer in me isn’t thinking “Why isn’t this working?”. Instead, I’m thinking “What would cause an installer to display a Disney html file?”

Some virus is hijacking the file and replacing it with the Disney file
The VMWare installer is hosed
The Real Answer

I wanted to explore the idea of something hijacking the file first. The question is… what file? This was easy to find. I simply looked in the VMWare workstation installer log and looked at what files it was unpacking and writing. Here’s a snippet:

20100507074429:INFO CPackageManager::GetFile: index.htm 20100507074429:INFO CHtmlUI::SetHtmlFile: Navigating from '' to 'C:\Users\Marc\AppData\Local\Temp\vmware_1273232669\index.htm' 20100507074429:INFO CHtmlDialog::NavigateToFile: Loading file C:\Users\Marc\AppData\Local\Temp\vmware_1273232669\index.htm?lang=1033&locale=1033

I opened that index.htm file and sure enough, it was the Disney html. So, off to see what activity was happening around index.htm. To do that, I used Sysinternals Process Monitor. This simply involved firing it up and adding a filter for index.htm to get rid of the copious noise that Process Monitor spits out by default.

Looking at the output, the answer became obvious. The first instance of an index.htm was in c:\<path_to_my_downloads>\index.htm. Then I saw the index.htm in the temp directory. Ahhh…. so the installer was creating a file in the same directory in which it lived and then copying it into the temp directory.

I’m sure you see where this is going.

I headed to my downloads directory, and there she was… index.htm, Disney-fied.

How does this happen?

You might be wondering how in the heck that Disney index.htm file might have gotten in my downloads folder. The answer is simple: I have kids. If you’ve ever watched a kid use an internet browser, you’ll see them mousing around and doing all manner of weird draggy things that inevitably involves downloading a bunch of web files (usually images, but occasionally they can manage html files) into wherever you have configured downloads to go. It’s usually harmless.

In this case, you can probably imagine the code in the installer: “if !fileExists…, write index.htm” …. “copy index.htm to temp\vmware_blahblah\index.htm”

But why would you, the installer writer (or the program that emits the installers), fail to handle the case when there could already be a file like that in the place where downloads go? It’s a stretch, and that’s why this is such a fun bug.

BOFs: An Opinion

noreply@blogger.com (Marc Esher) — Thu, 29 Apr 2010 01:26:00 +0000

Most conferences nowadays have BOFs, “Birds of a Feather” sessions usually held after-hours on one or more nights of the conference. These are informal get-togethers where folk sit around and talk about a topic. These topics are usually set in advance, either by conference organizers or existing speakers. Like most conference sessions, attendance varies. This is my experience with BOFs, at any rate.

This year, at cf.Objective, Bob Silverberg asked if I’d coordinate a Test-Driven-Development (TDD) BOF with him. I accepted. I expected a handful of people. I was wrong in this expectation. It started small but quickly got way too big for normal BOF comfort. See, the problem isn’t with the number of people who attend: I was thrilled to see so many people. I do not know “how well” we did but I give us a “C”. The reason for this mediocre assessment is the subject of this post.

A BOF should be a place where you, the attendee, go with either A) some interest or B) a burning question. If you’re in Camp A, then all you should expect from a BOF is exposure to ideas. If you’re in Camp B, then I think you should get something approximating an answer to that question. If not “the answer”, then at least guidance. But when BOFs get too big, it’s unlikely you’ll get that. Lots of things stand in your way, probably the most prominent of which is one or two topics (and people) dominating the conversation. It’s what humans do, and I won’t put upon a BOF some expectation that it transcend humanity.

In our BOF, we had at least three burning questions:

What is TDD and how do I get started?
How do I test database interaction (and perhaps other network-bound services)?
How does TDD improve API design? (this is what I wanted to discuss, by the way)

A fourth lingering one was “How do I test ‘the entire app’?”, i.e. how does unit testing differ from integration testing (note that this has nothing to do with TDD at all).

This is simply too many needs to be met in an hour in a traditional BOF.

What a BOF should be

So what should a BOF be? If you’re in Camp B, it should be small and focused. BOFs sprung up lo those many years ago as a reaction to typical eyes-forward conference sessions. They’re a way to bring people together around a topic where everyone gets a chance to talk, not just a speaker. They are a discussion. They attempt to imbue a sense of the “hallway conversation” that conference-goers love to rave about. When BOFs get too big, they have failed this mission. Why is this? Perhaps it’s simply that anything that gets too successful becomes a victim of its own success. Let’s fix this.

How BOFs should be organized

BOFs should not be organized ahead of time, especially by conference organizers or speakers. Oh, I’ll make an exception for “Meet the Team” events that you get at large conferences like Adobe MAX. Otherwise, they should be organic. This is simpler than it sounds. Here’s a recipe:

At the beginning of the conference, put a chalkboard or whiteboard out in some open space. At smaller conferences like cf.Objective, this is in the lobby adjacent to the conference rooms. In a larger conferences such as MAX, this is … the same thing.
Promote the chalkboard ahead of time, in emails to attendees, signage, Twitter, and other means. Make it obvious that this chalkboard thing is where you go to suggest BOFs
Organizers can put headers on the Chalkboard: D1, D2, D3, C1, C2, “The Bar”, “The Bar with fancy Tequilas”… etc. These are the available spaces. Call them D1:8pm and D2:9pm if you have multiple time slots
People interested in a topic can write it under one of the slots. People who see a topic they’re interested in can put a line or some other marker underneath a topic to signify their interest in attending. Nothing fancy. Numbers don’t matter. A BOF of two is just as important as a BOF of 10.
At the time of each BOF session, people meet in the room and ask around to see who’s here for what BOF. My gut says self-organization at start time will take no more than 5 minutes

In our TDD BOF, I wish we had course-corrected immediately… I wish we had realized the interests in the room and asked people if they wanted to break out into smaller groups. Unfortunately, at that time, it probably would’ve been too late. We’d have first had to figure out what those interests were and then hope that a leader emerged (by leader, I mean “the person who says ‘let’s go over to this set of chairs’). If we were more prepared, we could have had an idea of what kinds of things people might be interested in and when we saw the group get large (this didn’t happen until about 10 minutes in, by the way), we could’ve thrown out those topics for show-of-hands and then broken out. But you should already see why this is a failure in the making: it’s an inappropriate time for such a thing, considering by the time “real talk” started, we’d have been 15 minutes into it because ad-hoc organization is always more chaotic than if the organization had been happening – on the chalkboard – all week long.

There is a precedent

If this sounds strange to you… if this violates some sense of control that you feel needs to be exerted by those-in-charge, know that what I’m proposing here isn’t an original thought, and it’s not untested. In fact, entire conferences are run this way (Google “open space conference”). The entire Java Posse roundup is largely run this way, and it’s a smashing success (not because of attendance numbers, but by a far more important criteria… attendee reaction).

Imagine

You go to an eyes-forward conference and you have a list of topics to attend during normal hours. But you also have a burning question. You put it on a board. One other person comes. You talk for an hour about your shared interest. You might learn something, teach something, make a friend, make an acquaintance, learn “I can’t stand this person”, learn “I want to start a business with this person”. Maybe you yank out a laptop and code for an hour together. You never know where it might lead. It’s unpredictable, and energetic.

That’s the kind of spontaneity that is the hallmark of a great BOF.

Conference organizers: make it happen.

Changing ColdFusion Builder file encoding

noreply@blogger.com (Marc Esher) — Thu, 01 Apr 2010 21:50:00 +0000

A few of our source code files contain Spanish text strings and smart quotes (i.e. curly quotes) copied from MSWord and pasted into Eclipse. These “special characters” have never caused problems in CFEclipse, but they were showing up as little boxes in CFBuilder:

I knew it was a file encoding issue but couldn’t solve it. The default text file encoding (window – preferences – general – workspace -– Text file encoding) was Cp1252, which was the same as CFE. However, viewing the properties of a file showed that the file was UTF-8 in CFBuilder but still Cp1252 in CFEclipse.

To get CFBuilder to behave like CFEclipse with respect to file encoding:

Window – preferences – general – Content Types
Expand the “Text” tree
Select “CFML Source File”
Down at the bottom, change the Default encoding

Warning: I wouldn’t change this setting unless you’re experiencing this problem yourself. To be clear, this falls into the “if you must do dumbass stuff like copy text from Word into Eclipse and retain the funky characters, then here’s how to fix it”

Jira: Creating a custom description template

noreply@blogger.com (Marc Esher) — Thu, 01 Apr 2010 21:15:00 +0000

It’s hard work getting your team to enter bugs/issues into your issue tracker (“can’t I could just email you?”), and it’s even more difficult getting them to provide enough description to make the issue summary helpful. As the person who’s job it is to report issues to you, they need to “transfer context”. They know this bug intimately, either because they found it themselves or a customer/user reported it to them.

You – our heroic developer – don’t know anything about the issue. They want you to fix it quickly, and you want to fix it quickly… so why is it so damn hard to get them to understand that the more detail they provide, the faster you both get what you want?

I don’t have any good answers to that question, but I’m hoping this tip can help you incent them to give you more of what you need.

Editing the Description textarea

Often, you want users to enter the following:

a link to the page that’s causing troubles
relevant IDs (order ID, for example)
a description of the behavior they expected
a description of the actions they took that resulted in the unexpected behavior

Here’s what my description template looks like:

You probably have others. To Pre-populate your with a customized template:

Navigate to your jira install directory. Drill down to WEB-INF\classes\templates\jira\issue\field
Save a copy of description-edit.vm (to be safe)
Edit description-edit.vm, like so:

#controlHeader ($action $field.id $i18n.getText($field.nameKey) $fieldLayoutItem.required $displayParameters.get('noHeader'))

## setup some additional parameters
$!rendererParams.put("rows", "12")
$!rendererParams.put("wrap", "virtual")

## marc's custom hoogie
#macro( setDescription )
-- All your custom stuff goes here --
SYSTEM: Dev  |  Test   |   Staging |  PROD

LINK TO DEMONSTRATE BEHAVIOR:

ACTUAL BEHAVIOR:

EXPECTED BEHAVIOR:

STEPS TO REPRODUCE:
-- your custom stuff ends here --
#end

## let the renderer display the edit component

#if ($description=="" || !$description)
#set ($description = "#setDescription()")
#end

$rendererDescriptor.getEditVM($!description, $!issue.key, $!fieldLayoutItem.rendererType, $!field.id, $!field.name, $rendererParams, false)

#controlFooter ($action $fieldLayoutItem.getFieldDescription() $displayParameters.get('noHeader'))

Restart Jira

That’s it!

You can read more about customizing Jira templates here.

Mylyn: Creating new Issues from TODO comments

noreply@blogger.com (Marc Esher) — Wed, 03 Mar 2010 12:28:00 +0000

On Twitter, the Eclipse PDE lead commented on a neat feature of Mylyn that I hadn’t seen before: the ability to add issues/bugs into your issue tracker directly from the “Problems” view. I wanted to see if you could do this from the “Tasks” view if you entered “TODO” comments into your CFML, and sure enough, it works. As of this writing, CFEclipse supports TODO tasks but ColdFusion Builder does not. Please consider voting for this feature request.

What I love about this feature is that you can fairly seamlessly create tasks and attach your context for that new task in less than a minute. When executives yammer on about “Work Smarter, Not Harder”, Mylyn features such as this exemplify the “work smarter” mentality. The point here is that you reduce friction in two areas: 1) creating the task with relevant information, 2) sharing your context with the developer to whom you’re assigning the task.

Reduced friction == Increased Productivity.

Here’s how:

In Words:

In CFEclipse, enter a TODO comment in your source code, Something like
Open the “Tasks” view – notice your TODO is now in the that view as a new task
Right click on that table row, and select “New Task from Marker”
This will bring up the screens necessary to add the issue to your issue tracker (or enter as a “Local task” if not using an issue tracker)

To then attach that file as the “context” for the issue:

Activate the new task
Navigate back to your file that contains the “TODO”. This file will now be part of the context for the issue
In the Issue editor, go to the “context” tab and click the “Attach Context…” link. Follow the steps to completion
Note that this is the same thing I wrote about previously

Now, when the developer to whom the issue is assigned (even if it’s you) opens this issue, they’ll be able to retrieve the context and get working on the file you want them to work on without having to navigate a single tree.

If this ain’t Jedi effect smoothness, I don’t know what is.

Happy Coding.

Mylyn and Jira: Sharing Context with Another Developer

noreply@blogger.com (Marc Esher) — Tue, 02 Mar 2010 12:40:00 +0000

When working on a team of developers, it sometimes is useful to reassign an issue to another developer. In Jira, you simply hit the “Reassign” link; this is standard amongst all issue trackers worth their salt.

Wouldn’t it be nice if you could also copy your Mylyn context? If you could say “I’ve been working on these 5 files, and when I reassign this issue to you, you’re going to get the same view of the world as me so you don’t have to figure out what files are involved in this issue”?

Here’s how:

In Words

In ColdFusion Builder or Eclipse, In the Issue editor for a given issue, go to the “Context” tab
It will show you all the elements associated with your context
In the upper left, Click the “Attach Context” link. Follow to completion. This will attach your context to the Jira issue
When the other developer working on the issue activates it, Mylyn will recognize that they do not have any context for the issue yet and will prompt them to open your context.
When they do so, it’ll open up just the files you had attached to the context

Awesome!

Big Caveat: This relies on both developers having projects that are named the same

Mylyn and Jira: A Short Tutorial

noreply@blogger.com (Marc Esher) — Mon, 01 Mar 2010 14:16:00 +0000

Recently, I wrote about why I believe Mylyn is an indispensable tool for developers on the Eclipse platform – this includes CFEclipse and ColdFusion Builder users. I also wrote about my favorite Mylyn-related toolbar item.

Today, I created a screencast that quickly shows how to link Mylyn to a Jira instance:

In Words:

In Eclipse or ColdFusion Builder:

Window – Show View – Task Repositories
Right click in the view and select “Add Task Repository”
Select “Jira”. If it’s not in the list, hit the “Install More Connectors” button and install the Atlassian Connector from there
Hit “Next”
Enter the URL to your jira install, label it, and enter your credentials
Expand “Additional Settings”.
I like to set the “Refresh Configuration Automatically” checkbox and I limit the search results to 100. Do whatever you like
Hit the “Validate Settings” button at the bottom. Then hit “Finish”
It’ll prompt you to pull in a Query from Jira. These will be your saved filters, which will include Jira’s default filters in addition to any custom filters you’ve created
Select a filter or create a new query. Hit enter and wait for them to load.

Exult!

Once you’ve done that initial bit of setup, you’re ready to rock. You can now work with Mylyn from the “Task list” view (this will be your main Mylyn view). If you want to include other filters, do this:

From the “Task List” view, right click and select “New Query”
Select your jira install, and hit Next
You can create a new query with the query builder form, OR:
You can use a saved filter from your Jira Install. I like this method
Select the “use saved filter” radio button, and select a filter. If you’ve added one that doesn’t show up in the list, hit the “Update from repository” button
Hit “Finish”

Your filter will now show up as a new tree item in the Task List view

Happy Mylyn-ing!

ColdFusion Builder: Importing existing projects into a new workspace

noreply@blogger.com (Marc Esher) — Mon, 01 Mar 2010 13:16:00 +0000

In ColdFusion builder and CFEclipse, I like to keep my workspace “light”, meaning I keep only a few open projects in the workspace at any one time. When it becomes necessary to work on a previous project that isn’t in my workspace, I “import” that project back in.

This method is also useful when you have already created projects in CFBuilder or CFEclipse and you want to use them in a new workspace. The “long” way would be to start a new workspace and then to do File – New – Create Project… for all the projects that you’ve already created.

The fast way is to simply “Import” them. I whipped up a screencast to demonstrate this:

In words:

In ColdFusion Builder or CFEclipse:

File – Import – Existing Projects into Workspace
Type the directory where your project lives.
1. If you want the option to import ALL projects in your webroot, you could simply type the location of your webroot – for example, c:\inetpub\wwwroot. This will take a while since it will scan every directory for .project files
2. Alternately, if you know which directories your project lives in, you could do them one at a time
Hit Enter
Hit Finish

Special Thanks to Dan Wilson for suggesting this tutorial

Why Mylyn is Indispensable

noreply@blogger.com (Marc Esher) — Sat, 27 Feb 2010 23:12:00 +0000

Warning: This post isn’t meant to be an exhaustive exploration of Mylyn for ColdFusion developers; rather, it’s a quick dive into why I love it and would not want to work without it.

What is Mylyn

Mylyn is an Eclipse plugin that works with both ColdFusion Builder and CFEclipse.

From the Mylyn home page:

“Mylyn is a task-focused interface for Eclipse that reduces information overload and makes multi-tasking easy. It does this by making tasks a first class part of Eclipse, and integrating rich and offline editing for repositories such as Bugzilla, Trac, and JIRA. Once your tasks are integrated, Mylyn monitors your work activity to identify relevant information, and uses this task context to focus the user interface on the task-at-hand. This puts the information you need at your fingertips and improves productivity by reducing searching, scrolling, and navigation. By making task context explicit Mylyn also facilitates multitasking, planning, reusing past efforts, and sharing expertise.”

For me, this means:

Noise reduction
Easy context switching and context reconstitution
Easy “context sharing”

All of this happens in the context of “tasks”. This might be a bug, a new feature, etc. It’s great when you can hook it into an issue tracker like Jira or Trac, but Mylyn can work just as well without a formal issue tracker.

Let’s look at what that actually means.

How Mylyn benefits me as a CFML developer

Part 1: Noise Reduction through “context”

My main Eclipse workspace contains dozens of projects. I keep all projects closed except active projects. Even with only a single open project, the workspace contains hundreds if not thousands of files. Here’s a screenshot of the Project Explorer with me working on an MXUnit issue, with just that project active in my workspace:

As I’m developing, I probably only care about a few files at a time. But the project explorer doesn’t help me keep “just those files” in view.

This is the first problem Mylyn solves: it helps you keep all the files you care about in sight and hides the rest.

Here’s what that same screen looks like when I create a task for what I’m working on:

Notice how the project explorer now contains just the files I want to see. This is the context of the issue I am working on. As I open more files, they get added to the context. If I want, files can drop off of the context as I close them. Removing files from the context manually is a keyboard shortcut away (ctrl-alt-shift-downarrow).

Part 2: Context Switching and Context Reconstitution

I have no proof of this, but I’d bet that most developers – at least developers in a corporate environment – work on more than one issue at any one time. Perhaps you’re developing a feature, and you get an emergency bug that just has to be fixed right this second or oh my God we’re all gonna die. Perhaps you’re fixing a bug in some system and as you’re doing so, you stumble on a bug in another system that you need to fix first because it’s blocking your ability to fix the immediate bug you’re working on. Regardless of details, this “multi-tasking” appears to be common.

But I don’t think brains work that way. Brains can’t spawn threads that think about 5 things at once. You can’t be fixing one bug while simultaneously fixing the bug that’s blocking you from your immediate bug fix. We have not acquired such higher consciousness yet. So no… we don’t multi-task.

We context-switch. Very very fast, in some cases.

The inevitable problem with context switching is that as soon as you stop working on Thing 1, and move on to Thing 2, the process begins whereby you forget stuff about Thing 1. You forget what code you were working on. You forget what you were thinking. You forget what other files were associated. The mental bonds you had constructed around that Thing begin to decay.

When you return to working on Thing 1, loading all of that data back into memory takes time. This is why programmers are always complaining to managers about “Stop/Start Time”. It’s why, when a PM fails to fully spec a problem, and the developer ends up working on the same thing 5 different times till it’s done right, that developer gets grumpy.

Imagine, though, if moving from one issue to another, and back again, was a simple matter of a menu selection.

Imagine:

You’re working on Thing 1. You have 8 files associated with it. Your PM comes running down the hall, crying and blathering about how “The client is going to leave us right this very second if we don’t fix (insert dumbass unimportant thing here)!”.
She enters a bug for Thing 2: “If website font isn’t shrunken by .0003 Pixels, Client will Leave and we’ll all foreclose on our houses”.
You close all the files you’re working on, you open up the appropriate files for this new bug. You save the day and the houses of your colleagues
You close those files, hit a button, and presto, you’re back into your original bug… at the same file, perhaps even at the same mouse location.

That’s context switching. You’ve moved from Thing 1 to Thing 2. When finished with Thing 2, you moved back to Thing 1… right where you left off. You’ve reconstituted your original context.

This is the second problem mylyn solves: it helps you move from task to task and back again.

I recently wrote about how I do this context switching with my favorite Mylyn toolbar item, and I invite you to read that.

Part 3: Context Sharing

At work in particular, I often work on something and then sometime down the road, I need to work on something similar. If it hasn’t been too long, I’m likely to remember that thing I worked on. I’ll search Jira and find the issue number. Wouldn’t it be nice if I could take that “context” (remember… for us, Context means basically “a collection of files”) and copy it to my new Jira issue? With Mylyn, you can do that:

From the “task” editor for the original task, I go to the Context tab and click “Copy Context to…”. It pops up a task finder, I select or search for the new task, hit a button, and bam, done.

Now, I deactivate that old task, activate the new one, and this is what it looks like:

That’s right… just like the “source” task, except in a new task. My original task remains untouched, and anything I add or remove from this new one is unique to just this task.

This is the third problem mylyn solves: It literally takes seconds to transfer the “mindset” of a previous task to a new task.

Summary

As developers, we are expected to be more and more productive. By default, constant stop/start – constant context switching – is anathema to productivity. Mylyn helps mitigate the pains associated with this kind of chaotic working style, and that’s why it has become one of my most valued and loved tools.

The Best Eclipse Menu You’ve Never Heard Of

noreply@blogger.com (Marc Esher) — Tue, 23 Feb 2010 13:18:00 +0000

Where I work, a kind of schizophrenia is the norm. It’s rare that I can work on any single Jira issue, from start to finish, without being pulled off of that issue and onto another. It’s also rare for me to have fully defined requirements backing any single issue, and so even when I think I’m finished, I usually need to revisit. What I’m trying to say is that there’s a whole lot of start/stop/resume. It’s not ideal, but it’s how we roll.

Prior to Mylyn, this was maddening. Mylyn, though, makes it easy to get into, out of, and back into issues at the point where I left off. In earlier versions (I’m thinking around the Eclipse 3.3 timeframe), Mylyn would contribute a toolbar item that made it braindead simple to navigate previous tasks. It is a little purple arrow which looks like this:

When you want to work on a previously worked-on issue, you simply hit the arrow, pick your previous issue from the list, and you’re back in the context of that issue. It is, no question, my most frequently used toolbar item.

Around the 3.4 timeframe, it went missing from a fresh Eclipse install. Here’s how to get it back:

Window – Customize Perspective
Select the “Command Groups Availability” tab
In the “Available Command Groups” pane, scroll down toward the bottom and find the “Task Navigation Extras” checkbox. Toggle it. Save your windows
Exult!

Here’s what the Command Groups tab should look like:

While you’re poking around the Command Groups, note that you can also deselect stuff that you don’t use, which can aid in noise reduction. Enjoy.

--marc

Dropdowns: The Scourge

noreply@blogger.com (Marc Esher) — Fri, 19 Feb 2010 02:50:00 +0000

It doesn’t happen much that I think about HTML. In a weird convergence of events, though, I find myself reflecting on SELECT boxes, aka “dropdowns”. Specifically, how they are my own personal bane, and how application developers can redress the scourge they have put upon us.

This is my story.

Consider

This is a screenshot from Mint.com’s account setup screen, wherein you can enter your username, password, and answer security questions for the account which match the security questions from your bank. The typical workflow goes:

You answer your security questions at your bank’s website
You go to Mint, you set up the account (account numbers, usernames, blah blah), and you fill in the security questions
You hit a submit button, magic happens, and all your data gets pulled into Mint.
Glory ensues

Except: Step number 2 isn’t really step number 2. It’s like step 2 through 108. With the 106 steps being “read every $*%!ing option in the “security question” dropdown list 5 times, maybe 10, to try to find a match to the questions that you answered on your bank’s site. Oh: they’re not sorted; they’re not worded exactly; they’re in no way thematically related. It. takes. for. ever.

To wit, pain:

Notice the size of the thumb on that dropdown. Notice how the questions are ordered in no sensible manner. Imagine using this for real.

I dare you to guess how many choices there are in that one option list. Now multiply it by 3, because that’s how many questions you have. By the way, Intuit now owns Mint.

And you thought that was bad

You will not be able to read the content of this image. You will, however, see that it is an HTML dropdown, based on the fact that there is a scrollbar on the right side. The thumb is a few millimeters tall, indicating that the list is very long.

This is part of a time-keeping application that folk in my company must use. This dropdown contains a list of projects. You go to this screen, you pick your project, you pick options from other dropdown lists, you enter how many minutes you worked on that application for the day. Repeat as needed.

Now, I figure that most developers work on at most 5 projects at any one time. Do you know how many projects are in the list in this screenshot? Can you guess? Now, in your head, do the math…

5 / NumAvailableProjects = % of shit that is relevant

or something like that. It doesn’t matter. Math doesn’t matter. The size of that thumb in the dropdown is all that matters.

Your job as a user

Your job as a user of an HTML select is to pick an option from a list. Usually, you know exactly what you want. Or damn close. Probably you use this application every day. That means when you’re picking options from your Project list, you know 90% of the characters in the text string that makes up the OPTION. You’re not poking around, thinking “hmmmm, I wonder what options are available to me”. Most option lists you encounter aren’t a whimsical stroll through unknown possibilities (like, for example, “Which shipping method would you like?”).

Most of the dropdowns in your life contain XX% noise, and 1 thing you want. But you gotta get through the XX%, every single time.

When I was 15, I took a job as a meat room cleanup kid. I learned how to use a knife, how to clean a block, how to “bone meat”. Eventually, during college, I worked as a meat cutter. I “worked racks”, which meant taking racks of meat from the cooler to the case and back again. Meat rooms are designed for efficiency of meat movement. If it were still alive, a Delmonico could wheel itself from cooler to case… that’s how well most meat rooms are designed.

Imagine putting a 3-foot curb between the cooler to the meat room, then from the meat room to the supermarket floor. How, tell me, could you wheel that meat cart over a 3-foot curb? HTML dropdowns like the one above do the same kind of disservice to a user: they significantly impede progress. If you’re a manager-type, you won’t understand anything I’ve written here, so let me put it into Manager-ese: It “has a negative effect on productivity”.

By the way, that dropdown is courtesy of QuickBase, a wondrous, extensible, powerful, enterprise-y application. Sadly, I know nothing of its wonder. All I know is that I have to pick a single project from that enormous dropdown list and I hate the application for it. By the way, QuickBase is an Intuit application.

You need to know

Take off your user hat, and put on your developer hat: You need to know that as the application developer, all those fancy things you do to get your enterprisey app to meet all the needs of all your big-dollar customers – to make your app behave as a JeeJaw for Customer1 and a Fahoozle for Customer2 AND a timekeeping application for Me – all the magic you have to work doesn’t mean a damn thing to people like me. What matters to me is the dropdown. I don’t want to marvel at your skill. I want to fill out my timesheet and do my real job. But I can’t do that quickly because I have to pick one option from a 523-and-growing list of projects.

I know, I know. “But that’s not how we designed it!”

Whatever. You wrote it. You own the Problem. Only you can fix it.

How to fix it

The current SELECT element in html is broken. Even 15 years ago, people knew it sucked. Every time you have to choose a date from an old-school date picker (month, day, year), you’re suffering from the failures of the HTML fathers to predict the grievous abuses that website developers would inflict. Every time you fill out your credit card details in an order form and have to scroll through an interminable list of expiration dates, you pay for their mistake. Perhaps you have some authority in your projects to suggest a correction.

I started with what is, frankly, the only awful part of the Mint.com experience. I’m going to end with the part of the application I interact with most frequently and which I consider the current best model for what SELECT lists should become. In other words, If I were in charge of the HTML spec, dropdown lists would have this functionality built in.

I’m making this image big b/c I want you to see it right away. Notice how the dropdown list itself is a type-in box. It is built for filtering. It’s built for action. Your job as account maintainer is to look at a line item and assign it to a category if Mint.com got it wrong. You know exactly what it needs to be. Or close enough. You know it’s a “coffee”. You know it’s “gas” for your car. You know it’s a tax of some sort. Mint’s category dropdown perfectly addresses this need. You start typing in the type-in box, and it filters down to whatever options might match what you’re searching for. You could have 300 categories and subcategories configured, or you could have 1000. It does not matter.

This is what that same dropdown looks like as soon as I start typing the name of the category I want to put the expense in:

I do not need to know exactly how the category starts, either. This is critical: good searching lets you, the user, be “close enough”. For example, if I wanted to categorize something as “auto insurance”, but didn’t remember if the category was “car insurance or “auto insurance”, I’d just start typing “Insurance”. With a few characters of typing, it narrows down to the available options, and I can easily see that “Auto Insurance” is what I want. I arrow to the choice I want, hit “enter”, and I’m done.

The central concept is:

You know what you’re looking for

The standard HTML Select list is built around: “WE have no idea what you’re looking for. You choose”.

Let me break this down for you

Good

“You know”

Bad

“We don’t know”

As a developer, you should…

Find ways to anticipate odious dropdowns in your applications. When you find them, consider a Mint.com style dropdown.

If you have good suggestions for canned/pre-built/jquery plugin/whatever examples of this, please share them with the rest of us.

Until you do so, realize that every dropdown that is populated by a database query or some other external, non-hard-coded datasource has the potential to grow. Do not kid yourself thinking “it’ll only ever be a few elements”. It won’t. The thumb will get smaller and smaller, and some user will start to complain. And you know what:

They’re right.

Does ColdFusion Builder freeze for you? You can help

noreply@blogger.com (Marc Esher) — Tue, 09 Feb 2010 12:33:00 +0000

I’ve been using ColdFusion builder extensively at work recently. It is beta software, so my intentions for using it are at least in part to help ferret out bugs so that when it launches, it’s a stable product that we can all trust. Currently, it’s not quite there yet, and so I want to show you a quick and easy tip for helping the CFBuilder team diagnose problems related to stability. This is in the spirit of “every little bit helps”.

I have posted this bug, which contains a description of what I experience every time CFBuilder freezes/hangs. In addition, I attached a java Stack Trace to the bug, hoping to give the CFB team some insight into what Eclipse is doing at the time of the freeze. I think that seeing the stack trace is about as close as we can get to showing what’s happening under the hood when CFB freezes.

This assumes you have a recent JDK installed and on your path. You’ll know you’re good if you can open a cmd and type “java –version” and get information. If you can do that, proceed. If not, download a JDK, install it, and continue.

To get a stack trace, you need to know one thing: the process ID of the javaw.exe process for CFBuilder. In Windows, this is as simple as opening up Task Manager (ctrl-alt-escape), finding javaw.exe, and noting the value in the “PID” Column. If you don’t have that, you can get it by choosing “View – Select Columns” and toggling the “PID” checkbox. It looks like this:

Now then, to get the stack trace into a file, open a cmd and type

jstack <pid> > CFBuilderStackTrace.txt

This will pipe the output of the jstack command into that file. From there, you can copy the stack. Feel Free to add it as a comment to this bug.

--marc

CFUnited 2010: Two Topics You Do Not Want To Miss

noreply@blogger.com (Marc Esher) — Tue, 09 Feb 2010 10:00:00 +0000

OK, OK, I know. I normally don’t truck with such dramatics, but the two topics I want to talk about merit it.

I’m talking, of course, about the two topics I submitted this year to CFUnited 2010. I’m asking for the chance to earn your vote. If you want to skip my stump speech, simply

Go here and Vote

You click that link, you search for “Esher”, you click the checkbox beside it, and you fill out a little more info and submit the form. I cannot emphasize enough how much good you will be doing for this world of ours (if you don’t, the terrorists have won. Do you want that on your conscience? Didn’t think so). Vote for change you can believe in!

My Pitch

Session #1: Debugging like a Rock Star

Last year at CFUnited, I presented on “Automating like a Rock Star”, wherein I spoke on batch processing, keyboard launchers, ANT, and JMeter. I covered a wide range of subjects, all with an eye towards helping make your development life less monotonous and more enjoyable.

This year, I’d like to present with that same goal in mind: helping bring more joy to your day job. I know: there sometimes is nothing more joy-less than debugging. I don’t pretend to have magic pixie dust that you can sprinkle on your apps to make all your nightmares vanish. But I do have a bunch of strategies that have served me well. We’ll cover the most basic CF debugging – how to read a stack trace / tagcontext, how to trace through an object graph, etc – and move on to tricks for what is potentially the hardest debugging: when your app manifests bugs in production that didn’t appear during development.

Rock Stars don’t fill arenas by growing long hair and putting on makeup and pursing their lips (except Poison, but they rocked, so…). Rather, they work hard, practice till their fingers and noses bleed, and out-persist lesser musicians. This session aims to demonstrate how to out-persist your bugs.

Session #2: End-to-end Testing Roundtable

How many sessions have you been to at a conference where you sat in a chair, a dude like me walked up on stage, said who he is, and rattled on for 50 minutes?

The End-to-End testing roundtable is NOT that session.

This session will be “facilitated” by some of my best friends in the community. We have no idea whether this will work. It could be the most boring hour or two of your life. But man… it could be awesome! What we hope for is that we can harness the collective experience in the room to discuss wins, failures, challenges, ah-ha’s, and other manner of discoveries related to application testing. We’ll talk a little bit about the different kinds of testing (unit, integration, the blurry lines), and from there, it’s all about engaging the audience.

We have no slidedeck backup. We have no prepared demos. All we have is a hope that we can attract a room of people who want to teach and learn.

If this sounds interesting to you, we simply say, “Come”.

Sessions you can believe in

Look: if you have worked with people, or are one of those people, who struggle constantly with effective, productive debugging, then I’m asking you to check the checkbox for Debugging like a Rock Star.

If you want to take part in – or at least give others this opportunity to participate in – a one-of-a-kind session on application testing, then please check the checkbox for the End to end testing roundtable.

Go here and Vote

Thank you. And God Bless America.

Dissecting The Bug

noreply@blogger.com (bill shelton) — Sun, 31 Jan 2010 03:16:00 +0000

Shifting gears

Attending school forces me into a different mind-set, requiring me to slow down and really think hard about abstract concepts, rather than just fixing bugs as quickly as possible. I like this. It’s also very inspiring to sit in class and listen. It’s been a while and I’ve forgotten, but it’s kind of like being in a conference session without the glitz and knowing it’s going on for several hours, days, and months, vs. a mere 50 minutes. I was impressed, too, by my professor, Paul Ammann, who’s incredibly smart, skilled, and understands the practical needs of technology. This is key—having professors who don’t acknowledge what practitioners have to do to stay employed shows a lack of understanding. Paul, on the other hand, demonstrates a keen grasp of this important concept. Many students, myself included, enjoy abstraction, see value in it, but also need to apply concrete concepts in order to put food on the table and clothes on the kids. Case in point, almost to the minute my job agreed to pay for this class, I got a new project: “Bill, we want you to automate the testing for Project XYZ, our highest priority project”. Little do they know that this course is not about automating testing—it’s about good test design, which, of course, is fundamental to any good quality assurance efforts.

Ok, enough about me …

So, in the most recent class (#2), we discussed a single software fault in painstaking detail. It was dissection, the main concepts being _observability_ and it's antithesis. These are my notes, some reflection, practice, a little poetic license, and likely a mistake or two.

RIP Observability Model

This model articulates three abstract properties that must exist in a program execution in order for a software fault to be observable.

Reachability – the fault in the code has to be reachable
Infection – the fault has to put the program into an error state. This does not necessarily mean the program will return incorrect results—this is an important distinction! Just because a program is in an error state does not mean that it will always produce the wrong result. see below
Propagation – the program needs to exhibit incorrect behavior; e.g., incorrect outputs or behavior or throwing an unspecified exception

A program can exhibit zero, one, or two of these properties, but in order for it to be observable all these criteria need to be met.

Dissection

This code looks simple, and it is, but using simple code helps in grasping the RIP concepts. This is not the same code we covered in class, but it is in the book. The code has a bug; try to find it …

/**
*
* Effects: returns the index of the last element that matches y, else returns -1
*
* @returns int
* @throws NullPointerException
*/
public int lastIndexOf(int[] x, int y) {

  for (int i = x.length-1; i > 0; i--){
   if( x[i]==y ) return i;
  }
  
 return -1;
}

A typical JUnit test for this would look something like following and the test fails, but why?

@Test
public void testLastIndexOf(){
   int expected = 0;
   int[] x = {2, 3, 4, 5};
   int y = 2;
   int actual = theClass.lastIndexOf(x,y);
  Assert.assertEquals( expected, actual );
  }

The bug isn't too hard to find simply by looking closely at the code. But if this were buried in some stacktrace it would be much harder to locate—here we already know where it is, within a few lines of code. In the real world, we would simple change the > to >=, pat ourselves on the back for swatting the fly, and go grab some Fritoes. But with a slightly more rigorous approach, we have an opportunity to explore some interesting and useful concepts—state and what makes this particular defect observable or not. To be clear, the fault in the code above is the usage of the '>' operator.

The RIP model states that in order for the fault to be observable, it needs to satisfy all three criteria—Reachability, Infection, and Propagation. It’s also helpful to show cases (test cases) where the fault is not observable, yet puts the program in an error state. How can a program be in an error state and not output an error? These are the kinds of bugs many of us have wrestled with—the kind of bug that gnashes its sharp mandibles only under certain circumstances.

The questions to ask are (and these are loosely taken from the book) :

in which case(s) would the fault not execute?
in which case(s) would the fault execute yet not create an error state?
in which case(s) would the fault execute and create an error state, but not propagate?
in which case(s) would the inputs cause execution of the of the fault, cause an error state, and propagate?

If you're still reading my guess is that you want to know how you can execute a fault, be in an error state, yet not propagate. The other questions can be figured out without too much problem, but this one, might cramp the brain a bit. First, let's define what state is. State can be defined as a combination of variable and value pairs plus the program counter (PC) — this is where the code should be executing. It can also be viewed as a snapshot of the data plus the location of the program's execution pointer. Example:

S ₀ → S ₁ → S ₂ → S ₃ → S ₄ → S _{5 , where}S n_{represents some state of execution within the program.}

But, sometimes the program skips a state and goes from S ₃ → S _{5. This is the error state. If a some code is supposed to execute some line and it does not, that's skipping an intended state, which is bad.}

In S ₀ we’re entering the lastIndexOf(...) method and we have values for inputs x and y. In S ₁ we’re now at for(int i = x.length-1; ...) and our state now has some value for i. What if we have a single element array; e.g., x = [ 1 ] instead? Because it’s only one element, this gives i the value of 0, so the bug i > 0 kicks us out of the for loop, pushes us down to the return statement, and then returns the correct result, -1. The error state—and this is the take-home—is the fact the PC never reaches the if( x[i] == y ) statement.

An analysis of the state of the program when the error state was entered can be presented like this:

Expected State         Actual State
------------------------------------------------------------------------------------
 x = [ 1 ]             x = [ 1 ]    
 y = 2                 y  = 2
 i = 0                 i = 0     
 PC =  if( x[i]==y )   PC = return -1;  <-- error state

In more concrete terms, this is just saying the code didn’t do what it was supposed to, because it never evaluated the conditional expression if ( x[i]==y ). In RIP terms this is saying that Reachability was satisfied, Infection was also satisfied (an error state occurred), but Propagation did not occur. So, given the inputs, this software fault is not observable. Again, this is the kind of bug that, usually buried deep in some system, causes us pain. It's very difficult to find after the fact, but a simple unit test up front would catch it.

Today, one test I would write would be make sure that the y was present in every possible index of the array being searched:

Testcases:
x = [2,3,4,5]
x = [3,2,4,5]
x = [3,4,2,5]
x = [3,4,5,2]

Some array shifting algorithm would be useful here to make test case inputs easier to generate!

Chances are, in practice, you would never go into this much detail about such a small fault. This is taking an opportunity to think about bugs, or software faults, in detailed and abstract terms, which is not only a great exercise, but, it's also a useful and practical way to reason about the complexities of building quality into software. We're not constrained by a particular language implementation, or platform, and dealing with abstractions such as this transcends much of the industry noise and helps us to focus on the principals of software quality, rather than the software itself.

Next up: Coverage Criteria

Test and Be Happy!

bill shelton

Software Testing

noreply@blogger.com (bill shelton) — Mon, 18 Jan 2010 14:26:00 +0000

I “am” plugging this book, and for good reason …

Update (January 22, 2010) : The authors (Paul Ammann and Jeff Offutt) donate 100% of the book's royalties to the Software Engineering Scholarship Fund, a scholarship fund for software engineering students.

Admittedly, I am a bit biased, being both a student of the authors and a QA geek. To create even further bias, even though I bought a copy of the book for over a year ago, I never thoroughly read the preface until yesterday. I was both surprised and flattered to read my name in print, acknowledged as one of the students who contributed and helped with the book. I recall reading and commenting on the text before it was published and then being disappointed that I couldn’t attend the class before I graduated—it was a new course then and I had some graduation requirements to fulfill.

Coming from someone who contributes to the core MXUnit test framework, written a few blog posts, and spoken at conferences on testing and TDD, the book’s title may be construed as misleading. It can be used as an introductory text, but also presents practical, novel, and advanced concepts, such as coverage criteria (graph and logic coverage, input space partitioning, and syntax-based testing ). I suspect one of the main reason’s for the title is to make it accessible at many levels—and this is a big asset. Introduction to Software Testing can be used both as a text and as a test engineer or developer’s reference.

One of the goals of the book is to strike a balance between theory and practice. The book’s introduction establishes the context of testing in software engineering from a more formal process point of view. (It might be helpful or more accessible if the book were to also mention Agile methods, too, in which testing is an integral part of the process.) The book also identifies the fundamental weakness we all have to deal with in respect to testing—Testing can show the presence of failures, not their absence. This key point applies to both manual and automated testing and at all levels, unit through acceptance. In other words, based on your tests, you can’t prove that that your code does not have bugs. This is one of the main problems the book addresses through exploration of coverage criteria. Through coverage criteria, is it possible to describe and possibly measure the relative quality of your application? …

Now some 4+ years after receiving a masters, I’m going back to school and the first class I’m taking is SWE-637, Software Testing, the class I missed on the first go-around. I’m excited to dig deep, however, having a full-time job and being a parent drives me to try to work as smart as possible (that’s a constant learning process). Adding a graduate software engineering program to that, makes for very little time for anything other than that which absolutely necessary. With that, my goal for this semester is to focus on automated test design—how best to design automated tests in order to cover as many requirement criteria as possible in the least amount of space and time. I’d like to use functional testing as the primary medium and (as a bonus) develop a good grasp of the upcoming 2.0 version of Selenium (with Webdriver) as the test harness. But the focus will be on the principals of good test design rather than the tools.

Coverage criteria and test design is largely what the course and the book address. After focusing on TDD, unit testing, and building a testing tool, I still have the unanswered question of “what makes a test a good test?”. And how can I measure how much of a class, component, system, object, etc., is adequately covered by my tests? I know that when I write a lot of tests and take the time to reason about what I’m doing, I know the quality of my software is better. I know, too, coverage tools can let me know what parts of my code have not been exercised. Yet still, there’s that lingering question that tests can’t show the absence of defects …

Test and be Happy!

bill shelton

Using Eclipse MAT to track down a ColdFusion server Out-of-Memory Error

noreply@blogger.com (Marc Esher) — Mon, 11 Jan 2010 10:05:00 +0000

Recently, this came across my desk: “This process is causing ColdFusion to crash on the DEV server. It hasn’t crashed the developer’s local machine. Can you take a look?”

In this post, I’ll show how I came to the answer in this specific case. I’m putting this out there because the solution was both surprising and simple. I’m also writing about it because I want to show how I came to the answer very quickly, rather than spending hours going down a bunch of dark alleys.

The Short Answer

For the impatient, or the people who got here from Google, here’s the conclusion: ColdFusion Server Monitor can, indeed, cause your memory to spike, and crash your server, with “Profiling” turned on. It’s common knowledge that keeping the “Memory Tracking” turned on can crash CF; however, it’s rare that you hear about “Profiling” causing troubles. In my case, I believe the reason was setting the Database “Slowest Queries” and “Most Frequently Run” Queries to be very aggressive. For Slowest Queries, I had it set to give me the last couple hundred queries of 0 seconds. For Most Frequently run, I changed the default from 20 to at least 100.

To solve the server crash, I simply turned Profiling off.

The Journey

For those interested in seeing how I concluded Profiling (or, rather, my aggressive settings) was to blame, read on.

The Usual Suspects

I started by checking the usual OutOfMemory culprits:

Debugging on?

Memory Monitoring on?

Nope.

I went to server monitor and looked at the database stats: it confirmed my gut reaction, which was that the process was running thousands of queries. This isn’t necessarily a server killer in and of itself, but in my experience, OutOfMemory problems usually lead back to database access. So I went to the source code and checked out the queries that server monitor indicated were frequently run queries. I was looking for the tell-tale server killer: a couple of queries in a loop, and the loop having the possibility of running dozens of thousands of times. What I saw was perhaps questionable, but nothing shocking or easily identifiable as the offender. So, I put this in the back of my head and went to the next step in my process: get a heap dump.

By now, I’m about 10 minutes into the investigation. The 2 most obvious suspects weren’t implicated, and the 3rd most obvious was suspicious but not definitive. Now, at this point, JRun was up over 1GB of RAM, so a heap dump and parsing the dump would take a long time.

Getting a Heap Dump

I RDP’d into the dev server, using “mstsc /admin”. You might need to use /console instead. In my experience, using jmap over RDP usually requires one of these options.

I always have a JDK on the CF Server, so these instructions assume you either have a JDK installed or can install one.

Open a cmd, and type: jmap –dump:format=b,file=c:\jrun4\heapdump.bin <pid>, where pid is the windows Process ID of jrun.exe

this will put a heap dump file at c:\jrun4\heapdump.bin. If you want it somewhere else, just change the file param to the jmap command. Now, we need something to parse this file and show us what’s sucking up all the memory.

Analyzing a Heap Dump with Eclipse MAT

Before I describe the few steps I took, I want to point out Brian Ghidinelli’s excellent set of posts on analyzing memory problems in ColdFusion. My post is meant to be a quick and dirty description of how I found an answer to a memory problem quickly using Eclipse MAT. Brian’s articles are exhaustive, and I strongly encourage you to check them out if you want to *really* learn how to do this stuff.

Now then: I opened the MAT perspective in Eclipse (this is on 3.5), and I Opened the heapdump.bin file that jmap spit out. It spun for quite some time, and then I got an Out Of Memory error in Eclipse. Ironic, eh? This one was easy to identify, since I was running Eclipse with only 512MB of RAM, and the file I was parsing was over a gig. So I set Eclipse to run with 1350MB of ram by using the-Xmx option in eclipse.ini, and tried to open the file again. This time, it got pretty far, gave an error, but it still had generated some data. It didn’t give the full information one would need for an exhaustive analysis, but it gave me enough. Had I needed more information, I would’ve followed Brian’s instructions in this post for using Eclipse MAT to parse very large files.

Once I had some information on screen, I clicked the “Dominator Tree” link, and this is what I saw:

Go ahead. Open that file. Do you see that line at the top… the one with 22 million instances of a single class name consuming almost 1GB of Memory? Yeah… that’s the problem. His name is coldfusion.monitor.stack.CFStackFrame, and he was my huckleberry.

At this point in my investigation, I do not know exactly what that class is, but judging by its name, I’m pretty sure it’s related to the CF server monitor. I open server monitor, turn off just the “Monitoring” option, and re-run the application. It crashes within minutes.

I restart CF, open server monitor, and turn Profiling off. I re-run the application, and it successfully completes. I then turn Monitoring back on – to be sure that it’s isolated to Profiling – and re-run the application. It completes.

If there’s a lesson, it’s:

After you spend a few minutes investigating and ruling out the usual suspects, stop speculating. Obtain data, and let it guide you.

What would your CFML look like with closures?

noreply@blogger.com (Marc Esher) — Thu, 07 Jan 2010 12:27:00 +0000

In this article, I’ll describe my exploration of closures: what they are, why programmers love them so darn much, and what your CFML code might look like if they were to be introduced into the language. A paragraph worth of disclaimers wherein I basically admit “I don’t know what I’m talking about” will follow. But I figured that was no way to pull you in, so… without further adieu, let’s start.

The Problem

I’m learning that most advances in one’s thinking start with a problem. This should be common-sense. It’s not like we sit in our chairs all day thinking “I wonder if I can come up with a nifty solution to something today.” Rather, we have problems to solve, we think about the existing solutions, and every once in a while, we realize that what we currently can do isn’t what we should do. Sometimes, the solution hasn’t been born yet. This originality is rare and I’ll leave it to really, really, really smart people. Often, the currently-unavailable solution has already been born, just in other contexts. In my case, the problem I was trying to solve would have been neatly handled if CFML had closures, which I knew about from various other languages. So let’s talk about the problem, and why closures are a nice way to approach it. I apologize in advance if this specific problem seems confusing. I thought about drumming up a fake problem for sake of clarity, but I decided against that. Real-world problems, though harder to initially grok, have always helped me more fully understand things, even if it takes two reads.

I have a “Formatter” component. It’s a utility component that does some all-purpose formatting of financial data. Dollars, percents, that sort of junk. This is CF, not Java, and as such we’re not in the business of passing around “Money” objects when a simple decimal value will do. Consequently, it’s useful to be able to pass these values through a Formatter object for display purposes (think: barcharts, tabular data, etc) while still retaining their raw value for use in calculations and whatnot.

I have a “TableBuilder” component. Its purpose is to take arrays of raw data in an “addTableRow” function and turn them into tables for display. This builder accepts various formatting options, the raw data (one row at a time), and can when requested add additional rows/columns based on the raw data (doing other calculations, etc) as a convenience. When it prints the raw data, it uses the Formatter

I have a “ThingThatUsesTableBuilder” component. This component’s job is to build a part of a PDF document. It does all manner of boring things – put some text here, put some text there, add some junk here, draw a box there – and also it needs to plop a table smack dab in the middle. It will use the TableBuilder for that purpose.

So: ThingThatUsesTableBuilder –>uses TableBuilder—>uses Formatter

When this TableBuilder was conceived, it was only ever meant to display integer values. As in, if it received decimal values, it rounded them. It was a “Very Bad Thing”(tm) if decimals ever displayed in these tables. Litigation, torture, foreclosure, you name it. Consequently, we built that behavior into this thing. “If you are a decimal, ye shall be smoted into an int”.

Queue the old timey Detective music. And then “She” came along.

You know how it is. You nail down the specs. You argue incessantly about “will we ever need this to be more flexible?” “Avaunt! and quit my sight!” And so it goes. You build it to spec, and then down the road, things change. And you end up in my shoes:

“Marc, what we really need, for this one table, for this one client, for just the first row, is that the decimals should appear. And here’s all the behavior around decimals”. When you’re dealing with Other People’s Money(tm), this stuff can get tricky.

Fortunately, this formatting business can all be nicely tucked away into the Formatter object and your calculation “engine” (engine means “a thing with a shitload of ‘if’ statements) can remain blissfully unaware of the unwholesome demands being placed upon its sanctity. Unfortunately, the TableBuilder looks like this:

drawTextFromMacro(textMacroName, formatter.formatWholeDollar(data[col],true,false), startX, startY, startAmountRightX);

See what it’s doing? Except in this one case, we don’t want that. In this one case, we want something different… we want a decimal amount in the cell, for one call to addTableRow(), in a TableBuilder whose reason for being is to print whole dollars.

Existing Solutions

Myriad approaches exist.

Change the TableBuilder to not expect the data to be raw, but to instead expect the data to be formatted. This is perhaps acceptable if the TableBuilder didn’t also want to perform additional calculations on the incoming data. And so if we make this change, we must also change the input data to have those calcs done ahead of time. Small inconvenience, perhaps, but it does start to make the method signatures on this component get really hairy
Accept a formatter object as an arg to the function that does the drawing. This would work, too, since CF enables dynamic method calls via CFINVOKE. But in order to do this, I’d need to pass the formatter object, the function to be called, and the arguments to that function. So now my method signature just ballooned by 3. Blech.
Stop being lame with this “Formatter” object business, and create a different formatter for all the possible conditions I might encounter: WholeDollarFormatter, DecimalFormatter, PercentFormatter, and on and on. Then, I just construct an appropriate instance inside my loop that’s adding data, based on whether I need the row to be WholeDollar or Decimal Formatted, and pass it as an arg. All of the OO smarties right about now – who’ve gotten this far without wanting to go kick their cats b/c I’m so stupid for not having gone this route from day 1 – will probably say this is a good option. Yeah, probably.
Add other approaches here. The internets love to make suggestions. So pretend I thought of all your suggestions, and pretend they’re here, in bullet point #4. :-)
If your mind went to “FormatterStrategyFactory”, you need to go get yourself a beer

Where My Head Is

Lately, I’ve been thinking that maybe the strict OO that I have come to love isn’t all that hot. No, it’s not because some douchebag wrote a blog post about how he loves procedural CF and he can’t “get” OO. It’s more about how perhaps OO has run its course, and perhaps since the world is going multi-core, and parallelism is just the way things are, and since “state” is the root of all evil with respect to parallel computing, that maybe OO isn’t the best model in the new world. Enter stage left, Functional Programming. I’ve been doing a lot of javascript programming lately, dabbling in Groovy, and reading Programming in Scala. So somewhere in my brain, I have this ADD-raddled set of neurons that can’t get a moment’s peace, and as I hit this dumbass dollar formatting problem – and all I want to do is get to lunch, get through the day, and go the hell home – this thought pops into my head:

It kind of came unbidden. Up until that point, I had not done any serious investigation into closures, except what I knew about them from javascript. But it’s funny: in javascript, closures aren’t “a thing”. They are “the way things are”. It’s part of the language, like parentheses and colons and var and array.push(). If you’ve ever used jQuery or EXT, you’ve used closures and you maybe didn’t even know it.

All of this is to say that in my head, thanks to the fact that I can’t focus on one thing lately for more than a few hours, I had these other languages creeping in saying “If you were programming in ME, this is how we’d solve it and you’d be on your way out the door”.

I do not know if that is good or bad. Maybe it’s lazy? Maybe it’s impure?

But think about this: Wouldn’t it be sweet, if in my current position, I could tell my function: “Hey, Function, when you format this data, I want you to do it this way”

And maybe it would look like this:

tableBuilder.addTableRow(
  data=someArray,
  blah=whatever,
  formatter="decimalFormatter"
);

And you’re thinking: “But dude, that looks exactly like what you described above, with the different Formatter classes, each with a single “format” method or whatever. But it’s not. Instead, formatter is a function, not an instance of a DecimalFormatter object I’ve declared elsewhere. That’s right, a function. And when tableBuilder does its drawText(), it does it like this:

drawTextFromMacro(textMacroName, formatter(data[col]), startX, startY, startAmountRightX);

What your CFML might look like if it had closures

You should be wondering: “How did he define this function? What does it look like?”. And here, in imaginary CF, is the answer:

decimalFormatter = function(value){Formatter.formatWholeDollar(value,true,false)};

tableBuilder.addTableRow(
  data=someArray,
  blah=whatever,
  formatter="decimalFormatter"
);

What’s happening in this imaginary example is that I’m passing a function I’ve defined in “ThingThatUsesTableBuilder”, named it “decimalFormatter”, and passed that function into the tableBuilder.addTableRow() method. TableBuilder then calls that function as if it were some function inside of itself. It looks completely natural. But notice this: the call to the formatter(…) function has no idea about what that Formatter object is. It doesn’t know about its state. Hell, for all it knows, formatter() uses some other object to do its work. It doesn’t know, it doesn’t care. It just does it.

Another example: Maybe the call to addTableRow looks like this:

tableBuilder.addTableRow(
  data=someArray,
  blah=whatever,
  formatter="function(value){Formatter.formatWholeDollar(value,true,false)}"
);

See that: it’s defining the function on the fly. It uses the “guts” from that Formatter object that I discussed way back at the beginning, passing some additional params to control its behavior. Then, when drawTextFromMacro() gets that function as the formatter, it knows what the “Formatter” object is (this is the magic of closures), and it applies that formatWholeDollar() function to the data.

Which brings us to….

What is a Closure?

I will get this wrong. I will miss the nuance and subtlety and beauty and power. That’s why you go find the answers yourself. So I’m going to give you my definition, based on a lot of reading and experimentation:

Closures are function literals + party invites.

Let’s talk about literals. Literals are the things on the right. y = 5; y is an int literal. you = “rad”; “rad” is a String literal. MXUnitHombres = {frameworkJunkie=”Bill”, mockMaster=”Bob”, pluginDude=”Marc”, residentGrump=”Adam”, seleniumGuru=”MikeH”, cohibaSupplier=”MikeR”} is a Struct literal.

decimalFormatter = function(value){…..} shows a function literal on the right.

tableBuilder.addTableRow(
  data=someArray,
  blah=whatever,
  formatter="function(value){Formatter.formatWholeDollar(value,true,false)}"
);

shows a function literal, passed as an argument to some other function.

Party Invites: When you define your function literal, that function might do simple things to the incoming arguments and not need the support of any additional functions/objects/variables. In that case, the thing you define really isn’t a closure. It’s just a function. But when you need your function to “remember” things about the world in which it was defined – in my case, I want my function to “remember” what the Formatter object was – that’s were we get into closures. In my example above, where I created the decimalFormatter function, its contents included a reference to the Formatter object I created earlier. When I pass my function into other objects and functions, it will still work – even though the recipient of my function has no notion of this “Formatter” object. What I’ve done is I’ve invited the Formatter object to the party. I’ve said “Hey, you, you come along with me”.

At risk of offending the purity gods, I’ll say this. Think of it like if your function literal "kept knowing about all the variables you invited into it even after that function was passed into a place that had no knowledge of those variables”. Ben Nadel probably does as good a job as I’ve seen of graphically describing this, in the context of javascript.

So for example, look at that code up above, where I’m doing formatter=”function(value){Formatter.formatWholeDollar…}” as an argument to addTableRow(). In the refactored example, now that TableBuilder is no longer responsible for formatting, it doesn’t know anything about this Formatter object. It’s been relieved of that responsibility. How is it supposed to know what it means when it gets a function that calls this “Formatter.formatWholeDollar” thing? Well….

That’s closures. It knows about it because that’s how closures operate. They “enclose” or “close up” the stack around them and stuff em in a napsack and take it with them on their journey. They say to TableBuilder “tut tut, TableBuilder. Don’t fret. When you call me, I’m going to call this Formatter thing that I’ve got in me pocket, and ye shan’t worry, nor shall ye hurt him, because he’s mine and ye can’t keep him. So call me, I’ll do what I do, and go about yer business.”

Why is This Preferable?

In my case, this approach is preferable because it lets me very cleanly and easily encapsulate the behavior I want very close to the point at which I’ll use it. Immediately before I call the function that adds the row of data, I define the formatting it will use. Now… all the nastiness behind formatWholeDollar() is safely tucked away. We want that stuff out of the picture. We do want to control those last two args though (whatever they are) for the purpose of just this one row of data. So for this one row, we say “I want to call formatWholeDollar, with these two extra args”. And we do that by defining a function that does just that.

What’s it look like in real life?

As part of my foray into closures, I wanted to see what it’d feel like to do a rough approximation of my problem in some different languages. I chose the ones closest to me at the moment: javascript, groovy, and scala.

Here’s some javascript code that does kinda the same thing (stripped down for clarity). This is full HTML, so that muddies the picture somewhat, but I wanted to give you something you could copy/paste/run right in your browser. It defines a Formatter object, an array, a function that does stuff with the array and takes a formatter function. Then it defines two functions (dollarAdder and zeroAdder). In the HTML, the onClick attributes call the doStuffWithArray function, passing in the function.

<html>

 <head>
 
 <script>
     //ignore... just for printing stuff to screen
  function addRow(data){
   var newline = document.createTextNode(data);
   document.getElementById("content").appendChild(newline);
   document.getElementById("content").appendChild(document.createElement('br'));
  }
 
 
  values = [1,2,3,4,5];

  Formatter = new Object();
  Formatter.formatWithOptions = function(value,addDollarSign,addTwoZeroes){
   if(addDollarSign) value = '$' + value;
   if(addTwoZeroes) value += '.00';
   return value;
  };

    
  function doStuffWithArray(array,formatter){
   addRow("Doing stuff with " + array);
   for(var el in array){
    addRow("Using item " + array[el] + " as " + formatter(array[el]) );
   }
  }
  
  dollarAdder = function(value){return Formatter.formatWithOptions(value,true)};
  zeroAdder = function(value){return Formatter.formatWithOptions(value,false,true)};
 </script>
 
 
 </head>
 
 <body>
 
  <input type="button" value="dollar adder" onclick="doStuffWithArray(values,dollarAdder);">
  <input type="button" value="zero adder" onclick="doStuffWithArray(values,zeroAdder);">
  <input type="button" value="inline dollar and zero adder" onclick="doStuffWithArray(values,function(value){return Formatter.formatWithOptions(value,true,true)});">
  
  <div id="content"></div>
 </body>

</html>

Run that, click the buttons, and you’ll see it in action.

Here’s some groovy:

package formatters;
class Formatter { 
 def formatWithOptions(Number value, boolean addDollarSign, boolean addTwoZeroes){
  def newValue = value.toString()
  if(addDollarSign) newValue = "\$" + newValue
  if(addTwoZeroes) newValue += ".00"
  return newValue
 } 
}

import formatters.Formatter

class Main {

 static main(args) {
  def formatter = new Formatter()
  def values = [1,2,3,4,5]
  def dollarFormatter = {formatter.formatWithOptions(it,true,false)}
  def zeroFormatter = {formatter.formatWithOptions(it,false,true)}
  
  def main = new Main()
  main.doStuffWithList(values,dollarFormatter)
  main.doStuffWithList(values,zeroFormatter)
  main.doStuffWithList(values,{formatter.formatWithOptions(it,true,true)})
 }
 
 def doStuffWithList(list,fn){
  println "Doing stuff with " + list  
  list.each {println "using item " + it + " as " + fn(it)}
  println "Doing other stuff..."
 }

}

Looks awfully similar to the javascript, doesn’t it? The main difference is groovy’s “it” keyword, as opposed to javascript where you can name your parameter whatever you like. The other difference is that I think this groovy example more clearly demonstrates the “Party Invite” concept. I’ve defined the “formatter” instance in the main method. I’ve defined the dollarFormatter and zeroFormatter function literals in main, using that formatter instance. I pass in those functions as arguments to another function (doStuffWithList), which lives in an instance of another object (because I’ve created a new “Main”), and that other function, in a new instance of Main, should by all reasoning have no notion of what “formatter.formatWithOptions….” means. But it does, because we’ve invited him to the party when we defined the function. This is why doStuffWithList doesn’t complain. In the javascript example, it very well might work because of the global nature of the “Formatter” I’ve declared. In this groovy example, it’s clearly shown that that is NOT happening.

And finally, some scala:

package formatters

object Formatter {

 def formatWithOptions(value:Number, addDollarSign:Boolean, addTwoZeroes:Boolean):String = {
   var newValue:String = value.toString()
   if(addDollarSign) newValue = "$" + newValue
   if(addTwoZeroes) newValue += ".00"
   return newValue
 }
}

package main

import formatters._;

object Main {
  def main(args : Array[String]) : Unit = {
    val values:List[Number] = List(1,2,3,4,5)
    println (Formatter.formatWithOptions(values(1),true,false))
    
    //formatter.formatWithOptions(value,true,false)
    doStuffWithList(values,dollarFormatter)
    doStuffWithList(values,zeroFormatter)
    doStuffWithList(values, Formatter.formatWithOptions(_,true,true)  );
    
  }
  def dollarFormatter = Formatter.formatWithOptions(_:Number,true,false)
  def zeroFormatter = Formatter.formatWithOptions(_:Number,false,true)
  
  def doStuffWithList(list:List[Number], fn:(Number) => String ) : Unit = {
    println("Doing stuff with " + list)
    for(item <- list) println ("using item " + item + " as " + fn(item))
    println("Doing other stuff")
  }
}

I’m least comfortable with Scala, so it’s most likely I’m A) doing it a little wrong, B) doing it all kinds of wrong, or C) doing all this in 20 lines when I could contrive some angle-bracket-dash-colon-underscore-ridden beast to do it in a single line of code. The two main differences here are that I had to define the formatter functions outside the place where I was using them (i.e. I couldn’t define the functions inline with the rest of the code, like I could with Groovy and Javascript, though that’s probably b/c I don’t know what I’m doing), and the use of “_” as the argument name. Think of it as like “it” for Groovy, but with symbols instead, because Scala is queer for symbols.

Final Thoughts

I included examples from other languages because I wanted to give you the feel of closures. I do not intend to come across as one of those people who wants things in CF just because the cool kids have them. Still… I like closures, a lot, and I believe that if they were added to CFML, in time many programmers would look back on the days before closures and think “Remember them days, Buck?” “Yeah, Grady. Them days sucked.” Closures add a level of convenience that will be shown to exceed even the convenience of struct and array literals added in CF8.

The risk with closures is that it can invite a return to spaghetti, with large chunks of functionality stuck in-line. This is a Bad Thing(tm) and good programmers would be wise to avoid this trap. You’ll notice in my examples that I used closures more as a tidying-up. In fact, these examples show no original functionality. This is particularly important for testability. I trust that Formatter.formatWithOptions(value,true,false) and …(value,true,true) and …(value) all work correctly because they’ve all been unit tested. I do not care to test the closure I’ve defined because all it does is call functions on other objects. If CFML ever gets closures, you’ll start seeing all manner of bloggers railing against The Monster Closure. It will be good advice, so heed it.

I am most certainly not the first person to want Closures in CFML. One of my favorite CF dudes has been talking about it for a while, and Sean Corfield told me that he planned to formally submit a recommendation to the CFML Advisory Committee. CF makes things easier, and I’d like to hear your thoughts on how you might or might not use this feature were it added.

Finally, I welcome any comments, criticisms, dialog, corrections, clarifications, etc. This was a “journey” of sorts for me. I used it to learn a lot more than I knew before, but surely I’ve gotten things wrong and so I’ll gladly correct any misinformation.

--Marc

MXUnit Eclipse Plugin 1.2 Released

noreply@blogger.com (Marc Esher) — Fri, 01 Jan 2010 13:47:00 +0000

Happy New Year!

We’re happy to announce a new version of the MXUnit Eclipse plugin is available. The goal of this release is to simplify configuration for projects with respect to helping the plugin figure out the “cfc root” (i.e., turning c:\myproject\myapps\test\whatever\SomeTest.cfc into “myapps.test.whatever.SomeTest”). It’s always been a sore spot… this business of “What’s my webroot? Do I use the webroot or project properties? What if I have my project set up like this…?”

With this release, the “webroot” stuff is gone. The CFC root is now configured only at the project level. For current projects that live at the webroot (like c:\inetpub\wwwroot\MyApp”), this should not cause you to reconfigure anything. For projects that live in other places, you might need to spend a minute configuring when you update the plugin. Of my dozen or so Eclipse projects, I had to update one. But my hope is that this change results in a much clearer configuration path for new users.

In addition, the Plugin’s built-in help has been expanded and reorganized. Now, when you click on the green “Help” (?) Icon on the plugin view, you get a much simplified view of the help, designed to take you to the two most commonly asked questions: How do I configure this thing to get the correct CFC path, and what’s this RemoteFacade URL all about? It looks like this:

If you want more, there’s a link there to take you to the full help. When you click that link, you get taken to the introduction page for the Help. That looks like this:

If you click the “Show in Contents” Button toward the top right of the view, you get the tree view of the help (This is the same view you’d get if you opened Help from “Help – Help Contents -- MXUnit”). If you expand the tree view, you’ll see the new organization of the Help content for the plugin. It looks thusly:

I added new documentation for configuring projects that need an Application.cfc, for projects that test CF9 ORM components, and all new documentation for the new way in which the CFC path for a test file is derived and configured. In addition, I added documentation on some things that MXUnit or the plugin doesn’t do but which weren’t clearly specified. Finally, I expanded the “Resources” section. Notice too that there are now links to “Troubleshooting”, “Frequently Asked Questions”, and “Tips and Tricks”, all from our new Wiki (created and hosted by MXUnit hombre Adam Haskell (Side note: if you have any thoughts on what content should be added into the wiki, or how it could be better organized, we want to hear it!). I wanted to put this documentation on the wiki, and not directly in the plugin, so that I could easily update it without having to push new releases of the plugin. Plus, this way, more people can contribute.

As always, if you have problems with or questions about the plugin, the fastest way to get answers is via the Google group.

Test and be Happy.

--Marc