How Many Assertions in My Tests?

Friday, October 30, 2009

One Long Test

When I first started writing unit tests with JUnit, I didn't read any books, follow blogs, or otherwise attempt to be the best test writer I could be. I wanted to crank out code, crank out tests, and see the green bar. In fact, my tests looked a lot like this:

public void testGetRequestBeansByIDType() {
    Integer requestID=8;
    Integer statusID=4;
    Integer systemID=1;
    Integer processTypeID=1;
    String inList="0,4,5,6,7,8";
    
    String requestIDSQL = "Select count(*) as NumRequests from " +
        "requests where requestid="+requestID;
    
    String statusIDSQL = "Select count(*) as NumRequests from " +
        "requests where statusid="+statusID;
    
    String systemIDSQL = "Select count(*) as NumRequests from " +
        "requests where systemid="+systemID;
    
    String processIDSQL = "Select count(*) as NumRequests from " +
        "requests where processTypeID="+processTypeID;
    
    String inSQL = "Select count(*) as NumRequests from " +
        "requests where RequestID IN("+inList+")";
    
    try {
        //test requestID
        HashMap<String,Object> row = dbops.getRow(requestIDSQL,1);
        int count = ((Integer)row.get("numrequests")).intValue();
        ArrayList<RequestBean> requests 
            = dbops.getRequestBeansByIDType("RequestID",requestID.toString(),"");
        assertEquals(count,requests.size());
        System.out.println("RequestID rows is " + requests.size());
        
        //test statusID
        HashMap<String,Object> row2 = dbops.getRow(statusIDSQL,1);
        int count2 = ((Integer)row2.get("numrequests")).intValue();
        ArrayList<RequestBean> requests2
            = dbops.getRequestBeansByIDType("StatusID",statusID.toString(),"");
        assertEquals(count2,requests2.size());
        System.out.println("statusID rows is " + requests2.size());
        
        //test systemid
        HashMap<String,Object> row3 = dbops.getRow(systemIDSQL,1);
        int count3 = ((Integer)row3.get("numrequests")).intValue();
        ArrayList<RequestBean> requests3
            = dbops.getRequestBeansByIDType("SystemID",systemID.toString(),"");
        assertEquals(count3,requests3.size());            
        System.out.println("systemID rows is " + requests3.size());
        
        //test ProcessTypeID
        HashMap<String,Object> row4 = dbops.getRow(processIDSQL,1);
        int count4 = ((Integer)row4.get("numrequests")).intValue();
        ArrayList<RequestBean> requests4
            = dbops.getRequestBeansByIDType("ProcessTypeID",processTypeID.toString(),"");
        assertEquals(count4,requests4.size()); 
        System.out.println("ProcessTypeID rows is " + requests4.size());
        
        
        //test IN() statements
        HashMap<String,Object> row5 = dbops.getRow(inSQL,1);
        int count5 = ((Integer)row5.get("numrequests")).intValue();
        ArrayList<RequestBean> requests5
            = dbops.getRequestBeansByIDType("RequestID",inList,"");
        assertEquals(count5,requests5.size());
        System.out.println("IN rows is " + requests5.size());
        
        // test order by....
        ArrayList<RequestBean> requests6
            = dbops.getRequestBeansByIDType("StatusID","3","RequestID Desc");
        RequestBean rb = requests6.get(0);
        RequestBean rb2 = requests6.get(1);
        assertTrue(rb.getRequestID() > rb2.getRequestID());
        
        ArrayList<RequestBean> requests7
        = dbops.getRequestBeansByIDType("StatusID","3","RequestID asc");
        RequestBean rb3 = requests7.get(0);
        RequestBean rb4 = requests7.get(1);
        assertTrue(rb4.getRequestID() > rb3.getRequestID());
    } catch (Exception e) {
       fail("getRow should not have returned an error");
    }
   
}

If you’re looking at that thinking “What’s wrong with that?”, then please read on. If you’re horrified… join the club.

What’s wrong with that?

testGetRequestBeansByIDType, how do I hate thee? Let me count the ways

  1. If some assertion fails up toward the top of this test, none of the other assertions run. So what happens is you get yourself in this constant loop of “it worked last month, now it’s not working. Why?” and so you go fix the failing test, satisfied everything is OK, and then you run em again and another assertion in the same test fails. This infuriates me
  2. Notice how I’m using comments for each of the different tests?  I’m trying to communicate requirements, but I’m doing so in a way that doesn’t much help. If each of these little chunks were separate tests, the test case itself would be much more communicative in my opinion
  3. It’s been my experience that the more things I do in a single test, the more likely I am to create/change some state that modifies the behavior of the subsequent assertions
  4. When developing in this manner, just cranking out tests and not writing descriptive test names that describe the behavior under test, I’m missing out on one of unit testing’s greatest benefits: using it as an aide to clarify thinking. This is a huge topic, and one that I don’t want to get into here. Suffice it to say, anyone who’s done TDD for a while knows exactly what I mean. Anyone who hasn’t done it will only know what I mean once you get yourself in the camp of the people who have done it. You have to earn this answer

So what’s a boy to do?  Personally, I’ve become fond of BDD-style naming. I prefer long, descriptive test names; I prefer as few assertions as necessary in a single test.  This leads to some additional lines of code, but I use snippets for this so I’m not spending any more time writing more tests than I would be if I were simply writing more assertions. The resultant clarity is worth it.

Ahhhh, That’s Better

Here’s a more recent example:

<!--- basic tests to make sure the 'outer' logic is right--->
<cffunction name="recipientsWithAffectedLetterCode_Should_GetPhoneNumberCoordinates" returntype="void" access="public">
        <cfset var coordinates = "">
        <cfloop list="#affectedLetterCodes#" index="code">                    
                <cfset coordinates = getCoordinatesWithWCRBCD("JH89",code)>     
                <cfset assertTrue( StructCount(coordinates) GT 0,"Coordinates struct should not have been empty for #code# but was." )>
        </cfloop>
</cffunction>

<cffunction name="recipientsWithoutAffectedLetterCode_ShouldNot_GetPhoneNumberCoordinates" returntype="void" >                        
        <cfset var coordinates = "">
        <cfloop list="#unaffectedLetterCodes#" index="code">
                <cfset coordinates = getCoordinatesWithWCRBCD("junk",code)>     
                <cfset assertTrue( StructIsEmpty(coordinates),"Coordinates struct should have been empty for #code# but was not. It had keys: #StructKeyList(coordinates)#" )>
        </cfloop>
</cffunction>

<!--- more granular tests for the specifics of which number to get;
yes, this duplicates the logic in the production code. So be it. --->
<cffunction name="WCRBCD_JH89_gets_2011_PhoneNumber" output="false" access="public" returntype="any" hint="">
        <cfset var coordinates = getCoordinatesWithWCRBCD("JH89")>
        <cfset assertEquals("866.420.2011",coordinates.PhoneNumber,"")>               
</cffunction>

<cffunction name="WCRBCD_MK01_gets_9446_PhoneNumber" output="false" access="public" returntype="any" hint="">
        <cfset var coordinates = getCoordinatesWithWCRBCD("MK01")>
        <cfset assertEquals("866.356.9446",coordinates.PhoneNumber,"")>       
</cffunction>

<cffunction name="WCRBCD_NotMK01AndNotJH89_gets_7436_PhoneNumber" output="false" access="public" returntype="any" hint="">
        <cfset var coordinates = getCoordinatesWithWCRBCD("NOTMK01")>
        <cfset assertEquals("866.281.7436",coordinates.PhoneNumber,"")>       
</cffunction>

Ignore for a minute the weirdo codes (WCRBCD, JH89, etc) as they are domain-specific and well-known for people working on the project. The point here is that the names describe exactly the various expected behaviors of this single function under test (getCoordinatesWithWCRBCD()). When one test fails, I don’t have to spend time reasoning about whether previous logic or function calls are the culprit. I and other team members don’t have to learn about the behavior by reading comments because the function names themselves are the documentation.  I don’t have that “Fixed-this-assertion-and-now-another-one-is-failing” problem. If something breaks, I know very soon because we run everything in a continuous integration environment. Code changes, tests run, something breaks: I can go right into Hudson and see exactly the behavior that used to work and now no longer works. If multiple things break as a result of a code change, I see those multiple things, not just the first assertion that failed.

Is there a downside to this approach? Yes… if you’re typing out each function by hand, i.e. < c f f u …, then you’re going to be typing more. But if you’re typing out functions by hand, you’re doing it wrong! MXUnit comes bundled with a bunch of useful Eclipse snippets. Read the documentation, install them, and use them. Then, when you want to add a new test function, you type the word “test”, hit CTRL-J, and up pops this box:

testbox

From there, you type in your test name. For example: “thisFunction_should_DoFancyThings” and hit Enter. This code gets added to your editor:

test_result

Typing problem solved.

The Bottom Line

In the end, the fewer assertions you put into your tests  -- as opposed to fewer tests with more assertions -- the more descriptive, stable, and design-for-testability-helpful your tests will become.

Looping over an array in CFScript

Thursday, October 29, 2009

The other day, I was searching the cf9 docs for an equivalent of the <cfloop array=”#myarray#”> functionality introduced in CF8, and it appears that was not introduced. So it was back to ArrayLen() looping for me. Tonight, I saw TWiCF’s Micky Dionisio put this out on the Twitters, and I thought it was cool:

“#coldfusion 9 tip -grab a handle on the iterator of an array by calling someArray.iterator(). then u can use that in script FOR/WHILE loops!"

I never thought of that, so I wanted to give it a whirl.

<cfscript>
//pre-cf8
list = "one,two,three,four";
a = listToArray(list);
for(i=1; i LTE ArrayLen(a); i=i+1){
        writeoutput(a[i]);
}

//traditionally, with cf8+ syntax
a = ["one","two","three","four"];
for(i=1; i <= ArrayLen(a); i++){
        writeoutput(a[i]);
}

//with an iterator
//uncomment if you want to see what's available: writedump(a.iterator());
iter = a.iterator();
while(iter.hasNext()){
        el = iter.next();
        writeOutput(el);
}
</cfscript>

Warning: don’t try to skip the iter = a.iterator() and el = iter.next() steps in an effort to save a few lines of code! If you try to loop while(a.iterator().hasNext()), you’ll get stuck in an endless loop because you’ll be returning a new iterator each time. And each time you call iter.next(), you’re advancing through the array; thus, doing writeOutput(iter.next()); doSomethingWithThisElement(iter.next()); will give you very different results from what you’re expecting. You’ve been warned.

Thanks for the tip, Micky!

This Week in ColdFusion Episode #3


I posted my review of TWiCF episode #4 yesterday, and today on the drive home I listened to episode #3. The first half of the episode was a discussion of conferences, since the episode was recorded at the same time that Adobe Max 2009 was underway; the second half focused on the question “To Flex or not to Flex”, which basically means “Should I spend my time learning Flex, or should I learn a rich javascript framework instead?”. Or, perhaps it also means “If I were building a project that required X, Y, and Z, what does Flex give me, and what does <insert fancy JS framework here> give me, and why should I choose one or the other for my project?”. 

Conferences

The line of inquiry for the first part was “Is it worth spending the money to go to a ColdFusion conference?” I can picture some listeners thinking, “What? Are you mad? That’s the stupidest question I’ve ever heard”. But I have to say: I think it’s a completely valid question. Mike brought up his experience with conferences and I think he describes a feeling that a lot of people encounter: that feeling of “I’m here to bask in the glory of the presenter”… to be talked to, to be lectured. To soak up their wisdom and be grateful for this brief glimpse of their brilliance. Lo, a star!

I’d bet that more people than are willing to admit have had those Stuart Smalley moments. “Am I worthy?” “Am I good enough to be here?” Perhaps. Perhaps not. Regardless, I can understand the sentiment.

I want to say to the TWiCF guys – and to any listeners out there with whom those thoughts resonate – you will not find a more inclusive, caring, friendly, willing-to-engage community than the ColdFusion community. And I’m not just saying that as a guy who’s fairly outgoing. I attended cfObjective for the first time in 2008, and I was so grateful for the warmth and inclusion that I saw amongst people whom I consider heroes: Andy Powell, Simeon Bateman, Peter Bell, Brian Kotek, Bob Silverberg, etc. These people rock! And they love to talk about this stuff with people, no matter whether you’re a constant blogger or just a guy/girl who wants to learn some CF and possibly something beyond CF that maybe, just maybe, will become useful down the road. In fact, if I’m reading the pulse of these people correctly, I’d bet they’d be just as or more passionate about talking to people who really, really want to engage than to some super-smart know-it-all blowhard. And you get these small, intimate BOF sessions where it’s you and a handful of super awesome people simply talking about a topic you’re all interested in. Seriously: how often do you get to sit in a room – as peers – with Barney Boisvert, Joshua Frankamp, Laura Arguello, Oguz Demirkapi, Luis Majano, and no-names like me talking about Git in a session led by Peter and Sim? And it’s not because you’re a Git superstar, it’s because you just want to learn more about it face-to-face?  These are the reasons you pester your bosses to go to conferences.

This year’s CFUnited was outstanding, as well. The content was superb. The speakers were everywhere. It’s so easy to spot a dude who you saw speaking a few hours earlier, walk up to them, and start firing questions. Interestingly enough: one of the strangest takeaways for me from CFUnited this year didn’t happen as a result of me sitting in a session or picking some speaker’s brain. Rather, it came from a conversation I had after my session on automation with Sumit Verma. He asked me if I had ever heard of a certain Windows command for working with Services, and I hadn’t. So I emailed him afterwards, we chatted, and now what I learned from Sumit – a guy who wasn’t speaking but who was in my session – has had a profound implication for something I’m doing at work right now. How sweet is that? Teachers and Students? Speaker and attendee? Nah. The blurrier that line, for me, the better. It’s people who know things interacting with people who know other things. That’s where magic happens.

You always hear “You get out of it what you put into it”. It’s true. When you go to these conferences, try to engage. Walk up to people and say Hi. Even if you don’t like hanging out in the hotel bar, go sit down anyway. Order a coke. Eat peanuts. Whatever. Just open yourself up to people. When some funny-lookin’ bald dude says “yo, you wanna go hang out with us and smoke a stogie?”, say “yeah, sure”, even if you don’t smoke stogies. You don’t have to suck down a Cohiba! Grab your beer or your coke and you might be hanging out with Terry Ryan and Ryan Stewart and Adam Lehman, talking awesome stuff about what’s coming up in CF and the Flash Platform.

Notice how I spent three paragraphs talking about stuff that wasn’t sessions?  If the value from 5 pm to 12:00 am is this good, imagine how good the stuff from 9 to 5 is! Brian said that for one of the conferences he attended, he was “required” to give a debriefing to his team when he returned to work. I can’t recommend this enough, for several reasons:

  1. It encourages you to become a more active audience member. Rather than being a passive mouth-drooler, you’re more likely to at least take notes during sessions
  2. It provides you an opportunity to revisit the material afterwards. Consider: you’re going to go to one of these conferences, sit through 15-20 sessions, and no way will you remember it all. Going back through your notes – several times – gives you more time to think and interact with the material
  3. Your team members benefit from the time you spent at the conference. Surely it’s not the same as being there. But maybe something you say during your talk will inspire them to go check something out themselves (“What’s this CouchDB thing? Sounds interesting. I’m gonna go play with it”). Who knows, maybe you’ll present on something you learned at a conference, it’ll intrigue one of your team members, and you might collaborate on a small project down the road where previously you’d have going it alone.

I strongly believe that if you attend either cfObjective or CFUnited in 2010, it will be worth your while. Each conference has a great variety of tracks: ColdFusion, Flex, Air, Project Management, etc. Each conference is run by people passionate about what they’re doing. Each conference brings in wonderful speakers. Each conference LOVEs to get new speakers (Brian Carr… I’m talking to you! Suck it up… we need solid presentations on real-world ReST with CF).

The guys also discussed presentation formats other than the traditional “eyes forward” style. Dudes… I hear you. I don’t know what CFUnited has in store for 2010, but I can say that the people behind cfObjective are taking this very, very seriously. This will be a shake-up year. This will be a year when engagement is paramount.

XJS?

The whole way through the 2nd half of the episode, the guys talked about “XJS”. I was like “WTF is XJS?” Is this some new javascript framework I didn’t know about? They didn’t mention CF8 at all, so I wasn’t sure if they were talking about ExtJS, which is used for a lot of the CF8 Ajax widgetry. I was cornfused, as we say in Pennsyltucky. A look at the website shownotes afterwards cleared it up. They were talking about ExtJS. So my question is: How DO you pronounce it? Since I was introduced to EXT in 2007, I’ve always said “E-X-T”. Can I get a ruling from the line judge on how to pronounce this?

LCDS, push/pull, comet, and CF9

There are probably a dozen people in the world who haven’t been confused by “what is LiveCycle? what is LCDS? what is BlazeDS? What can you do with Flex and XXXDS? and with ColdFusion? What DS hoogie comes bundled with CF, and what are its license restrictions?”  I know I surely was confused by this. I think I have a handle on it now though. Bottom line: if you’re running CF8, and you want Push technology, you can use the “honor license” LCDS that comes bundled with CF8, and you can also use BlazeDS (free, open source). Apparently LCDS will scale beyond more than a few hundred simultaneous users, though I do not have confirmation or links for that. Also, I believe CF9 now comes bundled with BlazeDS, so you needn’t worry about licensing for your push/pull applications. In the real world, what this should mean is that you can drop messages into your Blaze-backed gateway, have them pushed out to all your Flex clients registered on that endpoint, and you’re good to go.

If you’re creating a 20,000-user online Flex-based push/pull poker game, well, you’re probably looking at some LiveCycle Data Services licensing (note: this does NOT mean you’re spending 250k on a LiveCycle ES license. LiveCycle DataSevices != LiveCycle ES).

If I’m one of the hapless masses  who is still putting out incorrect information, please, someone, set us all straight.

WAAAAAA, BOOOOOO

Mike started off the episode with a baby crying sound effect that almost caused me to wreck my car. Egads, man, that was too real. But funny nonetheless… pulled Brian right into that one. At the end, Mike pulled Micky into another gag, too, which I thought was pretty funny.  I hope Micky lets the air out of Mike’s tires.

 

Till Next Time…

Thanks again, gents, for enjoyable company on the commute home. Till next time…

Marc, out

This Week in ColdFusion Episode #4

Wednesday, October 28, 2009

The Skinny

I commute about 45 miles each way from Pennsylvania to the Baltimore burbs. Generally, I am not a fan of this commute. However, as one who likes turning the proverbial lemons into lemonade, I’ve found that podiobooks and podcasts are a great way to spend the 90+ minutes a day in the car. I try out all different kinds of podcasts, and more often than not I delete them from my iTunes after a few episodes. With the new TWiCF podcast, I think I’ve found a keeper.

Episode 4 of This Week in ColdFusion (TWiCF) was released on October 19. I had seen mention of this new CF podcast a few weeks back, and it came across my radar today, when I was in front of the computer with easy access to iTunes. So I subscribed and listened to this latest episode on commute home from work.

In short, I like it! It’s relatively quick paced, the guys have good podcast voices, and the content is fun and informative.

While listening, however, I wished I had a good way to take notes because I wanted to speak to a few items. Rather than hijack their comments thread, I figured I’d use this blog instead. Beats hearing about unit testing, don’t it?

Without further adieu, some points I’d like to make:

ColdFusion Builder Pricing

Where did Mike get that $250 number? I have it on good sources from my inside-Adobe operatives that the price is somewhere between “We ain’t telling you” and “STFU and quit asking”. (My quotes). So, please, don’t believe any numbers anyone says about CFB until you hear it from CFB’s PM, Adam Lehman.

Debugging in Eclipse

Let there be no question: you can use Eclipse, right now, for free, and do step debugging on your CFML. Provided you’re running CF8, all you need is the ColdFusion 8.01 Eclipse extensions. Once installed, it’s a matter of a 2-minute setup and then off you go. For an outstanding introduction to using the debugger, see Charlie Arehart’s CFMeetup presentation. While I’m at it… check UGTV often. It’s a veritable gold mine of CF content.

You can set “watchpoints” as well so you don’t have to scroll through gazillions of variables. I will say, though, that I personally hate being in a debugger. I figure if I’m in a step debugger, then either my app is to complicated or my unit tests are not helping me. Just sayin’ is all.

So, key words: CF8, Eclipse, Free, Easy.

Full CFScript and CFEclipse

The guys talked a bit about the enhanced cfscript support in CF9. In particular, the fact that you can write components without any tags at all:

component {
 public Order function init(required numeric id, array items = ArrayNew(1), String shippingMode="StandardShipping" ){
  StructAppend(variables,arguments);
  return this;
 }

 public void function dumpVars(){
  writeDump(variables);
 }
}

The problem in CFEclipse, as they mentioned, is that these files look like you’re editing in notepad when you open them in CFE. Good point, guys. I’ve emailed the CFE dev group to see what the level of effort would be to get the syntax highlighting working correctly.

Snippets WishList

One of the guys – I think it was Micky – mentioned a feature he’d like to see in Snippets, based on his work with Visual Studio. Essentially, he described the ability to type a trigger, and then have the cursor “follow” you as you “tab through” so you could type in what you wanted to type. I believe the Eclipse equivalent of this is functionality called “Templates”. Before I talk about templates and CFEclipse, though, let’s talk about what you get with the Snippets that already exist.

You can already get this behavior, or at least a “pretty good” version, in snippets as they’re currently implemented. I’ve found that for the (way too few) people who use snippets, this is something they don’t know about. I’m talking about “Variables”, and they are the cats pajamas. I wrote about this a while back, so I encourage you to spend 10 minutes sharpening the saw to save yourself hours cutting.

Now, onto Templates. Fortunately, Denny Valliant recently added an initial implementation of this feature into CFEclipse. Unfortunately, the CFEclipse team didn’t publicize it at all and so no one really knows about it. So it’s time to talk about them. Please read this post to learn about this sweet new feature with lots of potential for time-saving awesomeness.

The REST Discussion

I thoroughly enjoyed this part of the show. I’d like to see Brian give a presentation – with loads of examples, in particular from the Producer point of view – on designing and publishing REST-ful services. It’s easy to consume them… but how about architecting an application that produces the services? What strategies exist? What tools/frameworks? What big problems have you encountered? What if I want to do something more than just delete User 5? Instead I want to place an order for 13 copies of 5 different books and have them all mailed to my relatives in the 4 corners of the globe (OK, extreme, but you get the picture… simple scenarios = suck… I want more depth)?  What do you say, Brian?

Oh… and how do you unit test them? This is a blog with “unit” in its name, so I’d be remiss If I didn’t mention that.

What was Missing?

What was missing from the This Week in ColdFusion Podcast, Episode #4? Why… “this week in ColdFusion”, of course. I guess when I started listening, I was expecting more of a TWiT format, or perhaps at least more newscasty-ness to it. It is entitled, after all, “This week in…”. Consequently I expected more news items. I’d really like to have at least some time during the newscast devoted to keeping we CFMLers up to date on what’s happening in the community. News items, new projects, interesting blog posts, etc. Personally, I rely heavily on the Java Posse podcast to keep me up to date on all the things in the Java and Scala world that I don’t have time to investigate myself. Adding more news-i-ness into the TWiCF podcast would be a huge win for me.

Till Next Time…

Again, it’s encouraging to see another CF Podcast, especially one that appears to have a lot of promise. I hope Mike, Micky, and Brian can keep it up. I can only imagine how grueling it must be to produce a podcast, so kudos to all members of our community who devote their time to such endeavors.

I hope each episode is engaging enough to make me want to come home and write a review / response like this one.

Marc, out.

New in CFEclipse: Templates


This is part of an ongoing series on Timesavers. The goal is simple: short, easily-digestible posts designed to help developers get faster and more productive

The recent 4th episode of This Week in ColdFusion inspired me to finally get the word out there regarding this sweet new feature of CFEclipse: Templates. This has been in CFEclipse since June, but we never advertised it or blogged about it. If you’re a java developer who uses Eclipse, you probably rely on templates so heavily that you’d think they were just baked right into the IDE. You know how when you’re editing java, and you type “for”, and it pops up a bunch of suggestions? That’s what this is.

Denny Valliant’s initial implementation only came with two default templates, though I suspect more will come down the road. It’s easy to write your own, though, and here’s how:

  1. Window – Preferences – CFEclipse – Editor – Templates
  2. In the Templates window, hit “Add New”
  3. Type in a name, a description, and your templates. You can use “Variables” by using ${} syntax, and the contents in between the brackets can be any arbitrary name. Trust me, you’ll see how this works in a minute
  4. If you want custom stuff – like where to put the cursor or some other “standard” variables, you can insert them with the dropdown list
  5. Hit OK and get out of the preferences window. The window looks like this:

cfe_templates_1

Now, in your editor, I created a new array, went down a line, and started typing the characters that begin with my new template name. For example, in my screenshot, I had a template named “looparray”, and so I started typing “loop”, and in my editor, that template appeared:

cfe_templates_2

I hit “Enter” to accept the template, and now I’m in the template, and with each tab, I’ll move on to the next part of the template.

cfe_templates_3

In this example, I have an array named “stogies”, so I tell my template that my array is named “stogies”, and it replaces my “template variable”. I want my index variable to be named “stogie”, so I type that in, and it replaces it in the cfloop tag AND in the block of code that followed.

cfe_templates_4

This can save you a lot time right now, for free. This has the potential to be a huge timesaver, especially if developers write their own templates and share them with the community via the “export” functionality.  Hopefully, as the CFEclipse developers get more time, more templates will come bundled with CFEclipse.

Viva la TimeSavers!

Stackoverflow Dev Days D.C. 2009 Review


At $99 and just down the street, it was hard not to go. The State Theatre is a very cool venue, a restored old theatre turned into Rock club. In Falls Church, VA, my wife's home town, it was just about as close as close could be for me. I sat front row like I was at concert or something, and, as I was reading Twitter on my G1, I noted a tweet by Liz Frederick from Adobe who was also at Dev Days and sitting right behind me. Cool! It was nice to meet her, we had a pleasant chat, and I learned some things about how Adobe is reaching out to developers.

Executive Summary
: It was totally worth it for this local guy. I left wanting to buy a Mac, do iPhone development, manage my projects with FogBugz 7 (and figure out WTF bayesian statistics are), get more serious with Python, finish my Google App Engine projects, and front end all my web apps with jQuery. I met some super smart, nice, interesting people, had a good time, and left inspired. One cool thing was that most of the speakers were local folks. Not sure about Bruce Eckel, but the others were all from the general D.C. area. I think this was a good thing to do for everyone involved - the conference, the attendees, the speakers, and the community.






Here're my notes:

Joel Spolsky Opening Keynote - Great warm up video - I got a real good laugh right off the bat. Great slides. Joel is a skilled and entertaining presenter. Joel's keynote was about Simplicity vs. Features, with specific emphasis on usability. Spend that extra time doing things for the user. Don't interrupt them for stuff your app should be doing. Dialogue Boxes == bad. User's don't care how much work you did to make the app; be humble enough not to show off your effort. Hide the details of how it was built. Get out of the way and strive for elegance. Elegance always looks like it should be more than it is. He showed some an example Python spell check app that I remember seeing earlier this year - it was like 150 lines of code or something - spell checking in 150 lines?! Elegance. I get it. Thx.

Oh... here's a grin. I'm sitting in the front row and Joel is making a point about usability and he points to me and says, "Pick a gender!" ... huh? Me? ... "Yeah. Just say 'male' or 'female'" Me: "uh ... male". Then he goes into a story about "Bill" who's having an affair with his secretary who wants to blackmail him for $100K while Bill's wife is walking down the hall getting ready to bust them and Bill needs to cut a $100K check with QuickBooks but doesn't have the time to answer the stupid dialogue about upgrading the software because Bill's wife is about to open the door on the two lovers. Then Joel must have realized my name just might be Bill, so he asks, "Is your name, Bill?". Well, yeah, not that you could read my scribbled name tag in a dark room from 20 feet away ... But Joel's a pro and rolls with it no problem. I wasn't bothered by any of this at all, by the way. I was stoked to be out of the office around smart people.  I guess he remembered me later, because again he gestured towards me and used "Bill" to exemplify an idea. Ok, now if I can just figure out how to get some cool product front row to buyers ...

Dan Pilone : iPhone  - Dan says Objective C is not as bad as people say. It takes Apple 2 weeks to approve/disapprove an app, and they will test it and make sure it complies with rules and regulations. Even if it's been approved for release 1 and they missed something, like a minor licensing violation, it can get kicked out again in future releases. In short, you need to plan on this 2 weeks and dot your T's. There's money to be made but you have a short window of time to grab it and leverage the Apple store presence. There's a boat load of iPhone customers, so, lots of opportunity, but also lots of competition (like 100K apps in the iPhone Store) and it's growing rapidly. You can do the $0.99 app, which needs volume sales to succeed, of course, or your can go with the $20 app, which has less sales but might be more stable.

Scott Allen ASP.NET-MVC - Scott is clearly a great coder. Anyone who can write code on stage in front of 200-300 of probably the smartest developers in D.C. and beyond gets a big hand from me. This is me holding up my lighter for an encore - more! I don't code in C#/.NET but if I get the opportunity, I can only hope to be as skilled as Scott. And I will go right towards the MVC framework he described. One thing I personally would have liked to see, which he mentioned, was unit testing. Apparently, MVC generates test stubs and allows you to test in some interesting ways, like, defining a "context". I'll need to look into it, but I sensed an interesting approach to testing.

Joel Spolsky on Fogbugz 7 - I had no idea it did all those things - much much more than issue tracking. Evidence based scheduling ... bayesian something or other statistical intelligent learning (like I have any idea) with cool graphs ... wow. They're integrating their own version of Mercurial, too, which looks like Git from the command line, and it integrates well into FogBugz with code review capabilities and task management. Having an enterprise tool like this available where I don't have to administer it, just makes sense. I'd rather focus on getting my project organized and tasks completed vs. that AND learning to administer the beast. The usability that was emphasized in FogBugz is very attractive. Joel said he wants it to be as easy to enter tasks as typing in Notepad, and he have some good demonstrations. It also supports and seems to be geared towards Scrum.

I also "got" the idea of Stack Exchange as a way to capture domain specific knowledge. Joel's view is that the corporate wiki can fail because it's document centric and experts don't really want to write documents nobody reads. However, with a question-based environment, like Stack Exchange, specific questions can be answered and aggregated into a knowledge base. I like that idea. It's more like pull rather than push.

Lunch - Cheap box lunch & fine by me. Groups for various discussions were set up during lunch - agile, free software, start ups, etc... I wanted some sunshine and the seating did not really lend itself to round table discussions, for me. So, I sat outside in the sun, walked around the block, and talked with a PHP developer from Chicago on the corner of Washington Blvd.

Bruce Eckel Python
- Bruce talked about why Python makes sense and how the language develops vs. how Java/C/C++ evolved. Basically, it can do what Java/C/C++ do but with a lot less effort and do it dynamically. Also, the fact the Python is community driven and not the product of a company, makes it an appealing choice. He's working on a version on Thinking in Python, only it's under a different title. The interesting thing about this book is that it will be community driven. It sounds like an interesting idea.

Bruce is a forward thinker, to say the least, and when pinged he talked in abstract terms about what he sees as part of the future - human computer interaction and machine symbiosis. Not in a sci-fi machines taking over the world kinda way, but we are clearly dependent on machines today, and the future will be focusing on how this interface grows. I agree. Typing a keyboard is one weird way to communicate for a human. Speaking and audio with visual cues is how we, as humans, do it.

I think that might be Mr. Hankey's brother hugging the python.


The last time I heard Bruce was in 1998 at my first ever tech conference - SD '98. I worked as a volunteer so I could get free admission and one of my assignments was to support Bruce's Thinking in Java talk. I was so excited at the time when he said, "Hey, can someone go get some audio tech help?" I was on it, man ;-)   


Jonathan Blocksom Google App Engine  - I think everyone could identify with Jonathans' down-to-earth style. As humble as he was, it was clear to me that he is very smart ... Google smart. This was the one topic that was a review for me. I had covered all this with some App Engine prototypes I did over the Summer. But Jonathan discussed the AppEngine architecture, quota/pricing model, and how to get up and running. It was very cool that Jonathan could do an entire talk about distributed applications and not once use the term "Cloud". Good job.

Richard D. Worth , JQuery - A very well organized and detailed presentation. One of the main things I took away was a simple phrase about JQuery architecture: "Find Something, and Do Stuff". I love JQuery and it's been my goto for about a year, yet I never saw it like that. But it makes perfect sense. Richard clearly is passionate about JQuery and is willing to teach. He just created a poll (Poll Daddy) for a 1 or 2 day JQuery talk in D.C. in Nove,ber or December, so, keep an eye out. It will be well worth it. I'll be there.


Wrap up: There were a lot of people who flew in from out of town and came from more than an hour away. I felt fortunate that I could wake up and drive 20-30 minutes to do a cool 1-day conference. What strikes most about this, and it's not Dev Days specific, but it's about the concept of a cheap 1-day event that's on the road, vs. a multi day conference in some place like Vegas. I kinda like the idea of 1-day, but I'm not sure if I would do a road trip for just 1 day, maybe that's what surprised me about the number of folks from out of town. I spoke with a PHP developer who flew in from Chicago. I also connected with a Python developer originally from Northern Ireland and living in Philly. Chatting with Liz Frederick from Adobe was also a breath of fresh air. So, for this developer, getting out is important. But for me,I question whether or not the multi-day big events with evening schmooz-fests are the best environment for learning. They're fun, for sure, but I sense a better way to learn new stuff ...

It's funny, but I posed this question on Stakoverflow ... whether or not micro conferences like Dev Days are a good place to do peer-2-peer learning, and the Stackoverflow zealots quickly shut my question down, closed it, as "Non Programming Related". Whatever... So, I'll ask it here: How do you learn best and grow, and what inspires you?

“Look, Ma. No SQL!” – MongoDB and ColdFusion Part 3

Sunday, October 25, 2009

Addendum: The code examples below were built on earlier versions of MongoDB and will not work with 1.6+. Also, use Marc Esher's Github fork for the most current version of CFMongoDB.


Querying MongoDB documents with JavaScript and ColdFusion

In the first part of this series I presented my fundamental attraction to MongoDB, and in the last post I talked a bit about some of MongoDB's core concepts. In this part, I'll explore some approaches, challenges and a proposal for reading data stored in MongoDB.  The source code (still in a proof of concept state) is available at Github - feel free to fork it and make comments.

MongoDB's data is persisted as binary form of JSON (BSON) and the admin engine language is JavaScript. (We'll get to ColdFusion shortly) Let's work with the Blog document structure - JSON- presented in Part 1:

{ "_id" :  ObjectId( "65c5b8782745e04afc2abf00"), 
  "AUTHOR" : "bill_1" , 
  "TAGS" : ["Comics","Games","Python","NoSQL","Ruby"] , 
  "PUB_DATE" : "Thu Oct 22 2009 07:42:19 GMT-0400 (EDT)" 
  "BODY" : "Lorem ipsum dolor sit amet ..." , 
  "TS" : 1256177399955 , 
  "TITLE" : "Blog Title No.1 ..." 
}; 


The example code generates 1000 such documents. Imagine 1000 items with the author and title values incremented for each one from 0-999.

From MongoDB's admin console, you can query this structure like this:

> db.blog.find()

... which returns all the documents in the blog collection and is semantically equivalent to select * from blog

Here's some other basic examples for querying the blog collection. This should give you a feel for how to get data back from MongoDB:

Retrieve a blog entry by ID:
SQL: select * from blog where id = '65c5b8782745e04afc2abf00'
MongoDB: db.blog.find( {_id: ObjectId( "65c5b8782745e04afc2abf00")} )


Retrieve 10 of the most recent titles:
SQL: select top 10 * from blog order by pub_date desc
MongoDB: db.blog.find().sort( {PUB_DATE:-1} ).limit(10)

Retrieve 10 entries by a given author:
SQL: select top 10 * from blog where author = 'bill_1'
MongoDB: db.blog.find( {author: 'bill_1'} ).limit(10)


To retrieve data from MongoDB you use JavaScript and object notation. An interesting point is the syntax for search criteria. The find(...) method expects a JavaScript object with the criteria; e.g., {author: 'bill_1'}. For basic equality checks you pass in the name of the field you're searching and the value. To do comparative tests you can use the syntax:  db.collection.find( { "field" : { $gt: value } } ) for greater than. There's almost a full range of operators available, including full regular expression searches - Wahoo! Take a quick peak at the docs and come back ... http://www.mongodb.org/display/DOCS/Advanced+Queries . What do you think? That's the command line shell. Very cool. But how can we do that in a ColdFusion webapp? At the root, you wrap ColdFusion around the Java client:


//setup - no necessary for every search 
//create a mongo instance 
mongo = createObject('java', 'com.mongodb.Mongo').init( server_name , server_port ); 
//get a handle to the database 
db = mongo.getDb(db_name); 
//get or create a collection (e.g., Blog)   
collection = db.getCollection(collection_name); 

//build search criteria 
criteria = createObject('java', 'com.mongodb.BasicDBObject').init('TITLE','bill_1.*'); //regular expression 
//execute search 
results = collection.find(criteria); 


It's helpful to note that the primary interface for both writing data and querying data is com.mongodb.DBObject. This extends java.util.HashMap and represents a map of name value pairs that are saved to the datastore and also used as search criteria. In the example above note that you create a BasicDBObject with a name and value. This object is to the find method and used to query the datastor.

As for the syntax, I'd prefer something different. So, I here’s my attempt at a DSL in ColdFusion which looks like this:

results = mongo.collection('blog').startsWith('TITLE','Blog Title No.60').search();

This returns all documents whose title starts with the second argument. This is semantically the same as select * from blog where title LIKE ‘Blog Title No.60%'.

For other string related criteria there’s also endsWith(field,value), exists(field,value), regex(field,expression) and others. Since this is a DSL we can chain these together to better express a search:

results = mongo.collection(‘blog’).
                exists(‘BODY, ‘MongoDB’).
                eq(‘AUTHOR’, ‘bill’).
                after(‘PUB_DATE’,’09/12/2009’).
                search();

This is asking for all the blog entries by bill that were posted after September 12, 2009, with Mongo in the body. Now, I think this is clear, but maybe you don’t. That’s cool.  Here’s the proposed syntax. Again, this is all a proof of concept, so feel free to comment.


   Proposed DSL for MongoDB searches:  
  results =  mongo.collection('blog').   //optionally set the collection to search, otherwise it will use current collection 
             startsWith('name','foo').   //string 
             endsWith('title','bar').    //string 
             exists('field','value').    //string 
             regex('field','value').     //string 
             before('field', 'value').   //date 
             after('field', 'value').    //date 
             $eq('field','value').       //numeric 
             $gt('field','value').       //numeric 
             $gte('field','value').      //numeric 
             $lte('field','value').      //numeric 
             $in('field','value').       //array 
             $nin('field','value').      //array 
             search('title,author,date', limit); 

Proposed:    search( keys=list_of_keys_to_return, limit=num, start=num, sort={field=direction} );



The string and date searches all work as expected, but there are issues with the numerics and the conversion from ColdFusion to Java representations. Numeric comparisons need to support all numeric types. The numeric comparison methods are all prefixes with $. This is done to match the MongoDB JavaScript syntax and to get around ColdFusion's reserved keywords. The array searches have been proven, but need work.

One last goodie: $where(…) : You can also do adhoc queries with JavaScript like so:

results = mongo.collection(‘blog’).$where( ‘this.TITLE == ‘Blog Title No.1 || this.AUTHOR == ‘bill’ ’).search();

Important notes:
  • When using $where, you must prefix the item you are seaching for with 'this', which corresponds to the current collection.
  • MongoDB mentions that there is a greater perfomance demand for $where
  • Searches are case sensitive. This may throw some folks, but it might help to think in terms of JavaScript, which is case sensitive.
  • Case sensitivity applies to both keys and values. And note, too, that ColdFusion creates uppercase keys when creating structs, unless you create the struct using this syntax: my_struct["my_key"] = my_value;
  • There is no built in boolean OR, yet, except in $where

Feedback is welcome, of course.


Up Next: Storing Blobs, Indexes, Admin 101


Test and be Happy!

RESOLVED: VMWare 6.5.x Stopped Working on Ubuntu 9.04

Friday, October 23, 2009

Symptoms first occured after upgrading from 8.10 to 9.04 and included:
  • Start VMWare Workstation from menu: Nothing appears to happen. No errors. 
  • Start VMWare Workstation from command line and get the following error

billy@mxunit-ubuntu:~$ sudo /usr/bin/vmware
Logging to /tmp/vmware-root/setup-25257.log
modinfo: could not find module vmmon       
modinfo: could not find module vmnet       
modinfo: could not find module vmblock     
modinfo: could not find module vmci        
modinfo: could not find module vsock       
modinfo: could not find module vmmon       
modinfo: could not find module vmnet       
modinfo: could not find module vmblock     
modinfo: could not find module vmci        
modinfo: could not find module vsock       
/usr/bin/vmware: line 31: 25257 Segmentation fault      "$BINDIR"/vmware-modconfig --appname="VMware Workstation" --icon="vmware-workstation"    


Resolution:

billy@mxunit-ubuntu:~$sudo mv /usr/lib/vmware/modules/binary /usr/lib/vmware/modules/binary.old
billy@mxunit-ubuntu:~$sudo vmware
Solution from : http://ubuntuforums.org/archive/index.php/t-1212220.html



"Look, Ma. No SQL!" - MongoDB and ColdFusion - Part 2

Tuesday, October 20, 2009

Addendum: The code examples below were built on earlier versions of MongoDB and will not work with 1.6+. Also, use Marc Esher's Github fork for the most current version of CFMongoDB.



MongoDB Core Concepts

In the last post, I started to scratch the surface of MongoDB and focused on some of its efficiencies. I mentioned that querying would be next, but I think it might be better to briefly discuss some of MongoDB's core concepts first. Note that some of these will also apply to other schemaless datastores, like CouchDB and Google's BigTable. 

The biggest challenge getting your head around a schemaless datastore is shifting in how you think about data persistence. If you're like me, you've spent most of your professional career thinking in relational terms, and maybe have even formally studied set theory, and know what tuples are and who E.F. Codd is. The concepts of tables, rows, relationships, normalization, and structured query language (SQL) might be so ingrained, that it's truly difficult to think of data storage any other way. If you want what you got, don't change what you do. But, if you sense that there might be a better way to store and retrieve some kinds of data, you're certainly not alone.

Many of the schemaless/schema-free terms and concepts are either inspired by, or are borrowed from the relational model, yet there are fundamental differences - I'll touch on a couple of the differences later. For now, let's look at some of MongoDB's concepts and how they might relate to a relational mindset.

  • Database - Like in a relational model, this is a logical and physical storage area for data, which are Collections of Documents. A MongoDB server can have multiple databases. 
  • Collection - Similar to a table, a Collection is a group of Documents. 
  • Document - Similar to a row in a table, a Document represents an instance of some object in a Collection. In CFML, a Document object is a structure (or an array). A Document can contain other documents by either directly embedding them (see below) or by reference. 
  • Indexes - Like a relational database index, this is a Document attribute registered with MongoDB for query performance and query optimization. 
  • Queries and Cursors - Lastly, like a relational database, you query Collections. The query results are common programming data structures: Iterators or Arrays, but are referred to more generally as Cursors.

Example CFML/Java Implementation:

<cfscript> 
 person = {
  name = 'bill',
  profession = 'programmer',
  age = 'older than dirt'
 }

 mongo = createObject('java', 'com.mongodb.Mongo').init();

 db = mongo.getDb('my_db'); 
 collection = db.getCollection('people');
 doc = createObject('java', 'com.mongodb.BasicDBObject').init();
 doc.putAll(person);
 collection.insert(doc); //correction! thanks to marc e.
 criteria = createObject('java', 'com.mongodb.BasicDBObject').init('NAME','bill');
 query = collection.find(criteria);
 while(query.hasNext()){
  item = query.next();
  dump( item );
 }
</cfscript>

Note that the "get" operations above will create the object if it does not exist. This a true example of schema-freeness

Embedded Document Example:
This is much simpler than the name might imply. All you need to do nest one struct within another:


person = {
 name = 'bill',
 profession = 'programmer',
 age = 'older than dirt',
 address ={
   street = 'main st.',
   city = 'anytown',
   planet = {
     name = 'Earth',
     planetary_code = '0xFFB3540A'
   }
 }
}

If you've worked with XML, the nested document structure should be familiar. With MongoDB, Document objects can nest an arbitrary number of other documents. Check out the Schema Design guidelines for some ideas.

Key Differences:
From a developer's perspective, the key differences are in how you think about the data and how the data is retrieved. Instead of thinking about your data as a set of related 2-dimensional tables with rows and cells, you are free to think about data in terms of object structure or composition (hence the name Document). Storing the data is pretty trivial in MongoDB, and quite a relief compared to writing a complex insert statement. Retrieving data, though, is where stumbling might occur, and big picture consideration should be given when embarking on a new project and considering RDBMS or schemaless. Some people have recommended that the general guideline for deciding whether or not to use a schemaless datastore is to know whether or not your data is high volume and low value. In other words, if you have very complex data, such as financial data that needs to be frequently aggregated at a very atomic level (high value), or if your data not so complex and maybe, too, you have a lot of it (low value,high volume) e.g., a Blog. In the case of the former, maybe a relational database might be better; in the latter, maybe MongoDB.

As for querying the data, this is what you might be used to:

select  p.*, a.*
 from Person p inner join Address a on p.id = a.pid
 where p.id = 'some_id'

In MongoDB, given the person Document above, this would be done more like this:

person = db.people.find( {_id: some_id} ).limit(1);


Both say the same thing, semantically: "Given a person's ID, return the person and their address"

Up next, for real, Querying MongoDB in detail.

"Look, Ma. No SQL!" - MongoDB and ColdFusion - Part 1

Monday, October 19, 2009

Addendum: The code examples below were built on earlier versions of MongoDB and will not work with 1.6+. Also, use Marc Esher's Github fork for the most current version of CFMongoDB.



From the perspective of a developer who prefers to think in terms of real-world objects, relational databases tend to slow me down. I'm not making some broad sweeping statement that "databases are bad" - they're not. They're just another layer of disparate technology that I need to contend with. What if we could just write code and store and retrieve those objects quickly? ORM and caching technologies do this well and they're aim is integration with relational databases, and Adobe has nicely integrated Hibernate into ColdFusion 9.


Alternatively, there's been some very interesting work done in the area of NoSQL, or schemaless data persistence. These efforts provide relatively small, fast, and distributed key-value data stores, and allow applications to store and retrieve objects in an efficient manner - and programmers don't have to worry about creating and managing tables or schemas. These projects include (among others) CouchDB, Tokyo Tyrant, and MongoDB. In short, these can be viewed as efficient and huMongous persistent hash tables. I chose to work with MongoDB simply because I found it the fastest to get up and running on both Unix and Windows.


Quick example usage (full CFML source available here):

<cfscript>
  blog = structNew();
  blog.title = '';
  blog.text = '';
  blog.author = '';
</cfscript>

This is a very basic data structure (a CFML structure) representing a blog entity. We know what we would need to do to write this to a database - create the table(s), specify columns and datatypes, and write SQL to insert, update, delete, and fetch. Maybe we'll use stored procedures or DAOs, too. With MongoDB here's a simple example implementation - a proof of concept API:

<cfscript>
  mongo = createObject('component','MongoDB');
  //create the blog
  blog = structNew();
  blog.title = 'Look, Ma. No SQL! MongoDB and ColdFusion';
  blog.text = 'Rapid development with MongoDB ...';
  blog.author = 'bill shelton';
  mongo.put(blog);
 </cfscript>  

That's it - the entire structure is now persisted. For the jQuery and JavaScript folks, you're really going to like this. The stored data format (BSON - binary JSON) in string representation is plain old JSON and the MongoDB admin console is a JavaScript shell:

>db.default_collection.findOne()
{"_id": ObjectId( "58c5b878f16bdb4a94f4d300") , "AUTHOR" : "bill shelton" ,
"TEXT" :"Rapid development with MongoDB ..." , 
"TITLE" : "Look, Ma. No SQL! MongoDB and ColdFusion"}

How cool is that?

Adding additional fields or data. There's no need to create new tables or columns -  MongoDB takes care of this for you. If they don't exist they'll be created. Just add the elements you need to your code. Let's add some tags and comments to our blog entity. Note that these are not simple string properties - this makes the example a bit more interesting and shows how powerful MongoDB can be.

<cfscript>
  mongo = createObject('component','MongoDB');
  //create the blog
  blog = structNew();
  blog.title = 'Look, Ma. No SQL! MongoDB and ColdFusion';
  blog.text = 'Rapid development with MongoDB';
  blog.author = 'bill shelton';
  blog.tags = ['cool stuff', 'data persistence', 'hadoop'];
  blog.comments = [
    comment = {
      author = 'anonymous',
      comment_text = 'you suck at unit testing'
    }
  ];
  mongo.put(blog);
 </cfscript>  

Here we added an array of tags and array of structs ... comments. Again, our entity is saved to the data store. Let's read this back out and print it:

<cfscript>

 mongo = createObject('component','MongoDB');

 blog = mongo.findOne();// get the most recent entry
 writeoutout(blog.title);
 writeoutout(blog.author);
 writeoutout(blog.text);
 writeoutput(blog.comments[1].author);
 writeoutput(blog.comments[1].comments_text);

</cfscript>

Here's a basic example of how to update and delete data.

<cfscript>
  blog = mongo.findOne();// get's the most recent

  blog.title = 'My New Blog Title';
  mongo.update(blog);
  //if we want to delete it: 
  mongo.delete(blog);
</cfscript>

For someone who has embedded many lines of  SQL in Java and CFML and knows the pain with keeping all the parts in synch, this is very appealing.  Note the dot notation for traversing, too - It's familiar and fast. Also, my goal with all this as not to do a straight port of the Java driver. Rather, I tried for something extremely simple and fast for writing and fetching CFML structures to and from MongoDB datastores.

Though one of the big motivators for MongoDB seems to be geared towards creating very large distributed data stores, from a simple rapid development perspective, it's downright sexy.

Getting Started with MongoDB and ColdFusion
1. Download and start MongoDB - this is pretty much just downloading, unzipping, and running ./mongod. But, you might have to create a directory for the data. It's well documented and I was able to get it up and running in a couple of minutes on both Windows and Ubunutu.
2. Download the Java driver and put it in your ColdFusion server's classpath or use Mark Mandel's JavaLoader.
3. Check out the example code and the MongoDB wrapper for ColdFusion.

Note that there's a lot of other information about all this from both the developer's perspective and the admin perspective. I recommend reading up on use cases and schema design, as well as the other docs.

Next up: MongoDB query syntax .
 
Test and be Happy!

Concision in ColdFusion and Scala

Friday, October 16, 2009

I do not intend this post as some kind of Scala vs. ColdFusion shooting match. Rather, it’s about enhanced script support in CF9, but it was inspired by a nice post I read on the Scala language regarding a feature I’ve long-loved about ColdFusion: named and default arguments

I've been programming with ColdFusion and Java for a long time. Recently, I've started dabbling with Scala, and it's a language I'm starting to really like. One of the reasons I’ve taken to Scala is that in a number of ways it solves certain verbosity problems that ColdFusion solves as well. I decided to write this blog post after reading this article by Heiko Seeberger on named and default arguments in Scala 2.8. As I was reading, I thought, “Cool! Scala does something that CF has done for a long time”… something which has always irritated me about Java. And with ColdFusion 9, things get even better.

When we talk about verbosity, we’re talking about at least two things:

  1. language constructs
  2. language syntax

Sure there are others, but these two are what I want to discuss briefly right now. Let’s talk about constructs first.

Concision via Language Constructs

Argument Defaults

A traditional, pre-CF9 ColdFusion component (CFC) might look something like this:

<cfcomponent output="false">

<cffunction name="init" output="false" access="public" returntype="Order">
<cfargument name="id" type="numeric" required="true"/>
<cfargument name="items" type="array" required="false" default="#ArrayNew(1)#"/>
<cfargument name="ShippingMode" type="string" required="false" default="StandardShipping"/>
<cfset structAppend(variables,arguments)>
<cfreturn this>
</cffunction>

</cfcomponent>


Here we see one required argument and two optional arguments. The optional arguments have default values. This means you could create an Order in at least these ways:



<cfset items = ["boat","car","flower"]>
<cfset order = createObject("component","Order").init(1)>
<cfset order2 = createObject("component","Order").init(1,items)>
<cfset order3 = createObject("component","Order").init(1,items,"FancyShipping")>


Now, if you were to do this in Java, you’d have even more lines of code to write in your Order class. Java does not have defaults for arguments, so you’d need to write 3 different constructors. Heiko illustrates this using Scala 2.7 syntax in his article, but it’d look roughly the same in java, only worse. Your constructor signatures would roughly be:



public class Order {
public Order(int id){}
public Order(int id,String[] items){}
public Order(int id,String[] items, String shippingMode){}
}


Named Arguments



In the snippet above, I created the object by passing in 3 unnamed params. ColdFusion knows which arg is which based on position. But what if I wanted to create an order instance with the default argument for items, but “FancyShipping” for the ShippingMode? I.e. I want to pass params for arguments at positions 1 and 3, but not 2. Fortunately, ColdFusion makes this easy with named arguments:



<cfset order4 = createObject("component","Order").init(id=1,shippingMode="FancyShipping")>


In addition to making the code more readable, it affords you the ability to pick and choose which params you wish to pass, leaving the rest to the defaults.



Obviously, you can’t do this in Java, so you’d need to add a fourth constructor:



public class Order {
....
public Order(int id, String shippingMode){}
}


As Heiko points out, Scala 2.8 solves this Verbosity-by-lack-of-language-construct problem by providing named arguments and argument defaults. Ripped right from his post:



case class Order28(id: Long, items: List[Item] = Nil, mode: ShippingMode = StandardShipping)


And to create an order, you could do:



val order = Order28(1)

val order2 = Order28(1,item list goes here, OvernightShipping)

val order3 = Order28(1,mode=OvernightShipping)


Nice! As Sean Corfield pointed out, one difference between ColdFusion and Scala with respect to named arguments is that Scala allows you to mix named and unnamed arguments, as in the last example, whereas ColdFusion expects you to stick with one or the other.



Concision via Syntax



Looking at the Scala version (Please read Heiko’s full post to see it in all its beauty), we see how these two language constructs enhance concision when compared with Java. However, when you look at the ColdFusion code compared with the Scala code, you see that the Scala version is tighter and, in my opinion, simply nicer to look at. The primary reason is the difference in syntax between the two languages.



The ColdFusion version spends a lot of characters in tag names and attribute names. For instance:



<cfargument name="ShippingMode" type="string" required="false" default="StandardShipping"/>


Egads.



Prior to ColdFusion 9, you could’ve written this with a function in cfscript, but it didn’t afford you all the language constructs of the tag-based version. However, with ColdFusion 9, we get “full cfscript support”, which means tighter syntax. Check out the same CFC, but in full script:



component {
public Order function init(required numeric id, array items = ArrayNew(1), String shippingMode="StandardShipping" ){
StructAppend(variables,arguments);
return this;
}
}


Admittedly not as concise as Scala, but I don’t care… it’s a long way from the tag based version and is quite good enough for me. In addition, creating instances of this CFC gets nicer, too:



<cfscript>
order = new Order(27);

order2 = new Order(28, ["boat","car","flower"] ,"FancyShipping");

order3 = new Order(id=29, shippingMode="Good Enough Shipping");
</cfscript>


Not bad, eh? A big hat tip to the ColdFusion team for giving us named arguments and argument defaults since waaaaay back. And another hat tip to them and the CFML Advisory committee for steering the language, syntactically, in a direction that aids in readability and makes coding more pleasurable.

Why is CFExecute Hanging?

Thursday, October 15, 2009

The Problem

Wednesday: CFExecute is timing out, and I don’t know why

I get an IM from a coworker, Alan. One of the applications I wrote a while back isn’t working on our Dev server. Specifically, this application uses CFExecute to invoke Beyond Compare from the command line on Windows. The app would spin, and eventually, Alan would get a “Timeout period expired without completion of C:\Program Files\Beyond Compare 2\bc2.exe”. Windows Task Manager was littered with BC2.exe processes

I fixed this with little troubles and sent out an email to the dev group with instructions for fixing it (explanation below).

Thursday: CFExecute is timing out – again, but on a different machine – and I don’t know why

The next day, after the app was running successfully again on the Dev server, Alan emails me. The app won’t run on his machine now, and it’s giving the same error message. Me, well… I get pissy. “C’mon, dude, I just sent an email telling how to fix it”. It was early, I was cranky, and the “angels of my better nature” weren’t in the mood. In short, I was an a-hole, assuming Alan was being lazy.

He wasn’t. He tried everything I had suggested as a fix. It still wasn’t working.

Sitting at his desk trying to figure it out, I was stumped. I rarely get this stumped. I *never* get stumped enough to email a company for help (in this case, Scooter, the makers of Beyond Compare). And there I was, writing an email to Scooter.

But you know how it is. You’re a geek. You like puzzles. You never give up. So… one more trip over to Alan’s desk. “Let’s try a few more things…”

This is a post about debugging. And it’s a post about assumptions.

What we did to debug

Wednesday night: The Easy Fix

To debug the hanging bc2.exe on the Dev server took about 10 minutes. After a few piddles around (selecting different options in the app, restarting CF – ruling out the really easy stuff) I simply ran the same thing that CFExecute was running at the command line. This is CFExecute Debugging 101. In my experience, this almost always yields the answer.  When I wrote the app, I had it write its entire cfexecute command to a log file, so it was a cut-n-paste job to run it from the command line.

When I did so, I saw the problem right away: the license had expired for Beyond Compare on the Dev server. Instead of running the command I had given it, Beyond Compare was throwing up a popup asking for a new serial number. All those BC2.exe processes in Task Manager were those popups, swallowed by CF, but never completing.  Fixing the license problem fixed the CFExecute problem.

Thursday: The not-so-easy Fix

Attempt #1

After confirming that the problem on Alan’s machine was not license-related, we tried a whole bunch of things, mostly centered around answering the question: “Why does it work on one machine and not the other? What are the differences between these two machines?”

We suspected a permissions issue from the start. The BC command was looking at different network locations, so the first thing we did was change the script to look only at directories on Alan’s machine… remove anything network related. No Dice.

We always run CF under a special account (we’ll call it “cfservice”), and we thought “maybe the account got borked”. We weren’t taking that for granted as it’s happened before. So we tried running CF under the local system account. Nada.

We added additional logging into the BC command to get more info… it wouldn’t write the logs. But it would write the logs if we ran the command directly on CMD. WTF?

We went into Beyond Compare’s options and fiddled with a few important-sounding checkboxes. Still nothing.

So we’re at a point where, when running the command from CMD, it worked fine. But when run from within CFExecute, it was hanging.

What’s the difference between ColdFusion running it, and us running it from CMD? We were stumped.

At this point, we were certain that for some reason, Beyond Compare was throwing up a popup when run from within CF, but we couldn’t “see” that popup because, obviously, CF was swallowing it. We tried hooking a debugger up to the BC2.exe process, but we got nowhere fast.

Aside: while all this is happening, another coworker is talking about some other thing, “this app (unrelated) worked a few days ago. Now it’s busted. What’s going on?”. This was an app I had fiddled with “a few days ago”. I knew exactly what the problem was. But I didn’t step in yet. When you’re debugging, you stay focused. You ignore other problems even if you know the answer because when you’re debugging, you stay in the zone. There are times when you break from the problem and let your subconscious chew. For me, those times are toward the end of the day, when I’m spent, and I can let my mind wrestle with it on my commute home. But this was 10:00 AM. This, for me, is prime focus time, and a big no-no for me is pulling off of a problem during my peak mental energy.  When you commit, you commit, even if it means leaving someone else hanging for a little bit. Good teams understand this, and God willing you’re patient with each other and recognize when your coworkers need to focus. It’s a high form of respect among geeks. If you see one of your colleagues in the zone, and they sort of mindlessly blow you off, it’s not necessarily an affront. Maybe it’s that they’re running at full pace and stopping would put them at the back of the proverbial pack. Be patient. Do not assume the worst.

Attempt #2

I was about to give up, so I went back to my desk to do a little research. I drafted an email to Scooter. I held off, then went back to Alan’s desk.

Alan was still pretty sure it was permissions related, and at this point I was up for anything. But we had tried that route already. We ran it under cfservice. We ran it under local system. What else could it be? What’s the *delta* among running it from cfservice, running it from local system, and running it from CMD logged into the machine normally (under Alan’s account)?

OK, let’s try this: let’s take a cue from Michael Jordan. “You gotta be the ball”. So, let’s “Be ColdFusion”.

We logged out, logged back into his machine under our cfservice account (which, remember, CF runs under on our machines), and attempted to run the app from the command line. Bam. Popup… The damn license popup. How? What? Huh?

It turns out that Beyond Compare stores a license for each user on the machine, which we confirmed by poking through the registry. It explained why it would work when we ran it from CMD when logged in normally, and it explained why cfservice was hanging BC2.exe. We logged back in under Alan’s account, then did a shift-run as – cfservice on Beyond Compare, and sure enough we got the popup.

To fix, we simply entered the license for the application while running it from the same account as ColdFusion, which cured our ills.

In retrospect, it makes perfect sense. While you’re debugging, it never makes sense. This is learning, and it’s why I go to work every day.

Takeaways

  • If your application appears to hang, try running it on the command line first, outside of CF, and see if you get any popups
  • If your application appears to hang, and it works from the command line when you run it, try running it from the same user account as CF is running  under
  • Your coworkers can sometimes *appear* to be helpless, or at least appear to not be taking the initiative you wish they would take. Before you assume that’s the case, try instead to assume the best. Assume they tried; assume they tried hard. When you think you’re not getting the best out of people, ask yourself: Are you giving them your best, too? Not your best tech. Your best you.

--marc

The Nobel Prize for Software

Wednesday, October 7, 2009

The 2009 Nobel Prize for Software

From Galileo to the iPhone - Edward Tufte Connects the Dots

Tuesday, October 6, 2009

"Hey, have you heard of Edward Tufte?", Frank and I are standing in some random Northern Virginia parking lot and I'm swatting mosquitoes in the September humidity.

"Who?"



Frank enlightens. He describes a scientist and artist who wants to explain concepts of displaying information - graphs and other visual assets. Publishers can't get it right, or are not willing to publish to his standards, so, he prints his own books - eyebrows.raise(). I like that - not defiance, but someone believing in something deep enough they go beyond the status quot and make it happen correctly. Edward Tufte, aka ET, also builds large visual structures (statues) in Connecticut (in the town where Frank's wife grew up) - these not only present beauty, but are 3 dimensional descriptions of ET's analytical and presentation concepts.

 


Of the myriad of things Frank and I talk about - rapid bi-directional streams of consciousness - this one sticks. So, I find ET's web site and I see he's to be in Arlington, VA, in 2 weeks. I'm still a bit skeptical, but as I start reading more and more, I become quite intrigued. ET is thinking about information display at it's core, not just from the very narrow perspective of the web or the evil blood sucking spawn of Satan, known as PowerPoint, but from the perspective of history, of science, art, and cognition. When I noticed that when you attend his lectures ($380), you receive all 4 of his books, I was immediately sold. At a minimum I would I have 4 new books - wahoo! I registered. (The quality of the books is, by the way, simply amazing.) I have a tendency to coddle my books, too, especially ones of high quality like these. During the lecture, again, I was enlightened. ET was explaining a concept of presentation asks the audience to fold one of the pages in half, saying, "It's a book; it's meant to be 'used'". Pragmatic.

 



Summary
: Drop what you are doing and go. At the time of this writing, he's on tour now at various points in the U.S. ET is, of course, a seasoned master at presentation and for someone who periodically presents at conferences and other venues, it was great lesson to see a pro at work. Among many other 2D and 3D visuals, he showed both a 1st edition Galileo book with original drawings of Galileo's sun spot discoveries, and ET also demonstrated how and why the iPhone display works. Now, that's a perspective that transcends! There's plenty of other gems and gold in ET's writings and presentation, so, go for it.


Marc Esher asked me for the top 3 concepts I took away. I'll give you 4 plus a few more, trying to keep it as brief as possible. These concepts transcend the medium; that is, they apply to any information display, whether in print, on screen, or in person.

Moral and Ethical Responsibility: Presenting and consuming visual information is a moral activity. If you are presenting or receiving information in any way, be it on the web, in a meeting, or at conference, it is your moral responsibility to ensure the information is true and accurate. I get the fact that when you are telling people about something, that it better be right to the best of your ability, but as a consumer? As a consumer of information, it is up to us make sure that what is being presented is true and correct. We should hold the presenter intellectually & ethically responsible for what they show and tell. "It's better to be approximately correct, than to be precisely wrong."


Respect Your Audience: It seems that there is a dangerous assumption that as presenters we should know our audience. ET explains that claiming to know your audience puts the presenter in a precarious situation, creating a tendency to play down to an audience. So, it's better to respect your audience. Chances are your audience is as smart, if not smarter than you. Treat them as equals, as colleagues.


Use Successful Models: Instead of guessing at what works for your presentation, or re-inventing something, use something proven. For example, consider modeling your display after the New York Times Sports statistics or Google News. These are successful presentations that are used by millions of people each day.


Presentation Design: The problem with design is (1) designers are taught to introduce creative assets (pictures, images, things that dazzle the eye), but when displaying information, especially complex data, the idea is the opposite. We need to remove everything not related to or supporting the subject or content. (2) Use a super graphic to describe your content. This is a single large graphic that has all the information about your subject contained in a cohesive visual space. ET exemplified Charles Joseph Minard's pictorial representation of Napoleon's march to Moscow:




The above is a bit more philosophical. The following, though still abstract, tries to convey the principals of how to make it happen and what should be included in the display.

The Fundamental Principals of Analytical Design [Beautiful Evidence, pp120-135 Edward Tufte, 2006]
  1. Show comparisons, contrasts, differences
  2. Show causality, mechanism, explanation, systematic structure
  3. Show multivariate data; that is, more than one variable
  4. Completely integrate words, numbers, and diagrams
  5. Documentation: Thoroughly describe the evidence. Provide a detailed title, indicate the authors and sponsors, document the data sources, show complete measurement scales, and point out relevant issues.
  6. Content is everything: Analytical presentations ultimately stand or fall depending on the quality, relevance, and integrity of their content.

Displaying information is hard. As software professionals, I now see it's our responsibility (not just our job) to consider how the software we write is displaying the content our customers are trying to convey. Many of us are not designers, per se, and get wrapped up in performing complex computational tasks. We may end up feeling smart and get or create the occasional pat on the back for being so, but we might be just a bit more successful if we applied some of that analytical brilliance to displaying the information we compute. So many times I've heard or seen programmers say at conferences, "I'm not a designer, so, excuse my slides ...", implying incompetence at visual display. Well, this is me being your responsible and ethical consumer : You're more than a just a programmer and that statement comes across like you don't care or your audience is not worth the effort of trying your best. It's your responsibility to communicate your ideas to the best of your abilities and I know you can do it well. Ask someone you respect for advice. Ask colleagues or peers for feedback on your presentation. Then Rock the World with your brilliance.