Ajax Testing with MXUnit, Webdriver, and Firefox

Thursday, October 2, 2008

"Release early and often", is an Agile mantra. I usually wait until something is more stable, but I'm particularly excited about this effort and hope to get some feedback quickly - even better, get some help!

One of the goals we have at MXUnit is to be able to perform not just unit testing, but other types of testing as well - all inside of MXUnit and using CFML. One big gap and need I see is the ability to test ColdFusion applications via the the user interface and to test the UI itself. Testing has some catching up to do with the rapid adoption of jQuery, ExtJs, and other Javascript Ajax frameworks. One effort in particular that caught my attention is Webdriver (http://code.google.com/p/webdriver/). Simon Stewart's smart ideas behind it are to create a simple interface and to be able to use existing browsers. Webdriver has the ability to invoke Firefox and Internet Explorer natively as well as leverage HtmlUnit. HtmlUnit's main advantage is that it's fast. However, it does not handle JavaScript in the same way most user's browser do - it can, at best , emulate the browser. So, how can we then accurately automate testing in common browsers?

This effort, CFWebdriver Experimental (for the moment), allows you to write tests that call URLs, find page elements, write data to form fields, click buttons, and verify output. Again, the Webdriver interface is intended to be simple, which is very appealing.

Here's some example code:

<cfcomponent output="false" extends="mxunit.framework.TestCase"> <cfscript> function doGoogleSearch(){ driver = createObject("component","cfwebdriver.WebDriver").newInstance("firefox"); driver.get("http://google.com/"); q = driver.findElement("q"); q.sendKeys("mxunit"); q.submit(); link = driver.findElementByLinkText("MXUnit Blog"); link.click(); assertEquals("MXUnit Blog", driver.getTitle()); } </cfscript> </cfcomponent>

What this does is to start Firefox, invoke Google's home page, find the input named "q", enter mxunit into the input, submit the form, find the link text MXUnit Blog, click that link, then verify the expected text in the title tag. It works nicely on Windows and Ubuntu! Here's a screen shot of MXUnit and Webdriver testing Google's Ajax Search API example (http://www.google.com/uds/samples/apidocs/helloworld.html)


This is included in the experimental download .... repeat ... experimental. ;-) Visit http://github.com/virtix/cfwebdriver/tree/master and click the Download button. I'll get this in the mxunit subversion repository this weekend - having problems accessing it M-F/9-5.

The installation instructions are in the README in the download, but it's basically copying the Webdriver jars into the ColdFusion classpath, and making sure they are loaded first. I've tested this on CF8 multi-instance installations on Windows and Ubuntu, and Railo 3 on Windows.

This is the preferred method of installation as of today:

1. Copy /webdriverlib to WEB-INF/lib/webdriverlib 2. Edit WEB-INF/web.xml adding the path to the webdriver jars above to the BEGINING of the cf.class.path element: <context-param id="coldfusion_context_88"> <param-name>cf.class.path</param-name> <param-value> ./WEB-INF/lib/webdriverlib,./WEB-INF/cfusion/lib/updates,./WEB-INF/cfusion/lib,./WEB-INF/cfusion/gateway/lib,./WEB- INF/flex/jars,./WEB-INF/cfform/jars</param-value> </context-param>

3. Restart the cf instance

4. Try the example1.cfc test and the googleAjaxTest.cfc

Notes: The README should have enough info to get you up and running. The Internet Explorer implementation is very buggy when running inside of CF. It works sometimes and other times not. There's 2 CFCs (wrappers). These contains only some of the full functionality of Webdriver but should be enough to give you a good taste. If the CFC wrappers don't do what you need, you can always go down the java route and instantiate the java representations and work with those instead. Or better yet, contribute ;-)

Happy Ajax testing!



Marc Esher said...

this is really sweet stuff bill. i'm looking very forward to where you take it!

Rich Rein said...

Your example uses the firefox instance, but only the IE class files seem to be included in the download...?

Marc Esher said...

rich, the firefox stuff is all in the jars.

also, regarding firefox, WebDriver is looking in a hard-coded place for your firefox install. When I tried it out i had to change the path in that .cfc file because my firefox lives somewhere else.

Rich Rein said...

Marc - Yeah, that's what I get for looking at the file structure before actually trying to run it (I saw the folder structure for the IE driver, but not for FF).

After actually trying it, I get "An exception occurred while instantiating a Java object. The class must not be an interface or an abstract class. If the class has a constructor that accepts an argument, you must call the constructor explicitly using the init(args) method. Error : Unable to locate directory which should contain the information about Firefox profiles. Tried looking in: D:\JRun4\bin\null\Mozilla\Firefox". Where is the firefox path set up?

bill said...

rich, that's one issue that has been resolved with Webdriver, but is not yet in the main distribution. I sent a new build to Marc, who had the same issue (Marc, did it work?). let me know and I'll send you the same build and/or update the download. A workaround might be to make sure that the coldfusion instance that is loading those classes is started using your account, and not the local system account. You can do this by starting cf from the command line; ../jrun4/bin/jrun start yourinstance

thanks for hanging with this! keep me posted.


Marc Esher said...

bill, i did try the new jars, and i'm running CF under my account. i get this though:

Unable to connect to Firefox. Is the WebDriver extension installed, and is there a profile called WebDriver? To set up a profile for WebDriver, simply start firefox from the command line with the "ProfileManager" switch This will look like: firefox -ProfileManager. Alternatively, use the FirefoxLauncher support class from this project

This is definitely different than the error I was getting last night using the old jars.

Rich Rein said...


If you can update the download, that would be much appreciated. Otherwise, let me know a valid address and I can drop you an email.

bill said...

@rich, There's new jars in the download. give that a shot and keep me posted. Thanks for experimenting!

Also, as Marc mentioned, you may have to create a Webdriver firefox profile. I thought this was addressed, but I'm probably missing something. Anyway, to create a Webdriver profile open a command prompt and cd to your firefox install. execute "firefox -profilemanager" ... then create a profile called "Webdriver". This should work ...

keep me posted,

Rich Rein said...

For my purposes (addressing a defect that wasn't easy to "unit test" in my traditional understanding of the phrase, but that I wanted a test for so that I could be more confident in my fix), I switched to the htmlunit instance in my test code (and got my tests to work!!). I just downloaded the updated files, and will play around with that as time allows. Thanks!!!

Marc Esher said...

Man! I tried using firefox -ProfileManager, as the webdriver error message indicates, to no avail. but -profilemanager worked. thanks bill!

Marc Esher said...

sweet! i used firefox -profilemanager, created the WebDriver profile, and with the new jars you sent me the tests worked fine.

It's peculiar to me that webdriver forces you to create a profile... why not just create a default one silently? Any idea if they're intending to add that? the code says "you could use the FireFox launcher support class". But why even go through that rigamarole?

I wish webdriver had a driver.setIDoNotCareWhatYouDoAsLongAsYouWork() method.

bill said...


Both those firefox profile issues have apparently been resolved - having to create one and having trouble finding it. So, I'm not sure why, in your config, you had to jump through those hoops. Maybe you can send me (off line) your machine info and ff versions?

Funny, I had to create a profile on win in the older version, but on Ububtu no profile is apparently created at all! Might be the way Firefox is implemented in each OS.

FYI, there's a Safari driver and a Chrome driver is on the way. Also, for what it's worth, most of Webdriver's contributors are google folks ...


Simon Stewart said...

Bill, this is awesome work! :)

@marc: There's a system property you can set ("webdriver.firefox.bin") that tells the firefox driver where to look for the Firefox binary (note on Windows, you'll need the ".exe" on the end of the binary name)

You shouldn't need to set up the WebDriver profile yourself, WebDriver should handle that for you. Please feel free to log a bug on the site so we can help you more!

bill said...

Thanks, Simon! It's coming along nicely.

I'm curious, too, why the firefox profile was an issue with Marc. I'll get some more details and work that out. It could be something I'm doing, too, with the wrapper.


Mike said...

I finally got around to giving this a whirl and I'm having a little problem. When I try to run the test files I get this message:

Class not found: org.openqa.selenium.By

This seems to be a little different than the other problems in this thread. Could this be just a bad jar?

bill said...

Hey Mike,

It could be the way in which the webdriver classes are loaded. Can you send me a snapshot of the classes that are loaded in CF and what your environment setup is?


Mike said...

Okay, I got it working. Seems like I was making it a lot harder than it really is. I was mucking around with my ColdFusion install directly and screwing things up.

If you run the developer edition on top of IIS, the setup is really easy.

Download the cfwebdriver project from git and stuff it in your webroot with your other projects.

Then in the ColdFusion administrator under Java and JVM settings, put the path to the cfwebdriverlib directory in the box that says ColdFusion Class Path. In my case, that's c:/InetPub/wwwroot/cfwebdriver/webdriverlib.

billy said...

Mike, cool! Thanks for the install tip! I'll make to sure to note that and put it the verbose documentation ;-)

Let me know how it goes ...


Matthew said...

thank u r information

it very useful

u r blog Is very nice