On-line Survey Software Review

I recently installed and reviewed a number of third-party on-line survey apps for one of my clients. The products I reviewed covered a lot of ground, from simple-and-cheap to complex-and-expensive. I wish I had time to share everything I learned, but time's short, so I'll just pass on some info about the two strongest candidates from both ends of the spectrum.

WebSurveyor

This was the granddaddy of the survey products reviewed, in terms of cost, features, and complexity. The folks at WebSurveyor offer both hosted and licensed arrangements (I looked at the licensed version). Given the system's cost, this was expected to be the stronger product, and it was in many areas. However, in light of its cost, it also proved disappointing in some areas.

One significant difference between WebSurveyor and every other product I looked at is that WebSurveyor includes separate server- and client-side pieces. Every other product I reviewed was entirely web-based.

WebSurveyor’s server-side piece consists of two Microsoft SQL Server or MSDE databases, one of which stores user and basic survey information (e.g., name, user/owner, published file name, etc.) and another that stores survey results, and an IIS ISAPI extension that provides web-based administration services, including data retrieval and update services for the client-side piece.

The client-side piece is called WebSurveyor Desktop. It's a Windows app that provides users a fairly simple, but very complete, interface for creating, configuring, and publishing surveys on a WebSurveyor server on which they’ve been given an account. It installed easily and was easy to use, even without reading the documentation. Though I wasn’t able to confirm this, it appears you can install the client as many times as you like, on as many machines as you like, and point it at whatever WebSurveyor servers you want (in other words, licensing costs and restrictions only seem to apply to servers, not clients).

The server-side installation was suboptimal. The installation documentation, what there was of it, was weak and sometimes misleading. A problem I encountered during installation, when the client's tech targeted the wrong SQL Server instance, proved nearly impossible to resolve. After much spelunking through the logs, various config files, and the Registry, I was eventually able to get it working, though only after the WebSurveyor ISAPI extension apparently locked up IIS in a way that required a server reboot. In the end, I never could get the full compliment of server-side web-based administration features working. I could add users and check server statistics, but when trying to find a user, list expired users, or list surveys, I always got back an error message like this:

  Error Message: Invalid object name 'person' 
  Status Code: 10 
  Module: TActionFindUser.Process 

One important thing to note about WebSurveyor is that it stores all survey details (e.g., questions, answers, layout, etc.) locally on the client, one WSX file per survey. This file can be shared with others; however, once a survey is published, it is only of use when accessing the server with the original user's credentials. The significant point here is that there is no server-side storage of survey details. As a result, survey results, which are stored in the database in a non-normalized XML format, in a table auto-generated by the system when a survey is published, cannot be interpreted without the original survey’s WSX definition.

WSX files contain XML, and describe all aspects of a survey. They use CDATA sections to store various HTML values, including those for adopted custom templates (the custom templates themselves are stored as separate files on the local machine, allowing them to be used for multiple surveys). Templates represent entire survey pages and are comprised of standard HTML interspersed with custom HTML comment tags (e.g., , , etc.) which are replaced during the publishing process with HTML representing a survey's actual definition. If custom formatting of questions, answers, buttons, etc. is needed, it can be specified during their setup either explicitly or using templates.

All survey results analysis is done on the client (obviously, as the WSX file is required to make any sense of it). This means that the results, which are stored in the server-side database as XML, are actually brought back to the client machine for reporting. When the results arrive on the client, the app tears apart the results XML and writes the data to a temporary DAT file in a proprietary format, along with a separate IDX file. All subsequent client-side reporting and exporting relies on these temporary files.

Report formats can be customized using templates. The default report format is very heavily graphic-oriented (pie charts, mostly), though raw results are also available for review. In fact, the raw results can be edited and sent back to the server. During the setup of survey answers, a number of options are available for coding answers for identification and weighting purposes to improve reporting. Ultimately, the results can be exported to CSV or TXT files.

SelectSurveyASP

The folks at ClassApps have developed an amazingly complete survey product entirely in ASP with an enterprise license allowing for "unlimited installations within a single company or organization" for only $249.99 (I reviewed the Advanced edition; the Classic edition is even less expensive). Wow.

As best I can tell, SelectSurveyASP Advanced includes most (perhaps all) of the authoring and administration features found in WebSurveyor. Some features important to my review were its support for copying existing surveys, defining questions/answers based on custom SQL, support for question aliases (for improved reporting), question subtext, conditional branching, multi-page surveys (with piping of answers), and easy question reordering. My guess is that if you need it, it's probably here. But even if it's not, you get the source, and so can add whatever you want.

SelectSurveyASP is entirely web-based, so there's no separate client-side installation required. All survey authoring, user management, reporting, etc. is done through the browser. I found the interface to be very clean and easy to use, again without cracking the on-line documentation.

Unlike with WebSurveyor, SelectSurveyASP stores everything in a database, which can be SQL Server or Access/JET. Survey questions, answers, and responses are all normalized, making it easy to do custom data selection and reporting directly against the database. It also includes an Excel/CSV export feature, with some limited options for controlling the level of detail.

The server-side installation in IIS was simple (it is, after all, just an ASP web app). ClassApps provides a couple of SQL scripts to populate a SQL Server database (existing or new, your choice) with the system's necessary tables and metadata. Or, if you want to target an Access/JET database, they provide a complete pre-populated MDB file. The installation documentation was a simple step-by-step affair that was easy to understand.

The only problem I encountered during installation was with the SQL Server database setup scripts. The table creation script used uppercase for the table and column names, but the metadata population script used lowercase. For most folks, this probably isn't a problem. However, for me it was, because all of my client's SQL Server instances use the SQL_Latin1_General_CP_CS_AS collation, which is case-sensitive. This caused the metadata population script to fail, because the lowercase table/column names weren't found. The folks at ClassApps quickly responded to my support request with an apology and a confirmation that the ASP code's SQL used lowercase consistently. With this information, I was able to work around the problem by simply converting the table setup script to lowercase and running it (and the metadata population script) again.

For experienced ASP developers, the system's VBScript will be fairly easy to figure out. This is a good thing, because I believe they will likely be in it at some point to work around the system's greatest limitation: its survey presentation and layout options are extremely weak. While the system provides for adding and using as many survey styling templates as needed, there are very few options available via the web interface for adjusting their format, styles, element placement, orientation, etc. Instead, most survey presentation and styling is derived at runtime from code in TakeSurvey.asp. To get my simple survey looking the way I wanted for my review, I had to make a number of changes to TakeSurvey.asp:

The template definition allowed me to specify a graphic file, but it assumed its width and wanted to place the survey's title to the left or right. I wanted a cleaner, more conventional over/under graphic/title layout.

Even when page numbering wasn't included, the code included an empty <TR> in the header table, resulting in a big gap between the title and first question.

The code always forced the entire survey to the center with an explicit <CENTER> element.

The maximum width for surveys was hardcoded to 720 pixels. I didn't change this, but it's an example of ClassApp's use of unfortunate hardcoded presentation constraints.

Question subtext was rendered too close to the bottom of its associated question. To remedy this, I had to modify the line height for the subtext CSS style in the code.

The worst thing about making these changes to TakeSurvey.asp is that they then affect every survey. Honestly, I believe the system's entire survey templating subsystem would have to be expanded, or a better base styling coded up in TakeSurvey.asp, in order to make the system really usable. Not impossible, but kind of a drag given the system's other great features.

One other thing I noticed was that the system doesn't exercise good IIS Session timeout manners when users are taking a survey. I experienced unfriendly server-side ASP 500 errors when returning to complete a survey I started, but left open in the browser beyond the timeout interval. Again, not impossible to fix, but a drag nonetheless.

Perhaps when I have more time I'll post some additional details. Feel free to comment on your experience with these products, or any others you might have tried.