Test Run

Test Run now has a Twitter account. If you want to stay up with the latest development news, follow us there!

I have been receiving a lot of feedback in the New Year from new customers, perhaps because people out there are looking for ways to cut costs and keep things simple. Their feedback as usual has been invaluable, and most recently led me to revisit the design of the Manage Test Plans screen. The impetus for the change was to create a framework that allowed for me to surface more management functionality on the screen and to offer a more compressed layout that allows admins to take more information in more quickly and efficiently. I am not finished yet, so feedback is still welcome. Here is a before and after of the Manage Test Plans screen:

Before

After

What do you think?

With the slow and steady increase of customer usage of Test Run Online, the product has begun to feel a little sluggish and a number of you, my customers, had expressed their concern. On the up side, I guess that is bound to happen when you have over a hundred thousand test cases, and five thousand test plans - surely a sign of success by some measure. But the success of Test Run should never come at the expense of paying customers, so I have turned my attention towards performance and database optimization.

Having just completed a thorough analysis of Test Run's database architecture I will, over the course of the next several days, be rolling out a number of recommended improvements to the system. Several have already been deployed and the results are enjoyable - and difference is remarkable.

Of course, I am also working on other improvements to make Test Run more usable, and flexible. A number of users have been having difficulty with text formatting, and in working with them I realized that text formatting doesn't always make sense as a global setting. So, I just finished making text formatting, like the use of markdown and a rich text editor, a test case level setting. Now each and every test case can set its own preference for how it should be formatted.

Other improvements are also coming, so stay tuned.

Welcome the new version of Test Run: 1.1. Behind the scenes the product has undergone a great deal of change in the last twelve months and I am finally pleased enough with what has accumulated to release the first public beta.

More Information about this release:

• Detailed change log

Finding Help and Reporting Bugs

• Report a bug
• Find help from the community in our forums

Frequently Asked Questions

I am not sure what I found is a bug, should I still report it?

We want to hear any thoughts you may have about the application. Perhaps what you have found is not a bug, but if it doesn't work the way you want it to then something could be improved. So please, be sure to inform us if:

• an error occurs
• if the application appears slow
• you receive a javascript error
• you have a feature request

Do you have special support for Jira and other bug tracking systems?

Support for Bugzilla is almost complete. However before it is released, we wanted to finalize the developer interfaces between Test Run and the bug system driver with more thorough and public testing.

Next on the list to support is Jira because of the overwhelming number of requests we have received for it.

How long will the beta last?

Our hope is to wrap up the beta as soon as possible.

Can I subscribe to Test Run during the beta?

Absolutely!

Test Run 1.1 has been in development for a very, very long time. I admit. But the pay off will be worth it as there is a lot in this release to look forward to, and virtually all of it is the result of direct feedback from you - my users.

So, some time next week or two I hope to launch a beta of the next version of Test Run. I would like to first extend the opportunity to users of the downloadable version of Test Run to try it out and after an initial round of feedback and testing, roll it out to everyone using the online version.

If you would like to get a preview of this upcoming version of Test Run, drop me a line, and I will happily provide you with a copy.

For a complete list of enhancements that have been made in Test Run 1.1 to date, keep reading.

Continue reading "Preparing for the Test Run 1.1 Beta" »

I wanted to give a quick development update as we near a release of Test Run 1.1 Beta. I have nearly completed Bugzilla integration with FogBugz, but there are still a lot of kinks to work out in regards to allowing users of our hosted product to access their private instance of Bugzilla.

I have also been working on a number of UI tweaks to make Test Run easier to navigate and use. Without going into too much detail yet, here are a few screenshots to whet your appetite:

Summer is over, the traditional slow time of the year in the software industry, and I have been getting a lot of renewed interest in Test Run. Customers have been submitting bug reports and a lot of feature requests, and many of them have in all honesty gone relatively unanswered in the last several months.

Well, that is about to change. I feel there is a critical mass of feature requests to begin working on Test Run 1.1. So last night work officially began. I have started to assess a lot of different things about the product in an effort to improve customer support and the experience around finding help. I upgraded the Test Run Blog and Knowledge Base to use Movable Type 4.0, and I have migrated Test Run's forums off of my personal web server onto the production Test Run machines. This has the added benefit of dedicating Test Run's forums exclusively to Test Run. Users will no longer find content related to my other software projects. This is a good thing.

As for what features I am planning for this release:

* At the top of the list is the need to integrate with popular bug tracking systems. This feature will allow users to submit bug reports directly into Bugzilla and FogBugz from within Test Run.
* An import feature for Test Link which will allow people to easily migrate to Test Run from that case tracking tool.
* Test Plan printing - many users asked for the ability to print a test plan to make it easy to review plans offline.
* Rich Text Editing - several users have asked for me to integrate a WYSIWYG editor into Test Run, and I will see what I can do to accommodate.
* Naturally there will be lots of bug fixes as well.

I encourage users to submit feature requests to me (as a comment to this post if you wish) and let me know what you would like to see out of the next version of Test Run.

Development on Test Run continues at a fast and furious pace as we rush to keep up with all the wonderful feedback we get from customers. Most of the feedback is about a feature or two that people would really like to see. Sometimes the feedback is in regards to a bug that a user finds. And then occasionally a user will tell me something like this:

"I have worked at a lot of companies and have been responsible for reviewing and recommending countless test management tools. Test Run is far and away the best and simplest tool I have ever encountered."

God, I can't tell you how wonderful it is to get feedback like that. When it sometimes becomes really hard to create awesome software while continuing to work full time at another job you also love, feedback like that picks you up and keeps you going.

And what better way to reward a growing and increasingly involved and engaged user base then with a new release of Test Run?

Tonight I am happy to announce the latest version of Test Run which contains features implemented exclusively from customer feedback. Most notably, Test Run 1.09 has the the following new features.

Continue reading "New features: Tag Management and Time Estimates" »

So I have been working on a number of features requested by potential customers - oh the lengths I will go to to try and get someone to actually pay for the services! (It really is worth paying for honest!) One customer I have been corresponding with really admired the use of tags within Test Run and how they facilitated their filtering cases. However, they still found tags inadequate for searching test cases and asked if there was a way to offer a full-text search allowing users to find test cases by text found in the summary, detailed description, the notes and more.

A quick search on Google revealed that thanks to Lucene the feature is relatively easy to implement. So, a quick download of the PHP Zend Framework and presto: full text search is done.

I would have released the feature a while ago, but thanks to some issues with executing PHP from the command line it took a little longer than I anticipated to index all of the thousands and thousands of cases stored in Test Run. But I was finally able to get 100% of the test cases indexed and so search is now available online.

It may take a while for me to document the process to enable full text search, so users of the downloadable version will need to be patient while I wrap a few things up. In the meantime, users of the hosted version of Test Run can begin using full text search right away.

We have been hard at work this week making a number of small improvements to Test Run, and one big one. A number of customers have been expressing the desire to have more control over the order in which they execute their test cases. Until now Test Run always ordered test cases in reverse chronological order. What made this challenging for many users was the fact that some testing requires test cases to be executing in a very specific sequence. So to address their concern we implemented the feature to allow test planners the ability to order test cases however they see fit.

When we designed and developed this feature we wanted to make the process as intuitive and as simple as possible, allowing test planners to drag and drop test cases in what ever order they choose. We could tell you more about this feature, but I think it much better to see it in action.

Last night I released Test Run 1.04, a version that contained a minor feature enhancement enabling greater interoperability with popular bug tracking systems. Tonight I started work on Test Run 1.05 which contains a single feature: the ability to re-order test cases within a test plan. This feature request comes after a number of customers expressed concern that their process required tests to be run in a specific sequence. However Test Run's default sort order (the created on date) was insufficient. They wanted more control then that.

So in the next version of Test Run, customers will be allowed to specify the order of their test cases within a test plan using a simple drag-and-drop interface. Stay tuned...

A number of Test Run users, and wannabe users, use Jira, a bug tracking tool created by Atlassian. Jira is unique among other bug tracking systems in that the bug IDs it produces are not exclusively numeric. This made it difficult, if not impossible for those using Jira to integrate with Test Run's bug tracking capabilities.

So last night we released a few changes to enabled Test Run users to enter bugs into the system that can be any arbitrary string. So Jira users: rejoice. Test Run now works even better for you.

Continue reading "New feature: Non-numeric bug ids" »

A few customers with very large accounts began experiencing performance problems with Test Run soon after it was officially released. These performance issues were restricted to only a few accounts, however they posed a risk to all customers when and if Test Run ever came under severe load. Therefore we sought to address these problems proactively by taking the following steps.

First we have optimized the database to make certain searches and queries faster and more efficient. We also deployed support for memcached throughout the application to eliminate unnecessary database queries.

Right now these enhancements have only been made available to users of Test Run Online, but they will soon be made available to users of the downloadable version of Test Run as well.

In a continuing effort to make Test Run more secure for its users, Test Run Online has been upgraded to use SSL from a trusted and well established certificate authority. Now when you use Test Run your account information will be encrypted and secured.

After two years of development, Test Run 1.0 is finally here. Thank you to all the beta testers who have helped make this release and product possible. I honestly could not have done it without your ideas, patience, support and dedication. Thank you.

Most questions can be answered quickly by reading Test Run's list of Frequently Asked Questions. However, over the next couple of days and weeks we will be updating the blog to tell you more and more about Test Run and its capabilities. In the meantime, please keep reading to receive a brief overview of the some of the immediate changes you will notice when you login to Test Run 1.0 for the first time.

Continue reading "Test Run 1.0 is here!" »

It has been a long time since I have posted to this blog, but it is not because this project and product have been abandoned. Quite the contrary. I simply have been in a habit of posting only to announce releases, a habit I hope to break.

But not tonight.

Tonight I am happy to announce that the long awaited release of Test Run 1.0 is almost here. It is hard to believe that this release has been in development for almost two years. And that in the course of those two years Test Run has been downloaded almost 2500 times. Plus, the online version has had almost 700 sign ups, and over 16,000 test cases have been executed.

This release is the culmination of a tremendous amount of input from users provided over those two years. Without this feedback, Test Run would never have evolved to be so useful to so many companies. In addition, this release is the result of a wonderful collaboration with an incredible partner and design firm, Mule Design. Simply put, they are really good at what they do. And I believe it shows in the final product.

Long story short, in the coming days, users of Test Run will experience a number of changes. Most notably Test Run will be given a completely new design and interaction model to make test case management even simpler and easier.

Sadly for some users, this release will mark the end of the free beta period for Test Run. When Test Run 1.0 is released all existing users will be asked to purchase a subscription to Test Run Online, or a license for the downloadable version of Test Run which can be renewed annually.

There is so much to explain, and still many things to do. But I promise, to answer everyone's questions soon. So before I sign off for the evening to head to bed, I will leave you with one last treat, a sneak peak of the new Test Run web site.

Stay tuned. We are almost there...

From Sunday, October 15 through Tuesday, October 17, the Test Run service suffered an outage. This impacted the corporate website, the ability to download copies of the software, as well as the hosted version of Test Run. There was no data loss associated with this outage and the service has made a full and complete recovery. We sincerely apologize for any inconvenience and we are taking steps to ensure this type of outage does not happen again.

You may now return to your regularly scheduled testing.

Test Run has always offered top notch support through its private and secure help ticketing system. But as Test Run grows in popularity so does the need for community assisted support, and for collective wisdom to accumulate. A closed help ticket system does not permit that.

So we at Test Run have put together a Test Run Discussion Forum for all of our users to use - to ask questions, search for answers and to engage with fellow Test Run users.

If you have a question, try asking on the forums first. I will be there answering questions, so you will get the same level of support, but the good news is that if someone later has the same question you did, they won't have to ask it - they can just look at the answer.

Many users ask us about the roadmap for the Test Run Beta, and when we will begin accepting paying for Test Run. So we wanted to take a moment and answer the most common questions up front right here on the Test Run blog so that the answers are available to everyone.

Continue reading "Test Run Beta Update" »

It seems I got ahead of myself and "released" an update without updating the URL to download the latest version. For those that downloaded the latest version of Test Run, and got Beta 7, my apologies. You can now download the actual latest version of Test Run via the usual download page. Doh!

One of the hardest tasks of a product manager is striking a balance between developing one of the cool features that you have been talking about forever, or fixing one the many nagging issues that has been annoying your users for an equally long time. If you are lucky then you have the resources to devote to both and circumstances will give you the freedom to make that choice, because sometimes circumstances demand that certain issues are addressed.

Test Run faces this challenge like every other product. For the past several months a design team and I have been iterating on a new user interface for Test Run. The new UI will be a great leap in usability. Meanwhile, Test Run development has stagnated. Major bugs have pestered users and prevented wide adoption. So it is time to shift our focus back to development. While the design team starts to build out the new user interface, I have turned my attention back to the existing application focusing on bug squashing, and the polish necessary to prepare for a broader launch.

To be frank, I regret assigning the label “beta” to Test Run. It should have been labeled “alpha” as Test Run is still quite rough around the edges. But that doesn’t seem to deter people. More users download and sign up for Test Run everyday, a testament to the promise is has for enterprises.

At some time today, January 11, 2006, Test Run was downloaded for the 500th time. I can’t help but give myself a tiny pat on the back. Test Run started as a small experiment is growing into a small company. It is almost hard for me to believe.

We are pleased to announce the latest version of Test Run, Beta 7, is now available for download and is also available on the online version as well. This release is very significant and requires users of the downloadable version of Test Run to make some fundamental changes to their Test Run database, which is detailed later in this post. These changes however are required in order to meet the demands of Test Run’s growing international user base. Now in use in China and Europe, customers submitted feedback requesting support for the UTF-8 character set so that multibyte characters would be displayed properly. While the application is still shown in English, Test Run now allows users to enter virtually any character they wish into their spreadsheets and within Test Run itself.

Beta 7 also introduces native Excel spreadsheet support. What does that mean? In previous versions of Test Run users were required to import and export CSV files to and from Excel and Test Run. It was not technically difficult, but it was an extra step that just made integration with the most common QA tool more cumbersome. With Beta 7 one can upload and download Excel spreadsheets directly, with no other intermediate steps necessary.

Those are the two major changes in Beta 7, but there are a number of other feature enhancements and bug fixes as well.

Continue reading "Announcing improved international support, native Excel spreadsheet support and more!" »

Since Test Run was released less than a month ago it has been downloaded over 300 hundred times and currently has over 100 registered users using Test Run Online.

Thanks to all the feedback we have received as result of having so many users using the product the latest release, Beta 6, is a significant enhancement over Beta 5, to say the least. It's change log is the largest to date. Aside from addressing some long standing issues like better enforcment over user assigned permissions, a lot of refinement has being done as well.

For example, many users told me that they did not understand tags, which was unfortunate because mastering tags is one the keys to mastering Test Run. Tags enable users to easily generate test plans based upon a few simple keywords. To help educate users more about this feature, if a user attempts to create a test plan when they have not yet tagged any test cases they are presented with some contextual help documentation to lead them on the right path. On the other hand, if they have tags in their account, then those tags are listed for them to select from.

And that is only one example of how we have been trying to make Test Run easier to use. I wish I had the time though to detail every single enhancement contained in this most recent release, but it just isn't practical. If you are curious about a summarized list of enhancements, please continue reading this post, or take a look at the real time change log.

As always, thanks to all the user who sent feedback. I think you will all be happy to see your requests now in the product. Keep that feedback coming, and we will keep the releases flowing.

Continue reading "Beta 6 Released" »

In gratitude and in acknowledgement of the role open source plays in Test Run.s development, we are happy to announce that Test Run is available for use with open source projects completely free of charge. Furthermore, in pursuit of a belief that all businesses should think of how their products and services can best help their communities, a model proven not only successful but also profitable for Salesforce.com, Test Run is also available free of charge to non-profits and educational institutions.

More details can be found on Test Run's pricing information page.

It was not too long ago that Beta 2 was released. However, that release didn't have a lot of user facing features once you got past the setup process and it just didn't seem right to go a whole week without giving our users some candy. So we started focusing on arguably one of the most critical areas of the application: the import and export area.

In a multi-user environment, there is a real danger of users introducing conflicts or clobbering each other's data when importing test cases into Test Run. To address this we added the ability to preview the changes introduced by importing a spreadsheet of test cases into the application. Granted, it is a simple feature on the surface, but it is laying some important and substantial groundwork for more advanced importing and exporting controls, like advanced collision detection, record merging, and more.

Test Run Beta 3 is now available for download as well as online.

We received a good round of initial feedback and tackled some bugs that hindered some in the downloadable version of Test Run to get started more quickly. In addition we received some execllent feature request. It was our hunch that I would need to implement the features requests we received at some point, but it makes a big difference when a customer asks for them specifically.

Thanks to all of you that provided us feedback on Test Run Beta 1. Your help, input and insight is very valuable to us. Test Run Beta 2 is now available for download.

Continue reading "Beta 2 Released" »

I am happy to report that Test Run is finally available for download. Much effort went into making the setup and installation process as easy as possible. The first time you access a newly installed instance of Test Run you will be directed to a simple three step installation wizard. This wizard will ensure all your prerequisites are met, help setup the Test Run database for you, and allow you to create your superuser administrator account.

As always, if you have difficulty, please don't hesitate to contact us.

My Backpack To Do List is getting smaller and smaller everyday. Plus I have finally started implementing a much needed overhaul of the main site's design. The application's UI design isn't all that bad; it is simple and to the point, as it should be. But the main site needs to be much more succinct, navigable, and useful overall. Please let me know what you think.

I am probably a couple weeks from a public beta, but that won't stop me from really trying to educate people about Test Run and driving people to the site. If you are interested in participating in the Test Run Beta, then please sign-up for the Beta Newsletter and you will be the first to be notified when the public beta is available. Of course, I left the door wide open anyways, but until the beta official begins, I make no promises as to the stability of the application, and the integrity of the data stored within it (so use at your own risk).

But stay tuned. The public beta is coming.

Test Run, in one form or another, has been a pet project of mine for several years. But it wasn't until the past six months that I really believed what I had prototyped previously may actually be worth something. I originally wrote Testitool because the company I worked for at the time desparately needed some way for managers to gain a higher degree of visibility into the QA process. I spun off Testitool as an open source project, but do a really poor job of maintaining it.

I always felt that there was a real need for the kind of software I had created, but I simply didn't have the motivation to do what I knew had to be done in order to make it successful, easier to use, and more scalable: completely rewrite it.

But then, history repeated itself. I found myself, a project manager once again in need of software that would give those in my organization more visibility into the daily progress our QA teams were making at executing test plans. They also wanted a way to participate in the test planning process with the team and be able to add test cases to our test suite for future plans on their own. And finally, they wanted a tool that would automate much of the minutia that pre-occupied QA managers much of day. They prefered QA managers to spend their time actually writing test plans, not sending out daily reports, and making sure everyone knew what tests they were responsible for executing.

And so, I resolved to do what I had been putting off: make Testitool into a product worth using. Of course, I also had to change it dreadful name. Test Run was born. And over the past 6 months I have been nurturing this little piece of software, until now, where it is weeks if not days ready for a public beta.

I am excited. I believe in this software. Now it is time to see if others will too.