Test Run

Welcome the new version of Test Run: 1.1. Behind the scenes the product has undergone a great deal of change in the last twelve months and I am finally pleased enough with what has accumulated to release the first public beta.

More Information about this release:

• Detailed change log

Finding Help and Reporting Bugs

• Report a bug
• Find help from the community in our forums

Frequently Asked Questions

I am not sure what I found is a bug, should I still report it?

We want to hear any thoughts you may have about the application. Perhaps what you have found is not a bug, but if it doesn't work the way you want it to then something could be improved. So please, be sure to inform us if:

• an error occurs
• if the application appears slow
• you receive a javascript error
• you have a feature request

Do you have special support for Jira and other bug tracking systems?

Support for Bugzilla is almost complete. However before it is released, we wanted to finalize the developer interfaces between Test Run and the bug system driver with more thorough and public testing.

Next on the list to support is Jira because of the overwhelming number of requests we have received for it.

How long will the beta last?

Our hope is to wrap up the beta as soon as possible.

Can I subscribe to Test Run during the beta?

Absolutely!

Test Run 1.1 has been in development for a very, very long time. I admit. But the pay off will be worth it as there is a lot in this release to look forward to, and virtually all of it is the result of direct feedback from you - my users.

So, some time next week or two I hope to launch a beta of the next version of Test Run. I would like to first extend the opportunity to users of the downloadable version of Test Run to try it out and after an initial round of feedback and testing, roll it out to everyone using the online version.

If you would like to get a preview of this upcoming version of Test Run, drop me a line, and I will happily provide you with a copy.

For a complete list of enhancements that have been made in Test Run 1.1 to date, keep reading.

Continue reading "Preparing for the Test Run 1.1 Beta" »

The Test Run Beta is beginning to wind down. Feature development for 1.0 is almost complete - in fact the only feature actively being developed is support for account level permissions for anonymous users. This would allow each account to configure how they wish to handle anonymous access to their test plans and test cases - specifically to allow non-logged in users the ability to view test cases that others may wish to link to from bug reports and other systems and applications.

I find myself always saying this is the last feature... but I often get great feedback from users and testers that I cannot ignore. So please, don't hesitate to send me your ideas. It is never too late to get your ideas and feedback integrated into the product.

But just because there are only a few features left to implement for the first official release of Test Run, that does not mean there are only a few more features we need to test. In fact these final stages of the Beta will allow us to test an important component of the Test Run product and service - billing and payment systems.

In order to ensure that the entire Test Run experience is a good one, we will begin charging $10 for the downloadable version of Test Run and charging a one dollar per month subscription fee for users of Test Run Online. We appreciate any and all people willing to help us test this critical component of the Test Run product, and we want our testers to rest assured in knowing that they will be allowed to deduct all the Test Run fees they paid prior to the official release of the product from the cost of a commercial license or subscription. In this way the Test Run Beta is still "free."

Users evaluating the product can of course sign up for a Trial Account on Test Run Online which provides unlimitted access to a single user.

Again the Test Run team cannot thank those many users who have submitted help tickets and feature requests to us. I hope you are satisfied with how we chose to solve your many problems and use cases.

Many users ask us about the roadmap for the Test Run Beta, and when we will begin accepting paying for Test Run. So we wanted to take a moment and answer the most common questions up front right here on the Test Run blog so that the answers are available to everyone.

Continue reading "Test Run Beta Update" »

Despite having not posted any updates to this blog about Test Run releases, they have been happening. We have been iterating on Test Run quickly in order to continue to flush out bugs and tighten up the application. The recent release was significant enough however to warrant an announcement.

Continue reading "New Dashboard and Advanced Filtering" »

Over the past week or two we have been working with customers and beta users as we continue to refine Test Run's installation process. Over the past week and a half we have made a number of silent releases to these customers and the public to make sure that Test Run is constantly improving. Many thanks to these dedicated and patient users who download and install Test Run repeatedly to help us with our Beta process.

One of the great outcomes of this process is a detailed Windows XP Installation Guide that we authored to help people through the installation, setup and initialization process of Test Run. The document discusses installation on a Windows platform specifically, but many if not all of the concepts apply to Unix, Linux and Mac OSX systems as well, since Test Run is architected to run on any LAMP stack.

We have also closed to loop on a number of critical bugs that hampered people's ability to add and remove test cases from test plans in some cases. As always, these bug fixes and enhancements can be monitored in Test Run's real time change log.

In working with a recent customer, I learned just how difficult Test Run could be to install for some people. Geeks like me always run the risk of taking for granted having tons of software installed. So to address many of the problems users have reported having while trying to get Test Run setup, I released an interim release of Test Run called Beta 8.5. This release contains the following key features:

• a number of bug fixes detailed in the real time Test Run change log
• a greatly enhanced setup process, including:
  • more detailed in-app instructions on how to resolve failed dependencies
  • clearer language around failed dependencies
  • an extra step to allow users to confirm their PHP installation’s email settings
  • several bug fixes which failed to fully create the administrator’s profile

And that’s Beta 8.5.

Beta 9 development continues. We are working hard on this one, but the changes to the application are substantial and will take time to develop and test.

Earlier today I wrote about the challenges of choosing between new features and fixing bugs. Well, bugs unfortunately are a reality. And Test Run Beta 7 had tons of them, and honestly, I feel guilty about that. But with this most recent release I feel like we are starting to turn a corner as the major features I wanted to launch with get checked off the list and I have more time to devote to “polish” and detail.

Continue reading "Announcing Test Run Beta 8" »

One of the hardest tasks of a product manager is striking a balance between developing one of the cool features that you have been talking about forever, or fixing one the many nagging issues that has been annoying your users for an equally long time. If you are lucky then you have the resources to devote to both and circumstances will give you the freedom to make that choice, because sometimes circumstances demand that certain issues are addressed.

Test Run faces this challenge like every other product. For the past several months a design team and I have been iterating on a new user interface for Test Run. The new UI will be a great leap in usability. Meanwhile, Test Run development has stagnated. Major bugs have pestered users and prevented wide adoption. So it is time to shift our focus back to development. While the design team starts to build out the new user interface, I have turned my attention back to the existing application focusing on bug squashing, and the polish necessary to prepare for a broader launch.

To be frank, I regret assigning the label “beta” to Test Run. It should have been labeled “alpha” as Test Run is still quite rough around the edges. But that doesn’t seem to deter people. More users download and sign up for Test Run everyday, a testament to the promise is has for enterprises.

The single biggest feature request for Test Run has been for hierarchical test plans, or in other words, test case dependencies. Test case dependencies help streamline the test process by ensuring that QA engineers focus only on the tests that can be run. For example, imagine a sitation where test case A depends upon test case B. That means that test case A cannot be executed until test case B passes. Therefore, if test case B was executed, but failed, then test case A is “blocked” until test case B is re-executed and passes. In this state, there is no sense in an engineer even looking at test case A, right?

If that is at all confusing to you, then that is exactly why you need Test Run. Maintaining a dependency tree manually is cumbersome and tedious at best. Difficult enough that most organizations don’t bother tracking dependencies in such a way that they can be reported upon, or in such a way that QA engineers can track them while they execute their tests.

Test Run takes all the tedium and guess work out of managing test case dependencies. In Test Run, all one needs to do is identify a dependency, and Test Run manages the rest:

• Test Run provides an intuitive user interface for navigating dependency trees.
• Test Run prevents blocked test cases from being executed.
• Test Run highlights test cases that are blocking further testing progress.
• Test Run reports the percentage of blocked tests versus those that have not yet been run, or those that have been marked as failed or passed.

In the next couple of weeks Test Run Beta 8 will be released, and test case dependencies will form the centerpiece of the release. Development has already started, so if you have any additional feature requests, or particular questions or concerns about test case dependencies, now it the time. Leave a comment and let us know.

Normally I wouldn't bother posting about feature enhancements until after they have been released, but I have been working on some improvements this weekend that I wanted to share with Test Run's Beta testers. First, thanks to all those who are submitting feedback and feature requests. With each release I get more and more users, and as a result more and more feedback. So each release seems to be getting better and better.

Continue reading "Beta 7 Preview" »

An important aspect of companies today, and what I believe blogs have helped businesses achieve to a large extent, is an increased transparency into their business. Business owners are sharing with their customers more and more about their product development process and inner workings of their company. Doing so enables them to to build stronger and closer relationships with their users and customers.

At Test Run we feel that this transparency is also important and critical to our customers. That is why we just added the "What's coming" page to our website. On this page customers can view the current change log of our product as it is being developed. That way customers can see what bugs have been fixed, and what features have been added in near real time.

Rest assured that critical bugs, like security and data integrity issues, are resolved as soon as they are discovered in Test Run Online, while users of the downloadable version of Test Run will pick up those changes only after they download and install the latest version.

One of the reasons we opted to release this product early was to give our customers enough time to shape the product before it was officially released. If you are a user, or even if you are just evaluating Test Run for your company or use, we would love to hear from you directly:

What specific features are you looking for in Test Run?

Continue reading "Calling all users: we want your feature requests!" »

We am very interested in hearing from customers who have used the product and to learn what their impressions have been so far. If you are a user of Test Run and could spare a moment, please leave a comment and let us know what you think of our product.

Also, we am looking for users whose companies might be interested in becoming a reference customer to be listed among our other customers. If your company is interested your company will qualify for a free license of Test Run once it is officially released. Please contact us if you are interested and we will follow up personally.

We continue to refine Test Run well into the weekend and late night. Test Run Beta 5 is ready for download. You may notice a slightly different download process as I have finally made available the Test Run Beta license. Also, I have created a dedicated page for setup and installation instructions. All this to hopefully make things a little eaiser and a lot clearer.

Of course, Beta 5 contains a number of enhancements and bug fixes. Most notably a new "Active Bug Report." For developers and integrators, the ground work has also begun for an extensive callback framework that can be plugged into to create Test Run extensions and plugins.

It has been a busy Thanksgiving Day weekend for Test Run. There has been a flurry of activity and a few bugs reported, and a few discovered while fixing others. We have packaged up the lastest build of Test Run and made it available for download. And of course we have updated Test Run Online with the latest as well.

Also, now that there are plenty of more users we thought it best to started maintaining a detailed change log which will accompany every download and every new release. Keep reading to learn more.

Continue reading "Beta 4 Released" »

We have received a lot of questions of late about the Test Run Beta and Trial period, specifically: how long will the trial last and what kind of quality and service can I expect during the trial?

How long will the Test Run Beta Last?

We hope the beta period is as short as possible, however we can make no guarantee as to its length. The beta period is in place to ensure the highest quality software possible once Test Run "ships." We also want to leverage the beta period to ensure that Test Run in is the marketplace long enough to collect and integrate valuable feedback from potential customers.

What kind of quality and service can I expect during the beta?

During the Test Run beta period, we will take all reasonable efforts to ensure the safety, security and integrity of your data. Keep in mind that it is in our best interest to ensure the highest quality possible for our customers.

Are there benefits for participating in the beta?

Absolutely. All users that participate in the Test Run beta will qualify for special discounts and pricing when Test Run is officially released. To qualify, register for an account for the online version of Test Run, or sign up for the Test Run newsletter. You will be notified when the time comes on how to take advantage of any special discounts.

We encourage everyone to take advantage of the Test Run beta, and please don't hestitate to contact us with your comments and feedback!