A Walk Through of Camano and the Manual Test Runner for Quality Assurance Test Planning and Test Execution VSTS 2010 CPT (Rosario)

Introduction:

This is a walk through of the new functional testing tool codename “Camano” and the Manual Test Runner by Microsoft. These are very powerful tools put out by Microsoft that now allow you to efficiently plan and execute your QA Testing efforts. Also, all of your testing data is stored and integrate into Team Foundation Server. This is an amazing beneifit if you use TFS as one of your for project management tools. Camano is released with Visual Studio Team System code name “Rosario”. You can download a virtual machine image of Rosario from the Microsoft web site.

Camano Overview:

To start off I will go over all the entities in Camano. The most basic entity is a Test Case. A Test Case is work item type in that mostly consists of a sequenced set of steps and their expected results. A group of Test Cases make up a Test Suites, which also could be made up of other Test Suites also containing Test Cases. A Test Configuration is an entity that allows you to specify the name of the environment you will be testing in and provide a description for it. A Test Suite, a Test Configuration, and Test Settings make up a Test Pass. A Test Pass or a combination of Test Passes make up a Test Plan. A Test Pass can have its own “Global” Test Settings. I will cover each of these entities in detail later on. Here is a diagram of how these entities fit together that you can refer back to if needed.

Should you keep reading? What you'll get out of this post:

After highlighting some things regarding navigating through the application, I will create a couple of Test Cases, add them to a Test Suite, create a Test Configuration, create a Test Pass, create Test Settings, and create a Test Plan. I will introduce the Microsoft Manual Test Runner and use it to execute my Test Plan. I will then go over a report generated based on my testing activities.  

My examples, and the data I use in creating each entity, assume that I am testing the User Management functionality of a web site in Internet Explorer. I don't have an actual web site developed to test, so I will pretend I am executing my test cases and just worry about analyzing.

Before you fire up Camano:

First I start by creating a new team project from within Visual Studio. I used the CMMI template and finish the wizard. Once this is done we are ready to start working in Camano.

Navigating through Camano:

I took a screen shot of the Home Activity Center Screen below and split it up into 3 main sections to go over.

Section 1 lists all of the categories, or menu options in Camano. Once you click on one of the menu items, all of its sub menu options are loaded into Section 2. Other sub menu items are accessible by clicking on the little down arrow below each menu item. Section 3 lists all active windows you have open, or any unsaved screens. It took me a while navigating the application using the Working Progress section (Section 3), but its pretty handy once you get used to it.

Creating, Configuring and Planning your Testing Effort:

Start by connecting to the team project you created in Visual Studio. This is done by clicking the Home menu option and selecting Connect to a Team Project.

I will now start planning my testing effort. The first step will be creating the Test Cases.

First click on the Planning menu option.

Click on New Test Case.

Populate the Test Case's title, Actions or Steps, and Expected Results and any other fields you wish populated. I am going to create a second Test Case that will come in to play later on.

Click on New Test Case to create the second Test Case then populate the fields on the screen.

After creating the Test Cases, I proceed to create a Test Suite that will consist of both Test Cases created earlier.

Click on Manage Test Suites. Click on the Test suites, then click on the New Suites button and give it a name. I will call mine “User Management Test Suite” and reference it throughout the remainder of this post.

Once I created the Test Suite, I want to add or associate my Test Cases to it.

In the right pane, click on Add Tests. Make sure that the Test Case Suite you created is still highlighted in the left pane.

From the window that pops up, select the Test Cases and click the Add Tests button.

After creating my Test Cases and grouping them into a Test Suite, I will create a Test Configuration. A Test Configuration will allow me to associate the name and a description of my test environment with my Test Suite.

Click on Planning, then on New Test Configuration. Specify the Name and Description then click on Save and Close.

After creating a Test Configuration, I will create my Test Settings and then a Test Plan.

Click on Planning, then on New Test Plan, then click New Test Settings.

On the left pane, there are 7 options. I will briefly go over each one and and highlight what is needed to get through this walk through. I will not include screen shots for every option.

In the General tab, specify the name of the Test Setting.

In the Execution Criteria tab, select whether you want to execute the test locally or on a remote server. If you select Remote you must specify the server IP in the drop down box, or select it if it from a list if the list was automatically populated. Also in the Execution Criteria tab, you can enable the type of logging (Collectors) you want to happen.

In the Deployment tab, you can add files you wish to deploy with your test pass. These could be files used by the application your testing.

In the Setup and Cleanup Scripts tab, browse to script files that you want to run before and after the test run. These could be scripts to automate setup steps.

In the Hosts tab, select the Host Type (or platform) you want the test to run in (i.e. Your Machine, ASP.net, Device)

In the Test Timeouts tab, you can enable aborting or failing a Test Run its execution exceeds a specified time limit.

The data I used in the walk through:

• Under General, I set this Test Setting’s name to “Administrator Functionality Test Settings”
• Under Execution Criteria, I select “Execute Tests Locally” and collect “Event Log”, “System Information”
• Under Deployment, I disable deployment
• Under Test Timeouts, I Uncheck “Mark an individual test as failed if its execution time exceeds”

Once I am done, I proceed to create my Test Plan.

Click on Save and Close. This will take you to the New Test Plan Screen.

The Test Plan is where everything we've been creating so far comes together.

These are the Steps I took to create my test Plan:

1. Enter the Name, Description, Start/End Dates, and a State for the Test Plan. A Test Plan has to have a state of In Progress to be visible/usable throughout the application.
2. Select the Test Settings that were created earlier and saved
3. Specify a name for the Test Pass. A Test Pass is the grouping of your Test Suite, Configuration, and Test Settings. I called the Test Pass used "Test User Management".
   
   1. Select the Test Suite created in step two containing both User Management Test Cases.
   2. Select the Configuration Created Earlier containing the name of the environment we are testing in.
   3. Select the Test Settings created earlier (or different ones if applicable).
   4. Set the State to In Progress, and the Type as Non Recurring. This Setting will be used by Camano to determine if the Test Plan shows up in the All Test Plans Page, accessible from Planning --> View All Test Plans.
   5. Save and Close.

To summarize what I did so far:

1. I created 2 Test Cases to test the User Management section of a web site
2. I added those to a Test Suite
3. I created Test Settings and Test Configurations to give more detail about my test environment
4. I created a Test Plan to test the Administrative Functionality of a web site
5. I added my Test Suite, Settings and Configuration to a Test Pass under my Test Plan

Test Execution:

Click on the Testing menu option. In the left pane, right Click on the Test Pass (i.e. "Test User Management") and click “Start Testing this test Pass”

This will launch the Microsoft Manual Test Runner. This is a simple interface containing 3 expandable areas. In the first area is a list of all the Test Cases in the suite associated with the Test Pass. In the second area are the Steps for the Selected Test Case. In the third area are the Properties of the selected Test Case.

Select the Test Case you want to execute from the top pane.

Click on Actions, then on Start Test Case. This will Start the test case and, focus on one of your open windows and start recording. Since I haven’t prepared a web site for this demo, I ran a blank solution in Visual Studio to demonstrate how the recording behaves (indicated in the upper left corner of the browser window) I also launched MS paint and it also started recording my window there.

I am going to minimize my screens, open the Manual Test Runner and pass all my test steps. This is done by right clicking on each step and selecting pass. When I am done, I am going to click on End Test Case.

Notice that every time something is recorded and a test case is passed (or set to fail/inconclusive), a blue triangle is added. This is the a marker in the automation strip. When you double click on a marker, it and its corresponding end marker will change color and a menu will pop up. This menu allows you to Play the selected clip, save the clip for sharing, or preview the clip.

Notice that it also created and attached an xml file and an ActionLog text file under test case results.

After I am done, I can click on Actions and Publish (or alternatively click Publish Test Results listed under the Test Cases). This will gray out my test case and only allow me to run the remaining test cases. You are also able to create a Work Item Type Bug for any test cases that fail. This Bug will be stored in TFS and could be managed there.

Go back to my Testing Activity Center, click the Refresh Button, right click on my Test Pass and select “View Test Pass Details”. You will get the following report.

This report will tell you:

• How many test cases there are
• How many test cases were ran and how many weren’t.
• How many test cases passed, failed, or were marked as inconclusive.
• How many test cases were blocked and how many failed by regression, new issues, or known issues
• The test configuration name
• The test suite name
• The date testing should start
• The date testing should end

The report also allows you to assign test cases to TFS users.

Conclusion:

The Manual Test Runner is an independent tool of Camano, but they work together nicely. In a real world scenario, Test Cases can be prepared using Camano (or in Visual Studio), assigned to the testers that will be executing each test case, and organized into Test Suites and Test Plans. Once that planning effort is complete, each member of the execution team can fire up the Manual Test Runner and work with their own set of assigned Test Cases. Logging/environment settings and execution start/end constraints can be put in place for better control over the testing effort. The project manager can depend on real time reporting and work item customization to capture the data they need for keeping the porject on budget and on schedule.

Resources:

http://msdn.microsoft.com/en-us/vstudio/bb725993.aspx

http://ozgrant.com/2007/12/09/november-rosario-ctp-planning-a-testing-effort-with-camano/