When you look at your projects, what objectives and success criteria have you defined? Are you looking for a higher quality product, shorter cycles / faster time to market, or are you looking for cost reduction? Regardless of what you’ve set out to achieve you need to define metrics, capture, benchmark, and make changes to improve your process as needed. This may not be easy or even possible unless you can monitor your entire process and capture quantitative data.

Software development is a very complex process and there isn’t a single right way to plan, execute, control, and delivery a project.  The project documents and artifacts I use or produce throughout my development process could be very different from the ones you use or produce throughout yours. At the end of the day, we both produce software and want to do it efficiently. Tools can be used to help us do that, but there are a lot to choose from, so how do you know what to adopt? The tools you adopt to help you facilitate your process should be flexible and customizable to what you’ve established as your organization’s software development process. This is important so you don’t have to change your ways (kill your learning curve). This is also important so that you customize the tools as your process changes. The tools should also gather meaningful data about your process and allow you to access it easily at any time.

Team Foundation Server is the tool described above. It is meant to be used throughout the entire application development lifecycle. The more methodical your process is, the more value you’ll get out of TFS, but it’s also a good tool if your process is informal. I won’t discuss the detailed features of TFS in this post, but I will post another entry in the future to do so. In this post I’ll highlight some of the tool’s features, discuss how it meets the requirements stated above and how it can improve your software development process.

Firstly, TFS allows you to easily integrate your organization’s development process work flows into it. It allows you to create user interface screens that gather metadata about your requirements, development tasks, test cases, bugs, deployment tasks, and whatever other “work items” you’ve defined within your process. You project resources are also part of the TFS server and can assign/be assigned work items. These work item screens are defined and customized once and can be easily modified to evolve with your process. TFS also offers team development and deployment tools, collaboration tools, MS office document templates, and customizable SQL reporting on project health and project activities. All of these project artifacts and tools can be setup from a process template in a matter of minutes. This will save a lot of project setup time project after project. TFS meets our first requirement – customizable around your processes and increases efficiency.

Secondly, while project team members are using TFS, it will capture any actions they performed and store them into a MS SQL Server data warehouse. TFS is automatically capturing all of the quantitative data you need to measure your process. It is providing you with metrics on how many times a specific procedure was performed, how long each one took, and how many people it took to complete each one. It isn’t just doing that for a specific procedure, it is capturing those three metrics categories for your entire process that you’ve customized into the tool. All this data is in a database and can be queried at any time. TFS meets our second requirement – stores easily accessible and meaningful information about your process.

Knowing that TFS meets both requirements, you can use it to improve your development process as follows:

1. Customize TFS and integrate your current process into it
2. Evaluate project outcomes and set improvement goals
3. Mine for specific data that can help you measure your process
4. Analyze the data
5. Rework your process and easily re-integrate it into your TFS Process Template
6. Re-measure and check efficiency gains
7. Repeat

For my setup, I am going to start with a VPC running Windows server 2008. There are multiple scenarios for installing Team Foundation Server depending on your needs, but for the purpose of using this to demo the functionality of VSTS I will do a Single-Server Installation (TFS, SQL, and SQL Server Reporting Services (SSRS) all reside on once machine). For more information on the various scenarios, please refer to the TFSInstall guide available on the ISO image. Here is what I did.

1.       Download the Visual Studio Team System 2010 Team Foundation Server Beta 1 http://www.microsoft.com/downloads/details.aspx?familyid=58665C27-7023-4A50-B31D-C39CD86A415F&displaylang=en

2.       Download a 180-day trial of SQL Server 2008 Enterprise edition http://www.microsoft.com/sqlserver/2008/en/us/trial-software.aspx

3.       Login to the machine, and setup a user Account – create a standard user account with a password. If you are using server 2008, this will be under Control Panel à User Accounts à Manage another account à Create new account. This user willo be used during the TFS install. Please refer to the TFSInstall guide for more information on how many users you need and thier security level based on your install scenario.

4.       Install IIS, make sure IIS 6 Management Compatibility is installed, and make sure Windows Authentication is installed. You can do this from the Control Panel à Programs and Features à Turn Windows features on or off. If you are using server 2008, this will start up the Server Manager screen. You can install the features above under Roles à Web Server (IIS)

5.       Install the .NET framework (required for SQL Server 08 – the TFS installer will install .NET 4.0 Beta, but install 3.5 for now)

6.       Install windows installer 4.5

7.       Install SQL Server 2008 – Make sure you install Reporting Services so you can use the default installation of TFS

8.       Make sure you enable TCP/IP for SQL server (Start à All Programs à Microsoft SQL Server 2008 à Configuration Tools à SQL Server Configuration Manager à SQL Server Network Configuration à Protocols for <SERVERNAME> à Enable TCP/IP [Right click, enable])

9.       Make sure you start the SQL Server Agent (Start à All Programs à Microsoft SQL Server 2008 à Configuration Tools à SQL Server Services  à SQL Server Agent à Start the service [Right click, start]). This service to be automatic if it isn’t

10.   Install TFS using the default configuration. Once you pass the Readiness Checks, click Configure and wait for the installed to finish configuring the server. The server may detect errors during the readiness check. You have to resolve any errors before you can proceed. The steps above cover the majority of the prerequisites so you should have to do much debugging before you finalize the configuration process.

11.   Once you are done. Click next to finish, then click close. You have successfully installed and configured the Team Foundation Application Server.

12.   Next we will configure the Team Build Server.  

 

 

13.   Using the start menu, navigate to the Team Foundation Administration Consol. Expand the TFS server, and click on the Application Tier. Take note of the Uri. You will need this to finalize the Team Build configuration.

 

14.   Going back to the Team Foundation Server Configuration screen above, select Team Build Configuration and click on Start Wizard. Click Browse à Servers à Add… à and type in the Uri from step 13.

 

15.   Click Ok à Close à Connect, and you are done! You can manage the Team Foundation Build and the Server anytime from the Team Foundation Administration Consol accessible through the start menu.

 

 

 

 

 

 

Introduction:

This is a walk through of the new functional testing tool codename “Camano” and the Manual Test Runner by Microsoft. These are very powerful tools put out by Microsoft that now allow you to efficiently plan and execute your QA Testing efforts. Also, all of your testing data is stored and integrate into Team Foundation Server. This is an amazing beneifit if you use TFS as one of your for project management tools. Camano is released with Visual Studio Team System code name “Rosario”. You can download a virtual machine image of Rosario from the Microsoft web site.

 

Camano Overview:

To start off I will go over all the entities in Camano. The most basic entity is a Test Case. A Test Case is work item type in that mostly consists of a sequenced set of steps and their expected results. A group of Test Cases make up a Test Suites, which also could be made up of other Test Suites also containing Test Cases. A Test Configuration is an entity that allows you to specify the name of the environment you will be testing in and provide a description for it. A Test Suite, a Test Configuration, and Test Settings make up a Test Pass. A Test Pass or a combination of Test Passes make up a Test Plan. A Test Pass can have its own “Global” Test Settings. I will cover each of these entities in detail later on. Here is a diagram of how these entities fit together that you can refer back to if needed.

Should you keep reading? What you'll get out of this post:

After highlighting some things regarding navigating through the application, I will create a couple of Test Cases, add them to a Test Suite, create a Test Configuration, create a Test Pass, create Test Settings, and create a Test Plan. I will introduce the Microsoft Manual Test Runner and use it to execute my Test Plan. I will then go over a report generated based on my testing activities.  

My examples, and the data I use in creating each entity, assume that I am testing the User Management functionality of a web site in Internet Explorer. I don't have an actual web site developed to test, so I will pretend I am executing my test cases and just worry about analyzing.

 

Before you fire up Camano:

First I start by creating a new team project from within Visual Studio. I used the CMMI template and finish the wizard. Once this is done we are ready to start working in Camano.

 

Navigating through Camano:

I took a screen shot of the Home Activity Center Screen below and split it up into 3 main sections to go over.

Section 1 lists all of the categories, or menu options in Camano. Once you click on one of the menu items, all of its sub menu options are loaded into Section 2. Other sub menu items are accessible by clicking on the little down arrow below each menu item. Section 3 lists all active windows you have open, or any unsaved screens. It took me a while navigating the application using the Working Progress section (Section 3), but its pretty handy once you get used to it.

 

Creating, Configuring and Planning your Testing Effort:

Start by connecting to the team project you created in Visual Studio. This is done by clicking the Home menu option and selecting Connect to a Team Project.

I will now start planning my testing effort. The first step will be creating the Test Cases.

First click on the Planning menu option.

Click on New Test Case.

Populate the Test Case's title, Actions or Steps, and Expected Results and any other fields you wish populated. I am going to create a second Test Case that will come in to play later on.

Click on New Test Case to create the second Test Case then populate the fields on the screen.

After creating the Test Cases, I proceed to create a Test Suite that will consist of both Test Cases created earlier.

Click on Manage Test Suites. Click on the Test suites, then click on the New Suites button and give it a name. I will call mine “User Management Test Suite” and reference it throughout the remainder of this post.

Once I created the Test Suite, I want to add or associate my Test Cases to it.

In the right pane, click on Add Tests. Make sure that the Test Case Suite you created is still highlighted in the left pane.

From the window that pops up, select the Test Cases and click the Add Tests button.

After creating my Test Cases and grouping them into a Test Suite, I will create a Test Configuration. A Test Configuration will allow me to associate the name and a description of my test environment with my Test Suite.

Click on Planning, then on New Test Configuration. Specify the Name and Description then click on Save and Close.

After creating a Test Configuration, I will create my Test Settings and then a Test Plan.

Click on Planning, then on New Test Plan, then click New Test Settings.

On the left pane, there are 7 options. I will briefly go over each one and and highlight what is needed to get through this walk through. I will not include screen shots for every option.

In the General tab, specify the name of the Test Setting.

In the Execution Criteria tab, select whether you want to execute the test locally or on a remote server. If you select Remote you must specify the server IP in the drop down box, or select it if it from a list if the list was automatically populated. Also in the Execution Criteria tab, you can enable the type of logging (Collectors) you want to happen.

In the Deployment tab, you can add files you wish to deploy with your test pass. These could be files used by the application your testing.

In the Setup and Cleanup Scripts tab, browse to script files that you want to run before and after the test run. These could be scripts to automate setup steps.

In the Hosts tab, select the Host Type (or platform) you want the test to run in (i.e. Your Machine, ASP.net, Device)

In the Test Timeouts tab, you can enable aborting or failing a Test Run its execution exceeds a specified time limit.

The data I used in the walk through:

• Under General, I set this Test Setting’s name to “Administrator Functionality Test Settings”
• Under Execution Criteria, I select “Execute Tests Locally” and collect “Event Log”, “System Information”
• Under Deployment, I disable deployment
• Under Test Timeouts, I Uncheck “Mark an individual test as failed if its execution time exceeds”

Once I am done, I proceed to create my Test Plan.

Click on Save and Close. This will take you to the New Test Plan Screen.

The Test Plan is where everything we've been creating so far comes together.

These are the Steps I took to create my test Plan:

1. Enter the Name, Description, Start/End Dates, and a State for the Test Plan. A Test Plan has to have a state of In Progress to be visible/usable throughout the application.
2. Select the Test Settings that were created earlier and saved
3. Specify a name for the Test Pass. A Test Pass is the grouping of your Test Suite, Configuration, and Test Settings. I called the Test Pass used "Test User Management".
   
   1. Select the Test Suite created in step two containing both User Management Test Cases.
   2. Select the Configuration Created Earlier containing the name of the environment we are testing in.
   3. Select the Test Settings created earlier (or different ones if applicable).
   4. Set the State to In Progress, and the Type as Non Recurring. This Setting will be used by Camano to determine if the Test Plan shows up in the All Test Plans Page, accessible from Planning --> View All Test Plans.
   5. Save and Close.

To summarize what I did so far:

1. I created 2 Test Cases to test the User Management section of a web site
2. I added those to a Test Suite
3. I created Test Settings and Test Configurations to give more detail about my test environment
4. I created a Test Plan to test the Administrative Functionality of a web site
5. I added my Test Suite, Settings and Configuration to a Test Pass under my Test Plan

 

Test Execution:

Click on the Testing menu option. In the left pane, right Click on the Test Pass (i.e. "Test User Management") and click “Start Testing this test Pass”

This will launch the Microsoft Manual Test Runner. This is a simple interface containing 3 expandable areas. In the first area is a list of all the Test Cases in the suite associated with the Test Pass. In the second area are the Steps for the Selected Test Case. In the third area are the Properties of the selected Test Case.

Select the Test Case you want to execute from the top pane.

Click on Actions, then on Start Test Case. This will Start the test case and, focus on one of your open windows and start recording. Since I haven’t prepared a web site for this demo, I ran a blank solution in Visual Studio to demonstrate how the recording behaves (indicated in the upper left corner of the browser window) I also launched MS paint and it also started recording my window there.

I am going to minimize my screens, open the Manual Test Runner and pass all my test steps. This is done by right clicking on each step and selecting pass. When I am done, I am going to click on End Test Case.

Notice that every time something is recorded and a test case is passed (or set to fail/inconclusive), a blue triangle is added. This is the a marker in the automation strip. When you double click on a marker, it and its corresponding end marker will change color and a menu will pop up. This menu allows you to Play the selected clip, save the clip for sharing, or preview the clip.

Notice that it also created and attached an xml file and an ActionLog text file under test case results.

After I am done, I can click on Actions and Publish (or alternatively click Publish Test Results listed under the Test Cases). This will gray out my test case and only allow me to run the remaining test cases. You are also able to create a Work Item Type Bug for any test cases that fail. This Bug will be stored in TFS and could be managed there.

Go back to my Testing Activity Center, click the Refresh Button, right click on my Test Pass and select “View Test Pass Details”. You will get the following report.

This report will tell you:

• How many test cases there are
• How many test cases were ran and how many weren’t.
• How many test cases passed, failed, or were marked as inconclusive.
• How many test cases were blocked and how many failed by regression, new issues, or known issues
• The test configuration name
• The test suite name
• The date testing should start
• The date testing should end

The report also allows you to assign test cases to TFS users.

 

Conclusion:

The Manual Test Runner is an independent tool of Camano, but they work together nicely. In a real world scenario, Test Cases can be prepared using Camano (or in Visual Studio), assigned to the testers that will be executing each test case, and organized into Test Suites and Test Plans. Once that planning effort is complete, each member of the execution team can fire up the Manual Test Runner and work with their own set of assigned Test Cases. Logging/environment settings and execution start/end constraints can be put in place for better control over the testing effort. The project manager can depend on real time reporting and work item customization to capture the data they need for keeping the porject on budget and on schedule.

 

Resources:

http://msdn.microsoft.com/en-us/vstudio/bb725993.aspx

http://ozgrant.com/2007/12/09/november-rosario-ctp-planning-a-testing-effort-with-camano/